OpenAI dropped its most ambitious project yet this week with the launch of Sora 2 and its accompanying social media app, marking a watershed moment in generative AI technology.
It shouldn't even be that hard for them to do. Open source solutions allow for very easy training of character Loras for video models (it can be done on a 5090, or you can rent a Runpod instance of an RTX Pro 6000 and train one in a couple hours). If they can generate videos up to 60 seconds, paying for the training of a character Lora should be within reach at this point.
Then you'd just need the control to have the prompt know which character to place in which scenario for the video. It's only the multiple consistent characters in one scene at once that is tricky right now. But I think even that's doable if one really knows what they're doing with a decent custom setup.
Training a single character video character Lora works shockingly well and is very simple to do for the main image/video models using a tool called AIToolkit. It's very easy to setup from YT tutorials on Runpod as well because sometimes it does like more than 32GB of VRAM. Though it is possible to do locally as well on a 5090.
I've had some success prompting in a description of both characters and then using a character Lora to inpaint the specific details of each character with both Loras. Then if your resolution is high enough (720p+), Wan2.2 does a pretty great job at maintaining consistency of the character likeness. But the length of the video outputs is still limited to about 10s without quality degrading.
But all of that is very custom and doesn't always work 100% of the time and takes some finagling. It would be really nice to have the ability to take multiple character Loras and slot them in with a simple language prompt using the tokens from each.
I think this is great for all the reasons mentioned. Underserved communities will be able to have more content to their liking.
Buh bye gatekeepers!
There’s always going to be a place for live action shot films just like there’s still a place for plays. But the days of millionaire actors is likely drawing to a close.
By decentralizing the talent, if you will, it will force Hollywood and the entertainment titans to be more “honest” by keeping them accountable to the target audience. It’s almost like AI is a democratizing force. This is already fascinating to watch unfold.
They really, really, really need to focus on persistent characters. All the other flashy features are as nothing compared to the importance of that.
It shouldn't even be that hard for them to do. Open source solutions allow for very easy training of character Loras for video models (it can be done on a 5090, or you can rent a Runpod instance of an RTX Pro 6000 and train one in a couple hours). If they can generate videos up to 60 seconds, paying for the training of a character Lora should be within reach at this point.
Then you'd just need the control to have the prompt know which character to place in which scenario for the video. It's only the multiple consistent characters in one scene at once that is tricky right now. But I think even that's doable if one really knows what they're doing with a decent custom setup.
Training a single character video character Lora works shockingly well and is very simple to do for the main image/video models using a tool called AIToolkit. It's very easy to setup from YT tutorials on Runpod as well because sometimes it does like more than 32GB of VRAM. Though it is possible to do locally as well on a 5090.
I've had some success prompting in a description of both characters and then using a character Lora to inpaint the specific details of each character with both Loras. Then if your resolution is high enough (720p+), Wan2.2 does a pretty great job at maintaining consistency of the character likeness. But the length of the video outputs is still limited to about 10s without quality degrading.
But all of that is very custom and doesn't always work 100% of the time and takes some finagling. It would be really nice to have the ability to take multiple character Loras and slot them in with a simple language prompt using the tokens from each.
Staying LLM, this will be the hardest thing, as drift is inevitable without structure.
Endgame here is video games. This is 10 years away from it.
Less. Much much less. These things are progressing exponentially.
I think this is great for all the reasons mentioned. Underserved communities will be able to have more content to their liking.
Buh bye gatekeepers!
There’s always going to be a place for live action shot films just like there’s still a place for plays. But the days of millionaire actors is likely drawing to a close.
By decentralizing the talent, if you will, it will force Hollywood and the entertainment titans to be more “honest” by keeping them accountable to the target audience. It’s almost like AI is a democratizing force. This is already fascinating to watch unfold.