This was a just for fun project I did for a mates festival. Decided to use it as a vehicle for a bunch of AI tests that had been kicking around in the back of my mind. I used a whole bunch of different AI tools, and then tracked it all in with Flame Motion Vectors. Quite a fun project to do. Hope you like.
Here’s the before and after:
The main techniques used were:
You take frames from the video, upload to midjourney, prompt it to create a scene, it will create the scene, but with similar lighting, colour grade etc, Then you can paint the resulting image back in to the original video and track it in to the scene using motion vector tracking. If you’re tracking it in to the background with a person in the FG you can remove the person using Runway ML, then you get a clean motion vectors track of the BG
You can export stills and Use Runway ML object removal tool to for example paint someones clothes out the prompt it “Stripper” and it will create that, then bring back in to flame, difference matte to isolate the changed elements them motion vector track in.
Then the other more motion graphics looking shots were made using Runway MLs Vid to Vid tool, you can export 5 secs of video, upload to Runway ML and use a photo prompt to get a copy of the video vaguely in that style, then you can just Add/Screen on top.