AI projects in postproduction

Flame Between Renders
,
1

What are your experiences with AI projects?

Last year, we started offering active AI, which sparked a wide range of opinions within our company.

The footage we work with comes from various AI engines such as Comfy UI, Flux, SDXL, WAN Video, Kling, Runway, Veo, Midjourney, Facefusion, Hedra, Luma Dreams, etc.

Working with it and correcting errors or tracking it is awful. It takes a lot of time and effort to get the footage to a point where the customer is ultimately satisfied. So it’s definitely not a one-click solution. I get the feeling in my circle that customers now want to jump on the AI bandwagon to keep up and stay modern. Which is totally fine. However, clients are also slowly realizing that it takes a lot of effort to end up with a beautiful film. I hope this will regulate itself in the coming years.

I would be interested to hear your opinions and experiences.

2025: The Year in Review
2

Yup I’m fully involved in that, it takes longer than you think and costs more too. My AI day rate is £100 more than my flame day rate because of all the costs involved.

I’ve created an awesome technique in ComfyUI to retexture videos from say kling to be much higher fidelity. I’ll share it at some point.

Yeah tracking it can be tricky because it’s not following real world geometry, but syntheses does a pretty good job of averaging it out.

Even being fully immersed in AI, it’s so difficult to keep up. It’s really fascinating though.

9 Likes
3

This was full of good lab notes.

@RufusBlackwell Wondering what’s changed and what hasn’t in the last 6 months?

4

I just completed a job I hope to be able to share soon. I used photoshop generative fill to clean up still images, used those to train comfyui on a video (similar to nuke copycat). The resulting video was good but only able to get 720p results. So that video was then used to make motion vectors to manipulate the full 4k original source.

So yea… we’re using “AI” but it is the furthest thing from hit a button get a prize. And the hardest part is communicating this to the clients.

3 Likes
5

Nobody said cheaper or easier: It is just doable with a very reasonable price. When we started doing staff for commercials in 3D, it was ugly as hell…but it was possible. We are now in the same position, but with AI, and they have to start understanding.

6

I´ve been working for 3 months generating “B Roll” for a TV series about the Aztecs. Very difficult to get specific looks, specially with the specificity of this time period. Using like 10 different platforms to solve different problems. Still, have to correct problems from the AI generation in Flame or mix the result with live action plates.

6 Likes
7

I just read a fascinating proposition by one of my favorite writers on the AI topic (link below).

His premise is that because LLMs and any ML model in current use is based on statistics, and ultimately just provides a highly likely answer, it will be inherently wrong a certain percentage of the time, and no amount of GPUs or training will change that, because it’s inherent to the technology.

Furthermore, the failure of successful AI deployments, including the MIT study making the rounds in recent weeks, is actually not a failure of the technology, but a failure of us not understanding and using it accordingly.

We (companies and users) have built a life of technology that is supposedly deterministic and predictable with the results. We go to great lengths to test, create redundancy, and other processes to guarantee a successful output. We do not achieve 100%, but a common bar is 5x 9s (99.999% uptime) as an acceptable result.

Coming from world of thinking that way, we really can’t wrap our head around a technology that may only be 95% correct and will never be more.

But that is a failure of imagination, not a failure of the technology. We have to start thinking about AI differently, and deploy it in ways where it being right 95% of the time is still a winning proposition. Which only means we need some backup methods (most likely human for now) that be the backstop for the remaining 5%. So we benefit from getting fast results 95% of the time, and we figure out a way to make the other 5% work.

What does that mean for post production and Flame artists?

In short: we’ll be very busy for years to come.

If you accept this line of thinking GenAI tools will never generate perfect key art. But it will happily and more affordably generate elements of key art, or key art that can get there with some help from a Flame artist. Or background elements. Or various data channels like normals maps, and so forth.

We as artists support that of course.

Where this still fails is in the corporate leadership which rather just think the ranks and the keys to machines. They’re the ones that need to appreciate that to make the most out of AI, we should use it as part of our processes, not to replace everything and have the next TVC just ‘vibe-designed’.

Article: A Mistake folks make about AI

6 Likes
8

Yeah we’ve done some full AI spots, plenty of b-roll, a few pack shots, some lip sync swaps, a metric tonne of rescuing shitty stock backgrounds, more Beeble AI live action AOVs than I can remember, and now some Tunet mixed in, daily dose of FLUX and LoRA training in AI Toolkit for tons of proper nouns we service. All 7 Flames are Comfy servers using Distributed and I’ve never been happier to be all Nvidia. So far the 5000 Adas are a bit slower than the 5090s but at half the wattage I can delay the extra 20amp circuits I’d need. Plus banging on B200s in Vast is is not a bad way to spend a fiver.

6 Likes
9

God Damn I feel like this shit changed overnight. Or maybe over the summer.

10

Not quite up to the majesty of @RufusBlackwell yet: did this spot recently. This is the director’s version which has a lot more AI stuff in it. Mostly motion gfx stuff. It was disappointing the agency weren’t into it and posted a much toned down version of it. A lot of Kling, midjourney and Runway. Also all of the motion gfx stuff was the director. Only one cg shot where I pulled in a favour to build a flag on the moon because AI just “wasn’t getting it”. Loads of flame and after effects stuff too to tie it all together and an excellent grade by Bamf.

8 Likes
11

So much talk of AI everywhere… Unsure if this is jut a fad and everything will settle down back to normal or this is one of those moments where you either get on with the times or are left behind, way behind…

12

I feel like there’s two different ideas of “AI” in my world right now.

  1. The most popular idea is that AI can generate a commercial for you whole-sale. Work the prompts and generate some video. Bam you’re done.
  2. What I’m doing a lot of… and I think it is here to stay… is using AI / ML tools in my already wide-array of tools.

For instance… Things that are “AI” that I use but are not category 1:

  • Photoshop Generative Fill
  • Silhouette ML Mask, SammieRoto, etc.
  • Flame’s ML tools… Sky, Face, Body, etc.
  • Runway ML, other web-based tools… generate elements to be used as comp elements
  • Tunet / CopyCat for very specific situations

So I definitely think category 2 is here to stay. Category 1 might fade away but also it’s capabilities are changing daily.

5 Likes
13

Good summary by @bryanb.

My take is that both will survive but serve different purposes.

The first one (turn key prompt to video) will be used to flood the market with cheap clickbait and monetizing traffic with AI slop. I don’t think we have to concern us much with that. By definition they don’t want us there.

The second will continue to evolve and you need to be literate to remain relevant, or someone else will be. This is what will make modern video more budget friendly, but will need our specific skills more than ever.

Better to be a Flame artist than a camera person in the current market for years to come. They need more Flame type skills and less physical camera.

Though it may not be just Flame but other tools like ComfyUI and who knows. It’s the skills not the software.

3 Likes
15

Can I ask, in Cat 2, outside of Flame, how much more you use these tools compared to say OFX plugins, regardless of what you’re using them for?

Hoping they eventually make their way into Flame, like they usually do with better results ie ML TW.

1 Like
16

This is just my take, but I’m currently 50 50 between Flame and Comfyui. I don’t see how flame could possibly keep up with the furious innovation of open source Comfyui. It would be better to try to harness it by making flame and comfy as compatible as possible. I have no idea of the practicalities or legalities of that. But flame and comfy are a v powerful combo, but could do with being much closer aligned in terms of compatibility.

9 Likes
18

It’s a nice thought @RufusBlackwell . Imagine if they could segue…

What I’ve liked about AI stuff is that it feels more 2d in a way. I can shoehorn stuff together and paint over the joins.

What I don’t like is the unpredictability. Even when you do a test comp as a ref it still sometimes doesn’t get it (Maybe that’s due to my lack of expertise).

1 Like
19

That’s inherent to technology. It’s not deterministic. That’s the part we all have to get used to, and educate clients on.

1 Like
20

The Blender implementation of Comfy is truly a remarkable feat of community development, although the first thing I did was scribble down a script with GBTShat to translate all of the presets from Mandarin to English. Being able to capture the render buffer or viewports as inputs is bananas.

It does make you think how insanely powerful it would be to have Comfy in Flame. The Nvidia blueprint for Blender/Comfy seems equally batshit, although I don’t have a windows installation nor an a6000/4090 to fuck about with.

3 Likes
21

The lack of 4090 might change soon.

2 Likes
22

This is my latest experiment using AI generated video and Flame:

6 Likes