The worst ai bts thread ever

I recently finished this pretty juicy show that’s getting a bunch of traction on socials and figured I’d share it here and we could talk about what I can talk about.

Daylight exteriors are bg dmps
Stadiums are full background stadium and crowd replacements
Pitch to football stadium ground conversions
Jersey rebuilds
Fully generated aerials
Fully generated newscaster sets
Social bits are all fully generated
Zero cg
Only one shot received traditional roto

87 shots across 6 durations

Team of 8 Flame and AI and Design

A few weeks of early look dev

Basically a sub 20 calendar day turn around for all shots

Looks nice Randy! Two questions from me:

  1. For the stadium shots are you generating based on certain angles annd getting a locked off wide stadium and then using a traditional tracking workflow?

  2. Is any any AI related relighting happening across the shots (this is something I’ve dabbled with but never really ended up with something all that satisfactory for the particular shots)

Nice work!

Well, no denying ai is here, in mainstream production.

About the personnel-Is that 8 flame artists? Or a team of 8, including ai and designers? Could you clarify the distribution please? Thanks!

Edit: oh, and do you know how many shoot days?

Edit 2: what about prepro? What did that look like? How about bidding?

Thx!

Looks great. Can you share a ballpark budget?

Nope. No traditional tracking workflow here. We built character sheets of the stadium, working with Eric at Furlined, to figure out what kind of NFL stadium we wanted in the background.

We had to shoot in an L.A.-based soccer stadium that didn’t have the scale we needed for American football NFL games, so we had to do all the stadium replacements. We originally were thinking to do locked-off camera shots from inside the stadium or planar or match move and extract frames of the background, tile them together, and make AI panos that we would track in Flame. But there was no traditional compositing on those AI stadium interiors. They were all completely generated on top of video with no roto.

The team was four or five flame artists, and an additional four full-time AI artists. I can’t find enough flame artists that I know and love and trust that also know their way around AI enough. I’m desperate for them.

The AI artists all have backgrounds in design or CG or compositing. I’ve been doing AI centric spots, almost exclusively for the last 15 months. From real time AI character work posted to X in three minutes to three minute 100 shots fully generated long-form spots, there are few threads that are emerging that are perfectly clear to me.

AI is fucking hard. You have to have an expert level of storytelling or composting , or design, or CG chops. It’s such an amazing superhero power, but only when added to a core competency of one of those verticals.

I can’t think of a single shot I’ve delivered in over a year that we delivered straight from a gen. It always needs something:

  • split screen this
  • comp that
  • swap that
  • paint that
  • track that
  • restore that

For that reason, an AI artist who rocks up, never having experience in our business fixing pixels, has never worked.

It was a 2-day LA zone shoot to be next to talent. The job was originally only scoped for a 15 and a 30. We knew going into it that we wanted long form but it’s super risky for any client out there to rock up for these kinds of shoots and be willing to drop tons of money for multi-day shoots and green light huge piles of cash on a ridiculous story.

We cut the long form first. tease clients with that cut 30 and 15 lifts out of that while we sold the long form.

My memory is fuzzy. But we cut the long form internally approved it, confirmed those 70 something shots and started working on the 15 and 30. And then only after the long form was greenlit, I think just a day or two before we delivered the 15 and the 30, were we able to get a head start on the rest of the shots.

Pre-pros for the VFX were surprisingly simple. based on location scouting we spent a few days mocking up stadium exteriors and stadium interiors and dialed in a look pretty quickly.

All the creative bits from the prosthetic work, the statue, the aerials, the newscaster, social, the social matrix part, most of that was figured out before the shoot. It did take us forever to dial it all in, but the pre-process is just like the one we’ve been using for all these years. Once you know the location and the talent and the casting and the wardrobe and the camera language and the art direction and the production design, you gotta play mini department head of all of it when you’re generating this stuff or you’re just gonna waste everybody’s time.

I can’t disclose budgets. But if you look at a spot like this and you know anything about advertising, you know what a traditional schedule and pipeline and team and budget would look like and if you’ve done this for more than a couple years, you know that you haven’t seen a lot of this kind of advertising because it’s just been too cost and schedule prohibitive.

No AI relighting was necessary.

Bidding is definitely an art form with these kinds of shows. I’m embedded within the agency so it’s actually kind of straightforward. We generally know what budget parameters we have to work with and there’s an unbelievable back-and-forth about what we can squeeze out for those budgets. If we had X more or Y more, we could offer Z more. Blah blah.

I’ve worked on enough of these kinds of shows over the years with traditional pipelines to know what they should take and what they should cost. I’ve made more cool shit I’m proud of in the last 6 months than I have in the last 6 years.

That last sentence.

@randy Id be interested to hear if you are finding that you are doing more with the same budgets you were used to getting pre-AI or if the size of the budgets are smaller? The big question for me is in regards to where the client is saving thie money. Is it more on the production side or the post side? In TVC land, is it actually production suffering, that agencies are doing it with AI in-house instead of shooting, and that post production is similar in terms of time & budget except that more agencies are bringing that in-house now?

TVCs are very different to Features/Episodic, which we work on, but we are often now being asked, by clients willing to consider AI until you start talking about the legal considerations, to do a lot more VFX scope and shots for a lot less money (which I know is happening across the board). But I haven’t had clients looking to save money on the production side just yet, they think all the savings will be in their VFX budgets.

Truly impressive work there @randy. Out of curiosity, are you working on graded footage or have you figured out a nice workflow to maintain a log-ish pipeline?

Also, and I doubt you can share this, but a before/after would be amazing to see.

Love all of this. So, what was the one shot that needed traditional roto?

On the budget question, the short answer is: yes. :joy:

But I have a slightly weird perspective on this because I sit in the middle of all of it. Our clients have budgets and problems. Our agency writes scripts to solve those problems. Production companies figure out how to capture what we need, and then I work with production, our creatives and post to figure out how the hell we’re actually going to make the thing.

So I don’t really separate “production savings” from “post savings” as cleanly as maybe I once would have.

Shooting stuff is expensive. But shooting stuff is also important, and I don’t think the answer is simply “don’t shoot it, use AI.”

What having these new capabilities on the post side does is give you some pretty ridiculous superpowers when you’re making those decisions. You can do more with less, obviously, but more importantly you start to understand what you actually need to shoot, what you don’t need to shoot, and where it’s worth spending the money.

I’ve been saying this for years, even before AI: I think people who really understand VFX are uniquely positioned to help agencies and clients make those decisions. Now you add generative AI to that toolbox and the range of problems you can solve gets much, much bigger.

Basically: know when to shoot it, know when to VFX it, know when to AI it, and know how to AI/VFX your way to happiness. :slight_smile:

That’s much more interesting to me than “AI makes this 30% cheaper” or whatever.

On working graded vs. log:

Almost all of the generative work on this project was done on top of graded footage.

And yes, I know. Math is math. In a traditional VFX pipeline, working pre-grade is the “correct” answer. 2+2=4, and once I’m working on top of a grade, sometimes 2+2 does not equal 4 anymore.

But there’s a practical reason for it.

We’re a tiny team of X that 8x-10x freelancers when we need to. We’re basically a group of expert generalists, and color isn’t our core business. We rely on best-in-class color partners to do that work.

The amount of back-and-forth required to run all of this generative work pre-grade would absolutely destroy the schedule and budget.

And there’s another important difference: these AI models are effectively very comfortable with graded imagery. That’s the world they’re used to seeing. Working with the grade baked in is easier, faster, cheaper and, most importantly, produces results that are good enough to make the overall thing possible.

We still have all the camera originals, obviously, and we’ll ACES our way to happiness wherever it makes sense. But the generative work itself was overwhelmingly done post-grade.

Is it perfect? Nope.

Sometimes the color shifts a little. Sometimes we live with an imperfection. Sometimes a creative makes a decision with the colorist today and tomorrow I have to move something 3% because it makes the generative shot work.

But that’s the trade.

If I can go back to the creatives and say, “Okay, occasionally we’re going to have to accept some little imperfections or make some compromises — but in exchange we can make this entire long-form thing that otherwise simply wouldn’t have been possible,” that’s a pretty easy conversation.

Before/afters:

I would love to show some. I just have to be really careful about what I share. I’m going to see what I can put together that doesn’t step on any landmines.

And Nick — the traditional roto shot:

It was the touchdown catch.

Funny enough, that also turned into probably the hardest shot in the entire piece.

We used video-to-video to turn a completely different part of the football field into the end zone, and the models just hated the action.

Apparently generative AI finds the concept of a football player running without a football deeply offensive. :joy: It desperately wanted to put a football in his hands whether we asked for one or not.

We spent more time fighting that shot than probably anything else.

And it’s also one of the shots where, knowing what I know now, I would probably approach it completely differently. I’d likely do it with more traditional matchmove/projection techniques, maybe some light CG, and use the generative tools much more selectively.

Which is actually one of my favorite lessons from this whole project:

Just because you can solve something with generative AI doesn’t mean that’s the best way to solve it.

Sometimes the 25-year-old VFX technique is still the right hammer.

Amazing. This spot is like a Jersey Mikes sub sandwich made of VFX meats.

Quick question: If Danny DeVito was a sandwich which one would he be?

Fun fact: I’ve probably had more meetings with Accounts, Business Affairs and Legal in the last six months than I did in the previous 20 years of my career. :joy:

And weirdly, I’ve come to really enjoy that part of this.

Speaking generally here — not about the BA/legal discussions around this particular job — I think this is one of the more interesting and least talked-about parts of actually using AI in production.

Everybody comes to these problems from a different perspective. As artists and production people, we’re generally rewarded for making the impossible possible. Someone says, “We can’t do that,” and our instinct is basically, “Well, give me an afternoon.”

BA and Legal necessarily ask different questions. What rights do we have? What are the risks? What needs approval? What needs to be true before everybody is comfortable moving forward?

That’s not an obstacle to the creative process. It is part of the creative process.

We’re used to constraints. Budget, schedule, physics, talent, locations, licensing — production has always been about figuring out how to make something cool inside a box.

AI has just given that box some really fucking weird new sides.

I’m incredibly fortunate to have some really smart grown-ups on the agency side who are willing to get into the weeds with us. The question isn’t:

“How do we get around Legal?”

It’s:

“What would have to be true for everybody to be comfortable saying yes?”

Maybe that changes the technique. Maybe it changes a vendor or input. Maybe something needs additional approval. Maybe we build part of it traditionally. And sometimes the answer really is no, so we find another way.

The thing I’ve realized through all of this is:

The people who understand the constraints best eventually become the people who can innovate fastest.

Because once you understand where the real fences are — and which questions need BA or Legal involved — you stop wasting time being afraid of imaginary ones. You start designing the methodology around those constraints from the beginning.

It’s actually a lot like learning VFX. A junior artist can be terrified of breaking the image. An experienced artist knows which parts they can absolutely fucking destroy because they already know how they’re going to put them back together.

That’s increasingly how I think about AI production.

It’s not about eliminating risk, and it’s definitely not about ignoring it. It’s about understanding it well enough to make good decisions and still do cool shit.

Turns out the grown-ups are part of the pipeline now. :joy:

Bucket list entry achieved, being able to see him a few times a year for the last few years.