Making Rain - Flame's inference experiment

Flame Tools and Tech
, , ,
1

Hey guys! Sharing with you the workflow I put together using Flame + Inference node + Tunet models.

Some key points:
I only trained on 5% of the video (that’s just 104 frames out of 2084!)
Model render render the resto of the frames.
Pulling off like 95% effectiveness on the full sequence!

Just was just a experiment to use as example to show you guys the workflow.

00:00 - Introduction 01:22 - Footage 02:02 - Training set, 5% only 03:16 - Copy vs Learning 06:05 - Progress visualization between Epochs 08:04 - Convert trained model to Flame

25 Likes
2

I’m quite impressed with the copy vs learning part, this would mostly avoid to retrain in case of changes…
Curious to know if other training tools also understand this learning thing or will just copy? if this is exclusively of this TUNET tool

1 Like
3

Looks epic. love it!

1 Like
4

Nicee!
To my knowledge is the only one. I specific design the architecture to “push” the model to learn to do that. (And it does saved me a few times in real projects)

Down side that is also increase training time heavily.
That is why is possible to opt in or out for that in the training config.

4 Likes
5

Thanks Thiago. Great stuff as always.

1 Like
6

Stunning

1 Like
7

Very nice. It would have been interesting to see the trained model applied to shots that don’t belong to the training data.

1 Like
8

Nice work! This is gonna end up with VFX houses not posting before/afters…

9

You forget - there are no spoons - it’s all in camera…

2 Likes
10

that’s what i was thinking! but yes very impressive indeed @tpo !!! is this something that needs to be on an Nvidia card or can a Mac also run this in a somewhat timely fashion?

11

Tunet appears to be Linux only @TimC

1 Like
12

@TimC @hBomb42

Well, a knife cant hurt people on its own, a person with bad intentions using a knife who can.

But seriously, this is actually a much bigger question currently on AI space, private vs open source. Should a company or individual have a model without no one know that is possible to do? Or Should be open and everyone knows what is current possible to do?
I won’t pretend that i have the answer for this…

I have Tunet for around 3 or 4 years now, yet, i indeed removed lots of features for the open release, is there to raise a good debate or people use. Same for smilar models like simpleML or copycat.

@cnoellert you are right. Linux only, nvidia:( Tunet was never design to be used on artist workstation, more to be used on server or GPU farm. the exported/converted model should work in Mac, but i have not tested.

2 Likes