Hi all. Whats everyone using for AI tools. im using Runway ML - but it’s hard to get the output looking like the input - in terms of colour space and quality. this is for generating video, matting, getting rid of objects etc etc. is there a more PRO software out there. and if so - would be super cool to access it from within flame.
The two go-to tools at present (always an evolving situation) are Weavy.ai and ComfyUI. Weavy being easier to use, but slightly less flexbile. ComfyUI is the Wild West, but can do all sorts of crazy stuff.
Neither is integrated with Flame. You’re voicing a common request many of us have. How realistic this is anytime soon remains to be seen.
I’m not sure how we use the ‘PRO’ attribute relative to AI tools right now. I would say ChatGPT and prompting until you get happy is the ‘non-Pro’ approach. But beyond that it’s a matter of interpretation of what represents Pro? The quality? Color space handling? Cost? Flexibility? Predictability?
One could argue Weavy or Comfy are Pro, as you get workflows that let you control the result a lot more than repetitive prompting. Also you can repeat outcomes by using fixed seeds, something ChatGPT doesn’t afford you AFAIK. But Runway can produce Pro results by all accounts, if you have the resources of a big studio. As we’ve heard at the VES presentation in NYC a few months ago, there are significant gaps between the prosumer tools most of us have access to, and walled garden custom models and big server clusters big studios use. One trail to follow on that is the topic of quantized models, and how watered down the models are we actually get to use vs. their full potential.
2 Likes
thanks. I might have a go at Weavy - see what happens. I mean for example I was just about to embark on getting rid of someone on a zip wire. thought I would try in Runway - and boom - 5 mins later they were gone. good result - but not the right colour space. but still used it as it was easy to colour correct in. but 5 mins versus an hour or two.
1 Like
I feel we need a portal to all of these things within flame. not python and crazy stuff like that. just a way of sending prompting and the result being on your desktop
2 Likes
We would all love to see that.
Making that happen is non-trivial. As most of those tools run in the Cloud and require paid services, any such portal would have to use an API, for which there is no industry standard, so each would have to be custom integrated. You would have to sign up for API tokens with the service, add your API key, so that request can be properly accounted for. And that’s just one of the problems to solve.
It’s a fast moving space. If ADSK would integrate with Weavy today, by the time this comes out as a release Weavy is likely old history (already is as it was acquired by Canva). To integrate it the reverse way (Weavy integrating with Flame) is less likely than Resolve or any other more popular tool, and even then this isn’t the most important thing for them.
All AI tools are in a race against time. They don’t have time to worry about our needs. And us keeping up with their pace requires a bit of an agility that release cycles don’t afford.
It does bring me back to the fact, that having some sort of batch node that could execute python code and pass image streams back and forth, would make it a lot easier for the community to build the glue code, which we are capable and willing off.
One possible avenue for that is the existing PyBox node, which warrants further experimentation.
There’s a long thread here that examines some of this.
1 Like
I just signed up for Weavy - bloody brilliant. thats the future of post production right there,. we need batch to have access to all of this. nodes that go to AI solutions. nodes that are prompts. even a node that takes you to Weavy would do!
3 Likes
If anyone is inclined to upvote this, here’s a feature request to enhance PyBox so it can be used to trigger ComfyUI or other cloud-based AI tools if they have appropriate APIs.
PyBox is pretty close, but needs to be upgraded to work on whole sequences and to allow manual triggering of an external app run to avoid runaway token usage.
4 Likes
true - but PyBox is currently bound to python 2 and life-threateningly slow.
I 100% agree that this could/should be a gateway to AI in flame but I tried it and almost preferred to stop breathing.
4 Likes
Well, room for improvement to make it suitable.
2 Likes
‘room for improvement’ is engineering time and development cycles.
Personally, I’d be thrilled if development was all about making flame the mission-critical hub of a pipeline, and that there were spokes like machine learning tools for x, y or z, or nuke, or color-grading, or CGI…
But nobody else wants that - they want text or paint or markup or some other thing…
Getting Flow to work from camera to delivery would be monumental…
blah blah blah
So yeah, let’s get a dead inititative like PyBox to do something that can be done elsewhere with other tools…
3 Likes
1 Like
I don’t see anything incorporating with flame anytime soon. It’s not worth it.
comfy has major dev updates every week ish. Shit, the last few weeks has seen Nano Banana Pro, Flux2, Seedream 4.5, Topaz in APi form, and 15 other major updates I’ve forgotten about.
There is no such thing as Pro AI tools. The FAANGS are chasing billions not thousands.
4 Likes
correct
Totally agree on ComfyUI being unbeatable on that front. It’s a freight train in motion and nobody is stopping it.
What I envision is having (well, we already do) a bin of great ComfyUI setups for various tasks.
But able to run as part of Batch without having to manually move files around. Just add node to batch, select tasks and run.
PyBox already has the basics in place, but is missing some details to make it smooth and fast.
That’s all.
1 Like
Nobody is developing PyBox any longer.
And of all times - now is the time that we need such connective tissue.
I am mostly likely wrong but it appears that this initiative died in 2018 or earlier.
I’ve been using Weavy. Not great with colour space and a lot of the modules are hopelessly low rez, but I’ve gotten acceptable results at HD and 2k for a lot of things, and I used the roto tool today at 6k and it worked as well as any roto tool.
2 Likes
Well, they can pull the cobwebs off. It’s faster than starting from scratch. If there’s a better answer, not wedded to PyBox. Just looking for a way to get this done. And sometimes finding a fixer upper is faster than an empty lot.
1 Like
Wasn’t Baselight at one point using Pybox for their Flame integration? From what I remember, there were several limitations that were deal killers for us. Not sure if anyone is still using it.
We took a look at Pybox for some Nuke integration we had an idea for. Pretty quickly we determined it was a big nope, and forgot all about it.
It would be great if Flame became the ‘big connector’ for all things.
4 Likes
Upvoted! Thanks for doing this, @allklier
1 Like
sometimes you need to build a new spaceship instead of that barn find pinto…