I mean idea is good but additional subscription at that price is pretty crazy
I wonder what a studios actual savings would come out to vs the just “send it across the world” outsourcing method? I’m not sure who this is marketed to I guess is what I’m saying. Is it for studios that want to stop outsourcing and have artists just do in house roto? Or is it for the big roto factories (but aren’t they using silhouette?)?
Looks like it runs as a plugin inside Nuke, not as an app. That would rule it out for roto shops overseas. They’re waiting for BorisFX to do something similar which I would assume is in the works.
Could be useful if you just need a few masks and not enough leadtime to send it out?
I’ll check it out over the weekend and see what it does. The price is pretty steep though.
BTW: it seems to be based on an in-house research project Foundry started in 2019 together with a college and someone else. Finally leading to a commercial product.
yeah, the only reason I wonder about the market is that $500 a year is pretty steep for roto in comp being made more efficient! But then again in the scale of things, I don’t think it would seem wildly expensive if it were a standalone kind of thing, think I’m more shocked that the Foundry isn’t just saying look how rad Nuke is, we’ve got this awesome roto tool, but instead licensing it separately
Exactly.
I think all these shops are so cash strapped, if they see an opportunity to eek out a bit of extra margin, since it’s AI related and people these days pay money for anything AI, they’re just floating it to see what happens.
Gave it a go on two shots I recently had to roto. One I originally did with gMask, the old fashioned way. The other one I did with Sam2+VitMatte.
SmartRoto did fine one and, and surprisingly well on the other. The latter, which I had done with gMask was a dancer in dark room with neon light bars, and the arms and hands are moving over one of the lights. Lots of motion blur, hand twisting, and issues.
Basic workflow:
- Setup roto classic style on first key frame.
- Find a few more frames to anchor on as supervised keyframes.
- Interpolate between these.
Now for step 2 SmartRoto will help you. You pick the frame, hit a button and SmartRoto will try move all the masks in the right place based on analysis. In the simple shot it got that right 100%. In the hard shot it got about 80% and I had to do some tweaks. Fair enough on that shot.
Once you have these supervised frames, you let SmartRoto run to fill in the space in-between. If it starts drifting, stop it, make adjustments for another supervised frame and then resume.
Workflow is solid and straight forward. You do have full control, and if there’s an error it’s easy to fix - much easier than create in-fill or hold-out mattes for ML based mattes that go on tangents.
In terms of speed. Well, it’s as slow as most things Nuke. Not bad, but also not fabulous.
Does it help - yes, it takes care of a lot of the tedious in-between keyframe adjustments that take so much time. It will probably speed up roto of complex shot by 50-75%.
Is it worth $500? Depends on how much roto you do. If you spend a few hours a week with roto that has more than 2-3 keyframes per shape, then it probably is worth it. Otherwise not.
I do think they would sell at lot more at $125.
I’ll have another test case on Monday. A shot I just did with Sam2 and VitMatte, but which developed an annoying flicker. I’ll have to hand-roto it now. Was going to do it in Flame, but might try SmartRoto for it.
Oh - and the trial is longer than 30 days. It lasts until Oct 15.
PS: Just did that 3rd use case, and it definitely save some time. 8K plate, 38 frames, model holding a product with multiple finger occlusions. 4 supervised frames, filled in the rest in 38s without further adjustment needed. And only the first frame was manually adjusted, the other 3 supervised frames, SmartRoto created and I just verified.
That last case is the ROI test. This is a color job. I originally did Sam2+VitMatte Refinder in BL. It worked quite well across most jobs, and takes less than 20s to setup per shot (could also have been AutoMatte in Flame for arguments sake). But on three shots there was some chatter in a shadow section. So I created a augmented mask in BL and that took 3-4min. But then discovered that there was one frame that chattered differently and I needed a hold-out mask as well. Now I had a complex 3-mask boolean situation. Before long I spent 30+ min trying to address that. Or I could have spent 5-10min in Nuke making this smart roto and exporting a matte.
The ML Matte still wins. But at the first whif that the ML matte is trouble. It’s maybe not worth fixing, but going to an assisted roto tool like this as a known reliable path.
How does it compare to doing roto with Mocha and tracking shapes using proven method where mocha interpolates in between. If its a good track, it should be pretty good. Now with point ML added to the mesh which can used both for tracking and later in adjust panel. And off course mocha also has the Mask ML. Pricing wise its also more bang for the buck compare to this Nuke tool. Any insights how does it compare?
Yes, I was thinking about that too.
The way I understand Mocha’s system is that you start out with a spline on the first frame, and then it’s interpolated from there. If it drifts, you can set a new keyframe with a new spline, but it’s not one continuous spline that’s being key framed. You have independent keyframes and interpolation in-between.
Smart Roto actually works with the original spline the entire time, and just creates pseudo keyframes on that spline.
Somewhat similar result, but the data structures underneath are slightly different. I’ll have to do a more 1:1 test between doing a Mocha ML matte with extra keyframes and SmartRoto.
If you have to pick between them, definitely get Mocha. But I can see SmartRoto being a useful addition.
You can see here that each spline has full keyframes in Nuke. Only 4 of those I manually touched, the rest were all created automatically
Also, it runs on a complex multi-spline roto, not just the resulting shape.
This probably does make it more flexible in difficult situations. Mocha’s system is simpler, and faster if everything goes well. But if you hit a wall, the Mocha wall is harsher I think, because you have to fully pivot, vs. just take over what already exists.
Mocha like most roto tools these days if you tracked well can stabilize the footage and you only work on changing shapes right? So you only put keyframes where shape changes more dramatically and in between ones are interpolated automatically, so you end up with few key (keyframes) and rest is at least from what I can see similar to what I would think Nuke Smart Roto does. It may be assisted by ML but it works similar from the video demo I’ve seen. In Mocha you can link the shapes, you can ripple keyframes across the timeline, so you can adjust for any drift while you are rotoscoping and in the end you can do extra adjustments on top with adjust panel. The entire time you can choose what type of tracking you use, adjust shape, go frame by frame or let it run full speed etc. With point ML and Mesh tracking, including option to refine as you drive the tool, I wonder how much differnt it is in pratical day to day roto. By now Mocha is pretty battle tested and refine toolset.
In Mocha you can do multispline roto and track separately, together, or link to existing track if its in the same ballpark and only adjust shape.There is also auto key for ripple of keyframe adjustment across frames and the Ăśberkey which allows you to offset the positions of control points without destroying their keyframe data.
I still think there’s a difference. I just ran a Mocha Matte ML track. The process seems to be this:
On the first frame you pick your points, and it creates a spline. This spline is built around the edge of a Matte ML model, which returns a pixel render. Which is why you end up with 100s of control points. Boris is very clear that this is a one-frame spline (hence the function to turn that off and track it in traditional methods).
Then when you generate a MatteML, it takes this spline as a mask (similar to points editor with Sam2 in Comfy) to start generating ML matte shapes. Going forward, there is no spline, just updated ML matte shapes which are pixel based. The yellow line you see while doing that is likely just an edge detect. You can stop and set new points if there’s a problem, and the shape regenerates and continues on. But at no point do you have actual spline control points anymore.
The process is really pixel based, very much like Sam2 and VitMatte work in standard ML tools.
Also, you end up with auto generated control points on the first frame, which you have to massively clean up if you ever want to use that spline. And if you need a hold out matte beyond a negative point in the point editor, you have to track that separately ahead of time in a different layer.
In Smart Roto the spline has control points defined by humans. And you can build any hierarchy of spline shapes, with hold-outs, etc. The downside is you have to do that manually. The tool isn’t doing that for you, like Mocha is building the shape. But you do it for one frame only, not all the other frames. It does leave you a lot more control though for the extra sweat on the first frame.
It’s really that one tools is a shape/pixel based approach for ML mattes, and the other one is a spline based approach with ML assisting in the management of the keyframes.
Not that one is better than the other. They’re actually quite complimentary as you solve different situations. In that sense SmartRoto seems like a useful addition to the toolbox. Now we need more mileage to see how good it actually is.
And what I said here about Mocha is for their traditional MatteML tool, not the the mesh tracking and the AI point tracker. That’s a different and very cool new path yet.
Actually what SmartRoto reminds me a lot of is how KeenTools GeoTracker refinement works. You set pins on certain frames and then refine in-between. And if it’s off, you add pins and refine again. Same idea.
I didn’t mean the Matte ML when I was thinking of how does it compare to Nuke Smart Roto, but traditional roto process where Mocha has many refined tools and on the surface seems very similar. Matte ML is very differnt. Its closer to Magic Mask in Resolve/Fusion or Sam2 etc.
I can’t find best demo for it, but something along these lines is how I imagine comparing mocha to smart roto.
I see what you mean.
Yes, in that sense Mocha tracking (with its leading planar tracker) is very similar to Smart Roto. So the question would be - what tracking technology is Smart Roto using for these in-between frames. By all accounts it should be some ML assisted method, rather than a planar or some other tracker. In may in fact be related in technology to the ML point tracker from Mocha, which they haven’t explained on how it actually works.
So for Smart Roto to work better than classic Mocha, whatever tracking tech Smart Roto uses would have to be better than Mocha’s planar tracker. That’s a tall order, but in the age of ML not impossible. And it may be domain dependent.
Well it could be that smart roto is working similar to tools like magic mask, mask ML etc, but uses them to drive the shapes you draw. Like a hybrid.
There is an old article when they were researching this in collaboration with university etc.
I often use combo in similar way. Weather its in Mocha or I’ll just demo fusion version here because I’ve have some screenshots, but its the same idea what you can do in mocha, and possibly smart roto does it internally or something similar.
Basically, I would use ML masking to isolate subject and isolate any occlusions which later can be used as holdouts. For example a guy drinking beer. I would track the glass and hands with ML masking tools and than only track in mocha or with any planar tracker the isolated glass.I track the mask of the glass rather than glass which all the reflections. Mainly to stabilize it, so track need not be super precise for mask, just to stabilize. Than I do my tradional roto work on stabilized frames where I only have to add new keyframes to accommodate for changes in the shape of the glass. Once that is done I only apply matchmove to the roto so it follows the glass, and I apply Mask ML or Magic Mask to fingers as holdout mask. Something like this. Its a good blend of masking tools we didn’t have before and traditional roto and tracking for stabilization. I would imagine that smart roto probably uses masking to draw drive splines the artists made. While it might be ML assisted in that step, Mocha for example seems to be pretty good at doing that old way and now it has power mesh with points being driven by new ML point tracker and power mesh can be used to drive masks for roto but also convert to perspective track for match moving.
Yes, it seems to have lineage to that research. Though that wrapped in 2021, before the big ML boom, more than year before ChatGPT. And the chart of AI based research papers has an exponential curve in the time period in-between.
So while the initial project ended without a product, they have may have had new things to work with. Hard to speculate what’s under the hood.
Has it been that long already? Jeez. Time flies.
I think where I land after this interesting conversation - if you have a Nuke license, give SmartRoto a try for free. It’s definitely worth considering and it truly opens up a different angle in the workflow, so it’s not just another Matte ML tool. It may be useful or not, but you won’t know unless you try it.
I don’t have the license for Nuke, which is in part why I asked to see what is on the market at the moment. I know where was an interesting video at the start of the year, where at least I saw it being announced and demoed for the first time and was curious to see how it is after release.
Roto and Prep in the age of AI | a Visual Effects Society London Event Jan 14, 2026
there is at least this one other thing that gives you splines
there is a
They don’t list pricing on their site. Usually indication that they pricey and don’t want to scare you off, until they can talk to you.
There’s an on-prem solution, or you upload to the cloud. Could be interesting, but hard to judge from their site. I’ll download one of their samples and take a look later today.
Seems it likely might cost more than $500/yr if you use it regularly though.