Disclaimer: This post contains a link to a 3rd party ML model. This post and its poster do not make any warranties in regards to the model’s legal status and use. Anyone accessing and downloading this link must independently investigate the licensing terms of the model and training data and comply with them on their own.
Adding to the Flame ONNX pool.
A number of versions of the BitRefNet image segmentation model.
Good for general matting for color, or as a core/garbage matte, something that you augment strategically.
Note that this works similar to the Salient keyer in Image. It automatically picks the prominent objects, you cannot tell it where to look. And it will find some objects more easily than others.
3 different flavors, in 2K and 3K resolution (4K was beyond my memory).
Right now in a Dropbox folder. Once a bit more tested will add them to the portal.
Example on ActionVFX footage with a color shift operation.
There are some other examples at the bottom of the Github repo: GitHub - ZhengPeng7/BiRefNet: [CAAI AIR'24] Bilateral Reference for High-Resolution Dichotomous Image Segmentation. You may have seen the ComfyUI examples.
When using the Inference node, make sure to pick the right output. The most detailed matte is in the last output of the node.
6 Likes
Great to see this! Thanks @allklier for posting this and showing your interest in the onnx models…
I’ve already tested the BiRefNet onnx model and didnt post it here because the output of this node is not correct!. Well, the original BiRefNet is way more accurate than this one, maybe it has something to do with the operator used in the conversion process. Please do your own comparisons with the original one here: BiRefNet Demo - a Hugging Face Space by ZhengPeng7
Upload the same image and compare it agains the onnx model.
I’m trying to do some better conversion for this (its been three days working on this) in order to get accurate output since this is not just a core/garbage matte tool.
Edit: Also, where did you get these onnx converted models? did you convert them?
I have no tested the ones you uploaded but from what I can tell it is giving the same output as the one i mentioned above.
This BiRefNet is the new SOTA for matting as of the time writing…
Fair enough.
I’ve written my own code to do the conversion, and have done some comparisons with some of their test images. There are differences based on which ones of their epochs and variations you convert from their .pth files on the G-Drive they published. I used their cat picture (also from their G-Drive) in my comparisons.
It can exhibit very fine detail (like the Cat whiskers) similar to their results if you go to high resolution and use the bottom output of the Inference node.
(dis_590 3K model on the dropbox link)
My comment re: gargbage/core matte was more that I don’t expect any ML model to be more than that. Some are obviously better than others, but I don’t expect in the near term that we can generate final mattes via inference that don’t need additional work. Unless they’re relatively loose mattes for some light color work.
There was a good presentation by Ross Shain at a local meetup earlier this week on this very topic.
But this is a good starting point in order to get a final matte via ViTMatte, you can get a core matte with this, get a trimap out of this and give it to ViTMatte to get final matte. So this will be almost automatic! Have you tried ViTMatte btw?
