Connecting Flame with ComfyUI via OpenClip

[allklier]

In recent months there have been a multitude of ways to connect Flame to ComfyUI.

Here’s one more: Using OpenClips as the interchange - EXR file exchange, versioning, colorspace transform.

Why OpenClips - no software to install on Flame side, can work across systems if you use another system for ComfyUI. Can run independently from each other, and you automatically keep versioned records of everything you did.

The Flame side - Write nodes, and Import nodes. Just how we know them.

The ComfyUI side. A Read node, a version selector node, and (not shown in this screenshot) a ACES 2.0 color space conversion node to bring your Flame shots to Rec709 for AI to handle. You pick the path, you pick the version. It reads EXR files.

And the other end of ComfyUI - a Write node, which writes EXR files.

Metadata will be preserved if you connect the pipe.

And as little bonus, the Write node has a ‘publish’ features, which saves a snapshot of the ComfyUI workflow with the EXR sequence.

If you want to play with it: GitHub - allklier/ComfyUI_OpenClip · GitHub

Since it does pull in opencolorio and openimageio as libraries, it can break some python dependencies with other ComfyUI nodes you have loaded. I did have to do a bit of surgery to make everyone happy in the arena. So not entirely plug-and-play yet.

[CoryJohnson]

This is the way.

Love this idea.

[Ben]

This is nice. What sort of thing are you using comfy for and how good it the colour depth when you get it back?

I’m trying to figure out how I can use comfy but still confidently pass things to the colorist.

[allklier]

I have a considerable number of ComfyUI workflows. Object removal, or swap, I2V animation, in-painting - latest experiment is around fly away hair.

These openclips make integration with Flame easier. Unfortunately it’s not a magic wand around models that have been trained on 8bit material. That limitation exists either way.

But at least with EXR files I don’t add compression artifacts on top of it by using mp4 on the way in or out. Also easier to manage color spaces, and when you version a clip, just re-running everything.

[Ben]

I know it’s a bit of an ask but could you show us a bit of how you use it for a certain model. Maybe cleanup?

[allklier]

You mean the OpenClip nodes here, or ComfyUI in general?

Give me a bit more detail of what you’re looking for and I can definitely share something. I’ll have an example of the OpenClip nodes this Sunday on LogikLive.

[Ben]

A bit of both. So maybe a round trip if possible?

[allklier]

OK, give me a few days and I’ll have some examples.

[Ben]

Thank you

[tomothyhondrox]

This rules. Have you messed around with the LTX 8 bit SDR to 16 bit HDR workflow yet? It’s pretty new but I’ve been impressed at the amount of dynamic range it re-introduces, curious if there’s a way to fold it in here for maximum bit depth.

[allklier]

That’s not the Beeble one, is it? I tried that one, and it was a big disappointment. If this actually recovers details, it’s worth taking a look at.

[mattriley]

https://cloud.mercenaryvfx.com/s/F5WqWxW26KBfXX4

I did a short test with the LTX HDR processing in Comfy and was impressed. My test case was very specific (15 year old Canon 7D H.264 HD source). It had options to render to ACEScg and ACEScc - the ACEScc was more problematic in my test.

[allklier]

Will have to check that out then. Maybe I’ll make that my test case for Sunday.

[tomothyhondrox]

I’m excited to see what you think. I used on a Rec709 stock element the other day and was pretty blown away (no pun intended)

[kyleobley]

@allklier This is super cool, thanks for sharing. Out of curiosity, why not keep the alpha in the source clip and extract that from your reader as opposed to having to whole separate clip? I know the default nodes don’t support this but as you’re writing your own, seems like a nice thing to do to streamline things…if it’s possible, of course.

[allklier]

@kyleobley That mask was just an artifact of my test case. You can put the mask into the main EXR and it totally works.

Here’s an updated test case:

Flame batch. Taking an RED 3Log10 RAW file from Artlist. Doing a bit of downscaling to HD and limiting length.

Then a remove mask in Flame, pretty lose, passed in as alpha channel.

Two versions in the Write Node

v001: written as 3Log10 w/o any color management
v002: view transform to Rec709 before writing

Updated ComfyUI workflow.

Single EXR reader with image and mask. Also including the ACES transform to Rec709. Using the Flame ACES 2.0 config.

In the switch you can pick if you want color transform or direct, depending on if you selected v001 or v002 in the reader.

The Minimax Remover with the mask, and writing it back out to EXR for Flame.

That’s the result in Flame. Left is original clip import, right is EXR return import node.

The mask was to remove the orange balloon on the left. It overachieved and removed the right one too, even though it wasn’t supposed to do that. That’s why you always comp back

Now I did look into the LTX. One caveat, and the same caveat exists for Beeble - those are diffusion models, not discriminative models. So all the pixels will be re-imagined, not just improved. That’s were Beeble totally failed, it hallucinated new content that wasn’t there.

I’m keen to check LTX out, and have built the workflow in the same ComfyUI project, but I’m waiting for another 40GB+ of models to download, so won’t test until later today.

I don’t doubt that output from LTX might be pretty. The question is, can you comp it back over the original plate with good enough results to make it usable.

The alternative is to do an inverse view transform back to 3Log10 and just see where you land (see Flame batch). Of course there isn’t the original precision in it. But at least it’s the same pixels. Might depend on the circumstances, and in some it might be ok, and in others need additional work.

[allklier]

Well, the files downloaded pretty fast, so here is the LTX comparsion.

Added the OpenClip Writer at the end of the LTX processing. In theory we’re getting this in as linear / AWG color, but it’s not an except match. Not clear what the LogC3 references in LTX is, because the post process node definitely gives raw linear data.

Here’s the side-by-side:

Left is the LTX, right is the Rec709 output via EXR.

Currently tagged in Flame as Linear/AWG3. Comes in a bit to dark and too saturated. So a small color correction node that changes gain/gamma and tones down sat a tad.

More work needed to understand the proper color space journey there, so it comes out looking right.

And for reference: This all runs end to end in HD resolution in about 30s for 10 frames.

This workflow is 16fp end to end, with the exception of the ObjectRemove model which is 8 bit. But the overall structure maintains high bit depth as far as I can tell, and only specific models you feed through will degrade quality. More auditing may be required. It’s similar to Flame that Comfy defaults to 32fp, but doesn’t enforce anything, it’s really up to each node.

[allklier]

@Ben This is one example for you. This remove workflow is something I’ve used many times. Except here it’s wrapped around EXR in Flame instead of loading and exporting .mp4

[allklier]

Figured out the LTX color space - odd combination: Linear with Rec709 primaries.

And minor gamma/exposure/sat moves to bring into balance, but now with a lot more headroom.

[Fernandovfx]

@allklier

…he question is, can you comp it back over the original plate with good enough results to make it usable.The alternative is to do an inverse view transform back to 3Log10 and just see where you land (see Flame batch). Of course there isn’t the original precision in it. But at least it’s the same pixels. Might depend on the circumstances, and in some it might be ok, and in others need additional work.

I’m a bit wary of installing anything that might alter Flame’s dependencies; I use the Pinokio AppImage. I couldn’t quite tell from the screenshot—which IC-LoRA loader model node are you using? Thanks.