Flame project and HUGE ARCHIVES

I always create projects with Cache & Renders as Uncompressed/RAW. I’m beginning to rethink this…

Flame archives are always huge, then I compress them. But tonight I noticed something. When I look at my media (the grade folder, for instance) on the desktop, it’s 22GB. When I look at that same folder in my Flame (as I drag it to Archive), it’s 200GB. This makes sense now that I think about it.

What is your workflow? How can I set up my projects efficiently so as to not lose image quality without knowing what my sources will be when I set it up? When I’m delivering the archive along with the project I feel confident when I cache the media and send one archive. However that’s a huge waste of space and time waiting for the archive and then compressing it. Thanks!

2 Likes

If you have the original media local, there’s an archiving option that excludes caches and renders that could save you a bunch of space. You’d have to archive the original media along with your flame project, but that could probably go on the same LTO, no?

2 Likes

I find that’s the best option for me too. Just archive with cache off and you will see that the flame archive size will drop dramatically.

2 Likes

I’m not sure if it’s still prevalent with the new software, but if you delete hard commits, it used to clear up space. The hard commits added an additional render/hard caching of the elements/data stored inside the segments. ie just take the hard commits of your timeline and it should bring the compression archive lower.

1 Like

Hi,

There are a few points that might help you out.

Firstly, my advice is generally to work with the intermediates that match the resolution. Working at a much higher resolution when not needed only eats at your storage. For example if you have 10-bit ProRes, there is no reason to cache it as an uncompressed or RAW source. The media will still be 10-bit but with a much bigger overhead and much larger file sizes. In certain circumstances, Flame will adjust the cache format if the intermediate choice cannot contain all the data for the source file. So if you cache a 16-bit file into the project set to 10-bit ProRes intermediates, the 16-bit data will be cached in an EXR or RAW format depending on the project settings. This is why you see “Alternative” options when you set the intermediates in the project creation screen. So in terms of cache and renders, no quality should be lost and you’re making the most of your storage space.

When it comes to archiving, the story is very different because we aim to maintain everything as uncompressed RGB data. So when you do an archive estimate, regardless of your intermediate format, it will use calculations of uncompressed RBG to estimate the archive size. This is why your source media may be 20 gig but the archive will be 200gig. Any intermediates including cache and renders will be recached and stored as RAW RGB data.

Honestly, archiving everything uncompressed may not be a necessity these days with the quality of some compressed formats however that is how Flame currently works.

This is where the options to exclude source caching and timeline renders comes in because you can store just the metadata in the archive to keep the files size small and then it is your responsibility to backup the original source media. So you may land up with a 25gig combination of source media and archive as opposed to one archive of 200gig.

Now one point of confusion that I hope to explain in a video one day is more to do with renders. Excluding renders refers to the TimelineFX renders but NOT Batch renders.

TimelineFX renders can be flushed or ignored but you can ALWAYS re-render them as they are part of the metadata of a sequence. So I feel it is safe to exclude these renders knowing I can just render new ones if I restore the archive at a later date.

On the other hand, Batch renders are very different to TimelineFX renders. The rendered Batch clips have no connection to their Batch setup or the originating source media. You cannot un-link, un-render or flush processed Batch clip. It is considered as generated media and nothing more. Sure you have the setup but when you render/process the setup, it will create a new independent clip. This makes a huge point when archiving.

When creating the archive with exclude renders enabled, TimelineFX renders will not be part of the archive as Flame “knows” that they can always be re-created. For media generated as a Batch Render, there is no way to trace it in Flame or to an external source. Therefore Flame will ALWAYS include Batch Renders in the archive regardless whether exclude cache and exclude renders are enabled. Flame will preserve any data in the archive that CANNOT be linked or recreated. There is no choice in the matter.

In summary, you can shrink your archive sizes using exclude source cache and TimelineFX renders but it will be bigger than expected if there are lots of Batch renders in the project.

As always, this is my interpretation of how I was taught about archiving and feel to chime in if you have different experiences :slight_smile:

Grant

16 Likes

Grant, this detailed explanation was a huge help and I very much appreciate you taking the time to write it out this way. I try to keep updating my workflow and this helps a lot!

I’m concerned about creating a project too small for my source files, but it sounds like I don’t need to be? What is your rule of thumb when selecting a Preferred Format for a project? What exactly is the Preferred Format doing? What is the difference between ProRes 4444 and ProRes 4444/RAW? I see slight differences in the Alternate Formats but is there anything else?

I can modify the cache of a project after it has been created; could I change the preferred format to be lower after I’ve started and will that have a beneficial effect on my space?

Thank you!
Renee

1 Like

A post was split to a new topic: Reading Flame archives in older versions

This a great rundown on why archives are much larger than its media!. thanks Grant. The last time i used something other than uncompressed it seemed slower when working on a job, i thought everything was getting reprocessed all the time, but maybe i did not have the exact same res as the source and therefore it wasnt native. I really like the peace of mind using uncompressed because we use a lot of different media on the same project often.

1 Like

Hi Renee,

The meaning of Preferred format is that Flame will first try use this format as your intermediates for source caching, renders, etc. If you are caching/rendering to a resolution greater then the preferred format, Flame will switch to the alternatives which can (EXR/RAW/etc)

My rule of thumb is that the majority of your source media should be able to be “contained” within the preferred format you’ve chosen. Like I said, it wont be a problem if you picked a compressed format and some of the footage is 16-bit. This will force Flame to use the alternative formats which maybe EXR, DPX etc. Visually you wont notice a difference and there should not be any data loss.

The difference between ProRes 4444 and ProRes 4444/RAW is exactly as you’ve described. I believe that if you just use ProRes 4444, then the alternative choice will be EXR but if you use the ProRes 4444/RAW option then the alternative choice will be RAW RGB data. In terms of size, I havent seen anything major to have a preferred choice. Perhaps some feel more comfortable with EXRs as the alternative but its a personal choice.

I have never changed my preferred format in the middle of a project, please give it a try and let me know. I also do not know if it automatically flushes all the cache or you would have to manually recache everything. I would assume that if you go from RAW to ProRes/DNxHD, you may see a size difference. But once again, if you have media that is 16-bit that the preferred format cant handle, the cache source will probably remain at RAW or EXR. As I have said, I have never tried that so I can’t confirm or deny without further testing. :slight_smile:

Thanks
Grant

2 Likes

If all the media is cached the same and it works for you than thats great!

The real piece of mind for me, is if the source media is lost for whatever reason, you have a cached version in Flame and you can carry on working.

Thanks
Grant

archive size can also be greatly reduced if u manage your clips once built. ie consolidating handles for each clip. another huge space hog when archiving is your batch iterations. if you throw away all your old iterations and only keep the latest this can bring down the archive size dramatically

2 Likes

This also seems to be another vote for not caching the media for working, at least for my workflow (home, one machine, Mac Pro). I don’t cache for my jobs but I would cache the media on Archive so it was all contained in the Flame archive. I see now that was a huge waste of time (waiting for the cache and then the Archive), space (making huge flame archives that then needed to be compressed), and management (uploading oversized files to the server for storage). I’m SO grateful I’m having this conversation now so I can streamline my workflow! Thank you Grant!!

1 Like

if you absolutely need to send an archive and depending on how complex it all is , just hard commit everything so it doesn’t have handles, unlink it all and archive with unlinked media, this will lower it down to sometime manageable and sendable, send the original files that are small in size and relink everything on the other side. Only way I found doing it that doesn’t have an insane file size.

I forgot to add I put all the archives in folder with the originals and tar or zip it up and that will compress it further.

Hi so I actually made a flame request about something that this issue is covering, I think this is more of an issue now with more remote workflows and the requests for archives, would be nice to have something similar to AE (though its been ages since I used that) where they can consolidate project material into a folder and send that and the project file so its small in size, I think flame should have something like this where it you can package original material where it would copy it to specific location and you can just archive without the material then when opening the archive just point it to that folder or something so. Here is the flame request number FI-02412

5 Likes

This is exactly the sort of thing I’m after; even without needing to send an archive somewhere this would be a great way to handle the archiving.

1 Like

But why would this be any better? Do you really want to have to manage two separate systems for archives, one for setups and one for media? Wouldn’t it be better if Flame just had an archiving option that was “Archive and compress the shit out of this so I can upload it from home” option?

4 Likes

This thread is incredibly relevant with all the WFH right now (and the future), and one of the most important workflow improvements I hope ADSK bumps up the feature importance list… Bookmarking this thread. Will gladly upvote feature requests anyone has submitted to get more attention to this.

2 Likes

For some workflows it may be better to have an external media directory so there can be multiple iterations of archives without duplicating media. Media can remain in native compressed formats. Media can have the same paths to multiple machines.

Would be helpful if Flame had an option for relative paths within a project as well as absolute paths.

1 Like

@La_Flame
Is there a way just to archive the project with no media?

This would be great when I’m working remotely and I know that everyone has the same source media and we could relink what we need.

When I go to archive excluding renders and cache it still archives media.
I know I can archive just the setups, but I would want the entire workspace structure with no media.

thanks
Stu

1 Like