Flame 2024.2.1 … when archiving with no Media options selected, Flame calculates the Archive size to be appropriately small but then proceeds to archive all ~300GB of media that’s there … anyone had the same issue?
Media in the project is cached.
Thanks all!
This is exactly how we do it. Flame 2023.3.1
Works as expected. Not seen this issue. Sorry
Have you used a hard commit somewhere? It’s going to be archived as media no matter what your media settings are set to.
Does the project have bfx, especially as prerender nodes in batch?
or render nodes in batch instead of write nodes? Or commited media like Jason mentioned. That all will drag the cached material for them into the archive.
Hey there Igor,
I’ve helped a ton of people with this same issue. The root cause 99% of the time is “Flame generated media” just as some of the others have hinted at. Even though you’ve specified that you don’t want Flame to include any media, something has got to give with any media that Flame created itself. Flame generated media includes (off the top of my head):
These, combined with any media you have cached, contributes to the size of the archive. I have occasionally also seen timeline renders contributing to the archive size, where a simple “Flush Render” operation will free up that space.
The only time a Batch Render or otherwise Flame-Generated source is not included in the archive is when it is manually unlinked via Media → Unlink.
When I run into this during archiving, I’m often re-rendering my other artists’ / freelancers’ batches with Write nodes instead of Render nodes, and then replacing the media with the render that was written out to the file system. On top of that, I’m hunting down any instance of Hard Commits and just removing them completely. In all, once I know what I’m looking for, it’s a quick process but depending on the complexity of the job, this could be a big deal.
To narrow down the specifics, what I like to do is go into an empty archive, and instead of archiving the whole project, just drag in individual elements, one at a time. Maybe a batch setup, maybe a library, and see how much space it says it’s going to take up. That is your best indicator of diagnosing the problem. The only issue there is sometimes you may have iterations in a batch setup and depending on if you have those set to hidden or not you may not be able to see everything you need to see. The development team has mentioned that the concept of a Flame “media overhaul” is on the radar for the next release or two, which I think will be pretty exciting. But until then, this is what we’ve got from what I can tell!
There really should be an override button to rule them all, to write zero media to the archive, no matter its provenance.
A bit OT, but also being able to archive media in its original codec and not uncompressed DPX would be very welcome.
No hard committed clips.
All BFX on timeline.
Only one rendered clip inside a BFX but it’s a still frame and I trashed the original file and the render node.
Thanks all!
I’d try doing a Flush Renders on your BFX and then do those tests of dragging certain pieces of your project into an empty archive and see how much the estimated size jumps up to. Just remember that if you have multiple Workspaces, those are archived when you do a full project archive but to access that stuff you’ll need to be inside the Workspace.
Adding to this thread:
Exclude renders and cache also applies to openclip.
This is useful in many ways but a good example is if the files feeding your batch group are openclip.
You can cache the openclip sources files to improve responsiveness while working in batch.
These cached files do not get archived.
If you are using openclip on your timeline you can enable caching for all versions so that you can audition and approve any version and retain system/playback responsiveness.
These cached files do not get archived.
If you add a timeline fx on top of your open clip and render, this may or may not get archived.
If you add BFX, especially nested BFX, this may or may not get archived.
In my limited experience of using heroin BFX, anything that has an implied background clip will either generate media for an archive or fail.
“Only archive metadata”
Or just a flame native format to export timelines or bins like resolve… sigh.
we can dream
At this stage it is really only my timelines that I need an archive for
In a pinch:
Duplicate timelines.
Unlink media.
Archive.
Drop the mic’.
this is the way
Absolutely. Other than that I always go with batch iterations saves.
If you want to discover how to archive programatically, you can type this in your shell:
/opt/Autodesk/io/bin/flame_archive --help
This will print out the following information:
Registered thread 'NetMtor' [ 140482508222464 ]
Output log to: '/opt/Autodesk/log/flame_archive.log'.
MP: Using 32 cpus (32 visible)
USAGE: flame_archive <operation> <modifiers> <options>
Examples:
flame_archive -f -F /tmp/archive1 # Format archive file
flame_archive -a -P pj1 -W wksp -L lib1 -E /clip1 -F /tmp/arch1 # Add clip1 to arch1
flame_archive -a -E /projects/TestProject/a38310ac_53a82f6b_0006a1c1/a38310ac_53a82f6b_0006a1c3/a38310ac_53a82f77_000079d7/a38310ac_53a82ff6_000d6100 -F /tmp/arch1 # Add Wiretap to arch1
flame_archive -a monday -P pj1 -F /tmp/arch1 # Archive project prj1 in arch1 under session "monday"
flame_archive -a -P pj1 -B cliplist.txt -F /tmp/arch1 # Batch archive a list of clip to arch1
flame_archive -r -F /tmp/arch1 -S '/Archive\ Session\ -\ 2014\\/03\\/12\ 23:43:40/prj1' # Restore project prj1 from arch1
flame_archive -r -F /tmp/arch -S @<UID> -P pj1 -W wksp -L lib1 -E /fld1/fld2 # Restore clip having <UID> to project prj1 in lib1/fld1/fld2
flame_archive -r -F /tmp/arch -B cliplist.txt -P pj1 -W wksp -L lib1 -E /fld1/fld2 # Batch restore clips found in cliplist.txt to project prj1 in lib1/fld1/fld2
flame_archive -l # List the contents of local host
flame_archive -l -F /tmp/arch1 -u # List arch1 (with UIDs)
flame_archive -l -F /tmp/arch1 -x # List arch1 (in XML)
flame_archive -l -F /tmp/arch1 -S @H_-1518333780_S_..._893796 # List an entry in arch1, given a UID
flame_archive -l -P project1 -E /wksp/Libraries/MyLib/MyFolder # List MyFolder from project1
flame_archive -e -P project1 # Provide Size Estimate for project1
flame_archive -c -F /tmp/archive1 # Convert file archive
Main operation mode:
-a, --archive [session_name]
archive clips to a file archive. The selected entries
are grouped under an archive session. The default
session name is "Archive Session - <date>". This can
be overridden by specifying 'session_name'.
-r, --restore restore clips from a file archive.
-l, --list list dump of archive file or clips.
-f, --format format an archive file.
-e, --estimate estimate space of clips when archiving and restoring.
-c, --convert convert file archive to current format.
Operation modifiers:
To identify an archive (and a specific entry within that archive):
-F, --file <ARCHIVE_FILE_PATH>
-S, --source <PATH_TO_ENTRY_IN_ARCHIVE>
To identify a project (and a specific entry within that project):
-P, --project <PROJECT_NAME>
-W, --workspace <WORKSPACE_NAME>
-L, --library <LIBRARY_NAME>
-E, --entry <ENTRY_PATH_IN_PROJECT> or <WIRETAP_PATH_IN_PROJET>
-B, --batch <PATH_TO_FILE >
When using the -S or -E path option, you can separate the entry names
of a hierarchy with a '/' character. If a name contains a '/', you can
escape the character by using a '\\' in front of the '/' so it does not
get confused with an entry separator. Similarly, if the name contains a space,
the space can be escaped with a single '\'
You can also use the '@' symbol to specify a UID for any entry.
An entry with a matching UID will be searched recursively from
the current path location. A UID may be used at the begining of
the path, for a global search, or after specifying a few entry levels
using the '/' separator, in order to refine the UID search.
Here are some examples of possibile modifiers used to locate an entry,
using a path within an archive or project:
-P pjr1 # project prj1
-P prj1 -W workspace -L mylib # clip library
-P prj1 -E /workspace/myLib # idem
-P prj1 -L mySharedlib # Shared library
-P pro1 -E @H_-15183_U_771516 # recursive search of UID in project
-P pro1 -E /workspace/Desktop/Reels@H_-15183_U_771516 # recursive search of UID in reels
-F /tmp/myArchive # archive
-F /tmp/myArchive -S @H_-15183_U_771516 # recursive search of UID in archive
-P prj1 -E '/Date\ :\ 2015\\/21\\/12/Folder2' # using the '\\' escape sequence
# in front of the date '/' characters and
# a single '\' in front of spaces, for a
When using -B, you must specify a file containing a list of UIDs starting by @. or a
Wiretap path. The entries must be provided one by line.
Options:
-A, --archiveLibrary <PATH>, use PATH as the directory path for the archive library where TOC files are stored
-u, --displayuid with -l, display the UID when listing
-n, --name <NAME> with -f, use NAME as the name of archive when formatting, default is 'Archive'
-t, --comment <TEXT> with -f, use TEXT as the comment when formatting, can be omitted
-i, --capacity <SIZE> with -f, use SIZE for the capacity when formatting an archive, default is 1GB
-N, --normal, active 'Normal' mode, default is 'Compact'
-k, --linked, do not copy uncached sources in archive, default is to copy
-O, --omit <LIST>
<LIST> is a comma-separated list of at least one of the following:
sources : do not include source caches in archive (only relevant with --linked)
renders : do not include TimelineFX renders in archive
maps : do not include maps and ML cache in archive
unused : like 'sources' but applied to unused versions only
-p, --preVerify, verify source data before archiving, if illegal, stop archiving
-y, --postVerify, verify archive data after archiving
-g, --ignore, ignore preVerify failure and continue archiving
-m, --mode <MODE> with -e, one of 'All', 'MetaData', 'Video', 'Audio', for estimation
-x, --xml with -l, list archive file in xml format
-v, --verbose
-T, --toc with -f, -a and -l, the operation will generate the HTML TOC
-b, --background, with -T generate thumbnails in parallel
-V, --volume <VOLUME> use a specific volume
-G, --group <GROUP> with -f, use GROUP as the group of archive when formatting
-h, --help
Stopping managed threads.
I build archives with code snippets like this, where one of my bash scripts is populating the variables or modifying specific strings:
# check for the existence of the archive and create it if necessary
if [ ! -f "$flame_archive_full_name_path" ]; then
# Create a flame file archive (-f) with the following parameters:
/opt/Autodesk/io/bin/flame_archive \
-f \
-F "$flame_archive_full_name_path" \
-A "$flame_archive_full_name_toc_directory" \
-T \
-n "${flame_archive_full_name}" \
-i 25 \
-v
fi
# -G "${flame_archive_gid}" \
followed by:
# check if the archive version is current and update it if necessary
if [ -f "$flame_archive_full_name_path" ]; then
# Update a flame file archive (-f) with the following parameters:
/opt/Autodesk/io/bin/flame_archive \
-c \
-v \
-F "$flame_archive_full_name_path"
fi
# -G "${flame_archive_gid}" \
followed by:
# open the archive and append the project to it
/opt/Autodesk/io/bin/flame_archive \
-a "${flame_project_full_name}_${TODAY}_${NOW}" \
-k \
-g \
-O sources,renders,maps,unused \
-A "$flame_archive_full_name_toc_directory" \
-T \
-b \
-v \
-P "$flame_project_full_name" \
-F "$flame_archive_full_name_path"
The unsung benefit of archiving this way is to include all of the archive metadata in the directory of your job.
That means if you are one of the 6 people who looks at ascii or html archives, all of the the thumbnails etc. are available to browse in the job directory.
You can browse the contents and the thumbnails from any web browser that has access to the job folder.
The metadata is not abandoned in /opt/Autodesk/archive on some flame wrokstation that you just wiped to install a new OS.
You can also get more granular by using /opt/Autodesk/io/bin/flame_archive -a --help in addition to your posted /opt/Autodesk/io/bin/flame_archive --help