Recently I found out that some on my archives stayed veeeeery long at the “verifying clips to be included” stage. I made many tests and researches, spent hours trying to figure out what was happening. I finally discovered the cause. When starting the archive, Flame creates a compressed tar of the batches and it includes the onnx files when an inference as been used in the setups. It’s not a reference to the inference, it’s the model itself. Event worse, if I have x times the model in the batch setups, it includes x times the model… Even worse, it makes the same for every iteration. So it happens that this “verifying clips to be included” stage that looks as if Flame is doing nothing can last up to 90 minutes before the real archiving process starts.
Is there a way to not include these models in the setups (but only reference link to the model)?
@Flame Dev team, would there be a way to archive theses model only once if they are used multiple times and in multiple iterations? I iterate a lot and my archives not only are extremely long to start, but it also finally wastes a lot of space.
I also discovered that the compressing process uses only 1 CPU core which explains the super long delay.
There must be a solution. I really love using birefnet or its friends in Batch. Such a time saver! I wish there was a lot more flame compliant onnx models to be used!
I’m sure that now Flame team is aware, there will be a fix.
Would take a bit of tinkering, but a better interim solution would be a small python script which keeps the first instance, and replaces all other instances with a soft or hard link. That way your archive remains intact, but doesn’t waste space with duplicates.
I’ve been working on that this Afternoon. I already came up with something. I tested it and it works well, i’ll make it available for the community as soon as possible.
With the community in mind, why not spell out what could be improved and turn it into a learning opportunity rather than just a word game and some expletives?
The risk comes from the starting location of ‘find’ being the current folder. So you have to be mindful of that. But it will only delete .onnx files, so the risk of catastrophic damage is very limited. And it -i in the rm, so it would confirm every delete interactively, which one might tire of.
Keeping in mind that downloaded models are .inf and only inside the batch do they live as .onnx. You can always re-download them.
Small syntax error with the missing \ in ;
Anyone using terminal commands takes a certain amount of risk. That’s how you learn and get better. Even the most experienced engineers occasionally mistype and regret it. This wasn’t typing in DNS change commands for an e-com site, after all. Absolutely worst case you get to practice your Flame install skills.
I once ran a command similar to this to find and delete a specific type of hidden file. I got the syntax wrong, and ended up deleting our whole NAS overnight. Thank god for ZFS Snapshots.
The repeated ONNX copies across iterations sound like the bigger issue here. A 90-minute verification phase makes archiving hard to manage, especially on batches with lots of iterations. It would be useful if Flame treated the model as a shared archive dependency and stored it once, rather than embedding another copy for every setup. The single-core compression also seems worth looking at.
