ARCHIVING? Cloud/LTO - What is your workflow?

with canister your able able to search the archive directly on the system correct? something built in not just from creating the text file? It talks about catalogs and browsing the tapes without mounting so it seems like it, but they don’t have much for screen shots showing what it looks like.

Yes, they show up as a volume, but that’s only for the tape that’s currently mounted. The TOC files help you to find out what is on tapes that are on the shelf.

But I have a simple spreadsheet that tells me which job is on which tape, and then from there you can just select the folder in on the filesystem once you load it.

1 Like

gotcha, that makes sense.

The spreadsheet is a good idea. that way it can be shared across our offices and others could search for archived projects. appreciate the insight! I think I’m gonna test out the trial and see if it could work.

This simplicity is really appealing.

1 Like

me too

I’m another long time BRU PE user. Ugh, the OWC/Argest take over was a mess; like DCrites, I can’t import older archives either.

Two years ago I moved to Canister. I primarily use LTO for long term archives so the simplicity is fine… it does everything I need it to do. That being said, they’ve added quite a number of new features in just the time I’ve been using it.

These days, I’ve been creating asset catalogs with NeoFinder before I archive and remove the job from my NAS…years ago we used CatDV, but NeoFinder seems a lot faster and does a nice job providing a reference snapshot of everything (clips, graphic files, scripts, etc).

1 Like

The good thing about Canister is the company behind it. Paul Matthijs Lombert started it I want to say 5-8 years ago as a startup, originally created Hedge which was a nice nimble competitor to ShotPutPro and other camera offloading apps that used to be bulky and insanely expensive.

Since then they’ve grown their portfolio nicely and acquired a few other smaller shops, like Mimic and EditReady.

They keep moving at a good clip but a very customer friendly and innovative. What you like in a company that makes useful, not too expensive, not crazy complex workflow tools that just work for the everyday needs.

They acquired ScopeBox too, which I find useful.

Yes, that was part of the EditReady deal, they came from the same developer. I think ScopeBox was just along for the ride though. It had languished for a year or two at that time, and meanwhile OmniScope had come out with a lot of dev velocity and has eclipsed ScopeBox by a mile since. I used to have a ScopeBox license way back, but everyone I know has switched. The usability and additional features in OmniScope make a huge difference. I think there was a LogikLive on OmniScope not long ago.

I tried Amazon for cloud backup and was to much $$$
Switched to Backblaze B2 direct from QNAP app.
12 month retention set. Daily backup - Currently @ 150TB ish = $700 per month.
Also goto LTO8 daily using Archiware that has a full database to browse for restore.

Tapes a must as if you get attacked they can’t get to tapes on a shelf.

The other option - most cloud storage providers have implemented ‘immutable’ storage classes and most backup software can now use that.

When you backup to immutable storage you choose a retention period (e.g. 180 days), and those files cannot be deleted for that time period. This is specifically to avoid ransomware from encrypting or deleting your files.

Of course you have to be careful in using that, because you’ll be paying for those files for whatever period you set. No way to correct a mistake.

But yes, any offline storage on the shelf is safe from cyber attack, whether that is a tape cartridge or a naked HDD in a case.

For those in this thread that have been unable to import old LTO volumes into the BRU/Argest environment since the OWC acquisition, I have found a workaround, but it only works to import an entire tape.
Disable SIP on the Mac if it is enabled. Type csrutil status into the terminal and it will return either enabled or disabled. If it’s enabled, startup the Mac in rescue mode by holding command R after hearing the startup chime. When recovery mode opens up, use the terminal to type csrutil disable and then restart the machine.
Use the Argest backup Import tool to import the tape. It will still throw the “Can’t rename catalog file” error, but let it finish importing anyway. Once imported, open Argest backup and the tape will appear in the catalog, but the archives won’t. Just drag the entire hard drive icon to the restore pane and select your destination. It will successfully restore the entire tape to the destination you set, but into a hidden folder. Use shift-command-period to expose hidden files on the destination drive, and you’re good to go. Everything will be there in its place. The disable SIP step may be unnecessary, you can try doing it without, but I included it here because I did things in this order to be successful. I was hoping that disabling SIP would eliminate the renaming error due to permissions, but it didn’t. My suspicion is that this will work without doing that first, but I wanted to include it just in case.

Thanks for sharing David, that’s good info to keep handy.

I still have BRU running on an old cheese grater Mac for the occasional restore…all those tapes were from my old company, so luckily I only need to restore something about every 4-6 months at this point.

Every time I power it up I just cross my fingers, lol.

The one thing I’ll add is that you don’t have to do the entire tape after all. From the second archive on, on any given tape, the entries will say no catalog found. But if you drag the HD icons as described in my earlier post, it still knows the offset and is able to restore the proper stuff.

I actually wonder how much time, money and ressources have been wasted worldwide by autodesk choice to archive everything uncompressed , imagine all your archives where dwab EXR compressed or prores :pinched_fingers: we could just have our archives on google drive lol.

2 Likes

It’s definitely a major reason why I don’t archive sources and other files inside Flame, but separately. There would be a convenience to have the sources archived alongside the project, rather than having to relink things afterwards. Also less error prone. But the size just doesn’t justify that.

Keep in mind that I generally don’t cache sources since storage is fast enough, and the frame store is on the same NVMEs that the source media are on. So the only benefit would be faster decoding?

Simple test:

Fresh project, UHD, 16-fp. Import a single .mov camera clip from a Panasonic Eva1 - 23.46GB.

Put this one clip on the timeline and archive the project with sources. Which requires sources to be cached in order to be included.

Archive size: 985GB.

Somehow I doubt that the quality of the video increased 42x. It’s just stuffed with a whole lot of zeros. Not only that, it also increases my Framestore usage by 1TB at least temporarily and takes forever.

The problem here really is a system architecture that no longer reflects modern storage and workflows (see all the discussion about unmanaged media, openclip, etc.). We’ve made good progress uncoupling from that, but it has left the archive feature behind.

I think there are options for more efficient caching (the preferred formats during project creation), but since I’m not caching sources until archive they don’t seem to come into play. I tried changing the preferences in the project, but the results were the same.

What really would be desirable for the archive feature to be able to copy any clips from the MediaHub into the archive segments ‘as-is’ and bypassing the media cache if they’re not cached already. That would allow an archive to be created with all the materials of the project. Cached media could be treated as usual, and uncached media would be copied as is (similar to a .tar archive).

You could argue - well you chose to work unmanaged, it’s your job to archive all the other material you didn’t cache. Fair. And for the original camera sources, they are already properly archived. But every project also has all these little other source files, from some GFX, a plate here and there, etc. Being able to archive all clips and files without having to look through each folder makes it less likely that I forget one. And on the few occasions that you do need to restore an archive either because of system failure or because of an urgent last minute client revision - time is of the essence. Having confidence that I could restore an archive and the project would pick up right there without me having to chase an odd .png would be valuable. But there doesn’t seem to be any middle ground for the time being.

Your only two choices are bloat or roll the dice.

1 Like

This is your friendly reminder to upvote this Flame Feedback request:

https://feedback.autodesk.com/project/feedback/view.html?cap=5afe6c845cb3447ab36ccbd7f0688f84&f={8FBADB72-C1B4-44AD-B55E-888E3FDC17E4}&uf={E70B95F8-0989-454A-B763-915AA4A1E5C6}&a=v&t=1

“Archive media with project settings and compression, not as uncompressed.”

-Ted

2 Likes

Done and linked this trhead.

Adding a friendly reminder then when you want to upvote a feature request you have to click the green +1 button in the top left of the screen for your vote to be counted.

Comments are helpful, but I’m assuming the ranking of requests happens via the vote count, not the kind words of support.

1 Like