Backups and Long Term Archives

Flame Flame Questions
1

i need to improve and streamline how our projects get archived and backed up.

I need some advice regarding software solutions for running backups to an LTO drive.

Our two Flame suites have local storage on each.

Currently we have two Synology Raids (one onsite, one offsite) that hold the backups for disaster recovery purposes. These are big, and getting bigger all the time as (until now) we have stored everything we have ever done. Not deleted anything and still have rushes and all files for projects going back 16 years!
The graphics dept also uses this system for a working drive and backup.

The Synology use the Syncovery software to run the backups.

I used to use LTO 5 tape library, but the amount of tapes required to alternate on a weekly basis was getting to much to handle, and I was convinced by our IT dept to go the Synology route. I now realise that was not planned out as thoroughly as it should.

Now I want to evolve to a hybrid, where the Synology raids are used for Disaster recovery purposes, ie, backing up the daily projects and work files, and to look into getting a new LTO drive and using tape for long term storage.

The idea being that I can set up the backup system to no longer store everything forever, but to keep a 30day snapshot rota and a backup type that retains a duplicate of the current drives. Hopefully this will then delete any files no longer on the local drives and help to keep the data load to a more manageable size.

For software I have used Retrospect in the past but found it a little flaky once the backups got too large. Synology is not easy to understand, and after being setup by the install engineer, I have been stumped by some of its settings. Documentation for it seems non existent!

The main question is …What software are you using to run your backups?

Has anyone got alternate strategies for backups that might be more workable?

Thanks for any insight you might be able to offer…
Adam

2

What’s the tall amount of data you’re talking about here? And what’s the total amount of data you anticipate needing to archive every month? Or quarter? Or year? And are you sure the Synology NAS documentation is nonexistent? Because Synologies are pretty much the most well documented NAS ecosystem on the planet. Every YouTuber and its neighbor has a Synology how to video out there.

Do you have a physical local fireproof safe to store these LTO’s? Do you have a budget for multiple copies of a single LTO tape and it’s copies and offsite storage and transportation to and from offsite storage to rotate physical tapes?

3

i mistyped - its the Syncovery software that i found difficult to find documentation for.

We currently have 100tb on the synology’s. Probably adding several more terrabytes per month.
yes we have a firesafe.
yes we are fine getting multiple tapes and offsite storage is ok too.

I looked a Wasabi, as you mentioned it only yesterday! and whilst it looks good for smaller datasets, the cost of keeping our stuff online starts getting expensive. If i have to pull a project back in, i’d rather rely on the speed of a tape drive than the internet.

4

How often do you pull a project back in? What’s the range of project sizes? What internet bandwidth do you have?

5

hardly never! but it can happen.

that depends on whether we store the camera rushes and a Flame Archive or not. I’d prefer to have everything in at least one place. Its not like we can refer back to digibetas anymore.

we have a 100mb up/down connection.

6

Hi Adam,

I have older archives on LTO-5 and have used an LTO-7 for the last few years since it will read the LTO-5s. LTO-7 is the last generation that will recognize 2 generations back. The good news is that storage goes from 1.5 TB per tape to 6TB per tape. The transfer speed is also well more than double the LTO-5. I have used BRU software for years, and it has been bulletproof. However they went under during the pandemic and sold to OWC, and they have re-branded it as Argest back-up. It has it’s own proprietary format, but will also deal with LTFS. I have recently upgraded to the Argest software, and it works both on my archive station, (a 2013 Mac Pro running Mojave), and my M1 MacBook Pro.

2 Likes
7

Personally I went through a day of reckoning and walked away from tape archives completely. Drives last at least several years , and I haven’t had any spot come back in that timeframe. That loud, very slow, tape device from 1962, I don’t see the need.

1 Like
8

We did this for a while with spinning disks and it was great until it wasn’t. Drives last several years as long as you leave them powered on. If they’re off, the heads can (and did) get stuck to the platter. We paid something like 10 grand for data recovery, then switched to LTO and never looked back. It was a long time ago so I don’t remember how long that particular drive had been sitting on the shelf. If you’re doing it to SSD you might be fine? Idk.

9

Yikes.

10

Do spots come back after 2-3 years? Films, I can get it, but commercials?

11

I was bitten as well by a spinning disk, while I was transferring it to LTO no less. Sent it for recovery, and they were unable to do it. Only client data I’ve ever lost. @GPM I can see in your situation dealing with spots how it would be different. I can’t imagine too many spots needing updating a few years after the fact. I’ve found myself needing to restore 10 year old, very large archives, and LTO has yet to let me down. I imagine I’m not the common use case though.

1 Like
12

Wow, yeah. Thankfully my work is forgotten a week after it airs.

1 Like
13

They do in our sector…kids toys.

Whole new generation of kids needing new toys every few years, so I often have to revisit old commercials.

14
2 Likes
15

I think it was a grocery store ad that we paid the big bucks to recover, but things like end tags get pulled back out from time to time and updated, and just last week I was label replacing a job I originally did back in January of 2021. So…not often, but yeah.

16

During the lockdown we restored some 2011 digibeta projects just to get some scenes that just could not had been shot that time.
For normal projects and times it feels like most longterm commercial reworks are coming from in between 4 and 1 years. (At least here in Germany).
These still are rare, like a handfull of campaigns

17

we use yoyotta to LTO on a Symply-dual Lto 1U rack thing and have a pretty straightforward archiving plan.

We also use 2 synologies, one as a working drive and the other offsite, we use synos build in hybrid backup system, (looking into switching to snapshot replication down the line) with the most important stuff also going to synology C2 storage. (office data, project files, not rushes) and a nightly local backup to a external USB HDD for the latest projects.

after a project is done it lives on the “hot” synology storage for a bit but gets moved into a different “archive” share, once that archive share hits 11-12TB we throw everything on 2 LTO 8 Tapes, one goes into my basement in a element-proof safe the other in a safe at work. I also monitor humidity levels and temperature but they are damn constant in the basements, so no issue there.

This seems to work for us, In yoyotta you get searchable archvies/Reports so finding clips is a non issue.

But I also do not keep all the rushes, i usually collect my resolve project and only keep the used source files , if they did extremely long takes I also trimm them. unless the client wants to pay for us to keep their 12tb of out-takes then they can also do that of course :smiley:

Flame archives–> I dont do those anymore. too much of a hassle, to many issues, way too large, i save the batches and the renders myself MANUALLY (one mayor reason I dont use flame on every project, too much overhead when the project is done and sharing stuff across the internet is aweful imho compared to tiny nuke scripts… ) .

1 Like