Thank so much for this everyone. Perhaps I’m farther away than I thought from the crossover point.
I use Hedge Canister with my OWC LTO. It’s been working great for me so far and its super simple.
Mike
1 Like
Perfect timing on this Randy. I just brought this up at the office now that I’m on-site a bit more.
Thanks for asking the question I was about to come here to ask.
I’m trying to fall in love with the idea of LTO y’all. But in my situation, I don’t think it’s making much sense for my weird world.
First, I’m traveling more than I have ever and spend several months of the year away from home shooting mayo ads in non-exotic locations.
Second, the data I create doesn’t really belong to me, does it. It was shot on someone else’s camera, for someone else’s mayo client, for someone else’s mayo ad. My company’s name is only on the slate because its a faster to tell me the they need the broadcast mix on the web spot because WHO CARES. So why ought I to be saddled with this literal tech debt?
Third, I’m fortunate enough to have heard several anecdotal horror stories of MAJORCOMMERCIALVFXSTUDIO#11 spending 7 figures on LTO backup strategies that could be a studio-ending-event given the right combination of client and project and fuckup.
Fourth, my clients are more connected than ever. They’re likely on Lucid Link or maybe rockin’ the solid Nextcloud life or already Teraguccing into their on-prem or their cloud boxes. So, having a little tape in my basement or in someone else’s basement that I need to spend $6k for the privilege of writing to, twice, and manage, and inventory, and ask my technically minded teen to sift through the barcodes and chuck a tape into my loader when I’m in Mexico City struggling to tell the 1st AD I need a clean plate for the sour cream dollop shot bite and smile in my broken Spanglish, isn’t ideal.
For just a few thousand bucks, I can roll a decent upgrade to my local storage capacity and keep most projects around for a few months. Building in a small storage line item on the bids will make that work financially for me, no problem. Then I’ll slow roll it up to an S3 bucket with a DEEP_ARCHIVE storage class argument and, according to my calculations and verified with some test data, about $1.50/TB/month plus a few bucks, give or take for the number of objects as an up front cost, let it rot. If I need it back, that’s a billable event which likely will cost $95-$115 per TB for maybe 12-48 hours to be retrieved from Deep Archive to a S3 Standard storage class tier, and downloaded.
Or if they need the data and don’t want my services, I’ll hand them a key and they can pay for the restore.
And if they don’t need the data, it will self destruct after X months. Meanwhile I’ll be paying $1.50 per month or so per terabyte. Hopefully.
Why should I be subsiding our clients squatting on our storage? LTO or spinning rust?
1 Like
preach!
Joanne archives to a removable drive, bills it to the client and sends it to them at the end of the job. She keeps a copy for herself for a few months on a 16tb raid. It depends on who you are working for. Are you working for an editorial company? It’s on them to keep it in perpetuity. Are you working for a studio? Same thing. Is your client an agency or are you working direct? It might get fuzzier. You can send the archive to the agency, but they may not know what to do with it.
In my case I work for the editorial company as staff, so it’s important that I have a process that allows long term storage, fast retrieval and minimum dollars spent. Something I no longer do is keep stuff “forever.” If they haven’t needed it in 4 years, it’s history. I glean what I want for reels and such, but that’s about it.
Besides, you spend more than enough just for internet. You shouldn’t be the vault for someone who is capable of handling it themselves.
3 Likes
Yes. And other weird combinations of above.
It’s all a bit fuzzy and messy.
Sounds like for you and yours you’ve got it dialed.
the…
future…
mic drop…
LTO is a lovely but quirky technology with a niche application. It generally is meant for data centers and big scale. And in that case you can use gigantic robotic libraries that hold 200+ tapes unattended, have 4-8 drives they can feed for parallel operation, etc. etc. You have specialized software and people that know it inside and out.
It’s at that scale that the cost math gives tape a significant leg up over spinning disks.
For the rest of us, it’s something that can save a few bucks, but probably gets even by the complexity of it. So we do it mostly out of curiosity and because we can, not because we must.
The standalone tape drives and tools like YoYotta exist primarily for DITs, because you can dump multiple copies of the camera cards on set in parallel, send one off to the cave for safety, and one or two to the AEs for ingest. Again, mostly for large operations that go through gobs of data on set and may not have optimal cloud connectivity. You can see that clearly in the software design of both YoYotta and Canister. At scale LTO has better performance characteristics on set. My LTO8 drive can write at 450MB/s - that’s better than HDD unless you do RAID 0 which gets complicated for transit drives. NVMe is faster, but still pretty expensive at scale.
When you’re on the road like you describe, a sensible cloud and disk based strategy seems a better fit, and you can bill for it presumably. You may make the long-term archive their problem and wash your hands, but maybe keep a copy yourself you don’t tell them about for the invariable fuck-up that will happen.
Whether it’s cloud based storage or storage in your basement is a separate question. While most folks opt for cloud in any scenario where they may not be near home/office, it still remains an expensive option. Yes, you can go deep archive, etc. But the SLAs require consideration.
The other option you have - keep the primary storage in your basement. Just build out some rack mount NAS. Then you can rely on tools like Resilio or SyncThing which can sync folders at great speed without passing through cloud storage as they’re P2P. There are some firewall considerations, but if you have enough control you can overcome them. So while on the road, just setup a sync to the required folder, go for dinner and by the time you’re back you can play hero.
And regardless of what solution you use - any storage is only as good as your chance of retrieving the data when you need them. So practice and exercise restore every so often. Whether that is LTO, cloud, or a rack in your basement.
We had an old joke HP. The tape and magneto optical product line was called ‘HP SureStore’, interestingly enough it wasn’t named ‘HP SureRetrieve’. Everyone can store data. Not everyone can retrieve data when and where they are needed.
The absolute fastest way to store data is with this command, unless you care about retrieve
mv importantdata.mov /dev/null
1 Like
For your basement solution - I’d look along these lines:
RackStation® RS4021xs+ | Synology Inc. (since you’re a fan of Synology)
Or work with the Puget guys for configuring a good solution
Or just go for a gigantic DAS Array attached to P620 running Rocky instead of NAS infrastructure. This wouldn’t be SAN and the cost involved in it. Just big scale single server storage.
1 Like
We use Archiware P5 Archive.
We run it off a Mac mini that’s also our Resolve server.
It can run off a Synology system if you connect a LTO or Library to the NAS.
You can also use it for data sync, cloud connections, live backups etc. But we just us the Archive module.
We switched to Canister but moved back to P5. It’s a bit more of a pain to set up than Canister and its ugly. But the way it databases everything works better for us. It’s useful if you need to just bring back Toothpaste_logo_v4_final_final3.png. It tells you the tape then sorts it out.
1 Like
2 Likes
2 Likes
The interesting question now would be how much less tape would we use if Hollywood stayed at 4K resolution or maybe even HD?
1 Like
But it balances out when you consider that we don’t use video tape any more. And even when we were SD only we used way more video tape than we do LTO’s.
I’m onboard with this. I remember FLAME archives as barcoded DigiBeta tapes. Once I realised this we bought our LTO. When we left Deluxe I thought we’re in the physical free future, then I did some maths.
Also I’m in Australia so we’re not served as reliably by cloud, that UniSuper (pension fund) isn’t the first US company that forgot about our little country and turned something off.
2 Likes
Use case for LTO (or more so keeping older files)…
Just got an email from one of our main clients. Intern found that we attached the wrong .srt file to an edit we did in July of '22. Long gone from our RAID. But I was able to retrieve the correct file from LTO (5 tapes back from the current one in our archive series). Saved us from having to redo the .srt file.
Client happy.
2 Likes
@allklier - That’s a great anecdotal confirmation that your backup strategy is functional and useful.
(And you’re absolutely correct that it’s not really a justification for LTO over file stubbing and some other kind of tiered storage)
1 Like
Nice!!!
fwiw we use symply LTO drives (they are a very friendly company) we have their dual-drive rack thing .
we started with yoyotta and then also switched to hedge canister.
You want a dedicated machine for either, it has quiet heavy ltfs drivers and stuff you want to controll the macos version and blah blah, we have a dedicated m1 mini and archive straight off our NAS.
we are pretty happy.
they wont be as comfortable as a hdd, but its a lot cheaper.