No, and definitely not at the scale that would warrant their reaction.
Also keep in mind that this was one io module in one data center. Was that data in a storage class without replication? If so, occasional data loss is accepted because you didn’t pay for more.
So it could be LL being cheap and rolling the dice or just making stuff up on a random Tuesday because it helps their narrative.
If you have a single Synology unit, a drive goes bad and the controller botches the rebuild, that’s exactly what you end up with. That’s why critical data are replicated on two independent units in different data centers.
The whole thing just smells like BS.
BTW - I did ask the LL product manager who provided this detail after prompting them some clarifying question about redundancy and if Wasabi violated its SLA. He never answered. Tells you more yet. We’re not dumb.
Regarding cache corruption, much more likely a software bug.
Most apps run file caches by simply replicating files in a cache folder on a disk. The file system handles much of the data integrity logic. The only thing the app would do is decide which copy is most recent to know when and where to copy from. Like rsync for example.
LL uses a different architecture which enables it to get high performance even on small files. It maintains its cache as a large encrypted BLOB that can be transferred in larger chunks. Good idea.
But that means their code is essentially of the same complexity as any other file system logic like zfs with a remote/cloud component added. That is not for the faint of heart and you need your best people on it. Clearly no longer the case for LL Classic.
Easy to see where this ends. And that they need a sacrificial lamb to blame.
A week of production later and we are ordering a 24 bay NVME NAS and a bunch of mac minis now.
I am done with this BS.
Suite client uploading crashes our ISPs WAN connection (i suspect its too brutal for their traffic shaper or something… currently dealing with my ISP support…)
Suite Client crashes our 10Gbit Network card driver on Asus ProArt mainboards when pre-caching (both intel 13th gen and Ryzen 9th gen mainboards). So we had to run and buy a bunch of Intel X550 Cards …
Site-cache via NFS runs our flash based Qnap nas against 100% CPU load (NFSD), we switched to SMB and it goes out of memory instead (64GB ram) and then crashes all connections making it unuseable, so i stopped using it .
Suite client randomly Dies on Linux servers needing a full reboot causing service outages for us.
Suite Clients crash on heavy renders using Houdini on Linux and windows machines.
As nice as all of this sounds, it just doesnt work that well yet under the pressure we put on it it seems, its quiet sad.
So yay, no more cloud storage, will think about some way to 1 way mirror the NAS to dropbox or something for producer remote access or whatever…
Parsec/jumpdesktop is more than fine, thats not i
the issue I csn also spawn a lot of ubuntu desktop machines on my proxmox server so thats definetely OK.
There is however some use for having access to actual files like 100% sure perfect playback even when on a high latency connnection and stuff like that and then its always nice to at least be able to scale to freelancers workstations if we have to. Or you are on a plane or whatever else , need to stay flexible, as we are like 60/40 office/remote
But i can workaround these things way better than having the whole core storage system be a huge mess/pain.
there are a lot of tools like unison that can be easily made into syncing stuff and from lucid/suite instead of using it as a main storage.
I will not rest until ive got this properly sorted, modern, fast and nice.
The reason I suggested the native Mac Remote Desktop is that it allows for drag and drop directly from host to remote with no bullshit extra. Just drag from the finder on the remote machine to the finder on your local box and it’s done… simple as pie. The high performance mode is also pretty impressive on higher bandwidth connections. Plus… free.
Not that I thought for one second that you didn’t already have all of this well under control.
One of the many painful and expensive lessons here is that ‘pret a porter’ can be fraught with unknown / unforeseen problems and can sometimes be a false economy over ‘couture’.
Many people here have discovered the technological shortcomings and failures that were not outlined in the shiny sales brochure.
It’s why the instinctual model should be revered.
@ALan has overcome many of these road bumps, while maintaining a miraculously high level of uptime, and a 100% reliable delivery pipeline, and while I certainly don’t agree with a centralized stone+ wire workflow on any level, I admire the brilliance of this particular solution.
Mostly because it was designed, engineered and tested by one of our best artists, rather than designed, engineered and not tested by someone who never uses flame at all, let alone in production, let alone client-facing.
I digress (surprise).
These problems are not new, and there is no single solution.
At this inflection point my $0.02 suggestion would be to investigate something other than corporations that market NAS devices and begin to look at corporations that manufacture infrastructure devices or at least build a playground with their software.
@finnjaeger probably spends most of his waking hours juggling the same conundrums.
His tenacity is evident in his passion in his posts.
The LucidLinks and Suite IO players are classic tech start-ups that take a basic idea, iterate on it, try to give it mass appeal and scale for their VC overlords. But most are under-cooked ideas, because they still have their diapers on. And because of scale requirements they don’t typically go deep on one audience nor do they know for certain which audience is the bullseye, but try to offer you building blocks that mid-market players can then use to build their own pipelines. They usually come without tech sales and integration engineers, detailed and known tech requirements, etc. That is all up to the user to tinker with - as @finnjaeger has detailed in great detail.
There are other players who use some of the same infrastructure and actually build workflow solutions. At last year’s NYC FUG we had a presentation from Hammerspace. I don’t know much about them other than their presentation. But they seem to fall into the latter bucket of ‘solution provider’ rather than ‘infrastructure provider’. They test, they work out the kinks, they know what you need to succeed. Which may include pricey AWS S3 storage and egress fees.
I remember when I added an Avid Nexis how Avid has resellers that know how successfully deploy them. And Avid is very specific about what network switches have been qualified, what power conditions you need to provide, what networking should look like. Same deal. I ignored all of that, because it would have doubled the price, and I ended up with a disappointing end result. One of my few purchases I much regret.
The problem is - these solutions do actually work as advertised and come with tech sales people that fix it if it doesn’t. But all this comes at a cost. That’s usually an infrastructure tax the people who try LucidLink cannot afford, because they play in a different part of the market themselves.
And so we’re stuck spending late nights and non-billable time seeing if we can jerry-rig the Lucid Links into a workable solution. We’re making up budget with lost hair and sleep. And it’s frustrating. And you get angry at the lack of real interest the Lucid Links of the world take in your frustration and challenges as long as they can do a glitzy 3.0 launch and grab some headlines for their investors. You feel cheated. And you kind of are.
So Phil’s point is right. Try the LucidLinks with a big grain of salt. If it doesn’t smell right, jump ship early, not late. And investigate if there are solutions providers that may be within budget reach. Your own time is better used on billable hours than filling the gaps startups conveniently ignore in their chase for stardom.