Flame Linux Hardware - sanity check

This is in your home? I can’t wait for the Randy “Cribs” tour. I’m expecting no less than 3 X-boxes integrated in to furniture and plumbing.

1 Like

Only Fans is coming before Cribs.

Is that because you’ll be in Vegas next week? sounds specific.

Pricing in different locations is tricky, but in the spec listed above, whatever the actual “box”, the majority of the cost is going to be CPU / GPU / RAM / Storage, so IMO there’s a lot of value in going with a supported box like the P620 where you can just INSTALL_DKU and be mostly done (life’s too short to play Linux system integrator unless you enjoy that as a hobby…).

I wouldn’t say I’m in love with the P620: although better than your typical Supermicro box, the build quality doesn’t have quite the “premium” feel of a Dell or HP Z box, and the lack of working remote management on Linux means that you need some kind of external controlled PDU to power cycle the box remotely. But overall it’s a fast and competent Flame box that you shouldn’t have to mess around too much with to get going.

One caveat: not sure about 8.7, but Rocky Linux 8.5 doesn’t have the drivers to control the chassis fan speed on the P620, so if you have a bunch of hot cards in there (for instance a 100GbE network card), you’ll need to crank up the fan in the BIOS to avoid overheating, which can make the system louder than you might desire for a deskside setup. But that’s not unique to that system.

3 Likes

Good to know about fans thanks, hopefully that gets sorted soon but I won’t be running anything too hot that doesn’t have its own cooling.

I’m intrigued to see if 2024 can compete on a fully specced Mac now it’s M native. Probably not.

@ManChicken you were talking about P620 fans the other day?

Yeah - the BIOS fan setting didn’t seem to actually do anything that I could tell, at least anecdotally based on noisiness, from increasing it from 4 to 7. The board presumably takes care of its self based on CPU temperature, but I thought it curious that while doing a bunch of 3D rendering at 100% CPU usage, I never heard anything really spin up that I could tell from the other room.

That said I’ve never heard alarm bells or had a hardware over-temp shutdown or anything, so perhaps I shouldn’t be that bothered about it. There’s a newer BIOS from last month which includes a rather vague note in the changelog:

Optimized the FAN table to support more complex configurations.

so I’m going to update sometime next week when I can and see if anything’s different.

1 Like

Thanks for the advice all. I’ve gone and ordered the P620.

For storage I’m going with a HighPoint SSD7540 and adding 8x4TB of gen4 SSD. In theory 32TB at 28GB/s. I need all the space I can get on this job so going to live dangerously with RAID0.

I was hoping to install 2024 - any reason not to?

2 Likes

The licensing is horrid, or wretched, whatever you prefer. Otherwise there are some good stability improvements with Batch Paint, and we see no down side to using it as current in our productions.

1 Like

At least you spelled wretched right. :slight_smile:

2024 has been good for me in early stages of production. Typically if there was a big thingy we’d have discovered it by now.

1 Like

Hey I know this has been a while ago but what was your experience with the Lenovo in the end?

I’m thinking of buying my first ever big boy machine for home and just debating whether it’s even worth waiting for Apple to scrap their :poop: together with the M4 (which has a rather vague release date, by the way) or just go with tried and tested and grab a Linux - perhaps easier to swap parts too in the future…

Don’t know! Feel free to chip in, people!

all I can say is that after having 3 mac studios in production the best part is that they are just rock solid, they are very stable for us.

Would I like to have a big boy linux machine ? sure.

fits ours needs well, i am usually chasing absolute speed but the amount of trouble i have been having with all these pcs compared to the mac studios is just soo much less.

3 Likes

I agree with @finnjaeger. Unless you enjoy IT tasks, the Mac is an easier to use and operate system. Not that today’s Rocky 8.7 is too complicated, but it is more complicated than MacOS.

The performance is close enough on most tasks. There are still differences on anything ML related (like the updated timewarp, inference nodes, etc.). So if you anticipate using that a lot, minor nod to Linux then. On the flip side if you do have to run Adobe apps every so often for various assets, Mac has a big leg up there.

(Haven been running Flame on a Linux box and on a MacStudio for the last 2+ yrs side-by-side, tend run all my big jobs on the Linux Flame, but stay on the Mac out of convenience for the rest.)

1 Like

I think in terms of maintenance I think the biggest one is system upgrades especially when you use “exotic” hardware. stuff like raid cards and whatnot, I for some reason mostly ended up re-installing my linux boxes from scratch every bigger flame update.

and then also of course other software support, maybe neat video, resolve , nuke or whatever other tool doesnt work on that specific rocky version that ADSK wants …

Yes its gotten better with rocky, but its still a bit of a gap.

I am a pretty solid linux admin and even I would probably pick a mac studio over a linux box if I would go freelance right now as sad as that is. mac studio+ windows “gaming” pcs for all normal apps that dont require a quadro that you pay so much money for would be my ideal one man shop setup.

A linux box with a RTX gpu and gaming CPU would be like less than half price of the mac and there wouldnt even be a comparison in price/performance but I wouldnt be super happy having this as my main machine but I do want one of these as like a ML -Burn node lol

Exactly. I’ve done Unix admin since the early 90s, including coding. But it invariably eats time that’s not billable, or a time consuming issue creeps up when you’re on a client deadline.

If you do consider a Linux system, make sure to get one of the certified configs from ADSK website (P620, etc.). Anything else, ADSK support will wash their hands and you’re on your own, which adds pain points.

However, when you look at the certified configs vs. the price of a Mac Studio, the Mac looks pretty decent :slight_smile:

Everything else being equal, make a MacStudio the first machine you buy. That one you can always keep running or fix quickly. Then if you get lots of work, you can add a Linux machine as your companion for power hungry jobs. But if something happens to your Linux machine while on a deadline, you can switch to your Mac and deal with Linux issues once the project has wrapped and you have some breathing room. Gives you peace of mind.

And not a bad idea two have two systems, even if one isn’t as capable (could also be a beefy MBP).

Nothing worse than being on a deadline, and your only system craps out and you can’t get a replacement quick enough. That’s how I ended up with my current MacStudio.

I was on a deadline for a PBS show. This wasn’t Flame work, but something I was doing on my main Windows system at the time. It developed a hardware instability requiring new parts. I needed to find a solution in < 3 days or we wouldn’t make deadline, and we were coming up on the holidays. A replacement from Puget was at least 2+ weeks out, even with jumping the queue. No other reasonable system was available on a better timeline. Even Apple online was several weeks out. Except B&H was in-stock on MacStudios by pure chance. Next day shipping, switch from Windows to Mac with software and the show goes on… Shit happens. Need to have options.

@Miriam - if you’re in california, rent one fully configured workstation for one week, and a mac for a different week.

Make one side of legal (A4) notes about each.

There is no digestible way to burn money - you either buy a mac which Apple will retire for you, or you will buy a linux workstation.

There are plenty of used options in the linux market.
There are plenty of used GPUs for those used options.
RAM is cheap, storage is cheap, networking is cheap.
Power, AC, noise and footprint are considerations.
If the workstation will live in another room, cables can get very expensive.

If you want something truly miraculous then the latest mac workstations are that - low power consumption, high performance, good thermal management and acoustics, and they make great optical curiosities for friends and neighbors.

But the most important choice that has not yet been discussed is that you actually don’t need any of this stuff any more.

Allocate your money to opex rather than capex.
Use your laptop, a 5G hotspot, and a cloud flame for pay as you go.

1 Like

@Miriam - few weeks late. The Lenovo’s been great, dual booted it with windows as a Blender/Resolve box and it’s been rock solid. That said I recently upgraded Rocky for flame 2025 and couldn’t get the Highpoint SSD RAID drivers installed - this is largely due to Highpoint’s drivers rather than Lenovo but does highlight the increase in technical proficiency required to run a Linux machine. That said Flame still runs best on Linux with Nvidia GPUs, if you’re doing demanding work it’s worth it in the long run.

2 Likes

Hey @Ally_Burnett, are you per chance using HP7110 card? That specific card doesn’t have a driver for Rocky 9.3 yet. We did contact HighPoint in Taiwan about it by the way.

No, a 7540 - it doesn’t have a specific driver for Rocky either but my understanding is it should work on 9.3 - Highpoints instructions and documentation is terrible though so I gave up after 2-3 hours of getting nowhere. Might ask back here when I get back into it.

You don’t have to use any hardware raid on the Highpoints. Every machine I’ve ever plugged the thing into reads the indiividual NVMEs. Plug it in…

For every NVME stick…

sudo fdisk /dev/nvme1n1

g
w

Then

Install ZFS

sudo yum install https://zfsonlinux.org/epel/zfs-release.el8_5.noarch.rpm

sudo yum install -y kernel-devel

sudo yum install -y zfs

Usually then you do this to start the service…

/sbin/modprobe zfs

Make your framestore folder mountpoint

sudo mkdir -p /Volumes/MVP/

Create a sriped RAID O equivalent…

sudo zpool create   -o ashift=12   -o autoexpand=on   -o autoreplace=on   stone /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1 /dev/nvme6n1

sudo zfs set mountpoint=/Volumes/MVP stone

Then choose this as your Framestore in Setup App and Robert is your Mother’s Brother.

5 Likes