Two enclosures - raid 10 openZFS Internal NVMe - L2RC Internal NVMe - ZIL More RAM

I’m back and available for a little while. The shouty people in the senate chambers are great wallpaper when the volume is muted. Is your backup complete? Want to teamviewer/FaceTime/phone call?

@randy That was a fun phone call. Once again, good job maintaining this safe space for nerds. Thanks. Good luck with the zfs experiments. Let me know if you want me to post anything more to this list.

Password: pool: macguyver state: ONLINE config: NAME STATE READ WRITE CKSUM macguyver ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 disk10 ONLINE 0 0 0 disk4 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 disk11 ONLINE 0 0 0 disk5 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 disk13 ONLINE 0 0 0 disk6 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 disk14 ONLINE 0 0 0 disk7 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 disk15 ONLINE 0 0 0 disk8 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 disk16 ONLINE 0 0 0 disk9 ONLINE 0 0 0 errors: No known data errors randymcentee@dxs-flame-01 ~ % Well, I got the 2 Thunderbay enclosures into OpenZFS. Took some fiddling with…of all things…Big Sur System Preferences security and the kext nonsense and csrutil and something about the nvram boot something because kext wouldn’t load. Read speed are solid at 2,000MB/s according to Blackmagic, writes are slower in the mid 400MB/s if I recall. I’ll write a bunch to it tonight and see what happens next. The other whacky thing was just dealing with all the disk names, but I finally figured out how to get them all and match them up as individual mirrors. To add the L2ARC, its as simple as:? zpool add $poolname cache $diskname …right? And I’d just add 4 disks which are my NVME disks, right?

@randy Excellent news! The system specs for each enclosure state 1150MB/s so you’re doing well. Also of note, if you were previously running raid 5 and then copying to another drive you would have Been running at half that speed. So you’ve cheated the system, you’ve got more authoritative parity and your read speeds have doubled. (raid 5 is one parity disk so you’d have had 5x8TB per enclosure and one parity then all backed up again to the second enclosure - now you have 6x8TB in each enclosure working synchronously!) Great job! As for the write speed - yes it’s shit - that’s spinning disks for you - the way to mitigate is to get the m2 NVMe sticks for each enclosure. That way writes get buffered (and as a neat byproduct they organize the data on the spinning disks much better because of the queue control) So two big improvements today. Your writes are probably doubled (or more) and your reads are doubled (or more). So now comes the litmus test: Do you add NVMe as four individual cache devices? (Which is the recommended way) Or do you raid 0 on the high point and use that as one cache? TBH it doesn’t matter either way. If the hardware raid0 fails it doesn’t corrupt your files in the zpool, remember it’s just a detachable cache - like a pressure washer on a garden hose. If zfs cacheing doesn’t seem fast enough then use the hardware RAID. If zfs cacheing is fast enough then use 4x individual sticks and let zfs decide. Or you can try what we talked about on the phone and partition the hardware raid and use part for cache and part for ZIL: Make a hardware raid of all four NVMe of say 7.2TB and use the remaining 800MB for ZIL And remember, you can add and detach cache and log devices in seconds with a single command line so you can experiment quickly. I’m so happy that you’re trying it. You have many options (and all the other ones we talked about by phone) Don’t forget to link aggregate your network connections too. And look out for cheap 40Gb Ethernet NICs on eBay to run point to point between the two macs. Endless weekend fun. How’s the gin?

This conversation has been utterly fascinating.

@randy I sent you a pdf by email of some simple notes that I have used in the past to make openZFS volumes for Mac. I’m unable to upload it from my phone or from the computer where I made it. Would you kindly pop it in this chat for me? Thanks. It’s been fun talking with you about this stuff. I hope Alan can chip in with his experience of openZFS on centos. (Otherwise you’ll be stuck with me gibbering on about halves and terabytes of ram and oracle infiniband)

Ah Actually your write speed has depleted from over 500MB/s That’s disappointing. Your read has only increased by 20% Gin Makes your brain don’t remember properly no more!!! :slight_smile:

So $110 will fix the read speed, or maybe the NVMe partition will fix the read speed.

Today was amazing. Last night was the first time I’ve had the pleasure of connecting with @philm and its as if I was lost in a big city, stumbled into a stranger that turned out to be the mayor and got an incredible tour of OpenZFS. Ill take care of the pdf @philm…I got you. Many, many thanks.

OpenZFS crash course

Flame Hardware Questions

Topic		Replies	Views
RAIDS (which ones do people like?) Hardware Questions	57	4840	May 3, 2023
Highpoint 7505 and Lenovo P620 incompatibility Tools and Tech p620 , highpoint , 7505	41	1788	June 9, 2024
45Drives HomeLab product line Hardware Questions storage , stonewire	34	2362	October 20, 2024
OWC Thunderblade Hardware Questions external-drive	31	2549	July 10, 2023
SMB vs NFS Hardware Questions nfs , smb	34	3785	June 16, 2024

OpenZFS crash course

Related topics