# OpenZFS crash course

**URL:** <https://forum.logik.tv/t/openzfs-crash-course/2380>\
**Category:** Hardware Questions\
**Created:** [February 13, 2021, 2:24am UTC](https://forum.logik.tv/t/openzfs-crash-course/2380 "2021-02-13T02:24:47Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 2:09pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/21 "2021-02-13T14:09:24Z")

</div>

I’ll have to drag out a laptop so that I can search the pdf for the switch, but I’m pretty certain that it will support link aggregation and it has four ports for rj45 corresponding to your four 10g ports on your two Mac pros

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 2:23pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/22 "2021-02-13T14:23:08Z")

</div>

I’m looking on owc website for the thunder bay 6 to check what NVMe is compatible but I don’t even see the model on the website.  
Perhaps you can drop the support team a note to ask what NVMe is compatible?  
The pdf says that it supports 1xM2.NVMe per enclosure  
The smallest capacity on owc website is 240GB for $55 each.  
A worthwhile upgrade.  
Oh  
Ram is pricey right now - forget the ram upgrade  
It’s a luxury not a necessity

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 2:24pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/23 "2021-02-13T14:24:59Z")

</div>

Neither the cache disks or ram are absolutely necessary immediately.  
One of the good things about openZFS is that you can start with the basics and add the go faster stripes later.

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 13, 2021, 2:27pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/24 "2021-02-13T14:27:37Z")

</div>

Yeah I’m more in the mood for an afternoon and cheap improvement. 👍

> **[owc-thunderbay6-sales-kit.pdf](https://www.owcdigital.com/assets/dealer/slicks/Product_Spec_Sheets/owc-thunderbay6-sales-kit.pdf)**
>
> 1644.65 KB

…says m.2 nvme.

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 2:31pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/25 "2021-02-13T14:31:57Z")

</div>

Great that’s the same pdf that I’m reading here.  
Buy 1x OWC aura 240GB m2.NVMe per enclosure when you want to spring for them.  
They’re $54.99 each right now.  
Probably cheaper if you buy western digital blue from amazon.  
They will act as a write cache which is part accelerator and part security against data loss through power loss.  
You currently have a Mac boot disk per Mac Pro right?  
And a high point raid card with flash drives in it?

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 13, 2021, 2:35pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/26 "2021-02-13T14:35:36Z")

</div>

Alll correct. Standard internal apple boot off the 4tb internal and high point filled with nvmes for the framer store.

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 2:37pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/27 "2021-02-13T14:37:14Z")

</div>

Ok  
Is everything backed up?  
System disk is time machined?  
Flame framestore ready to be wiped?  
Every flame project backed up?  
Reliable archives in place?

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 13, 2021, 2:50pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/28 "2021-02-13T14:50:08Z")

</div>

Framestore is ready to be wiped.  
Sys disk cloning now. Ready in a bit.  
Flame projects backed up.

Is this major surgery?

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 2:51pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/29 "2021-02-13T14:51:56Z")

</div>

Not at all  
It’s very straightforward  
But it’s good practice to make upgrades or side grades reversible

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 3:00pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/30 "2021-02-13T15:00:17Z")

</div>

Make sure you have all the latest security and system updates on the boot disk before you install the openZFS tools.  
It’s now 7am here  
The house is starting to wake  
The dog will require attention soon  
And I’ll head out for a run.

The 30,000 foot view is:  
Back everything up  
Check the backups  
Install openZFS  
Wipe the thunder bays  
Start making disk pairs (Thunder Bay 1 disk 1 & Thunder Bay 2 disk 1)  
Add mirror pairs 1 through 6 to the zpool  
Add the high point raid as a cache.  
Generate some test data  
Copy it into the new zpool  
Bond the Ethernet connections  
Transfer some data over the network

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 13, 2021, 3:01pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/31 "2021-02-13T15:01:41Z")

</div>

same here. lemme mull this over and find a way to dip my toe.

merci monsieur.

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 3:16pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/32 "2021-02-13T15:16:16Z")

</div>

All good  
A fresh system will give you the options you require:

Either:  
SoftRAID (which you can do)  
APFS tiered storage (which you can do)  
Or openZFS (which seems new to you but you’ll soon be a master of it)

HFS or APFS volumes are only good for Mac but an openZFS volume could be attached to a windows machine or Linux or FreeBSD or oracle.

The last consideration is whether or not you want a fixed framestore of NVMe (8TB?) or if you want an 8TB fast cache on a 48TB mirrored frame store.

The only other thing you want to consider is to bond your 10Gb ports to get 20Gb between your flames.

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 3:18pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/33 "2021-02-13T15:18:32Z")

</div>

And you can easily set all these scenarios up in half a day and test them all

First things first  
Make your backups  
Check your backups  
Archive your flame projects  
Archive your flame setups  
Check your flame archives

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 13, 2021, 3:39pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/34 "2021-02-13T15:39:06Z")

</div>

Copy that. Thank you for the generous wisdom. This is an area which I know nothing about. It all sounds cool, and better, but I’m pretty much guaranteed out of my element on this one. Ha! That’s never stopped me before, but, the only thing I know about zfs is watching Linus tech tips talk about how great it is. So, yes, its all new to me and as a serial over-complicator…well, you get the idea.

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 3:57pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/35 "2021-02-13T15:57:46Z")

</div>

Linus is great.

OpenZFS as a concept is less complex than hardware raid.  
It’s less complex than a SAN.

It’s very reliable on Linux.  
It’s reliable on macOS (until apple break it like they break so many things - shake, aperture & QuickTime for example)

The oracle only version is superior, but that’s not openZFS, it’s oracle zfs and expensive.

The optimal expected outcome of this exercise is a a good volume for you to do local and shared flame.

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 5:30pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/36 "2021-02-13T17:30:17Z")

</div>

I’m back and available for a little while.  
The shouty people in the senate chambers are great wallpaper when the volume is muted.  
Is your backup complete?  
Want to teamviewer/FaceTime/phone call?

---

<div class="post-metadata">

**Author:** ![philm](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/philm/32/9286_2.png) [@philm](https://forum.logik.tv/u/philm)\
**Post date:** [February 13, 2021, 8:50pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/37 "2021-02-13T20:50:02Z")

</div>

@randy  
That was a fun phone call.  
Once again, good job maintaining this safe space for nerds.  
Thanks.

Good luck with the zfs experiments.  
Let me know if you want me to post anything more to this list.

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 13, 2021, 9:10pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/38 "2021-02-13T21:10:57Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 14, 2021, 6:14pm UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/39 "2021-02-14T18:14:06Z")

</div>

A post was merged into an existing topic: [SoftRAIDers, how do you SoftRAID?](https://forum.logik.tv/t/softraiders-how-do-you-softraid/2374/45)

---

<div class="post-metadata">

**Author:** ![randy](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.logik.tv/randy/32/6122_2.png) [@randy](https://forum.logik.tv/u/randy)\
**Post date:** [February 14, 2021, 12:33am UTC](https://forum.logik.tv/t/openzfs-crash-course/2380/40 "2021-02-14T00:33:30Z")

</div>

Password:  
pool: macguyver  
state: ONLINE  
config:

```
NAME STATE READ WRITE CKSUM
macguyver ONLINE 0 0 0
  mirror-0 ONLINE 0 0 0
    disk10 ONLINE 0 0 0
    disk4 ONLINE 0 0 0
  mirror-1 ONLINE 0 0 0
    disk11 ONLINE 0 0 0
    disk5 ONLINE 0 0 0
  mirror-2 ONLINE 0 0 0
    disk13 ONLINE 0 0 0
    disk6 ONLINE 0 0 0
  mirror-3 ONLINE 0 0 0
    disk14 ONLINE 0 0 0
    disk7 ONLINE 0 0 0
  mirror-4 ONLINE 0 0 0
    disk15 ONLINE 0 0 0
    disk8 ONLINE 0 0 0
  mirror-5 ONLINE 0 0 0
    disk16 ONLINE 0 0 0
    disk9 ONLINE 0 0 0

```

errors: No known data errors  
randymcentee@dxs-flame-01 ~ %

Well, I got the 2 Thunderbay enclosures into OpenZFS. Took some fiddling with…of all things…Big Sur System Preferences security and the kext nonsense and csrutil and something about the nvram boot something because kext wouldn’t load.

Read speed are solid at 2,000MB/s according to Blackmagic, writes are slower in the mid 400MB/s if I recall.

I’ll write a bunch to it tonight and see what happens next.

The other whacky thing was just dealing with all the disk names, but I finally figured out how to get them all and match them up as individual mirrors.

To add the L2ARC, its as simple as:?

zpool add $poolname cache $diskname

…right? And I’d just add 4 disks which are my NVME disks, right?

[Previous page](https://forum.logik.tv/t/openzfs-crash-course/2380.md?page=1)

[Next page](https://forum.logik.tv/t/openzfs-crash-course/2380.md?page=3)
