Buy a second one as a spare.
1 Like
Buy the second one for ceph
Thank you guys so much.
This has been super informative.
I have two servers that are 2U rackmount 12x12TB SAS with a hardware RAID controller, RAID-6, and a 40 GbE NIC. XFS file system. I’ve benchmarked these at over 2 GB/s no problem. Would be happy to sell one or both if anyone is interested, looking to downsize.
1 Like
12tb hdd or 12tb ssds?
What does your software setup look like…Proxmox and VMs?
12 TB HDDs - spinning disks. The boot drives are 2x 960 GB NVMe set up as RAID-1.
I’m actually just running Ubuntu on the bare metal as I don’t have much of a reason for a bunch of VMs at home.
1 Like
i hope you dont mind me contiuning this thread a little… Ive had a fun time creating my next server. i got myself a dell poweredge and added proxmox and two VMs, one ubuntu and one truenas. So the thing im a little stuck on here is: I have built a zfs Storage in proxmox. Do i do this in proxmox or do i create the storage pool inside of truenas? I feel like isnt it better if proxmox held it and so its VM agnostic? then i could choose any OS to share it to my flame and network as opposed to creating it within a VM. It seems a zfs pool cannot be shared via smb/CIFS nfs, but im still learning about directories and how to share it on my network. Anyone has good recommendation on how to proceed here? where is it best to actually create the zfs storage?
here’s what I do:
the proxmox virtual environment (pve) gets a little bit of storage from the boot disk. if i have the correct host, i make a mirrored zfs boot drive.
this is useful for ISO and container templates.
at this point i attach a portable drive (or a mirror) in case my pve blows up. (portable-mirror-zfs).
i store my virtual machines and containers here.
then I attach disks that I pass through from proxmox to my truenas vm.
it’s better to do a disk pass through and let truenas govern the file io.
if you make the zpool with proxmox and then pass it to truenas to govern, you’ve doubled your processor and memory load.
if possible, I pass real network interfaces to proxmox and truenas respectively.
this provides a separation of host, nas, and volumes, which will permit you to move to a failover setup when you’re ready.
I tend to use nfs to mount the truenas storage, but it also works with cifs.
my most reliable results are when i add mount points using /etc/fstab
.
└── datacenter
│
└── my_pve
│
├── 101(truenas_scale)
│ ├── truenas-mirror-pool-1
│ │ ├── home-dirs
│ │ └── SNAPSHOTS
│ │
│ └── truenas-mirror-pool-2
│ ├── PROJEKTS
│ └── RESOURCES
│
├── local(my_pve)
│ ├── ISO Images
│ └── CT Templates
│
├── local-zfs(my_pve)
│
└── portable-mirror-zfs(my_pve)
├── VM Disks
│ └── 101(truenas_scale)
└── CT Volumes
thanks Phil, i have my boot on a mirrored zfs drive for ISO and containers. Id have to read up on the portable mirror drive but if i skip that for a minute i need to follow the dev/disk how tos to get my PCI and drives as passthrough to my truenas. thank you! i knew i needed someone that has actually have this up and running to get a clear answer. Tutorial videos are plentiful on these installs but i wasnt sure what needed to create the pool. I have one windows, one flame linux, one mac laptop and one ubuntu server to share this pool t. id love to stick to nfs sharing but its been tricky in the past to get all systems working consistently, ill try again with truenas, im very hopeful this will be good! Thanks for the direction!
1 Like
For the portable drive, either add some fault tolerant NVMe/usb drive(s), format as zfs from the my_pve disks menu. Then save your vm and ct here.
If the host blows up, this zpool can be mounted on another pve.
New_pve
Same truenas_vm
Same truenas_pool
once you’re happy with your proxmox/truenas configuration you may want to take a look at building a pxeboot server and a freeipa server (amongst many other types of server).
pxeboot is very useful for installing operating systems on real hardware without making USB sticks.
it’s particularly useful if you need to roll forwards or backwards.
the freeipa server will help you to manage identities - you can use it to automatically mount home directories on heterogeneous physical and virtual devices that use heterogenous operating systems.
it works with macOS and linux, but I haven’t tested it with windows since I no longer own any windows licenses.
it’s some way off but I’ve already started LOGIK-NETWORK which is a series of scripts and utilities, designed to automate the basic construction of a network of services for small to medium sized firms.
I’ll open source it in the same way as LOGIK-PROJEKT.
What guide did you use for authenticating macOS to FreeIPA?
Attached is a guide for Windows. I’ve used it, it works.
join-windows-system-to-freeipa-realm-without-active-directory.pdf (3.2 MB)
Hey @ALan - I started off reading this guide
And followed up by re-reading the same notes on this page
Since getting it to work (macOS12 on an older MacBook) I’ve had some success by experimenting with the following:
adding a symbolic link to /Users/$USER from /home/$USER on linux.
and the reverse on macOS but using /etc/synthetic.conf to create the root level symbolic link, e.g.
home Users
In some of the tests I mount truenas.example.com:/mirror-pool/home-dirs to /home on linux and /Users on macOS, but this is very problematic for mac laptops.
you already know tihs, but for those that don’t it’s because the home directory for users is different on macos and linux - freeipa deafults to putting user home directories in /home.
It works nicely for my old mac-pro workstations, all of which are too old to run flame, sadly.