Hi Mike, Easily understood video. I did this operation several weeks ago, but neither my RAID level nor my storage have changed.when i return to config storage->config physical hard drive, i don't see any more drives. however the new drives are present in Storage->physical drives. Any idea how to fix this?
The display has been working great. There was once about 10 months into ownership that the display backlights flickered in a pattern until I rebooted. It happened after waking my computer from display sleep. I took it in to the Apple Store and they ran a battery of tests on it and couldn’t find anything wrong. They said they also updated the firmware, and that might solve it. I was a bit nervous about it, so I ended up getting AppleCare for the following 2 years, just in case. However, it hasn’t happened in the almost 3 years since. Best theory at what happened was that while the Mac was in display sleep, the power flickered earlier that same day. The monitor was plugged into a UPS, but I’m thinking maybe the power blip was slow enough to get the display into a bad state, but fast enough to not make the UPS switch to battery mode. Hard to say for sure. Otherwise the XDR has treated me well. The Dell 32” 6k display would be a valid consideration these days, because it’s so much less expensive. But it wasn’t out at the time I was purchasing. Even so, I’m glad I ended up with the XDR.
I have a question regarding expanding raid - I have RAID10 on a live server, with 4 disks. I've added 4 exactly the same disks to the server and I would like to expand this virtual disk, but without loosing any data. What are the steps fro this? I see 4 new physical disks under configutaration but they are non-raid. When I choose convert to raid I'm getting information that all raid configuration creabed in OS will be removed - will this break my current raid? as I have live OS there with 2 virtual machines within.
I've read mixed reports, and it likely depends on the version of IDRAC and the model of PERC card you have. Here's a link to a discussion on it for the PERC h730: www.dell.com/community/en/conversations/poweredge-hddscsiraid/expanding-raid-10-on-dell-r730-with-perc-h730p/647f5b7bf4ccf8a8de646da4 It sounds like it might be possible from the PERC CLI interface at boot, but it might not work from the web interface. Depending on how much downtime you can afford, it might be worth trying. Before attempting it, I would update both IDRAC and the PERC firmware to the latest versions available. And make sure you have recent backups, of course.
It changes your redundancy calculations slightly. For instance, you could configure 8 disks as a single raidz2 vdev or two 4 disk raidz1 vdevs. Both setups could have a maximum of 2 disks fail, but with the raidz2 setup you could have any 2 disks fail, while in the raidz1 setup the failed disks would need to be in separate vdevs. In some scenarios you can only have a single disk fail before losing the pool. What you gain by having more vdevs is performance, because you stripe I/O between all vdevs. The two 4 disk raidz1 vdev setup mentioned above would have roughly twice the performance of a single raidz2 with the same 8 disks. The dual raidz1 setup would also be quicker to rebuild, because you only need to resilver 4 disks instead of 8 when replacing a failed drive.
This is so bad design, but I managed to randomly get across the progress bar. Storage->virtual disk->hit the pluss sign and it should say Operational State Reconstructing Progress XX%
You sir are a gentleman and a scholar. I was very near a panic reboot of the host to see if my changes would go through then. I completely missed the rescan of the adapter. You saved me putting this off until we could get an outage window and for that you have my thanks! Will definitely be checking out your other videos!
I’m really happy with the server after using it for years. One of the SSDs died about a year in, but it was part of a RAID mirror so Dell was able to replace it next business day with 0 downtime and no cost to me. I can’t comment much on the noise because it’s in a colo, but during setup it didn’t strike me as particularly loud…
@@mike_pj thanks Mike. I am buying one tomorrow . I ask about the noise because my rack is inside my house and AMD servers are particularly noisy . Cheers
What issue are you having with CHAP? What's your Hypervisor? I run ESXi 6.7, and I have my TrueNAS-SCALE-22.12.1 storage network VLANs locked down with both IP / Network access and using Unidirectional CHAP. Are you setting your CHAP password at the root of your Storage Device, and letting it propagate down to the individual connections?
Thanks, the sandals are Keen Arroyo II. They are really comfortable if you're looking for a wide-width shoe. If you have narrower feet, try the Arroyo III. www.keenfootwear.com/p/M-ARROYO-II.html
Thank you. It is rare yet gratifying to find a video that shows exactly the steps one is trying to accomplish. This video was that for me. I'm on a Dell R640 (2 redundant ones actually) with an integrated H740P Mini and this gave me the confidence I needed to navigate this. I'm using the iDrac 9 Express BTW and it seems to allow everything you are doing as well. Thanks for taking the time to share this. It was completely helpful! And shout out to the person that described viewing the progress in the vdisk detail... that also put me at ease :)
Thanks for your video. I need your help regarding the same issue with us: we are unable to show drives in VM ESXI 6.7.0 after RAID configuration on a Dell server. while inserting a new HDD on the server, our VM is not showing storage. Once we ejected the HDD, it was working fine. The Dell team stated that the ESXI version needed to be upgraded. please help However, we can corelated version with new added HDD.
I actually have a router running pfSense and do the VLAN routing there. The Mikrotik can route packets at wire speed if you don't have filters, but I wanted to make sure some VLANs were isolated and I didn't want to take a performance hit. I ended up posting a couple of videos on my pfSense setup. First was an unboxing here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-_x84qTSfsU0.html and then I posted a follow-up here: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-OBNnVC7FmHU.html
Hey Mike, how has this computer aged over the last 3 years? I'm looking to get an intel mac desktop for development work, and this seems like the best option but there's not much online about how well they've aged.
So this is still one of my main Macs in the office, but I'm not using it as a desktop. I've been running VMware ESXi on it and it's been a pretty good tool for running an half dozen or so virtual machines (a couple with pretty high load). I think it'd still do pretty good as a Mac desktop if you want to stick on Intel. The 2019 MacBook Pro with an i9 offers a little more performance, but the noise is annoying. The Mini is a lot quieter and feels like a better Mac. It's about 60% of the speed of a current Apple Silicon 8/10 core Mac in single core benchmarks. Probably the bigger drawback would be the slower integrated graphics. It was "fast enough" 3 years ago when I stopped using it as a desktop, but I think that might be the first bottleneck you would run into. Then again, with Intel Macs supporting external GPUs, if that does become an issue for you, it'd be pretty easy to correct. If you have any other questions about it, let me know. Good luck!
Love that desk, lots of real-estate and pretty! I also like the labeling you did for the ports, very clean! I hope my Brother label maker can make them that small. I use MikroTik switches in my rack as well, they are an excellent value and don't have to worry about licensing fees. Looking at your channel, I have came across your channel before when you did "Ditching FreeNAS" (which like many I didn't agree with) but see you're back on it. I run TrueNAS on a custom built computer I did back in 2019 but plan to move it to a Dell PowerEdge R730xd. I have a 24U too but mine's a Tripp Lite. I like your channel, hitting right on target with me, subbed!
Thanks! Yeah, while I'd love to have a standing desk, I don't want to give up all the extra surface area of this one. For the labels, I printed a 2 line label and cut it in half. It's a little more difficult to peal the backing, but it got me the size I needed. I've had much better luck with TrueNAS the second time around. I've added an LFF MD-1200 DAS to the rig too, so I have 8 internal SSDs in a mirrored vdev and two RAIDZ2 vdevs in the DAS. While still not as clean as Synology's DSM, It's working really well for me at this point and I use it for all my server and VMware storage. I'm also thinking the R730xd is a good way to go for TrueNAS. If I ever need more than 8 SSDs of storage, I'll probably upgrade to whatever R7x0xd is the best bang for the buck at that point. Having the faster 2.5" media internal and a DAS with the slower 3.5" media makes a lot of sense in my mind.
What kind of internet service is available there? Is the house wired for a network? You'll probably have something like Starlink by the time you retire. Add to the shopping list: Fire extinguishers and bear spray. I bet it will be a pretty smart home when you have some more time. 🧯🐻👀
I’m planning to go into more detail of the networking in a future video. The house was built in ‘86, so it isn’t prewired for anything. Fortunately we are in an area where we can get cable internet, so we have a 200/20mbit connection. I had them drop the cable in a central area of the house, so just a single access point covers the entire living area both inside and out with good speeds in every room.
I boot VMware on that Mac Mini and run between 5 and 10 virtual machines on it. At the time, VMware didn't support the internal SSD, so I installed the OS on the flash drive and boot off of that. Now they have driver support for the internal SSD so fortunately that USB stick went away.
My rack isn't quiet by any means, but I've never heard this switch make much noise. There are fans in it, and I believe they ramp up when the switch is first turned on and booting, but during active use I never really hear them. I only put about 40W of POE load on the switch though, so if you're using more of the 500W of capacity you might start hearing the fans spin up.
@@mike_pj Thank you for your response. My use case with PoE is not crazy. I guess I might use at most around 40W as well. ( some RPi + UniFi AP ) In the video, you used CRS328 switch. In your opinion, what do you think about using it as Router including little firewall feature?
@@gatsbylee2773 I would shy away from using it as any kind of router. I didn't do speed tests to verify, but I've heard that adding any kind of filtering will push you out of the hardware accelerated mode and slow things down. It can do things like VLAN routing okay, assuming you don't need packet filtering.
Hey I have dought. I am planning to make a nas with truenas. I want 4tb storage from my two 2 tb hard disk which raid must I use ? I don't want mirror. because it gives me half the storage if I mirror it. I only use these hard disk to save my old pics and family videos. I am willing to risk it.
If you have 2 disks, the only redundant option is to mirror. If you don't want to mirror, then I would set them up as two separate volumes. That way, if one disk fails, you only lose half of your data. Make sure you have a backup though, even if you mirror. I'd be crushed if I lost my old pictures and family videos.
If you have 2, 2tb, then YOU should get 2 more 2TB drives and have them on zfs Raidz2 You will have total 4tb and 2 drives redundancy so you dont loss anything when 2 drives fails
I just had truenas fail spectacularly losing all the backup data. I thought by boot thumb drive failed, but when I reinstalled on an SSD and tried to connect to my data, it failed again, locking the entire system. If I reinstall and create a new pool, it's fine. All drives and all components test fine, even through Truenas, so my only logical conclusion is that Trunas trashed itself. Not what you want on a critical backup system. Luckly my windows server 2016 system literally built from trashy spare parts still works fine. So I'm probably going to change this system to Windows server. The bright side of this is that it's much easier to administrate, if you have a problem, you can find a solution because someone has already had that problem and when you do ask a question you don't get some self righteous geek asking "why would you want to do that in the first place?".
Unfortunately I purchased this at the wrong time. I'm in a 3-4 month backlog due to the ongoing chip shortages. I don't mind though, I have an RB4011 and plan on using the CRS328 only as a switch. Living in a smallish apartment so I don't have a great deal of devices, aside from a Raspberry Pi cluster, my personal PC and the TV, but it's a great way to save money by investing now for future use.
You'll definitely have some future-proofing there, and especially with the 10Gb ports. Good luck for a shorter wait! There's so much hardware that's hard to get these days…
I hVe 3 different sized hard drives. My question is on TrusNas what is the easiest way to maximize my space on them? Mirroring will eat a lot of space and striping is not recommended. Should I just make 3 different pools or?
Ben's got it there… I'd recommend same size-vdevs in a pool (and same-size disks in a vdev). The other advantage of going with 3 different pools is if you have a disk failure, your other 2 pools are still accessible.
Thanks guys. I bouch an old school matx chassis last year for my NAS. I decided to populate the 2 5.25 bays w a 3 bay hot swap drive. Hence why I asked about it. Thinking about adding an extra ssd for caching.
@@TheBobcat1978 Having an SSD for L2ARC can help in certain situations, but you're better off adding more RAM first if the server will support it. With 48GB of RAM on my NAS, I consistently see cache hit rates above 90% (almost always in the high-90s). If you do go the SSD route for a cache, make sure you get a drive with a high write endurance. I've had cache SSDs in my Synology die in less than a year due to running beyond the rated TBW.
It's pretty quiet relative to everything else in the rack. The only time I can hear it is when the CPU is pegged and the fans ramp up a little. In typical usage it's reasonable.
Unfortunately I don't have experience with iDRAC 8, so I'm not sure why you aren't seeing an option for migration. I'm assuming you've inserted the new disks into the server? Best bet is to search online for iDRAC 8 User Guide. Dell's site should show up 1st in the list.
Wonderful video, very interesting, missionary trip. We need to go in every corner of the World to spread the Gospel. Bet you will never forget that trip ❤❤❤
Hi, I am trying to get a supermicro my budget is about $1000-1500. For Pfsense.. What would you recommend ? I am thinking if I can have possibly SFP28 ports also on it or is it even possible ? Otherwise SFP+10gb at-least four will be perfect !
Many of the Supermicro boxes have PCIe slots, so you can add any network card that is supported by pfSense/FreeBSD, or VMware if you are virtualizing it. However, if you really want to route/filter at 25G, you're going to need to spend a lot more. For instance, Netgate's fastest 1541 firewall appliance will only do L3 forwarding around 18Gbps…and it costs just under $3k. (I recommend checking out their product list to get a feel for how much performance you can expect from different hardware.) If you want to route in pfSense as fast as possible, then you want a box with really high single-threaded performance. That probably means a higher end Epyc or Xeon server. But it might not be worth it over speccing out a router that's just fast enough to handle your edge traffic, and then combining it with a solid L3 switch with SFP28 ports to handle your internal VLAN routing with ACLs.
Looks like I posted just a couple hours too soon! STH just posted about Intel's new Xeon D-2700 and D-1700 CPUs. Skip to page 3 to see the Supermicro SYS-110D-16C-FRN8TP with dual SPF28 ports. Looks like these systems are still a few months out, and I'm not sure if they're fast enough to saturate 25Gbit, but it'd probably be a pretty good platform to build on. www.servethehome.com/welcome-to-the-intel-ice-lake-d-era-with-the-xeon-d-2700-and-d-1700-series/
@@mike_pj Wow Thank you so much Mike for this Break down of things this is really really helpful. I wonder what the pricing will be on the SYS-110D-16C-FRN8TP haha WOW cant believe this was just about an hour ago thank you letting me know about this latest tech... :) But Like you said it be probably upwards of 2k... Sounds like..
@@na1067 I know…timing right!? It looks like a pretty sweet setup. They sell as barebones servers, so maybe it'll come in below $1500 before RAM and storage.
@@mike_pj That be sweet what is the best place you recommend to buy from ? Once they are available to order ? thank you so much for your expertise on it really appreciate a lot thank you!!!
It's a Norco C-24U. I'm not sure Norco is around anymore, but if you come across one I would recommend it. It's a bit too shallow for an R710 (I had to remove the rear door in order to mount those servers), but other than that I have no complaints.
Yeah, the 24mm isn't a bad way to go, but with the crop factor it can still be a bit too tight. I often find myself recording on a 16-35mm to get a "normal" focal length while creating videos.
Great tip labeling the disk. I’m having an issue now I have two sata doms mirrored in OS and I can’t figure out which one is bad. No way to label being internal. Any thoughts?
That's a tough one. :-) You might just need to shutdown the server to find out. Do the internal SATA DOMs have any serial numbers printed on them? If not, you could always mirror a third disk for the boot volume and then guess. All else being equal, I'd think ada0 would be the first SATA DOM slot, and ada1 being the second, if there are markings on the motherboard. Make sure before trying anything to download a backup configuration file from TrueNAS. That way if something goes wrong while trying to replace the bad SATA DOM, you can always just start fresh and re-import the configuration. The storage pool(s) should be left untouched in that situation (but you should always have a backup of that too).
Hi, Great tip labeling the disks. I’m having an issue now using proxmox with two Sata doms mirrored for the OS. There internal no way to label. One is bad and I can’t figure out how to tell which one it is. Any thoughts?
Ugh…that's quite a problem. Unfortunately I don't have anything to suggest. If your motherboard or manual has the slots labeled, I would guess the lowest /dev id might match with the first slot on the MB? But that's not always the case. Hope you were able to get this resolved!
Speaking about compression ratio, ZFS datasets, used to back NFS shares, by default use recordsize of 128KB, that provides plenty of possibilities for data compression. ZFS ZVOLs, used to back iSCSI LUNs, have default volblocksize of 16/32KB in TrueNAS, that may compress data less efficiently, but on other side may be more efficient on random rewrite operations. So it may be a bit unfair to compare space usage of NFS and iSCSI running at the default settings, since those are different. But it may be also just a difference in space reporting, as you have said. I'd look on space usage numbers reported by ZFS pool, since that is what matters at the end of the day.
That's a good call-out…the larger block sizes for datasets might be what's giving the extra efficiency during compression. I wish I had checked the ZFS pool usage before switching to iSCSI. I wasn't expecting a difference in the storage used, so I didn't make a point of checking before starting the migration.
@@mike_pj One more difference is that up to very recent time ZFS on FreeBSD did not support puncturing holes in existing files. So VMware could free space from VM only by deleting its whole disks. For iSCSI it can theoretically be more flexible by using UNMAP if VM guest deleted some large files. I haven't checked, but I guess NFS may also grow that ability in later releases.
Interesting…that would lead me to think that iSCSI should be more efficient with storage instead of less efficient. There's just so many details that come into play when calculating disk usage on modern filesystems that support CoW and snapshotting.
Speaking about write speeds, NFS on the specification level defines several levels of write cache volatility and recovery (replay) procedures in case of NAS crash. Using that VMware NFS client requests all writes to be immediately written to non-volatile storage, and ZFS with sync=standard respects those requests, that that is why it is slow(er). Setting sync=disabled in that case would be roughly equivalent to VMware not setting that flag. I.e. ZFS itself is not in any more danger in case of sync=disabled, only VM data would revert back few seconds in case of NAS crash. iSCSI (and SCSI in general) on the other hand is much less specific about the cache control, since internal disks are not supposed to crash or loose power without the host, and while TrueNAS iSCSI and ZFS respect all existing SCSI cache control primitives, VMware just does not use them. So iSCSI with sync=standard is roughly equivalent to NFS with sync=disabled, and to reach reliability of NFS with sync=standard iSCSI actually needs sync=always, that would also make it slow(er), unless your pool includes very fast SLOG device.
I appreciate the extra insight. So do you use NFS for VM storage then? I agree that I'm giving up some protection by using iSCSI, simply because like you said the connection is emulating DAS and is assumed to be online as long as the host is online. I was torn between the option of having iSCSI or NFS with async writes disabled. What ended up swinging me to iSCSI was the post linked below, where the point of ZFS metadata being written asynchronously could present a problem. I'm not sure if it's really an issue, or if it's more of an issue than the protections that iSCSI gives up, but I leaned that way regardless. www.truenas.com/community/threads/sync-writes-or-why-is-my-esxi-nfs-so-slow-and-why-is-iscsi-faster.12506/
@@mike_pj Some people use NFS for VM storage, and I did it also in some cases, for example, because in case of NFS each VM disk is stored as separate file, that allows to easily restore single VM from ZFS snapshot just by copying respective files from command line. With iSCSI it is more difficult, since there is VMFS in between. But iSCSI is more simple protocol, that makes faster in many scenarios even with sync=always. Speaking about metadata, the phrase from the forum "it also turns off sync writes for ZFS metadata. This may be hazardous to both your VM's and the integrity of your pool and ZFS filesystem." is not true. sync=disabled makes asynchronous only requests sent by user. For example, if user created some file or changed its size or permissions (which are metadata operations), those may be lost in case NAS crash. But pool's internal metadata, such as mentioned "free block table" are always handled same way no matter how sync property is set.
That's good to know. It would make sense that ZFS would write all its internal metadata using as safe a method as possible. I can't imagine the vmdk file-level metadata being written asynchronously would have much affect on stability in the case of a lost connection. Sounds like in most situations it's 6 of one, or half a dozen of the other. It would have probably made sense for me to just stick with NFS and turned on async writes at the time. Now that I'm on iSCSI, I'll probably stay with it. While it's not as convenient, I can rely on VMware's snapshot support for individual VMs.
Thanks a lot for the video, you open my mind on possibility of virtualizing of firewall on dedicated machine which only has networking VMs on it. Can you check how much power it is using? What is the noise level of that server?
Noise is reasonably quiet at idle, but the fans will spin up if you peg the CPU. It's still fairly quiet compared to most 1U servers because it's a low power board. Speaking of power, I would need to unplug it to measure the power usage. I checked the IPMI and it didn't support power monitoring. It showed that 15W was being used for the CPU and 2W for the memory though. I saw one website claim theirs was running at around 30-35W. That sounds about right.
I had the same issues with FreeNAS a few years ago, continuously burning through mirrored boot flash drives until they finally both died at the same time. Definitely a bad recommendation for the time, although they don't recommend it any more.
I'm glad they removed that from the documentation. They also now offer a way to select a pool for logs to be stored on, which is a big improvement from before when logs would just be lost when booting off a USB stick.
