What Is ZFS?: A Brief Primer 

I love how a 30+ minute video is a brief primer on this channel. No, I'm not being sarcastic, I actually love that.

I'm an engineer with almost 30 years experience in the storage industry at a low level (the bottom 3 or four turtles in the stack you alluded to) with other Sun Microsystems engineers who worked on most things you're talking about, including the full fat ZFS. I properly enjoyed your video and really like the fact that you mentioned that computers barely work. Storage reliability is one of the most complex computing problems out there in a world obsessed with the higher parts of the IT stack assuming everything Just Works (tm). Latency, throughput and reliability quite often can be mutually exclusive practices. When you test for any issues and see some race conditions in the lab every 6 months... you can bet your customer base (depending on its size) will see that kind of issue in 6 days. Storage is Hard, like... properly hard. :) Thanks for your video. It brought me back to why I do this.

Excellent, this comes up literally the same day I buy three 6TB drives for my first ZFS setup. :D

Oracle acquiring Sun is classic example of some evil corp buying a really good company and destroy it.

Wendell:"Brief primer" Me: "Really? Brief?" Wendell: "30 minutes" Me: "Oh, ok! That's very brief for ZFS!"

Time to take notes!

I’m so glad people like you are working so hard to educate us; I owe you. If you are ever in London, my house is your house.

I love ZFS, it is the best FS I've used, at work and at home. I really like the snapshot functionality

It's pronounced ZFS not ZFS.

I use ZFS everyday, we use it on most of our storage devices, and I'm still learning new stuff about it on a regular basis. Sun really did some great work.

As a viewer from the overseas. I appreciate your Aloominum vs. Al-u-min-i-um - I mean Zed vs. Zee distinction.

Thank you for this. Having someone like you explain it is so much more digestible than reading a ton of technical documents.. Would love to see a video like this for *XFS* ! But if you don't want to that's totally understandable.

2:50 oh the days we've had SUN-BLADE workstations with Solaris OS , SPARC processors and the rest .... such nice systems to work on ... some still survive till today (almost 20 years later)... only a few of us original IT guys know how to use and administer them so their days are numbered (in preeeeeeetty small number).

Good stuff Wendell. For months I poured over reviews of raid cards to run a raid 5 array. Then I was introduced to FreeNAS and ZFS, and when I found out all was needed was a decent hba, I was sold. ZFS is crucial for data protection.

I enjoy these types of videos way more than tech news on the other channel, I would really like if the focus would shift towards teaching material such as this!

I'd love a video on ZFS tuning. So far, all articles etc. on this topic have been a complete mystery to me, the documentation is just not very helpful to me in this regard. It only talks about hundreds of tunables without any comprehensive explanation (that a person without a deep understanding of the source code could understand) what they mean.

This is probably the best ZFS explanation I've ever encountered in my life.

Thank you Wendell, this is one of the best introduction to ZFS, it's clear and concise, it's easy to understand from a beginner's perspective. Would love to see more of this type of video!

Wow - thanks for the shot at 13:00 of your old offices. It made me realize how long I've actually been watching your videos! Boy, how time flies...thanks, Wendell!

Thanks for this great fireside chat Wendel, you are the best

I started using ZFS a few days ago, and I was really blown away at how incredibly simple it was. I wanted to configure four identical disks in RAID 10, and this was the only command I had to use: $ sudo zpool create NAME mirror VDEV1 VDEV2 mirror VDEV3 VDEV4 And that was it. No need for formatting or lenghty initialization processes. It just worked right away, and performance is excellent. I'm sure ZFS can get really complicated for more intricate setups with caching and tiering and tuning, but for my simple use case it was refreshing to see how easy it was.

2 years later--ZFS is still rock solid and I have to say the only time I have lost data, is when I tried to use the command line to move data instead of TrueNas native data transfer. Probably error on my part. I understand more now about ZFS, and love freeBSD which natively supports it and so I have not done anything with BTRFS. Great Informative video !

OMG, PBS couldn't have done a better job at explaining this. I was with DEC back when ... This was the type of information that was available daily. It's great to see it again. Hit SUBSCRIBE, "YES"

Excellent high level video. Just migrated from a Synology to my home build Ubuntu, and needed to understand what the ZFS hype was all about.

I live in a country with 2 to 14 power-fails/week and I had a lot of garbled music files, so I'm very happy with ZFS and its Copy On Write. I even have my dataset with music stored with copies=2, basically introducing a mirror for that dataset in an otherwise striped datapool.

Great high-level overview video Wendell! I wonder if you do a next installment could you fit unRAID (XFS/BTRFS) in there with a similar approach? Like basic overview, differences, applications?

Thank you, chill background music.I enjoyed this.

17:55 *Hard drive can be a silent carrier: ASYMPTOMATIC !!!!* LOLOLOLOLOLOLOLOLO +1

Would love to hear a comparison between ZFS, WAFL, and Nimble CASL

Awesome video and explanation of ZFS. Now I have the bug to want to learn more. Thanks so much. 👍

Im amazed on how simple you manage to explain this complex topic. Thanks for all the effort you put in to your videos! Btw: I think its pronounced ZFS ..

you said asymptomatic in such a way I just knew this was uploaded this year

"Well, that machine caught on fire-" Me: **LP0 ON FIRE!**

Love ZFS as a gamer. I took my gaming laptop with two ~500GB SATA SSDs, created a mirrored bpool, and then created a striped rpool. Of course, I recommend no one run this way, but it’s fast. Also, with built in compression, potential exemption of datasets from snapshots, and dataset level case sensitivity, I can create separate sets for flatpak, Steam, and Lutris that are fully under my control with increased operability of case-insensitive programs :) No idea how to use it on a server level, but as an insane home user, I love it.

You have a good voice for explaining. Kudos.

This is so informative and well-explained! Respect man!

This was so incredibly helpful and educational. Thank you so much!

"Brief" 31:49 Wendell!

Thanks! Been doing flashes of OpenBSD & Ubuntu, both of which offer 'experimenting' with ZFS. Gonna try them out today

While other people go for 19:9 video aspect for phone users, you are going the other direction (1.66:1). Just shows how special this channel is :)

I used to go to a swap meet in CA with my dad on weekends where he would buy things like 8" floppy drives and huge 68000 cpu-based servers that were basically rescued from a landfill. Two or three weekends later he would triumphantly show me a working command line on an amber monochrome monitor. I miss my dad.

TLDR: Basic Raid: If a file corrupts, even when there is a non corrupted file on a another drive, you can't get it (unless you send the drives to recovery) ZFS: automatically gets the non corrupted file from the mirror device Also ZFS hasn't caught upto NVME yet (@24:22), to be fair RAID is an array of *"Inexpensive"* drives

I put butter on everything, just not my file system. BTRfs is catchy, like you can't say "I can't belive its not ZFS" it doesn't roll off the tounge

love ZFS in FreeNas/FreeBSD. when BTRFS gets more reliable I will learn and use it. Thanks for the video W. :)

Taciturn is the perfect word to describe the speaker in this video! We are all wondering, when will he come out of his shell and speak more at length?

Checksums give you 1 bit of information - whether or not there is an error somewhere. Error-correcting codes tell you where the error is. With binary data, when you know where the error is, you can invert it - therefore removing the error.

I mentioned this on your other channel, but I've been watching your videos since you had your last channel, hiding behind your mic. Love your tutorial videos. You guys need to quit hammering old people over there. I've built two Freenas servers, a Linux system with GPU passthrough, a PFSense router and many other projects. I'm not as computer illiterate as Ryan's parents.

Love your videos, Wendell! Thanks!

Thanks Wendell, I love these, now when is the next one? ;-)

Thank you for this video, you did a good job dimystifying zfs

12:29 It’s a *proprietary* computer on a card. And an inflexible one at that. Which is why I try to avoid hardware RAID.

In a round about way ZFS was a pioneer in software defined storage where managing storage at a low level went from hardware to software. You now see the same principles applied in many other applications and technologies like HCI for instance.

Ha! I did the raid 5 recovery last week, that was fun.

Here I am almost a year later watching this video. Why? I bought a QNap NAS to play with, and I'll eventually buy another one to back up to. Thank you once again Wendell for putting out these, and I'm now subbed to this channel in addition to the L1T channel. :)

Pro tip: you don't need the music in the background. If you really feel you need it use it for cut scenes in between segments instead 😀

Well, now that I have absorbed the news that there is no data santa, I feel I should thank you for this new relationship with reality. Though life was so much cosier wallowing in my ignorance. Seriously, thank you.

Love your work my friend! thank you so much for your splendid explanation

Cool, that corrupted picture you used was of an FRC competition from 2015 if I remember correctly. Winnovation was one of our regional teams when I was in high school.

It sounds like ZFS works best with a large number of slow storage drives. Could you use like … 48 micro SD cards with ZFS to create one really nice and reliable total storage area? (Not sure what the technical name for a combined virtual storage drive is. Is there a generic term, or does it change from file system to file system?) Actually, how would you use an silly number of micro SD cards as storage for a single computer?

Very well explained, insightful video. Thank you for this.

That background music.... 🤔 You shoud totally do a spinoff where you explain technology with your hair combed way back, wearing a maroon robe holding a pipe in one hand and bourbon in the other.

Any hope for more enterprise RU-vid videos?

It's funny how people often say "bit rot is a myth"... and many of those people are just out of highschool, basing their anecdotes on a single decade of experience with relatively high quality storage.

I have a 12 bay ZFS file server appliance, and i have experienced bit rot, but thanks to ZFS Z3 i have never actually suffered a problem do to it, and it fixed the problem, if i get more than one on a drive, I'll do a scrub to make sure no other drives have corruptions, then swap the drive with a replacement

Fun fact: Sun Microsystems basically reimplemented NetApp‘s WAFL

Great video ofc, but would love to have a similar video about btrfs

Thank you for the excellent video. I am considering switching from Windows 11 to TrueNAS for my home file backup server.

Makes me proud to have had an X4150 back in middle school :P

Thanks. I like this format and appreciate your videos. I'd watch a 30min on XFS as well if you end up doing some other file systems! I've been running UNRAID servers for many years and would like to peek under the hood a bit more.

Thank you so much. Eye opener!

OMG FINALLY, BEEN WAITING FOR THIS ALL MY LIFE, NOW PLS COMPARE IT WITH WHAT "OPEN MEDIA VAULT" USES, I NEED TO KNOW WHICH IS BETTER !!!!

Obviously ZFS is more mature than BTRFS, because you can’t tell me that whoever came up with the name wasn’t thinking “butterface” as in a lady with a nice body, but an ugly face-which isn’t exactly the worst description of BTRFS as a file system…

Thanks Wendell, great as always! If possible I'd like to see a LVM deep dive as I'm not yet understanding it totally. I always thought LVM would be unnecessary but with what you now said, it makes much more sense now. Also have you or anyone else tried out a recent version of bcachefs?

22:30 You can always se the "copies" property to e.g. 2. By doing so you're telling the filesystem that it should store every file within the given dataset 2 times, on 2 different physical location on the disk. So if one of them is on a location with full of bad sectors, the fs will still be able to retrieve the file from the other location.

Thanks man, this was both informative and your voice is like unix ASMR :3

More zfs videos. Can you make a gluster plus zfs video? Or another scalable zfs solution

Might be more a level1linux flavor video, but can you do another deep dive like this explaining CEPH?

The awareness that spinning rust was horrifically unreliable isn't new. There have been numerous filesystems that tried (and failed) to address this. The issues that held back development of anything like ZFS for many years was a) developing a robust filesystem takes at least 5 years to get all the bugs out b) NetApp were famously protective of the patents around WAFL

8:21 drives. ZFS START: 17:41

Make a video about linux kernel CPU scheduler and I/O scheduler, that would be interesting

Great video. Thank you

Excellent overview! Thanks :)

Maxtor actually sped up their spinning rust by enabling ALL read heads, not just one. 8 platters increase the read and write speed 16 times. And caching an entire cylinder reduces latency time.

I recall looking at Sun workstations in 87 and early SparcStations in 89-90, they started around $5K I think and were out of reach for me. Fast forward to first dot-com boom and man you should have seen what people spent on Sun servers.

Great insights. Have you done vid on how long solid state storage lasts versus hard drive (spinning rust)?

Fountain of knowledge as always. Great explanation Mr Wendell. Not that I understood much of it :-)

Whoops. I thought I was already subscribed. I quite enjoy your personality.

ZFS is so much better than the filesystems of old that It's not even funny. Sure, It's not the single fastest or most efficient but It's no doubt the most intelligent, reliable and useful storage solution you can easily setup on your desktop or server today. It does use more resources than say ext4 but the benefits are well worth the cost in RAM and CPU time. That being said, It can also be set up to run in a resource constrained environment without much hassle. You should also have mentioned copy-on-write, snapshots, clones, zvols, compression, encryption and the seemingly endless number of configuration options available. Learn about ZFS and use it, It will solve problems you did not know you had.

I'm interested in how the Windows port of ZFS turns out. Recently I've been tinkering with the next best thing (kind of) - exporting a ZFS ZVOL on FreeBSD as an iSCSI target and mounting it on a Windows PC. Effectively NTFS on top of ZFS. Surprisingly easy to set up, too.

Again, imagine just talking with Wendell for an entire day.

What I am interested about is actually the development of bcachefs. It's built on top of bcache but (oversimplified) want to be a ZFS with the performance of an ext4. It's pretty young (with few developer(s)) and isn't mainlined, but can be mainlined into the Linux kernel, is actually stable (compared to btrfs) and a pretty "clean" (for a FS) codebase. Definitely something we should keep an eye on.

Regarding scrub speed - 24 drive storage: CentOS 7, OpenZFS 0.8.4-1.el7, 1 pool, 1 drive pool global spare, 2 vdevs raidz2 (11 and 12 diskas), 90TiB of data ~= 2 days for a scrub (while other machines are writing about 50MiB/s data to it over the network)

Awesome video!

Great stuff as always!

I used ZFS for big pools for for than 7 years, with raid cards, but zfs was awesome, I did had a few problems and all where solved, even some bad IT guys messing with my big pools, couldn't make it fail hahaha! I did tried many things, for clusters, like GlusterFS and all the others you know. ZFS is THE best.

I love ZFS on my QNAP NAS (I'm using QuTS hero) and everything related with it (ZFS cache processes, inline dedup, compression etc.). 💪👍

Good stuff Wendell !

There is error correction encoding at the driver level, so the driver can usually tell, if the data is corrupt. Of course, this may fail too, or just the electronics can fail, but it is not so simple for a drive to fail without notifying it.

Parity RAID won't help you with drives returning bad data (aka "the drive is lying"), it only helps with drives not returning any data (aka "bad sector" or "drive died"): Even with parity you have no checksumming... Patrol read would just detect that something doesn't match, it cannot repair the data because it doesn't know which drive is lying when data doesn't match.

Here is an example of bitrot: you pay for google photos and notice that really really old photos start to have gray lines or half of the picture is gray. It doesn’t what device you use to try to view those photos. Has Googles datacenter experienced bitrot? I have since added Nextcloud with a zfs backend to the place I backup, no more trusting in google to ensure your data is reliable.