Synology's Hybrid RAID (BTRFS+SHR) deep dive - Can I trust it? 

Thank you, appreciate it. 🙏

Appreciate the comment, thank you!

Thank you, appreciate it!

SHR will use the underlying MD driver to perform this, so its the standard MD process to move from one RAID type (RAID5) to another (RAID6). The TL:DR on this would be that its a very slow and long process where the parity needs to be recalculated for every bit on every drive in the array from the data of all the other disks. I am performing a SHR-1 to SHR-2 migration currently on a NAS, going from 5 18Tb drives to 6 and this looks to take around ~30 days to complete. And this isn't Synology specific, it would presumably take a similar time on any Linux box with MD running. And looking at utilization during this process, the NAS is not CPU or memory bound, it just looks to be IO wait. I also performed a disk replace to upgrade a disk from 6Tb to 20Tb in a different SHR-2 array, and this took about 15 hours. And this operation is a pure copy operation. I will likely summarize both of these into a video once the migration is complete for those who have the same questions and to see how the process looks in advance of people doing it themselves. Hope this helps

I would think they can fork it, but I don’t know the specific terms for the btrfs code.Many commercial products are built on open source code bases.

ext4 doesn't offer scrubbing unfortunately, this is primarily a capability of BTRFS and ZFS (and some less known FS's). But yes, the parity logic is only really to recover from disk errors / failures and not data corruption. There may be system tools that would do the job on ext4, but not sure off hand. Not sure how fast data rots on unpowered magnetic media. Its certainly way better than on NAND but maybe 10 years significantly increases the chance that enough bits got flipped that you have a problem. Maybe its time for some shiny 20tb disks running SHR with BTRFS and copy that data over. :-)

@@sometechguy Haha - yeah, a pack of new 20TBs and Btrfs would certainly be a bump in reliability :-) This is just purely archival data from a former business (I tested cameras and lenses for 20+ years), so there’s really no justification or budget for putting any money into it. We’ll see, maybe in 5 years I’ll just buy a pair of 36 TB drives for $150 each, make two copies and forget about it 😂 Thanks for the super-quick answer, btw!

​@@DaveEtchells Scrubbing raid is different from scrubbing BTRFS data checksums which can be of use for identifying damage even on a single partition, though at least two btrfs volumes are needed for repair. The problem with an offline HDD is not bit-rot, the issue is the mechanical parts get stuck and control boards fail so they just won't physically spin up after a few years on the shelf. Best to keep them in a machine that gets booted up every 3-6 months. No controls or fine mechanics is part of the reason tapes are still used, along with the cost per TB when working at very large scale, (Cost for a tape drive is prohibitive at small scale.) but even tapes get copied around every 5→10 years to refresh the magnetic strength before it becomes unreadable, often coupled with checksums and duplicates for repair of detected faults.

This is really interesting, thank you. I will take a deeper look at ReFS. Seems it’s the future replacement for NTFS?

@@sometechguy It's gonna take MS so long to roll it out Windows will no longer be relevant.

I think torrents should be fine. But yes, probably don't use Copy on Write on the volume for the torrents downloads directory as the files will be constantly growing, and that could result in constantly creating new files. But you can put the completed files on a different volume with CoW enabled. This isn't something I have done, but it seems reasonable. On Synology you can do this not enabling data checksum when you create the volume, but cannot be edited later. Does that help?

Old write-copies should only persist if you tell it to take a snapshot, otherwise they should become free space simply by changing the inode link/pointer to the newly written file. Might be something else going on. You can probably setup a subvolume for torrents with the configuration tuned to what you need. Also, slowly writing a file(slow growing) should be different from modifying a file, but I'm not an expert with CoW file systems let alone the file handling code in your torrent application.

@@mytech6779 oh I have learnt so much since then. At that time I was using the array function in unpaid. It basically sandwiches many single drives with a party drive over then. Thus constant io bottlenecks and small 1tb drives filed easy. Since then I have those 16 1.2TB drives in volume together. So they work as one large drive thus now in zfs2.

Sorry if it didn't have what you wanted, but I would say that all videos can't be all things to all people. MD uses standard RAID semantics, and I do have a video that covers how the various RAID types organise data, that explains things like RAID1, RAID5 and RAID6 which the SHR implementation use under the hood with MD. I also have another video with higher level on how SHR organises its data to maximise drive usage. For reasons of time, i wanted to keep the video to a specific topic. There is always more that can be added, and some people want more detail, and some less. On the other EXT4 point, you are correct and mandatory was the wrong word. I have tried to make it clear from the video title and description that this firmly about BRTFS and SHR, where BTRFS is the recommendation for many reasons I mention in the video. But it is, as you say possible to implement SHR with Ext4. I just wouldn't personally.