How Do I Backup All My Servers??? 

So freaking relatable, bro

I liked this comment and I have full knowledge that those prayers do, literally, nothing

White wine.

Lol 😆 ✅️

*Thoughts and prayers

And Jeff does a lot of hand talking

😂😂

Not switched on so 0 risk lol

1. Jeff checks Linus' systems before passing them on to others. Linus built a new system for a big guy from Oregon and because of Corona and as Jeff lives practically (!) next door to the guy, Jeff delivered it in person instead of Linus. 2. Linus has visited a data recovery company, so he is aware just how much he can screw up before actually losing data. LMG actually suffered a fire in the server room. There was something about a bolt. Not lightning but a regular bolt.

@@mikkelbreiler8916 The closest LMG came to losing all their data was a raid controller dying while he was prepping the new backup solution. There's a very interesting video about that. But yeah, the UPS frying is also something that happened.

Those 2 events are 100% unrelated for sure

LOL

Insurance is sometimes nice!

Or the data center fire in france recently ... Coincidence for sure

@@gari5961 that one definitely wasn't the reason for me to automate my server backups for the server I got at that company in that city and luckily didn't burn down. Nope.

As he states in the video. One port is used for direct connection to one another.

@@mormantu8561 I'd be surprised if the switches aren't full fabric switches, ie, they can switch 100% of the aggregate port bandwidth simultaneously, and that's almost all switches nowadays. That means that "keeping data off the network" is completely irrelevant, because the network as a whole will never be congested. Additionally, none of these arrays can saturate a 10gbps link, let alone two, so aside from whatever overhead SSH would cause you (shouldn't be much), aggregating the dual 10gbps links via LACP 801.11ad gives you the same performance, plus the benefit of fault tolerance.

@@jttech44 But we don't know anything about his networking infrastructure. And I agree fully with what you are telling. But without knowing the rest there's really not a lot we can remark on.

alternatively, pay for 1RU of space in a nearish DC or setup a "cloud" sync to somewhere like backblaze B2 or amazon glacier.

if your car is in a closed storage...same building... get a ssd server for your car and a network port in the garage... if something happens im sure i rush away with my car. its not the best but also a solution to help the family with backups if you do a visit. :) greatz from Germany have a nice Day

Tape drive storage is an ol' reliable and you can store the tapes off site at a friend's house or a climate controlled storage unit (nearby or by mail for geographically separated backup) or a safety deposit box at a bank. Tapes aren't that big in physical volume each, but this is a case where sneakernet may still outstrip network duplication for offsite backup. Plus it's a true offline storage which would protect you against certain types of malware. That said, enterprise grade tape drives can get expensive fast. A newer LTO7 or LTO8 drive is north of $3,000. An LTO5 drive on eBay is like $400 reliably, not counting the HBA or rack mounting, but the LTO5 tapes are like $15 each if you buy a 5 pack - though the newer varieties of tapes are cheaper per terabyte yet more expensive per tape. I'll use the old LTO5 tape format for the rest of this reply, since that's what I'm using. Bear in mind you likely won't get the full capacity claimed on tapes if you write video files. That's because video files compress poorly to tape media and video formats already try really hard to be efficient plus when you add in a modest capacity loss if you enable LTFS and an LTO5 tape will hold a bit more than 1.2 TB. (Or 1.5 TB if you leave LTFS off, but then you need to do more tracking of what's on a tape rather than just mounting and reading the filesystem.) A tape rotation plan would be needed too if you plan to reuse tapes... Aaaaand then you're back on human scheduling. A harder problem for a RU-vid channel or someone running a home lab than a general user, clearly. Source: I have an LTO5 external SAS tape drive hooked up to my home/lab server. It's enough for me, but my RAID 5 data array is only 2.7 TB in usable size, so... Your problem is significantly bigger than mine. Though watching you gives me the itch to buy more servers. 😉 Some other people have suggested you can try network duplication to an offsite location, but maybe you can try with some rate limiting to try and share your uplink bandwidth with your other use cases? The initial sync will still be crazy long (or even get longer), but it might not necessarily cause you daily discomfort while it runs in the background. You might also need to break that rsync job into batches. 😬

As an lto6 user, lto7+ for for 40tb+ arrays is my vote. Especially if using a simple tape drive, and not an autoload library. I'd love to go that route, but would probably need some friends willing to split the cost of the drive, and only make periodic tape backups

I calculated rhat for this exact use case (off site raw video backup) AWS Glacier is the cheapest option (cheaper than buying your own lto drive and managing everything). You'll need these Backups close to never. I'm just not sure if AWS Glacier stores the data redundant. I think I wouldn't even use on site backup If the videos are years old if Glacier is redundant.

how much tho? i thought about using it and my array is 2 TB of 12 TB and growing

@@someguy9321 $5 per month per TB. (USD)

you spelled licks wrong

Yep, I noticed that as well. D'oh!

@@KaneHusky That was so he could show his wife the video for plausible deniability.

Even with the bad counters he should be fine. With the error rate of 1 per 10^14 bit (all consumer hdds), and a pool made of 10TB+ drives having the checksum errors after a full scrub is within drives specs, normal behaviour really. Hence the one needs zraid2 indeed.

That's also what I am doing. First upload took a while but now my off site backups are safe in Norway :p

Or If you have physical access the offsite location and server, you make the first "big" replication on your local LAN, then move the server to the intended location and then start copying snapshots during the night xD

I'm thinking that Backblaze offers that as an option for the initial backup.

This BUDS for you!

So that's a setting, not a piece of information.

i know, right, one of mine has 9 3TB drives ranging from 6 to 7.5 years of age.

Me too mate, my Nextcloud is on my 3TB HDDs lol, although im in the process of setting up a back up system for that pool alone, should the worst happen, as Jeff says, RAID is not a back up lol

Yeah, most but not all were unreliable. The only two 3TB drives (Seagate Barracuda) i ever bought are now about 9 years old and still working in my backup NAS. I guess i was really lucky there. They aren't constantly powered on though, so power-on hours sum up to

I think he said “SAS” disks? My 2x 3TB SATA disks run just fine. Purple series.

@@Charlie8913 Absolute terrible experiences with 3TByte HDDs from Seagate. Over the years I have collected an impressive amount of broken hard drives (100+....have been collecting for quite some time) and there are at least 5x a 3 TByte drive. Almost all of them died within 1 year. Only one lasted about 1,5 year. They left such a sour taste that I don't want them, even when offered for free. In my personal (n = 1) experience 1, 2, 4, 6, 8 and 10 TByte drives are far, far more reliable than those dreaded 3 TByte drives.

@@geroldmanders9742 I've never had a hard drive fail yet, but I don't put constant wear and tear on them and I buy slightly used enterprise drives with minimal hours on them. I only back up the data I truly need.

Pfsense just removed Wireguard due to critical security issues. You might wanna wait on Wireguard.

Gah, if Rambo had jumped on the desk...

wasabi as offsite backup site with 6$/TB is good alternative.

I've been using 40Mbit/s upstream for remote backups for over a year, I have no impact from this. The job needs about one and a half hours and is done while I sleep, so it'S really not a problem

That's a terrible idea if you need to back up as much as he has to. Upload times are horrid and if you need to download the backed up data, it can take forever.

@@cederian but offsite backups are for your last resort when downloading. Uploads will be a pain at first though since American ISP's are horrible with upload speeds

The great thing about RU-vid is you can rewind the part you don't understand and take notes. Use that as a basis for researching what you need to learn more about.

True, I have a backup server running on a 2 cores/4 threads celeron and it's doing fine.

I think his electric bill costs more than your computer. I just sold my 32 core 2990wx and moved all my day to day tasks to a J5040 10w 4 core 4 thread. It's not so bad for most things. (it was £200 new and i sold my 2990wx with board, ram, cooler etc for £2000. I will save £1000 a year in electric. Now i'm happy to just wait for tasks becomes i just think about better things i could spend that money on (like my kid)

@@KillaBitz Yeah

@@KillaBitz Then you had no need for the 2990wx in the first place if a 4c 4f could do everything you needed.

Snapshots are incredible for more than just backups. If you ever get ransomware'd, one click brings every file back. Pretty freaking cool.