Rsync Backup on Linux 

One recommndation I would make for running rsync from a cron job and that's to prefix the command with "nice -n20" just in case you're doing something else when the cron job launches. That way it will give rsync the lowest possible priority on your CPU and won't interrupt whatever you're doing. If nothing else is going on, it'll give it full CPU usage, but if you have another process taking up CPU time, that'll get higher priority.

TimeShift using rsync is awesome. Restoring my system was so fast my eyes were bugging out of my head in amazement.

Rsync is fantastic. It’s my favorite backup tool. I’m glad you covered it, and covered it so well. Backups have come a long way (I still remember backing up data on 5.25” floppies and even tape drives), so there’s no reason not to properly backup one’s data, and this video explains it all extremely well. Thank you!

You can use the --dry-run ( 2 dashes) option to run rsync without actually doing the backup so you can see what it's going to do.

Good video. I used to live with rsync and scripts with it. One option I would add is -z (compresses file data as it is sent). Big speedup over slower networks with very little overhead: -z, --compress With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. Note that this option typically achieves better compression ratios than can be achieved by using a compressing remote shell or a compressing transport because it takes advantage of the implicit information in the matching data blocks that are not explicitly sent over the connection.

Been needing this- and with YOU doing the video- I can learn it.. too many RAMBLE- or talk so fast I can't catch it... GREAT JOB!!

The one thing I always do before that line is check for running processes and/or dismount databases otherwise a lot of critical files will be open and not backed up just because you left apps open on your machine.

Thanks Chris, I was always a bit confused but -a so I took the time to crack it down: -a, --archive archive mode; equals -rlptgoD -r : recursive -l: copy symlinks as symlinks -p: preserve permissions -t: preserve modification times -g:preserve group -o:preserve owner -D: same as devices specials (don't really know what this is) x So now I see what you were saying, -a is a sensible way to copy. ------ -AX i don't really understand what they are useful for, even though Chris explained it.

As someone mentioned, -x would naturally exclude other files systems but also --exclude=/home/titus/{ltb/*,256nvmw/*,.cache/*} would save a few keystrokes... and don't forget --dry-run when test different rsync arguments, especially when using --del (being short for --delete).

If you want a GUI to rsync, try, Grsync, it's a pretty thorough GUI

Thank you, I am using rsync for the first time. I have tried before with no luck, but your piece was clear. I have used Grsync before. I thought there was a manual for rsync which told what all the options were.

I have been using Linux Lite and Peppermint os Linux for about 4 years now and it's time to upgrade a few machines. I added a new 4TB WD Red Drive to my Master heaviest duty fastest PC that was running Linux Lite 14.04 LTS and added the new drive via fstab to mount to the /media/ folder at boot time. I then watched another one of your videos that showed how to set up a Samba Shared Folder and after a little tweaking, I was able to accomplish that. I then went over to my old Toshiba Laptop that has a 250GB HD in it and modified the fstab to auto -mount to the new shared folder on a 2TB Partition I made on the new 4TB drive on the Master PC. Working on the command line is a little tedious, but I managed to watch this video a couple of times and was able to read between the lines a little bit about how to get the rsync configured using my specific folder, share device and backup names. I'm still not totally convinced I did it correctly, but it seems to be attached and able to create, read and write files and folders to the Master PC Shared folder on the new HD. I followed your command line almost exactly, but got a little confused about whether to use the actual Samba shared folder name or the actual names and tree structure to point to the folder. Anyway after a little testing and tweaking, it appears to point to the right folder seems to be working. I fired off the backup job and went to dinner, and when I came back everything was standing still, but there were no errors on the screen. Everything just seemed to stop for no apparent reason. There was plenty of space on the Master drive to store the files, and I really could not see any reason for the pause. Nothing I could do would move the job along. I went into task manager on the source PC and rsync appeared to be in use, but after waiting almost 30 min, no change in status of the backup job in terminal. I tried to kill the job with a Ctrl-Z in terminal, and that didn't yield any results. I finally closed the terminal and then restarted a new terminal and tried to run the rsync job again, but this time to a new directory. I got an error that I Googled, and it said that either the target drive was out of space (which it was not as it was a new 2TB Partition that I placed on the 4GB Drive), OR the connection simply timed out. I believe this was what happened. The author suggested adding --Timeout=120 to the rsync command to prevent timeouts from occuring for 2 minutes. But, I wanted to kill the existing job and start over, so I went into task manager and tried to kill the tasks from there, but no dice either. I don't believe I started task manager as a Root User. Unfortunately, after a few minutes the laptop locked up and mouse froze. The only way to shut it down was a hard power off and restart. It came up just fine, so I tried to run the job again, but this time to a new target directory and using the additional --timeout=120 command line variable, and it kicked off and has been running for over an hour just chugging along backing up this old laptop. It's using wired ethernet connection through my gigabit network, but this old Laptop has only a 100Mbit connection, so this will take a while as the drive was about 90% full. The first job backed up about 10% of the laptop's files when it stopped working, so I could tell by checking available disk space on the target drive on the Master PC that so far I have backed up much more data than the first time. My goal it to basically replace Peppermint OS 8 (Ubuntu 16.04) with the latest version of Peppermint OS which I believe is Peppermint OS 10 Respin. I was planning on re-partitioning the Laptop HD into multiple partitions as outlined in one of your other videos to make this job easier next time I update. I will have to do the upgrade on the Master PC as well because it is running ver 14.04 LTS of Linux Lite. It was my very first Linux installation back in the day, and I DID originally put multiple partitions on that one for the root, boot, swap and home directories. In fact the entire OS is on an SSD drive, and my data is on a 2nd traditional HD, so that one should be easier to update I hope. Anyway, thanks a million for your hard work on these videos! It sure helps out people like me that enjoy a little challenge, but really love the stability of Linux compared to the old Windbloz Software we had to endure for many years in the corporate world. Thanks Again from Central Florida!!!

Great vid as usual. If I may, I'd like to add the fact that this method (RSYNC) allows for really big data backups with no real limit beside the destination storage space available. Compare to say windows server backup software which are usually limited to 2tb. Which I have encountered multiple times in various clients businesses. They usually use this builtin windows tools to perform backups. But this tool like many others have a data limit of 2tb due to the process they used for the transfer.

I am personally using deja-dup and I am very satisfied with the backup function and the simple GUI application.

I appreciate the time you take to make these videos. You're content has helped me out. You mentioned going over cron jobs but didn't. I'm sure it could be a video all in itself for somebody who hasn't done it, but regardless thanks for this.

Thank you, Chris. Timeshift has been great when I needed it. But I think I might also do an rysnc cron-job as well.

I think you should probably have mentioned it was fast because you'd already run the command and copied all of the data over. Subsequent runs will only copy the deltas. Also, something to remember, if you use --delete and your source gets wiped, when you next run your backup job the destination will also get wiped.

Love your stuff mate :) - also greetings from Australia :)

To see how long a command takes to execute, prepend it with 'time'. For example, to see how long an rsync takes, you would execute the following: time rsync -aAXv .......

Yes, so glad you did Rsync! I've been using it for years ever since someone on linuxquestions.org told me about it. It's incredible!

It’s a wonderful tool for general bulk copying, backups etc. It was created as part of Andrew Tridgell’s PhD thesis: he invented an algorithm to answer the question “given two nearly-identical files on different machines, how do you find the differences between them without (doing the equivalent of) copying the entirety of one file to the other machine?” I have also used it to upgrade the hard drives in my machines, without reinstalling the OS: just rsync the installation to the new drive, fix up /etc/fstab, reinstall GRUB, and you’re good to go. Linux is so neat that way ...

Great video!! I love using rsync. Easy to set up and implement through Cron.

I love rsync. It's not as old of of a tool as one might think. I was recently reading about the brilliant young student who created it. It was his thesis. Very cool stuff

Oh, nice. Am a bit new to backups, did clone my nvme recently in timeshift with rsync. It seems really great. Maybe I can use this tutorial later for NAS.

Thanks so much! Good to know about the exclude option!

Something along --backup-dir=${your_dest}/$(date +"%Y%m%d")/ is useful for simple incremental backups because the other day, you will need the file clobbered with the latest snapshot... --one-file-system (-x) can be used instead of --exclude of all those mount-points (exept bind-mounts)

I haven't actually used rsync, but from a quick read of its man pages it seems that you can use it to backup to a remote device over SSH without having to mount a samba share locally. Useful if you want to run backups on a laptop when you're out and about.

The problem with Rsync is that when u wanna back up on ext hard drive it needs to b a Linux partition. And I wanna Dual boot windows on ma laptop for ma lil brother but can't find a way except for the traditional method

Question (relatively new user here, Linux Mint 19.1): my home folder on my desktop is less than 5 Gigs (some files "unreadable," if that matters); to save my personal data, firefox and GUI settings (in-case of a hard-drive melt-down) can I not just merely copy my home folder off the desktop, dump it on a thumb-drive, and call it "Done?" So that, in the event of a hard-drive failure, I can just dump all my settings and tweaks stored in the home folder on the thumb drive into another system running a fresh install of Mint 19.1, or whatever, and carry-on? Or is the Rsync route recommended for ease of re-installation? My understanding is that TimeShift (which I use) is sorta hit-and-miss if you're using it to save to an external drive to save personal data and profiles to re-upload to a fresh install. Seems to be a lot of differing points-of-view here...

I can't become a patron, so instead I just watched 2 ads 'till the end to support this channel. Great and useful video, as always!

Hey dude, i really like your videos and i learn a lot with them. One question, assuming I have snapshots in timeshift using rsync from a system without luks encrypt, if I reinstall the system with encrypted hd am i able to restore the system without losing the encryption?

Great video! Question: How do you skip destination files that are read-only? I've been trying to figure out how to not overwrite read-only destination files since I am syncing source code between a local Linux box and remote Linux build box. Regarding software repositories, they will mark the file read-only and when you check the file out, they mark the file writable. The problem that I am having is that when I specify the source and destination directories, rsync will overwrite read only files that are not checked out because I am specifying a project path and may only have a few files checked out locally, I checked the files out on my remote box that I want to work with. I have not found a way for rsync to not try and sync read only files at the specified destination. I really like rsync, but it keeps overwriting destination that are read-only files unless I check out the entire source branch at the destination. Is there a way to check the destination files first and not overwrite any files that are read-only? Again, my usage is to sync only modified local files and not overwrite read-only destination files. If by chance I am modifying a local file and the destination file is read-only, I would like rsync to stop and return an error code that I can see within my bash script.

I know I am a bit late in asking this question but here I go. I am trying to create linux (ubuntu) root / directory (ext4 file system) backup on a zfs file system (TrueNAS) via rsync and ssh. I use the options -aAXHvz but the preserve access control list option "A" fails by throwing "protocol incompatibility" error. I can't seem to find a way around this. Without using the A option the file/directories are saved with invalid ACLs and are messed up on restore. Is there anyway to solve this problem?

Hey Distrofriends, I have a question. I just tried to use the new Green-Recorder app in Wayland ( in the Fedora 30-Beta ). I wanted to do a demo showing how I use my computer. But the final published video is barely viewable. The resolution is just awful. And it only has a window mode, cutting out the top panel, which is needed in the video to show how everything works. I tried to make the same video in simple-screen recorder and the final published video was just a black screen. The music was there and the mouse & cursor worked great, LOL ! So which screen-recorder should I try next in Wayland. Or am I just stuck having to show how Fedora 30-Beta works in Xorg ??

thank you very much and God bless you for sharing your knowledge. Greetings from Split/Croatia

This is essentially a file copy which I also do for my data drive. Does it support something like time machine for Mac or File History on Windows? I use that on my work PC to go back in time on my desktop and dev folders.

What format was the backup drive? I didn't think NTFS drives supported all of the ACL and permission things that rsync would try to copy over?

Mounting remote filesystems..? I've always had issues with smb shares, so after all my systems went over to linux I eventually stopped using it, and only use sftp/scp - which mounts easily in the file browser, handles ownership and permissions correctly, etc. I've also used netcat to copy large amounts of data extremely fast (also using an archive stream, no encryption). I don't have much experience with NFS (network filesystem) though. Have you ever used it, and is it better, more/less difficult than other linux methods?

What, if anything, would you do differently to backup/restore an encrypted /home folder?

Is there any way of not sending files that weren't changed? Like a comparison of the hash or modified date of the file?

Thank you for the video. I am running linux mint 20.1 now, after my 18.2 crashed. is restoring just running rsync in reverse? Could I restore by graphically moving files from the extHD to the OS?

What is the easiest way to compress your files before a back up? something like mkdir (backup folder name); cp -r / ~/(backup folder name) | gzip | rsync ....?

I used rsync and hard links on Windows XP to do a file based full backup onto an external disk on daily basis. Each day folder contained the full filesystem but only consumed incremental more space due to the use of hard links. Downside was it only worked on local ntfs external harddisks but not on fileshares. And rsync was only available via cygwin library back then which limited to a maximum path depth of 256 chars.

So, without the --delete option, rsync would only every copy over new files from destination to source but never delete anything in the destination? Kind of like a "non-destructive" sync instead of an actual SYNC? One thing I haven't really gotten, yet, is the difference between -a and -A. In the man pages it says that -a includes the -p switch which preserves permissions. What extra does -A do, then? Is this a difference between permissions (-p) and ACLs (-A) ?

Excellent video. Love rsync.

Could you please explain the following case. I have virtual server Ubuntu on AWS, I need to backup data from it to my local FreeNAS which is behind NAT. So I need to pull data from remote rsync to local rsync. Is it possible to do it? Will it work behind the NAT? I need a bit more information about network interaction. Which part will be an initiator in this scheme?

The grsync tool is useful to help generate command lines. It provides a nice GUI to plan an rsync and can execute and even save the resulting command.

I also use rsync, without --delete when I just add new things to my main storage and then with --delete when I try to mirror folders. I have faced a problem a few times. When using sudo rsync, instead of rsync, some files and my home folder (ICE.authority) changed ownership and I had difficulties booting. I had to go to CLI and then chown the folder.

I use rsync and ZFS send/receive for my backups. Rsync operates on changes of complete files, while ZFS send/receive works on changes on record level. If your datasets (folders) are compressed ZFS will only send the compressed and changed records. I think BTRFS has the same type of send/receive facilities. I use rsync for: - USB 3.0 connection or; - folders with files say < 2 GB or; - folders where the receiver has no snapshots or a different period for own snapshots (e.g. 12x monthly). I use ZFS send/receive for: - USB 2.0 connection or; - folders with files say > 2 GB or; - folders where the receiver shall have a copy of the original snapshots. I have two backups one backup is my laptop 1 TB SSHD and the other has one 3.5" HDD (IDE; 320 GB; USB 2.0) and two 2.5" HDDs (SATA; 160 and 320 GB; USB 3.0). For the laptop I use SMB shares, that I mount after connecting the laptop. I only have 100 Mbps routers, so for that backup I connect laptop and desktop directly with one Ethernet cable at 1 Gbps. That is possible by selecting in the "edit network", the IP4 tab and select for the DHCP settings "shared to other computer". It will function as DHCP server/router for the second computer. I use the laptop Ethernet with this setting, since I'm too lazy to keep changing this setting each time. For normal operation I always use the laptop WiFi.

I don't think, that "-A" for ACLs covers what you explained... While it's true that it syncs ACLs, ACLs are not the unix access attributes (chown, chmod) you explained but they are really ACLs. ACLs are an extra list of access rights beyond chown/chmod which you modify with setfacl/getfacl.

When do you recommend the use of the --checksum option?

can rsync be used as a replication, some sort of scheduling done on it?

Timeshift is great but limited to linux native file systems. I need rsync to copy my root dir/timeshift backups of my NAS server to the zfs mirrored pool

I hope you can post some more videos llike this Just ignore the linux zaelotes. Most of us we are pragmatic users. Thank you for yours videos.

Thank you for the video! Worked like a charm! I am now trying to do the same with a network mounted driver ( from a Windows machine ). The files transferred successfully, but I got a few error before the transfer started: 'rsync: symlink "/mnt/SG2000/Backups/12-1-2021/bin" -> "usr/bin" failed: Operation not supported (95)' What do ?

log said: some files vanished before they could be transferred - what's that about?

Thanks for the video Chris, would you be able to make a video of how to perform restore in case of system total failure? Thanks.

Hi Chris, i really like your videos, i am using your rsync with --no-perms otherwise it gives me errors, do i get permission problems when i get them back from backup?

Exercise for the reader: for file transfers as a non-privileged user, instead of “--archive”, nowadays I like to use“-AlprtX”, while for root, I like to use “-ADgloprtX”. (This is separate from the deletion and verbosity/progress options.) What do these options do?

How quick was it before you were just doing a comparative backup?

I love the way you say F-stab instead of FS-tab lol

This is great. I'm fairly new to Linux, and have been working for a couple of days now to learn enough about rsync to create a reliable backup/restore routine for backing up my home directory. All the articles I've read so far, are either too complicated, missing info, or just not what I want to do. So this is a nice run down. One thing I don't understand yet, is what the wildcard is for in the --exclude string. Perhaps someone could enlighten me. Thanks. Per your request: One thing you missed is to explain how to restore. Do you just switch the source/destination, or is there more to it? What if you only want to restore some things? Thanks.

Great video! Can rsync be used to back up to an off site server as well?

sys and proc are temp system state directories that are built every time the machine is started up. They can be excluded. Had to on my rbpi4 or it stalled out trying to access active system files lol. Also take a look at Urbackup. Its a great open source agent-server style backup solution.

Can you make something but sync to a remote server ie- OMV NAS with SMB or an rsync server.

Does it do incremental backups?

What NAS do you use?

Grsync provides a nice graphical front end to rsync. Highly recommend to those who want a gui.

Can you show us how to do a full restore using rsync?

Could not enter folder /run/media/hisforever/grsync. plarse help

6:13 building on that, I generally use a tmpfs ramdisk for my downloads.

You could use time shift and save to an internal drive then rsync the time shift backup to a nas.

Can you make the font bigger. I can barely read it.

Thanks, Chris!

I think this is what I need. DejaDup isn't behaving well with ElementaryOS. And Tardis was backing up the whole home folder including the download folder.

Is your Storage on a FreeNAS CIFS Share?

My rsync is installed still isn't working Cmd not found it shows

nice. how to copy from linux to windows using rsync ?

So how do you restore it then?

Hi Chris you missed the rsynç GUI grsync its great and very easy to use

So this is for backing up personal data (you demonstrated backing up the home folder), but for a system backup, like before doing a system-wise arch upgrade, you still need timeshift, right? So if that upgrade fails you can restore from the timeshift backup, but the rsync backup won't do that?

why destination doesnt have ip?

Is there an rsync option, or other method for creating an archive copy of something before -delete from the destination? Example would be a documents folder with very important stuff, something gets deleted, rsync mirrored, and then you discover you really need it, but can't restore it.

Look at grsync if your worried about using the terminal but be careful you still can delete file and folders

would recommend covering rclone for offsite backups to different services (drive, onedrive, AWS, etc) it is kind of a cousin of rsync. also we use rsync for local backups at my job which i set up, those backups then gets then archived and encrypted then uploaded offsite through rclone. i have also used rsync to "hack" scan to email to work again at work on an old and perfectly working canon MFC from early (i think, as it was here before i started working there) 200X which just does not have support for TLS1.2 or TLS1.1 so it cannot send emails from itself anymore.

Just in case anyone has the same problem : when I tried mounting my SMB share, it failed for me. It turns out my NAS was only configured to use SMB 2.0 , not 3.0 . When I changed it to allow 3.0 , everything worked.

Fine intro to rsync. But how do you restore a Linux system?

I first create a btrfs snapshoft so that filesystem is frozen and no files chages can happend while im doing rsysnc to my NAS

Good video.

Have you used Rclone? It is supposed to be like rsync but for cloud storage.

awesome, grsync is a gui to it. please mention that a lot of commands have guis for them too :D

Can you via rsync do incremental backup or restore a backup from the specific point in time?

I usually dont back up steam. 200 gigs is a bit overkill. Most of these games are downloaded pretty fast anyways.

Where did you get that conky template ?

Good video. Thanks. "Eff"-"Stab" though... is that really how people pronounce it? I have few people that I get to regularly speak with about linux. It's mostly typing. But I always pronounced it "FS-Tab" in my mind. (as in, File System Table)

Why not just use backintime?

Do a mini script that does backups with rsync and does logging in /var/log/backups. Then you can make a systemd service with a timer to automate and you are golden.

rsync is great to use, however, if you want to back up to cloud, I strongly recommend restic with backblaze, i use it for all my systems and it works amazingly well - restic is FOSS, backblaze is $0.005/gb/mo, with the first 10gb free. - Restic also does full dedupe, compression and encryption.

with timeshift you can't backup to network drive

Need a video for RSYNC AND CRON SCHEDULING