Linux/Mac Terminal Tutorial: How To Use The rsync Command - Sync Files Locally and Remotely 

One of the best rsync tutorials ever. Thank you.

I really appreciate your approach -- everything you said was about rsync -- no rambling on an on. A short description up-front, then useful examples following a planned presentation. Informative and straightforward.

right out of the gate, glad there's no stupid background music!

Very clear and I liked the illustration for --dry-run. I'm also in awe of your typing skills (almost 18 minutes and not one typo!). But I'd be careful of calling rsync a backup tool. While rsync makes a copy, proper backups should be versioned so that if you accidentally damaged a file a couple days ago, you can get a clean copy from the backup made three days ago. If you rsync nightly to a destination copy, the damaged version of the file would already be synced by the time you realized it was damaged. You can add versioning to rsync (and --link-dest=dir can make it very space-efficient on the backup host) but you need to add a process for maintaining versions. Basically, to get seven days of versioning, you should rsync to a rotating day-of-the-week destination folder. But just rsync'ing to a single destination folder (as shown here) would not be versioned and would not be a good substitute for proper (versioned) backups. But a great explanation of using rsync.

I have a lot of experience using rsync, and found some things to do around rsync that get around issues, especially to do with "big" files. How big is big is debatable, it depends on disk size, bandwidth, etc. Anyway... Firstly if a big file is moved from one location to another, as rsync (with the --delete option) does not search for every file to delete before copying files somewhere earlier in the tree, it is possible if the file is big enough that the big file is transferred and fills the source disk before it gets chance to go to the part of the tree to delete where the file used to be. Solution... Do a one time pass of purely deleting files before running the normal rsync, ie: rsync -av --delete --ignore-existing --existing SOURCE DESTINATION Secondly, if the connection is of great distance and data transfer rate is slow, it is possible the rsync TCP connection is dropped before it completes copying a big file. rsync will then delete the half copied file, and the next run suffers from the same issue, and the file never gets copied. Solution... Use --append-verify so that if a timeout occurs, the next run of rsync continues from where it left off. However --append-verify will not copy files of the same (or smaller) size, this can happen with a logfile that might change content and be the same size every run. Or with repodata XML files. Solution... Once the --append-verify run of rsync successfully completes, THEN run a normal rsync (without --append-verify) to ensure files that have different content or are smaller, get copied properly. However this runs into the problem of the file that needs to get copied is BIG and doesn't make it across and it's too easy to get into a situation where --append-verify is ok, but the final rsync is not ok, repeatedly, and a big file never gets copied. Solution... monitor this normal rsync (ie the one after the --append-verify) and if it fails delete the last file being copied on the destination system, and let the next run of this (say) cronjob start again and --append-verify will copy over the big file (in several steps). FYI: I recommend the use of --conntimeout and --timeout to make rsync timeout if the connection hangs or has trouble starting. You might want to put a "timeout" command to call rsync anyway, in case rsync gets into a hung state internally. Finally, this does not solve the issue of a timeout happening if the time it takes for the directory tree to be completely read is too long. rsync will half die, and it's difficult to see why. In that case, split the transfer into multiple sub directories trees, so the directory listing is not too big for rsync to end up timing out during startup.

For those who read comments, before watching: it is exactly the tutorial you were looking for: good vid quality, good English, good examples. Thank you, Corey Schafer!

Ok, so its now 23rd of September 2020 (Year of the Covid-19) and I am 100% not a Linux guy for sure, but I am getting there with this kind of help. I have just followed this tutorial for Ubuntu 20.04 and have to say, it is still very much current. Thanks Corey, I have subscribed and you will now be my no 1 go to for my tutorials, if you have what I am looking for. Superb examples all the way through. Oh, and this is the first review I have ever felt the need to give in all my time on youtube. thumbs up mate, thank you from the UK.

Corey, I have been learning about coding and linux for the last three years, and I always come to your channel first to see if you have the content I am looking for. I just donated $20 through PayPal. I know it`s not much, but at least a small tolken of my appreciation. I just bought a Raspberry Pi and looking forward to your upcoming tutorials on this topic :)

Hey Corey I really like your videos, concise and quite in-depth, but I wish you included the --include and --exclude since they get quite confusing.

this is a great tutorial... i really like how you emphasized the --delete option. That would be catastrophic in a moment where you're trying to improve efficiency. Thank you!

Quick tip for Mac OSX terminal users - when doing rsync (or scp for that matter), you can just drag folders for FROM and TO into your terminal to save time typing. So type your rsync command followed by a space, then drop your origin folder onto terminal and path will appear, (plus an unseen space) then drop your destination folder and enjoy.

Wow, I'm a huge fan after viewing that. Been having a hard time understanding rsync, but love all your examples! Please keep adding videos like these!

Thank you so much. By far the best tutorial I have seen about rsync. Great job!

Have been trying to read about rsync since last few weeks with nothing to stick in my mind. Thanks for being crisp and clear.

recently picked up an old nas and nas is new to me. the nas does have apps and one is a sync utility but it's confusing and over documented. Your video was crystal clear. Thank you.

You just saved my day with this tutorial. Was fed up completely and thought of getting rid of rsync and implementing custom solution. This tutorial helped me realize my silly mistake and made my day.

Excellent tutorial! I did not know about the --dry-run parameter. I also like to exclude some folder when I am backing up my PC. I use after the --dry-run: --exclude={/sys/*,/tmp/*}.

Thank you for these. A good place to start learning a command, because the man pages are convoluted and impossible to read, but with you, it's all pretty clear

Fantastic video! Imagine the hours I could have saved doing drag and drop manual backups if I’d seen this before! The dry run is such a useful feature and gets rid of the ‘clenching’ anxiety when you’re not sure that your backup is going to do what you thought it might. Thank you.

Well, definitely this tutorial was exactly what I was looking for. It took me maybe half an hour to whatch it and write down the concepts, but I'm sure I won't forget your lessons. *Very* good ide to advice to be careful with the --delete flag. I'm looking to do bidireccional syncing and it seems like this is exactly the tool. Already liked your video. Thanks a lot!

thank you, I come back to this video to brush up the commands and their meanings :)

Man, I had to do a hw assignment on rsync. This video literally answered all of the questions that i had. Thanks a lot.

I’ve never felt so excited about watching more videos, if your other videos are as well put together and informative and easy to understand as this I’m going make some popcorn and tea :) good work and thanks for information it will be very helpful.

Corey your videos have helped me a lot...especially the ones on Rundeck..Ty!

One of the best explanatory video on rsync, the presentation was good and the commands are clearly visible

Perfect tutorial. You explained exactly what I needed to know.

After a long time, I am satisfied with what I have learned. So many things but you made it simple and easy to understand.

Love your videos.. I need come back again, and again through time..

Great video. Ditto to the comments made by someone that there is no background music. A lot of information presented quickly and clearly...

Damn.... EXCELLENT tutorial. I'm speechless.

Beautiful video! Great demonstration. Really showed me what I need to know to get rolling with it. Much appreciated

nice video, didnt feel it was 18min, very clear thx.

Awesome! I did my personal scripts and binding to specific shortcuts so I can sync some directories with just two or three keys. Thanks for the video it provides me with all the information that I still need to accomplish that.

Great tutorial! Much appreciated. I like your style. Direct, to the point, and accounting for all the things someone needs to understand who's new to this. A natural educator.

Fan !! Take a bow , what an explanation, helped to get a task done 😎🥂

Hey mate, great video... subscribed! I will be checking the rest of the videos that you suggested. I started using rsync but now I understand it better. Great explanation! Cheers.

Excellent video! Thank you!

Excellent step by step rsync demonstration video from a simple way to a complex way. Very useful, thanks a lot, Loïc.

Nice presentation. I really understand this topic. Thank you.

Don't forget to use the -X flag to preserve extended file attributes!!! "rsync -aXP" Also some GUI frontends / alternatives: FreeFileSync (www.freefilesync.org ), Unison (www.cis.upenn.edu/~bcpierce/unison/ ), Synkron (synkron.sourceforge.net ). Be aware that they don't preserve xattr though.

this is what we call a greaaat job ,thanks man ,i really appreciate this work

Thank you. very useful to my migration process.

Thanks man, helped me understand rsync much better.

Great video ! The number of options for rsync could be quite intimidating. It should not discourage people from using it because it is a great tool. It is a good candidate for an alias with tailored defaults for specific scenarios.

Liked, Subscribed and thump Upped. Great video man. Keep it coming.

Wonderful presentation Corey!

Very clear demonstration thank you

Awesome tutorial. Clear, direct, and very easy to follow.

Excellent description! Love the visual on the right.

Thank you for this awesome tutorial. 👍

Awesome video tutorial. Thumb up and subscribed!

Wow. This video was so smooth. Great job buddy! (y)

One other option I like to use is --stats. This gives a summary of files/folders which have been created, deleted, modified, etc. along with some other info.

You have a gift sir thank you very much for your videos it was extremely useful

Crystal clear! Thanks!

Your videos are too good. Thank you.

wonderful video is possible .if I have a home folder with subfolders, in them there are different files. now I want to copy only the files in all folders to a destination folder.

Thank you, this has helped me a lot.

Great tutorial. Very helpful!

Very good rsync tutorial. Thank you.

such a great video good examples and everything important explained Thank you!

Wonderful video, thanks.

I heard about the legend Corey for Python and decided to check him out later now I'm here looking for Rsync to make backups offline and here we are on his channel and subscribed to him

Good job Bro ....Thanks a lot ..thanks

This helped out immensely. Thanks so much!

Great video. Thank you!

VERY GOOD! > U got me back into [terminal] > now to re-learn how to colorize my terminal and all that stuff. Thanks again. #HHNET

Thank you for this tutorial

Would love to see you explain rsync excludes

Thanks my project done ❤️

Thank you so much!!!!

bruhh just lost my entire external drive to a MARS ransomware attack and I had no backups on my Pi. This is so helpful

Very clear, thanks.

fantastic tutorial!

Best tutorial ever

Please make Java tutorials after this series. You have some of the most well explained content I have ever had the pleasure of watching. Keep it up!

the best teacher, as usual!

This is what i wanted ThankYou

Wow!! Thank you so much!!

I am seeking a backup solution that supports "full" followed by incrementals. And although rsync does this, it lacks the following functionality, which I think is critical to a viable backup solution: -- If, for example, you schedule a daily incremental backup, and you are routinely editing various files, then your rsync backup will have your latest edits. This is where it is potentially a problem, because you will have only your latest version of your files. If you did not realize that you screwed up a file, then rsync will have made a copy of your bad file. Now you have no way to recover your file, from the other day, prior to you screwing it up. The same is true if you use the --delete option. If you absentmindedly deleted a file, and a week later you discover your mistake, well... rsync made an image (which is missing the file that was deleted), so now your backup drive will also no longer have that file. Not using the --delete option is not a solution, because if I do intentionally delete files, then one day if I need to do a restore, I do not want files that I intentionally deleted to be loaded back to my computer. I currently use Acronis' True Image for my Windows PC. But I have filed papers and will soon be divorcing myself from Windows, in favor of a Linux OS. But I am new to Linux, and want to be able to undo my screw-ups, via a comprehensive backup solution. True Image does a full backup, followed by incremental backups. But their design differs from rsync, in that each backup stands on its own merit. Each incremental backup is an exact image of the time that the backup was run. In other words, if you run a daily backup, then you can restore your computer from any of the endless daily backups, and your computer will return to exactly as it was on the date of that backup. So you never have to worry about not catching a mistake right away. If you deleted a file, you can go back, day-by-day (assuming you did a backup each day), until you find your file in one of the incremental backups. Same thing for deleted files. Go back far enough in your incremental images, and you will find the file that you had deleted. And these are not full backups (except for the first one). The daily incrementals are tiny. They are based off of the first full backup. Yet, True Image organizes the images in a way that you can pick any backup ever done and your files will be in that archive exactly as your computer was on the day of that backup. To my knowledge, rsync does not include the above functionality. Is there a Linux based program that does the above? By the way, True Image is Linux based. When you boot from their rescue media, you can escape from their GUI and run a bash session. But the Linux based True Image is available only when you boot from their rescue media (used when you cannot boot Windows). Their normal app seems to be available only for Windows. I do not want to be forced to boot from rescue media every day, in order to perform backups as I described above. So I am seeking a Linux based backup solution that will run while my Linux box is running in its normal boot mode. Thanks for any help.

Thanks for this video, really rsync is a great command and I have been using it a lot lately. If you could do a tutorial on linux networking commands then that would be of great help as well. Thanks.

Great tutorial Corey! Subbed

Excellent 👍👍👍👍👍

Very helpful! Thank you!

thanks! It is good explained, very useful

This Video is Great !

Very useful

thank you very much

This is way better than the stupid syncthing

Thank you very much for this detailed tutorial. However, I would like to ask how to use rsync between 2 servers to local machine..i.e a server that is accessed through DMZ to your local machine.

Great Tutorial! I have learned very much! Q: If i made cron job on daily basis for rsync the directory. And in the directory there is lots of content which will take around 2-3 days to complete. But my cron job will run on daily basis. So will my server get crash or get load or anything will going wrong ?

Thanks for yet another tutorial! Deja vu 6:19 - 6:38 ^^

I would suggest to use Systemd timer units instead of cronjobs. Also would like to see --include-from and --exclude-from covered there as most likely you don't want to sync .cache; .Trash and other garbage.

Superb video. Just subscribed. Does rsync notice if the file sizes are different. For example, if I’ve added more photos into iPhoto. The file would be there b the size is different and copy it over?

Nice Job! Best tutorial for rsync that I found so far. Can i get help with a simple question? Suppose i need to sync main folder/upload/jpg/png but i want to exclude "upload" which contains jpg and png folders? How do i do that?

Hi Corey, thanks for your extremely nice explanation. could you make a video for the same content in python, since nowadays we are doing everything with python and would be helpful if u help us.

07:50 --dry-run

Was hoping you would talk about the include and exclude directives and wildcard matching... that is hard to understand!

Hey, thanks for this nice tutorial. But I would like to ask you, if it is possible to use it, as I want. I have a Rpi running as a web server. I would like to make my pi run every 2 hours some rsync script, that would copy my Windows-shared folder \\pcname\user to \\pcname\backups\pc. I also edit already existing files and want the files to get updated. What command should I use for that?

Perfect.