Introduction
Many network administrators often invest in robust and high-cost commercial tools to perform backups, synchronizations, and data mirroring. However, rsync offers these same functionalities for free, with the advantage of being highly customizable.
rsync can be easily integrated into scripts, allowing the automation of complex tasks and the creation of custom solutions to meet the specific needs of each environment. For example, it can be combined with cron to schedule automatic backups, send email notifications, or generate detailed logs that facilitate monitoring operations.
Main Features
Rsync (remote sync) is a widely used command-line tool for synchronizing files and directories between two locations, either on the same system or across a network.
It is known for the following features:
- Efficiency
- Compares source and destination files, transferring only the differences.
- Offers data compression during transfer.
- Security
- Uses SSH for remote transfers.
- Preserves permissions, owners, groups, timestamps, and other file attributes.
- Allows simulation of operations without making real changes.
- Flexibility
- Allows excluding files at the destination that no longer exist at the source.
- Offers granular control over the bandwidth used during transfer.
- Supports recursive directory synchronization, including subdirectories and files.
- Versatility
- Can be used for backups, copying files, mirroring servers, websites, for bidirectional synchronization, and more.
- Is easily integrated into scripts, enabling automation of complex tasks.
- Is available on various platforms, including Linux, BSD (FreeBSD, OpenBSD, NetBSD), Windows (via Cygwin or WSL), macOS, and Solaris.
- Reliability
- Has mechanisms for verifying the integrity of transferred data, ensuring that files are copied correctly.
- Displays detailed reports, including the number of files transferred and the total size of the data.
Syntax and Basic Usage
The general syntax of rsync is:
rsync [options] source destination
1 - Example for copying local directories:
rsync -av /source/ /destination/
- -a (archive): Preserves directory structure, permissions, attributes, timestamps, and enables recursive mode.
- -v (verbose): Displays details of the transfer.
Rsync treats source directories that contain a trailing slash (/) differently. If the slash is present, the command copies only the contents of the source folder to the destination. Otherwise, the source folder itself will be copied to the destination.
Example:
- rsync -av /source/ /destination/: Copies only the content of /source/ to /destination/.
- rsync -av /source /destination/: Copies the folder /source (and its content) to /destination/.
If the destination directory does not exist, rsync will automatically create it during the operation.
2 - Copying Files:
To copy a file while preserving its original name:
rsync -av /source/filename.txt /destination/
To rename the file, specify the new name in the destination path:
rsync -av /source/filename.zip /destination/new-name.zip
Sync with a Remote Computer via SSH
To synchronize files from a local directory with a remote computer via SSH:
rsync -avz /source/ user@server:/destination/
- -z (compress): Compresses the data during transfer.
To copy files from a remote computer to the local system, reverse the source and destination:
rsync -avz user@server:/remote/path/ /local/path/
If the SSH server is configured to use a port other than the default (22), specify the port in the command using the -e option:
rsync -e "ssh -p ssh-port" -avz /source/ user@server:/destination/
Replace ssh-port with the port number configured on the server.
Ignore Files or Directories During Copy
To avoid copying temporary files, use the command:
rsync -av --exclude='*.tmp' /source/ /destination/
- --exclude: Allows specifying patterns to omit files or directories. In this example, no files with the .tmp extension will be included in the copy.
To omit a specific directory during synchronization:
rsync -av --exclude='dir/' /source/ /destination/
This command ignores the directory dir/ (and all its contents) from the copy but includes all other files and directories of /source/.
Delete Files at Destination That Do Not Exist at Source
To maintain an exact copy of the source at the destination:
rsync -av --delete /source/ /destination/
- --delete: Deletes files at the destination that are not present at the source. This option is useful for keeping the destination as an exact copy of the source, removing files that were deleted or renamed at the source.
Simulate Operation Without Executing It
To test a command, use:
rsync -av --dry-run /source/ /destination/
- --dry-run: Shows what would be done, but does not execute any changes. This option is useful for testing commands and checking the impact before applying them.
List of Parameters
Below is a list of options to be used in the rsync command and their functions:
- a (archive): Equivalent to -rlptgoD. Enables recursive mode, transfers special devices and blocks (e.g., sda, tty), preserves directory structure, permissions, attributes, symbolic links, and modification dates.
- --append: Continues a transfer where it left off, in case there is a partially copied file at the destination.
- --checksum: Verifies the integrity of copied files. Significantly increases execution time, as it calculates the checksum of all files at the source and destination. Useful when continuing abruptly interrupted transfers.
- --delete: Removes files at the destination that are not present at the source. This option is useful for keeping the destination as an exact copy of the source.
- --dry-run: Simulates the operation, showing what would be done, but without executing any changes. Ideal for testing commands before applying them.
- --exclude='PATTERN': Ignores files and folders matching a specific pattern. Allows the use of wildcards, such as * and ?.
- -e: Specifies the remote shell to be used and allows including additional parameters. Example: -e "ssh -p 22" to specify a custom SSH port.
- -h: Displays output in a human-readable format. Instead of showing transferred data in bytes, displays values in KB, MB, GB, etc.
- -P (--partial --progress): Shows a progress bar and preserves, at the destination, partially transferred files. To continue an interrupted transfer, combine with the --append option.
- -q (quiet): Suppresses messages, displaying only errors during execution.
- -u (update): Does not overwrite files at the destination if they are newer than those at the source.
- -v (verbose): Displays details of the transfer.
- -z (compress): Compresses data during transfer.
Practical Example
Below, a directory structure and files will be created to demonstrate the functioning of rsync in practice. A folder called rsync/ will be created in the user's home directory, and inside it, the subfolders source/, where various files and subdirectories will be stored, and destination/, initially empty, where the files and folders will be copied from the source.
The script below creates the structure used in this example, allowing you to run the same tests. Enter the user’s home directory, cd /home/user/, create a file for the script: vi create-rsync-structure.sh, and add the content:
create-rsync-structure.sh
#!/bin/bash
# Creates the directory structure mkdir -p rsync/{source,destination} mkdir -p rsync/source/photos/cache mkdir -p rsync/source/documents/backup # Enters the "source" directory cd rsync/source/ # Creates example files touch file1.txt file2.log file3.tmp photos/cache/photo_cache.txt documents/notes.txt
# Creates binary example files with specific sizes dd if=/dev/zero of=photos/photo1.jpg bs=512K count=1 dd if=/dev/zero of=photos/photo2.png bs=512K count=3 dd if=/dev/zero of=documents/report.pdf bs=512K count=2 dd if=/dev/zero of=documents/backup/old_backup.zip bs=512k count=4
The dd command in the script creates files that occupy disk space to enrich the tests. Add execution permission to the script and run it to create the structure that will be used in the examples:
$ bash
chmod +x create-rsync-structure.sh ./create-rsync-structure.sh
After running the script, files in various formats will be created and organized in different folders.
Directory and File Structure Created
rsync/ ├── destination └── source ├── file1.txt ├── file2.log ├── file3.tmp ├── documents │ ├── backup │ │ └── old_backup.zip │ ├── notes.txt │ └── report.pdf └── photos ├── cache │ └── photo_cache.txt ├── photo1.jpg └── photo2.png
The following commands will be executed in the rsync/ folder within the user's home directory. Access the base directory, inside it you will be able to view the folders source/ and destination/.
$ bash
cd /home/user/rsync
ls
destination source
To make a complete copy of the source/ folder to destination/, use the command:
$ bash
rsync -avh source/ destination/
Note that in the output of the rsync command, the transferred files and folders are listed, along with information such as bytes sent and received, transfer speed, and total size.
sent 5.24M bytes received 222 bytes 10.49M bytes/sec total size is 5.24M speedup is 1.00
To display the structure of files and directories in the source/ and destination/ folders in tree format, use the tree command:
$ bash
tree source/ tree destination/
Repeat the command rsync -avh source/ destination/ and note that only a few bytes will be transferred. This happens because rsync checks whether the files at the destination are identical to those at the source. On the second execution, no files will be transferred, as no changes have been made at the source since the first synchronization.
Delete all files in the destination/ folder to start a new test. Then, repeat the rsync command, now using the -z option to enable compression.
$ bash
rm -rf destination/* rsync -avhz source/ destination/
Note that, in the output, the transferred data was compressed, and only 954 bytes were sent, instead of the 5.24MB from the first command.
sent 954 bytes received 214 bytes 2.34K bytes/sec total size is 5.24M speedup is 4,488.77
Add some text to the file file2.log. Then, repeat the rsync command and check that only the file file2.log will be copied.
$ bash
echo "New entry" >> source/file2.log rsync -avhz source/ destination/
Delete the file file2.log at the source and execute rsync with the --delete option. This will delete the file at the destination, maintaining an exact copy of the source.
$ bash
rm source/file2.log rsync -avhz --delete source/ destination/
Creating a Script with Rsync
In the previous examples, you can see the basic behavior of rsync and already get some inspiration to use it in your daily life. For instance, you can maintain an exact copy of your website hosted on the internet on a local computer.
However, imagine that there are files or entire folders that you do not want to copy. The script below copies the contents of source/ to destination/, except for files with the extensions .tmp and .log, as well as the cache/ folder and its contents. The script also uses the --delete option to remove files at the destination that do not exist at the source.
synchronize.sh
#! /bin/bash rsync -avzh --delete source/ \ --exclude "*.tmp" \ --exclude "*.log" \ --exclude "cache/" \ destination/
Create the synchronize.sh file and add the script content. Then, grant execution permission and run it from the rsync/ folder in our example to test its functionality:
chmod +x synchronize.sh ./synchronize.sh
In the previous examples, rsync is being used to copy data locally. Remember: for remote copies, just add user@server: before the path of the remote folder.
Large Files and Slow Connections
If a transfer is interrupted, the default behavior of rsync is to discard the partially copied file. To prevent this, use the -P option. This option preserves the partially transferred file at the destination and displays a progress bar during the operation.
rsync -avP source/large-file.zip destination/
When using the above command, if the connection is interrupted, the partially transferred file will remain at the destination. To continue the transfer from where it stopped, use the --append option. If you repeat the previous command without this parameter, the copy will start from scratch again.
Another useful option in scenarios where failures occur during the transfer is --checksum. This option compares the files at the source and destination to confirm their integrity. The default checking mechanism of rsync queries the file size and the last modification date to determine if the file has changed, but does not verify the integrity of the data.
To continue the interrupted transfer of the previous command, use:
rsync -avP --append --checksum source/large-file.zip destination/
The --checksum option is not mandatory and will significantly increase the execution time of the script, depending on the size of the file to be verified.
Bidirectional Synchronization
Imagine a scenario where you need to maintain two copies of the same files in two different locations, such as at home and at work. During the week, you edit the files on the company computer, while in the evenings or on weekends, you work on the files at home.
Rsync will replace the files at the destination if they differ from those at the source, unless you use the -u option. This option instructs rsync to not overwrite files at the destination that are newer than those at the source.
Although this approach solves part of the problem by avoiding overwriting newer files at the destination, rsync is a unidirectional synchronization tool. When executing rsync source/ destination/, the files at the destination are updated with the latest versions from the source, but the reverse does not happen automatically.
To keep both sides updated (source and destination), you need to run the command twice, reversing the source and destination:
$ bash
# Updates the directory at HOME with files modified at WORK rsync -avu user@work:my-files/ /home/user/my-files/ # Updates the directory at WORK with files modified at HOME rsync -avu /home/user/my-files/ user@work:my-files/
Conclusion
Rsync is an indispensable tool for system administrators and users who need to manage large volumes of data efficiently. Its incremental synchronization capability, coupled with support for secure connections via SSH, makes it a popular choice for backups, data replication, and maintenance of distributed systems.
References
- https://rsync.samba.org/
- https://download.samba.org/pub/rsync/rsync.1
- https://linuxize.com/post/how-to-use-rsync-for-local-and-remote-data-transfer-and-synchronization/
