Syncronizing File Trees

Introduction:

There are many different reasons why you might want to syncronize file trees. If you are running DNS you would want to keep

your dns listings in sync across several servers. The same holds true if you are running backup webservers. As you can

imagine there are several reasons and applications for syncronizing file trees.

In this tutorial we will discuss syncronizing file trees using a very popular and powerful tool for moving files around.

This application is called rsync.

rsync can be used in six different and distanct ways as listed in the man pages. These six methods are as follows:

  1. for copying local files. This is invoked when neither source nor destination path contains a : separator
  2. for copying from the local machine to a remote machine using a remote shell program as the transport (such as rsh or

    ssh). This is invoked when the destination path contains a single : separator.

  3. for copying from a remote machine to the local machine using a remote shell program. This is invoked when the source

    contains a : separator.

  4. for copying from a remote rsync server to the local machine. This is invoked when the source path contains a :: separator

    or a rsync:// URL.

  5. for copying from the local machine to a remote rsync server. This is invoked when the destination path contains a ::

    separator.

  6. for listing files on a remote machine. This is done the same way as rsync transfers except that you leave off the local

    destination.
    I'll only be looking at copying from a remote rsync server (4) to a local machine and when using a remote shell program (2).

    Installing rsync:

    The first thing to do, as always, is install the port from the ports tree. Sinc syncronizing requires a local and remote

    system we will need to install rsyn on both local and remote systems

    # cd /usr/ports/net/rsync
    # make all install clean

    Now we must configure rsync to do what we want. In most cases you are going to have a primary server and a backup server.

    The primary server will be the one that has the most recent copies of the files that need to be syncronized with the files on

    the backup server. In our example we are going to do a simply test using our www folder (apache web server data files) and

    syncronize them with files on our backup server. There are several different ways this can be accomplished and can be

    applied to several different situations. You can also apply this to syncronizing your DNS data files. Remember that rsync

    does have the ability to execute sh commands on remote servers. This is extremely using, especially if you wish to rebuild a

    tinydns data.cdb.

    Alright let's start configuring

    Configuring rsync

    As mentioned before we are going to do a sample configuration to syncronize our www folder on our primary server with our www

    folder on our backup server. The primary server will be called primary_server and the backup server will be call

    backup_server. Clever naming scheme don't you think.

    Let's create our rsyncd.conf file. See man rsyncd.conf for more details.

    # vi /usr/local/etc/rsyncd.conf

    Here is a sample of what to put in your rsyncd.conf:

    uid = nobody
    gid = nobody
    use chroot = no
    max connections = 4
    syslog facility = local5
    pid file = /var/run/rsyncd.pid

    [www]
    path = /usr/local/www
    comment = All Data in the Apache www folder

    Again, please use the man paages for rsync.conf to see these and other available options. Now let's start the rsync daemon,

    verify it is running, and the verify that you can connect.

    # rsync --daemon --config=/usr/local/etc/rsyncd.conf
    # ps -aux | grep rsync
    # telnet localhost 873

    At this point the rsync daemon should be running without errors. You should note that if you wish to specify user id you

    should consider removing the world readable bit.

    # chmod 640 /usr/local/etc/rsyncd.conf

    If you wish to create user accounts and use the auth users and secrets file options also chmod your secrets file as well.

    For example, if you have the following two lines in ou rsyncd.conf file:
    auth users = user1 user2
    secrets file = /usr/local/etc/rsyncd.secrets
    You should also have a rsyncd.secrets file with corresponding user names and passwords:
    user1:secretpassword
    user2:anotherpassword

    Alright, now let's test to make sure rsync is working corectly by doing a test connection from a remote client. From your

    remote client, where rsync is also installed, run the following command:

    # rsync -avz primary_server.com::www /tmp

    The above command connects to primary_server.com and syncronizes the files in the www group in our primary server's

    rsyncd.conf file and stores those files in /tmp.

    If you wish to run this as a service you can either add it to the /etc/inetd.conf file or, my personal preference, create a

    startup sh script:

    # vi /usr/local/etc/rc.d/rsync.sh

    !/bin/sh

    case "$1" in

    start)
    if [ -x /usr/local/bin/rsync ]; then
    /usr/local/bin/rsync --daemon --config=/usr/local/etc/rsyncd.conf && echo -n ' rsyncd'
    fi
    ;;

    stop)
    kill `cat /var/run/rsyncd.pid`
    ;;
    *)
    echo "$0 start | stop"
    ;;

    esac

    Now you must set the script to be executable and start the service:

    # chmod 755 /usr/local/etc/rc.d/rsync.sh
    # /usr/local/etc/rc.d/rsync.sh start

    All that is left if for you to read the rsync and rsyncd.conf man pages and configure for your own needs.