{"id":5639,"date":"2024-06-19T02:06:00","date_gmt":"2024-06-19T06:06:00","guid":{"rendered":"https:\/\/www.both.org\/?p=5639"},"modified":"2024-06-14T09:13:08","modified_gmt":"2024-06-14T13:13:08","slug":"how-my-easy-home-made-backup-program-saves-time-space-on-the-storage-medium-and-network-bandwidth","status":"publish","type":"post","link":"https:\/\/www.both.org\/?p=5639","title":{"rendered":"How my easy, home-made backup program saves time, space on the storage medium, and network bandwidth"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n 1 <\/span>\r\n<\/div><\/div>\n

Note: This is a long, in-depth article. <\/strong><\/p>\n\n\n\n

Nothing can ever go wrong with my computer and I will never lose my data. Riiiiight.<\/p>\n\n\n\n

I’ve experienced data loss for a myriad of reasons, many of them my own fault. Keeping excellent backups has always enabled me to continue with minimal interruption. There are many ways in which data can be lost, compromised, or corrupted.<\/p>\n\n\n\n

This article discusses the Bash backup program I created to prevent catastrophic data loss and facilitate easy recovery. Performing regular backups and ensuring that data can be restored from them is a critical part of any security plan. We’ll start by describing the problem and my solution, and look at two ways to download and install the program. We’ll then move on to running the setup program to prepare both the computer and the USB backup storage device. Finally, we’ll perform a simple backup of the root, home, and \/usr\/local directories.<\/p>\n\n\n\n

The main configuration file works without change for what you’ll have done up to that point. However, you’ll undoubtedly need to backup more than just your own home directory. So we’ll then discuss the rsbu.config file and how to configure it to backup other directories on your local host and selected directories on remote hosts as well.<\/p>\n\n\n\n

The Problem<\/h2>\n\n\n\n

Although data loss is a problem, the related problem is finding a backup system that meets all my requirements. Some were way more complex than I need, while others required tape for backup. In the past, I found that tape seldom works well for various reasons. Most other media specifically designed for backups have become obsolete.<\/p>\n\n\n\n

My backup program must meet the following requirements:<\/p>\n\n\n\n

    \n
  1. Be easy to use.<\/li>\n\n\n\n
  2. Have a command line user interface that is easily understandable.<\/li>\n\n\n\n
  3. Not have a graphical interface.<\/li>\n\n\n\n
  4. Be easily configured.<\/li>\n\n\n\n
  5. Maintain the configuration data separately from the executable code.<\/li>\n\n\n\n
  6. Create backups that can be restored by any user.<\/li>\n\n\n\n
  7. Be able use internal hard drives or external USB drives for backup media.<\/li>\n\n\n\n
  8. Use space on the backup medium efficiently.<\/li>\n\n\n\n
  9. Minimize the amount of time needed to make a backup.<\/li>\n\n\n\n
  10. Backup all hosts on the network to a single backup device.<\/li>\n\n\n\n
  11. Have an integrated help function.<\/li>\n<\/ol>\n\n\n\n

    No open source backup system whether free (as in beer) or not, that I ever considered met my requirements. Don’t get me wrong, there are some really good ones out there. But none had all the features I needed nor met all my requirements. So I decided to write my own backup program.<\/p>\n\n\n\n

    Just a quick work about terminology. The local host is the one on which the rsbu program will run. The rsbu program can perform the backup of the local host. It can also backup all remote computers that are in the same network. In this case, remote means “not the local host” and distance is irrelevant. The remote hosts can be a meter or many kilometers away so long as they are on the same network. Of course there are ways to circumvent the same network requirement but that’s a network issue.<\/p>\n\n\n\n

    Why hard drives?<\/h2>\n\n\n\n

    I chose spinning disk hard drives (HDD) rather than solid state devices (SSD) as my backup medium. The primary reason is that SSDs can begin to experience data degradation<\/a> after about a year without power applied. That makes them unsuitable for long-term, off-line backup and archival storage. Those rust-covered spinning disk HDDs can store data for decades without degradation. They are also still less expensive than SSDs.<\/p>\n\n\n\n

    Testing has shown that spinning disk HDDs typically store data significantly longer than SSDs with no power applied. However, there’s no guarantee that data on any given device won’t begin to degrade sooner.<\/p>\n\n\n\n

    I’ve used many types of tape and removable cartridge formats such as the Bernoulli Box and Zip drives for backup in the past, and all had less than perfect results. One reason is that the storage media on all those devices is open to the air so that the read\/write heads which are mounted on the device into which the cartidge is loaded, can access the recording medium. This left them susceptible to contaminants that, even as small as a speck of dust or a hair can cause what IBM used to call Head\/Disk Interference (HDI) which would damage the disk and the heads. USB HDDs are sealed and are not susceptible to airborne contaminants.<\/p>\n\n\n\n

    Why write it in Bash?<\/h2>\n\n\n\n

    I use Bash all the time, in part because it is the default shell for most Linux distributions, including Fedora, the one I normally use. However I also use Bash because it is an excellent shell programming language and it has a large number of features to make life easier for the Lazy SysAdmin.<\/p>\n\n\n\n

    Features like tab completion, command line recall and editing, shortcuts like aliases, and more all contribute to its value as a powerful shell. One of my favorite Bash features is that, although it uses Emacs mode for command line editing by default, that can be changed to Vi mode so that I can use editing commands that are already part of my muscle memory.<\/p>\n\n\n\n

    But if we think of Bash solely as a shell we miss much of its true power. While researching Bash for my three-volume Linux self-study course1<\/a><\/sup>, I learned things about Bash that I never knew in over 20 years of working with Linux. Many of these new bits of knowledge relate to its use as a programming language. Bash is a powerful programming language, one perfectly designed for use on the command line and in shell scripts.<\/p>\n\n\n\n

    About rsync<\/h2>\n\n\n\n

    Before I wrote my backup program, I had been experimenting with the rsync command which has some very interesting features that I have been able to use to good advantage. My primary objectives were to create backups from which users could locate and restore files quickly without having to extract data from a backup tarball, and to reduce the amount of time taken to create and the backups.<\/p>\n\n\n\n

    I wrote about Using rsync for Backup<\/a> in a previous article so you should read that for additional background about why rsync is an excellent choice for creating backups that meet my requirements. It also explains the rsync<\/strong> commands that will be used in the rsbu<\/strong> Bash program. I’ve repeated two of the most important attributes here.<\/p>\n\n\n\n

    The bottom line is that rsync<\/strong> can be used to synchronize two directories or directory trees whether they are on the same computer or on different computers. It can create or update the target directory to be identical to the source directory. The target directory is also freely accessible by all the usual Linux tools because it is not stored in a tarball or zip file or any other archival file type; it is just a regular directory structure with regular Linux files that can be navigated by regular users using basic Linux tools such as a desktop file manager. This meets one of my primary objectives.<\/p>\n\n\n\n

    One of the most important features of rsync is the method it uses to synchronize preexisting files that have changed in the source directory. Rather than copying the entire file from the source, it only copies the blocks that have changed of the source file. This saves an immense amount of time and network bandwidth for remote sync. For example, when I first used my rsync Bash script to back up all of my hosts to a large external USB hard drive, it took about 3 hours. That is because all of the data had to be transferred because none of it had been previously backed up. Subsequent backups took between 3 and 8 minutes of real time, depending upon how many files had been changed or created since the previous backup. I used the time command to determine this so it is empirical data. Last night, for example, it took 3 minutes and 12 seconds to complete a backup of approximately 750GB of data from 6 remote systems and the local workstation. Of course, only a few hundred megabytes of data were actually altered during the day and needed to be backed up.<\/p>\n\n\n\n

    The rsync command has a very large number of options that you can use to customize the synchronization process. For the most part, the relatively simple commands that I have described in that previous article are perfect to make backups for my personal needs. Be sure to read the extensive man page for rsync to learn about more of its capabilities as well as details of the options discussed here.<\/p>\n\n\n\n

    Install the code<\/h2>\n\n\n\n

    An RPM and a tarball are both available for download so you can install using the method of your choice. Regardless of which method you use, the files for rsbu are installed in the standard locations defined by the Linux Filesystem Hierarchy Structure (FHS)<\/a>.<\/p>\n\n\n\n

    From an RPM package<\/h3>\n\n\n\n

    Download the code for rsbu<\/strong> from the link, rsbu-02.06-00.noarch.rpm<\/strong><\/a> into \/tmp or another directory of your choice. This RPM installs all of the files required for rsbu<\/strong>. This includes rsbu.conf, the configuration file for rsbu. All user configuration is performed in this file. It also includes rsbu-setup script that provides tools to prepare the host computer and to prepare external USB media to store the backups.<\/p>\n\n\n\n

    Install this RPM as the root user with the following command.<\/p>\n\n\n\n

    # dnf -y install rsbu-02.06-00.noarch.rpm<\/strong><\/code><\/pre>\n\n\n\n
    From a tarball<\/h3>\n\n\n\n
    If you don’t have a Red Hat, Fedora, or other RPM-based system, download the tarball from rsbu-02.06-00.tar<\/strong><\/a> into \/tmp or another temporary location of your choice. This is the tarball equivalent of the rsbu RPM for those who don\u2019t have an RPM-based distro. You must be root and in the root directory (\/) to extract the files into their proper locations.<\/p>\n\n\n\n
    Make \/ (the root directory) the present working directory (PWD). Assuming that you’ve downloaded the tarball to \/tmp, use the following command to extract the contents of the tarball into the correct directories. The PWD must<\/strong><\/em> be the root directory for this to work properly. I used the -v option so that the tar<\/code> command will display the files it installs and their locations.<\/p>\n\n\n\n
    # tar -xvf \/tmp\/rsbu-02.06-00.tar<\/strong>\netc\/\netc\/systemd\/\netc\/systemd\/system\/\netc\/systemd\/system\/backup.service\netc\/systemd\/system\/backup.timer\netc\/ssh\/\netc\/ssh\/ssh_config.d\/\netc\/ssh\/ssh_config.d\/01-permitrootlogin.conf\nusr\/\nusr\/local\/\nusr\/local\/share\/\nusr\/local\/share\/applications\/\nusr\/local\/share\/applications\/rsbu\/\nusr\/local\/share\/applications\/rsbu\/gplv3.txt\nusr\/local\/share\/applications\/rsbu\/README.rsbu\nusr\/local\/bin\/\nusr\/local\/bin\/rsbu\nusr\/local\/bin\/rsbu-setup\nusr\/local\/etc\/\nusr\/local\/etc\/rsbu.conf\nroot@testvm1:\/#<\/code><\/pre>\n\n\n\n
    Preparation<\/h2>\n\n\n\n
    Regardless of which installation method you used, the rest of the instructions are the same. All tasks must be performed as the root user.<\/p>\n\n\n\n
    Read the README<\/h3>\n\n\n\n
    I know — nobody reads the documentation, right? Well, I ask you to please do that.<\/p>\n\n\n\n
    Be sure to read the \/usr\/local\/share\/applications\/rsbu\/README.rsbu document before performing setup or backups. It’s as yet incomplete but I’m working on it. What’s there, along with the instructions here will get you going.<\/p>\n\n\n\n
    Check the code<\/h3>\n\n\n\n
    Before doing anything else, be sure to at least look at the Bash shell code for rsbu<\/code> and rsbu-setup<\/code> to get a feel for what they both do. It doesn’t matter if you’re not a programmer. You should do this anyway. The code is well commented and you should be able to sufficiently understand what it does as you read them.<\/p>\n\n\n\n
    Initial testing<\/h3>\n\n\n\n
    Now do a quick check to ensure that the code has been installed correctly. Just view the Help.<\/p>\n\n\n\n
    root@testvm1:\/# rsbu -h<\/strong>\nrsbu - Performs backups of local and remote hosts using rsync.\nIt also uses the link capability of rsync to minimize storage usage\nfor unmodified files for series of daily backups.\n\nSyntax: rsbu -[l|L|c|b|h|u]vd <Device number> s <host number> f <file name>\noptions:\nb              Backup data to the selected Backup Media. Implies -u.\nc              Check the contents of the Backup Media.\nC              Clean the contents of the Backup Media leaving only the most recent\n                backup which is renamed using todays date. Removing old Cruft.\n                New option. Added 2022-09.\nd              The number of the backup device. Use -l to list devices by number.\nf <filename>   The fully qualified path to an alternate configuration file\n                You are not likely to need the -f option.\nh              Print this help.\nl              List the backup devices and their respective numbers.\nL              List the hosts to be backed up and their respective numbers.\np              Prepare the backup device. Requires device number and device name.\nu              Unmount the backup device after completing backup or check.\nv              Verbose mode.\nV              Print version number and exit.\n\n\nThe configuration file is \/usr\/local\/etc\/rsbu.conf. Make all configuration\nchanges in that file.\n\nSetting a 1 day retention in this file creates a single\nbackup of each host without dates. Additional backups simply update the\nexisting set of backup files and do not create a backup history.\n\nroot@testvm1:\/#<\/code><\/pre>\n\n\n\n
    If the Help is displayed as shown above, the code has installed correctly.<\/p>\n\n\n\n
    Preparing for the rsbu setup<\/h3>\n\n\n\n
    I’ve written a Bash program to perform much of the setup for you. You will need to use the rsbu-setup<\/code> program once to configure your computer and the external USB disk drive you will use for the backups. You can use the rsbu-setup<\/code> program as many times as needed to prepare additional disk devices.<\/p>\n\n\n\n
    You will need an external USB drive for this. You can use a USB thumb drive for testing but you should definitely use a USB hard drive with spinning disks for your production backups for the reasons described above. Be sure that the device you use for testing has enough space to hold the data from \/root, \/home, and \/usr\/local.<\/p>\n\n\n\n
    The rsbu-setup<\/code> program performs the following tasks while providing the user with descriptions of what it’s doing and multiple opportunities to opt-out of the process.<\/p>\n\n\n\n
      \n
    1. Notify the user to plug in the external USB backup drive.<\/li>\n\n\n\n
    2. Create the \/media\/Backups mount point.<\/li>\n\n\n\n
    3. Select the most recently plugged storage drive.<\/li>\n\n\n\n
    4. Add the new mount point to \/etc\/fstab.<\/li>\n\n\n\n
    5. Delete all existing partitions on the drive.<\/li>\n\n\n\n
    6. Create a single Linux partition that fills the entire drive<\/li>\n\n\n\n
    7. Create EXT4 partition on drive.<\/li>\n\n\n\n
    8. Creates the PPKP for SSH.<\/li>\n\n\n\n
    9. Installs the PPKP on the localhost.<\/li>\n<\/ol>\n\n\n\n
      This script can be used to prepare a new hard drive for use as a backup device without performing any of the tasks required to prepare the system itself. Use the -p option for this.<\/p>\n\n\n\n
      View the rsbu-setup help.<\/p>\n\n\n\n
      root@testvm1:~# rsbu-setup -h<\/strong>\n\n################################################################################\n# rsbu-setup - Performs setup tasks for the rsbu backup program.               #\n#                                                                              #\n# This script can be used to prepare a new hard drive for use as a backup      #\n# device using the -p option. You can also use the -c option to prepare the    #\n# host computer.                                                               #\n#                                                                              #\n#                                                                              #\n# Syntax: rsbu-setup -[h|g|V] -cpqv                                            #\n# options:                                                                     #\n# c            Prepare the computer. Creates the \/media\/MyBackups mount point  #\n#              and adds the mount point to the \/etc\/fstab. file. Doesn't       #\n#              prepare the USB device.                                         #\n# g            Print the GPL license statement.                                #\n# h            Print this help.                                                #\n# p            Prepare a new external storage device for use as a backup       #\n#              medium. This option doesn't prepare the computer itself.        #\n# q            Quick mode. Skips built-in wait for drive to be identified.     #\n#              Use this option if the drive has already been plugged in        #\n#              for more than 30 seconds or so.                                 #\n# V            Print the software version number.                              #\n# v            Print verbose status information.                               #\n################################################################################<\/code><\/pre>\n\n\n\n
      This will give you an idea of the capabilities of this program. It also tells us that it was installed properly.<\/p>\n\n\n\n
      Start the setup procedure. Insert the USB device in a USB slot. Let’s identify the device just inserted so we can be certain we have the correct one during the setup. The data about the USB device should be the last entries in the dmesg output data stream. Look for the serial number to identify this specific device. Further down in this data, you’ll find the ID assigned to this device, sdb in this instance.<\/p>\n\n\n\n
      [42218.246900] st: Version 20160209, fixed bufsize 32768, s\/g segs 256\n[42218.403572] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found\n[42245.308870] usb 1-1: new high-speed USB device number 2 using xhci_hcd\n[42245.641903] usb 1-1: New USB device found, idVendor=abcd, idProduct=1234, bcdDevice= 1.00\n[42245.641919] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3\n[42245.641926] usb 1-1: Product: UDisk           \n[42245.641931] usb 1-1: Manufacturer: General \n[42245.641935] usb 1-1: SerialNumber: 1404190300490149481200<\/strong>\n[42245.688320] usb-storage 1-1:1.0: USB Mass Storage device detected\n[42245.688685] scsi host6: usb-storage 1-1:1.0\n[42245.688775] usbcore: registered new interface driver usb-storage\n[42245.692777] usbcore: registered new interface driver uas\n[42246.736852] scsi 6:0:0:0: Direct-Access     General  UDisk            5.00 PQ: 0 ANSI: 2\n[42246.737871] sd 6:0:0:0: Attached scsi generic sg2 type 0\n[42246.740778] sd 6:0:0:0: [sdb] 15435960 512-byte logical blocks: (7.90 GB\/7.36 GiB)\n[42246.741613] sd 6:0:0:0: [sdb] Write Protect is off\n[42246.741622] sd 6:0:0:0: [sdb] Mode Sense: 0b 00 00 08\n[42246.742372] sd 6:0:0:0: [sdb] No Caching mode page found\n[42246.742383] sd 6:0:0:0: [sdb] Assuming drive cache: write through\n[42246.754919]  sdb: sdb1\n[42246.756793] sd 6:0:0:0: [sdb] Attached SCSI removable disk\nroot@testvm1:~#<\/code><\/pre>\n\n\n\n
      Performing the rsbu-setup<\/h3>\n\n\n\n
      Use the c and p options to prepare both the computer and the external storage device. Whether or not you’ve connected the USB storage device, you’ll get this reminder. You could also examine this code to see how it works. If you do so, you’ll notice that this code checks to see if various tasks have been performed before. If they have, it will skip those tasks. This is intended to prevent things like installing a PPKP that supersedes a previous one, thus breaking PPKP communication for other hosts.<\/p>\n\n\n\n
      root@testvm1:~# rsbu-setup -cp<\/strong>\n\n##############################################################################\n#                               ATTENTION!!!                                 #\n#                                                                            #\n#  Plug in the external USB hard drive you will be using for your backups.   #\n#                                                                            #\n##############################################################################\n\nHave you plugged in the external USB hard drive? (ynq)<\/code><\/pre>\n\n\n\n
      Type y<\/strong> and press Enter<\/strong> to continue. The rsbu-setup program then displays the following message while it extracts the data about the USB device from the dmesg<\/code> data stream. Once itfinds the device it displays the information it has found about it and asks you to positively verify that the device it found is the one intended to be your backup storage. The term “General UDisk” is almost always used for the device model regardless of the vendor.<\/p>\n\n\n\n
      Have you plugged in the external USB hard drive? (ynq) y<\/strong> \n\n##############################################################################\n#                                                                            #\n#   Please be patient while we identify this new USB hard drive.             #\n#                                                                            #\n##############################################################################\n\n##############################################################################\n#                                                                            #\n#                A new device has been plugged in.                           #\n#                        New Device = sdb \n#                                                                            #\n##############################################################################\n##############################################################################\n#                                                                            #\n# Verify that the information on the external storage device, especially     #\n# that the device serial number matches that printed below.                  #\n#                                                                            #\n##############################################################################\n##############################################################################\n\n##############################################################################\n#                                                                            #\n#    WARNING!!   Be sure that this information is correct!   WARNINIG!!      #\n#                                                                            #\n#----------------------------------------------------------------------------#\n#                                                                            #\n# The new storage device is:                                                 #\n#                                                                            #\n#      Vendor: usb 0xabcd \"General\"\n#      Model: \"General UDisk\"\n#      Serial ID: \"1404190300490149481200\"\n#                                                                            #\n#----------------------------------------------------------------------------#\n#                                                                            #\n# WARNING!!!    If the serial number printed above does not match that       #\n#               printed on the physical label on the storage device, DO NOT  #\n#               CONTINUE this procedure. Enter n or q below.                 #\n#                                                                            #\n##############################################################################\nAre you POSITIVE that the device shown above is the correct one for your backups? (ynq)<\/code><\/pre>\n\n\n\n
      Compare the serial number displayed here to that from the dmesg command. If it’s the same, you can proceed.<\/p>\n\n\n\n
      Press y<\/strong> and Enter<\/strong> to positively verify that this is the correct device. The program gives you one more chance to be absolutely certain that you’re going to let it delete all the data on the USB device.<\/p>\n\n\n\n
      Are you POSITIVE that the device shown above is the correct one for your backups? (ynq) y\n\n##############################################################################\n#   WARNING!!   WARNING!!   WARNING!!   WARNING!!   WARNING!!   WARNING!!    #\n#                                                                            #\n#                                                                            #\n#    !!THIS PROCEDURE WILL DELETE ALL OF THE DATA ON THE USB HARD DRIVE!!    #\n#                                                                            #\n#                ARE YOU REALLY SURE YOU WANT TO CONTINUE?                   #\n#                                                                            #\n#                                                                            #\n#   WARNING!!   WARNING!!   WARNING!!   WARNING!!   WARNING!!   WARNING!!    #\n##############################################################################\nARE YOU REALLY SURE YOU WANT TO CONTINUE? (ynq)<\/code><\/pre>\n\n\n\n
      If you are sure you want to continue, press y<\/strong> and Enter<\/strong>. This process can take several minutes for large devices so be patient. This will complete preparation of the disk and begin preparing the computer by generating a Public\/Private keypair (PPKP)2<\/a><\/sup> for SSH. This is required because SSH is used by rsync even on the local host. Without this PPKP you’d need to enter the root password even when performing only a local backup. You’ll need to install the public key for the root user on all other computers that will be part of your backup procedures.<\/p>\n\n\n\n
      ARE YOU REALLY SURE YOU WANT TO CONTINUE? (ynq) y<\/strong>\n\n##############################################################################\n#                                                                            #\n# Please be patient.                                                         #\n#                                                                            #\n# This may take a few minutes depending upon the size of the backup          #\n# device being prepared.                                                     #\n#                                                                            #\n##############################################################################\n\/dev\/sdb: 2 bytes were erased at offset 0x000001fe (dos): 55 aa\n\/dev\/sdb: calling ioctl to re-read partition table: Success\nCreating new partition on sdb\nChecking that no-one is using this disk right now ... OK\n\nDisk \/dev\/sdb: 7.36 GiB, 7903211520 bytes, 15435960 sectors\nDisk model: UDisk\nUnits: sectors of 1 * 512 = 512 bytes\nSector size (logical\/physical): 512 bytes \/ 512 bytes\nI\/O size (minimum\/optimal): 512 bytes \/ 512 bytes\n\n>>> Created a new DOS (MBR) disklabel with disk identifier 0x74631234.\n\/dev\/sdb1: Created a new partition 1 of type 'Linux' and of size 7.4 GiB.\n\/dev\/sdb2: Done.\n\nNew situation:\nDisklabel type: dos\nDisk identifier: 0x74631234\n\nDevice     Boot Start      End  Sectors  Size Id Type\n\/dev\/sdb1        2048 15435959 15433912  7.4G 83 Linux\n\nThe partition table has been altered.\nCalling ioctl() to re-read partition table.\nSyncing disks.\nmke2fs 1.47.0 (5-Feb-2023)\nCreating filesystem with 1929239 4k blocks and 482384 inodes\nFilesystem UUID: d9eda845-5855-4f1a-b2be-6ed3e9ba82e6\nSuperblock backups stored on blocks:\n        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632\n\nAllocating group tables: done\nWriting inode tables: done\nCreating journal (16384 blocks): done\nWriting superblocks and filesystem accounting information: done \n\n##############################################################################\n#                  Backup partition space usage data                         #\n##############################################################################\n# Device label =   MyBackups\n# Mount Point =    \/media\/MyBackups\ndf: \/media\/MyBackups: No such file or directory\ndf: \/media\/MyBackups: No such file or directory\n##############################################################################\n\n##############################################################################\n#                                                                            #\n# Your new backup device has been prepared for use with the rsbu program.    #\n# The device has been unmounted so you can remove if from the USB slot.      #\n#                                                                            #\n##############################################################################\n\n\n##############################################################################\n#                                                                            #\n# Preparing testvm1.both.org for rsbu backups.\n#                                                                            #\n##############################################################################\nCreating fstab entry\n\n##############################################################################\n#                            !!!ATTENTION!!!                                 #\n#                                                                            #\n#  During this portion of configuration you need to create a Public\/Private  #\n#  KeyPair (PPKP) to provide encrypted communication during the backup       #\n#  procedure.                                                                #\n#                                                                            #\n#  You will be asked to enter a file name in which to save the RSA key.      #\n#  DO NOT enter a name for the file. Just press the Enter key to accept      #\n#  the default file name.                                                    #\n#                                                                            #\n#  You will be asked twice to enter a passphrase. Do not enter a passphrase. #\n#  Just press the Enter key for an empty passphrase.                         #\n#                                                                            #\n# Refer to Wikipedia for details of PPKP.                                    #\n#  https:\/\/en.wikipedia.org\/wiki\/Public-key_cryptography                     #\n#                                                                            #\n##############################################################################\n\nGenerating public\/private ed25519 key pair.\nEnter file in which to save the key (\/root\/.ssh\/id_ed25519):<\/code><\/pre>\n\n\n\n
      The response to this prompt is to press Enter<\/strong> because the generated file name displayed in the prompt, \/root\/.ssh\/id_ed25519, is fine. Then press Enter<\/strong> for both passphrase prompts. We’re not using passphrases as that would require the SysAdmoin — you — to enter the passphrase every time rsbu begins the backup of any host, local or not. That would prevent performing scheduled, unattended backups.<\/p>\n\n\n\n
      Enter file in which to save the key (\/root\/.ssh\/id_ed25519): <Enter><\/strong>\nEnter passphrase (empty for no passphrase): <Enter><\/strong>\nEnter same passphrase again: <Enter><\/strong>\n<\/code><\/pre>\n\n\n\n
      The public and private keys are stored in this section of the code.<\/p>\n\n\n\n
      Your identification has been saved in \/root\/.ssh\/id_ed25519\nYour public key has been saved in \/root\/.ssh\/id_ed25519.pub\nThe key fingerprint is:\nSHA256:EVx8ujEcl9MQ76OnD+bLt9lJvOWXHwAmlr+VHzJMSLc root@testvm1.both.org\nThe key's randomart image is:\n+--[ED25519 256]--+\n|       ..o..o*   |\n|        ..= B.o  |\n|        .= O E.  |\n|        ..O +..  |\n|        S  = Bo. |\n|          . o.*..|\n|           .+ .=o|\n|           + +o=*|\n|            =+++*|\n+----[SHA256]-----+\n\n##############################################################################\n#                                                                            #\n# We now need to install the public key for this computer. You will first    #\n# need to respond with 'yes' and press the Enter key when asked if you       #\n# want to continue connecting to the host. Be sure to use the word 'yes'     #\n# not just 'y'.                                                              #\n#                                                                            #\n# Next enter the root password when requested.                               #\n#                                                                            #\n##############################################################################\n\n\/usr\/bin\/ssh-copy-id: INFO: Source of key(s) to be installed: \"\/root\/.ssh\/id_ed25519.pub\"\nThe authenticity of host 'testvm1.both.org (192.168.0.101)' can't be established.\nED25519 key fingerprint is SHA256:i4XXP20X4d+5A8mIprOBq9oVNH2JGo5TGBu2vF7ICWM.\nThis key is not known by any other names.\nAre you sure you want to continue connecting (yes\/no\/[fingerprint])?<\/code><\/pre>\n\n\n\n
      Be sure to type the word yes<\/strong> and press Enter<\/strong> to continue.<\/p>\n\n\n\n
      Are you sure you want to continue connecting (yes\/no\/[fingerprint])?yes<\/strong>\n\n\/usr\/bin\/ssh-copy-id: INFO: Source of key(s) to be installed: \"\/root\/.ssh\/id_ed25519.pub\"\nThe authenticity of host 'testvm1.both.org (192.168.0.101)' can't be established.\nED25519 key fingerprint is SHA256:i4XXP20X4d+5A8mIprOBq9oVNH2JGo5TGBu2vF7ICWM.\nThis key is not known by any other names.\nAre you sure you want to continue connecting (yes\/no\/[fingerprint])? yes<\/strong>\n\/usr\/bin\/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed\n\/usr\/bin\/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys\n\n############################### W A R N I N G #################################\n# THIS COMPUTER SYSTEM IS PRIVATE PROPERTY. USERS HAVE NO EXPECTATION OF      #\n# PRIVACY. USE OF THIS COMPUTER SYSTEM IS SUBJECT TO MONITORING OR OTHER      #\n# REVIEW BY THE OPERATOR\/OWNER OR OTHERS. UNAUTHORIZED OR IMPROPER USE OF     #\n# THIS SYSTEM MAY RESULT IN LEGAL PROSECUTION AND CIVIL AND CRIMINAL          #\n# PENALTIES.                                                                  #\n#              USE OF THIS SYSTEM CONSTITUTES CONSENT TO MONITORING.          #\n###############################################################################\n#    Continuing the login process constitutes acceptance of this policy.      #\n###############################################################################\n\nroot@testvm1.both.org's password: <Enter the root password><\/strong>\n\nNumber of key(s) added: 1\n\nNow try logging into the machine, with:   \"ssh 'testvm1.both.org'\"\nand check to make sure that only the key(s) you wanted were added.\n\nroot@testvm1:~#<\/code><\/pre>\n\n\n\n
      Testing<\/h2>\n\n\n\n
      Testing is an important aspect of installing this new software. So let’s start with some basics. As root on the local host, testvm1 in my case, login to the local host. Be sure to enter yes<\/strong> and press Enter<\/strong> when asked if you want to continue connecting. You must perform this step or the backups of the local host won’t work.<\/p>\n\n\n\n
      root@testvm1:~# ssh testvm1<\/strong>\nThe authenticity of host 'testvm1 (192.168.0.101)' can't be established.\nED25519 key fingerprint is SHA256:i4XXP20X4d+5A8mIprOBq9oVNH2JGo5TGBu2vF7ICWM.\nThis key is not known by any other names.\nAre you sure you want to continue connecting (yes\/no\/[fingerprint])? yes<\/strong>\nWarning: Permanently added 'testvm1' (ED25519) to the list of known hosts.\n\nroot@testvm1:~#<\/code><\/pre>\n\n\n\n
      After verifying that this works, type exit<\/strong> and press Enter<\/strong> to close this child shell session.<\/p>\n\n\n\n
      There are a couple additional things to check before performing the simple backup configured in \/usr\/local\/etc\/rsbu.conf. The first command below shows the revised \/etc\/fstab with the entry for \/media\/MyBackups. The second test lists the actual \/media\/MyBackups directory.<\/p>\n\n\n\n
      root@testvm1:~# cat \/etc\/fstab<\/strong> \n\n#\n# \/etc\/fstab\n# Created by anaconda on Mon Jun 10 11:23:45 2024\n#\n# Accessible filesystems, by reference, are maintained under '\/dev\/disk\/'.\n# See man pages fstab(5), findfs(8), mount(8) and\/or blkid(8) for more info.\n#\n# After editing this file, run 'systemctl daemon-reload' to update systemd\n# units generated from this file.\n#\nUUID=8a1ca039-dace-4628-b4f8-4dd7a5d79647 \/                       ext4    defaults        1 1\nUUID=4dbfc467-4ea3-4e0b-8260-30a922319de1 \/boot                   ext4    defaults        1 2\nUUID=3bf9194e-54b0-4e66-ab37-b04b715c2a6f \/home                   ext4    defaults        1 2\nUUID=75bafb60-fa9b-4c51-9526-bf31b59ebd03 \/tmp                    ext4    defaults        1 2\nUUID=8a0d9c1e-c552-4aa5-84a8-76f82fbd07cf \/usr                    ext4    defaults        1 2\nUUID=ee939988-bd3f-4723-9d99-10e1e2baffcd \/var                    ext4    defaults        1 2\nLABEL=MyBackups   \/media\/MyBackups    ext4    noauto,defaults      0 0\n\nroot@testvm1:~# ll \/media<\/strong>\ntotal 4\ndrwxr-xr-x 2 root root 4096 Jun 11 09:43 MyBackups\nroot@testvm1:~#<\/code><\/pre>\n\n\n\n
      Now let’s verify that the rsbu program is reading the rsbu.conf file. We’ll list the backup device and the configured hosts and the files to be backed up. The lowercase l (el) option lists the configured backup devices. In this case we have only a single device which is device number 1. We also have only a single host. The hostname for the localhost is determined by the script which obtains it from the $HOSTNAME environment variable. I have set it up this way so that during this initial installation and testing phase, it won’t be necessary for you to make any changes to the rsbu.conf file. We’ll get to that later.<\/p>\n\n\n\n
      root@testvm1:\/usr\/local\/etc# rsbu -l<\/strong>\nDevice 1 = External USB hard drive\n\nroot@testvm1:\/usr\/local\/etc# rsbu -L<\/strong>\nHost 1 = testvm1;--exclude .gvfs --exclude Cache ;:\/root :\/home :\/usr\/local<\/code><\/pre>\n\n\n\n
      This looks correct and, with the exception of the hostname, should be the same for you.<\/p>\n\n\n\n
      Performing a simple backup<\/h2>\n\n\n\n
      It’s now time to perform an actual backup as a final test. The default configuration in \/usr\/local\/etc\/rsbu.conf will backup the \/root, \/home, and \/usr\/local on the localhost to the USB drive that you prepared. The b option tells rsbu to perform a backup based on the details in the rsbu.conf file. The backup device is automatically unmounted at the end of the backup.<\/p>\n\n\n\n
      root@testvm1:~# rsbu -b<\/strong><\/code><\/pre>\n\n\n\n
      Mount the backup device again, and verify that the backup has been performed. I like the tree<\/code> command for this as in Figire 1.<\/p>\n\n\n\n
      root@testvm1:~# mount \/media\/MyBackups\/<\/strong>\nroot@testvm1:~# tree \/media<\/strong>\n\/media\n\u2514\u2500\u2500 MyBackups\n    \u251c\u2500\u2500 Backups\n    \u2502   \u2514\u2500\u2500 testvm1.both.org\n    \u2502       \u2514\u2500\u2500 2024-06-12-RSBackup\n    \u2502           \u251c\u2500\u2500 home\n    \u2502           \u2502   \u251c\u2500\u2500 dboth\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .bash_history\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .bash_logout\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .bash_profile\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .bashrc\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .cache\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 abrt\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 applet_dirlist\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 gstreamer-1.0\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 registry.x86_64.bin\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 imsettings\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 log\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 log.bak\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 obexd\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 sessions\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 tracker3\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 files\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 errors\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 first-index.txt\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 http%3A%2F%2Ftracker.api.gnome.org%2Fontology%2Fv3%2Ftracker%23Audio.db\n<SNIP>\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 http%3A%2F%2Ftracker.api.gnome.org%2Fontology%2Fv3%2Ftracker%23Video.db-wal\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 last-crawl.txt\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 meta.db\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 meta.db-shm\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u251c\u2500\u2500 meta.db-wal\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 ontologies.gvdb\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 rss\n    \u2502           \u2502   \u2502   \u2502   \u2502       \u251c\u2500\u2500 meta.db\n    \u2502           \u2502   \u2502   \u2502   \u2502       \u251c\u2500\u2500 meta.db-shm\n    \u2502           \u2502   \u2502   \u2502   \u2502       \u251c\u2500\u2500 meta.db-wal\n    \u2502           \u2502   \u2502   \u2502   \u2502       \u2514\u2500\u2500 ontologies.gvdb\n    \u2502           \u2502   \u2502   \u2502   \u2514\u2500\u2500 xfce4\n    \u2502           \u2502   \u2502   \u2502       \u2514\u2500\u2500 notifyd\n    \u2502           \u2502   \u2502   \u2502           \u2514\u2500\u2500 log.sqlite\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .config\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 abrt\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 dconf\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 user\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 imsettings\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 pulse\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 cookie\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 Thunar\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 Thunar\n    \u2502           \u2502   \u2502   \u2502   \u2502   \u2514\u2500\u2500 uca.xml\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 user-dirs.dirs\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 user-dirs.locale\n    \u2502           \u2502   \u2502   \u2502   \u2514\u2500\u2500 xfce4\n    \u2502           \u2502   \u2502   \u2502       \u251c\u2500\u2500 desktop\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u251c\u2500\u2500 icons.screen0-1256x957.rc\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u2514\u2500\u2500 icons.screen.latest.rc -> \/home\/dboth\/.config\/xfce4\/desktop\/icons.screen0-1256x957.rc\n    \u2502           \u2502   \u2502   \u2502       \u251c\u2500\u2500 panel\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u251c\u2500\u2500 launcher-17\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u2502   \u2514\u2500\u2500 17180401151.desktop\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u251c\u2500\u2500 launcher-18\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u2502   \u2514\u2500\u2500 17180401152.desktop\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u251c\u2500\u2500 launcher-19\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u2502   \u2514\u2500\u2500 17180401153.desktop\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u2514\u2500\u2500 launcher-20\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u2514\u2500\u2500 17180401164.desktop\n    \u2502           \u2502   \u2502   \u2502       \u251c\u2500\u2500 xfconf\n    \u2502           \u2502   \u2502   \u2502       \u2502   \u2514\u2500\u2500 xfce-perchannel-xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 displays.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 thunar.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 xfce4-desktop.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 xfce4-keyboard-shortcuts.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 xfce4-notifyd.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 xfce4-panel.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u251c\u2500\u2500 xfce4-power-manager.xml\n    \u2502           \u2502   \u2502   \u2502       \u2502       \u2514\u2500\u2500 xfwm4.xml\n    \u2502           \u2502   \u2502   \u2502       \u2514\u2500\u2500 xfwm4\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 Desktop\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 development\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 .bash_logout\n    \u2502           \u2502   \u2502   \u2502   \u2514\u2500\u2500 .config\n    \u2502           \u2502   \u2502   \u2502       \u2514\u2500\u2500 mc\n    \u2502           \u2502   \u2502   \u2502           \u251c\u2500\u2500 DavidsGoTar.ini\n    \u2502           \u2502   \u2502   \u2502           \u251c\u2500\u2500 ini\n    \u2502           \u2502   \u2502   \u2502           \u2514\u2500\u2500 panels.ini\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 Documents\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 Downloads\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .gnupg\n    \u2502           \u2502   \u2502   \u2502   \u2514\u2500\u2500 private-keys-v1.d\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .ICEauthority\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .local\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 share\n    \u2502           \u2502   \u2502   \u2502   \u2514\u2500\u2500 state\n    \u2502           \u2502   \u2502   \u2502       \u2514\u2500\u2500 wireplumber\n    \u2502           \u2502   \u2502   \u2502           \u2514\u2500\u2500 stream-properties\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 .mozilla\n    \u2502           \u2502   \u2502   \u2502   \u251c\u2500\u2500 extensions\n    \u2502           \u2502   \u2502   \u2502   \u2514\u2500\u2500 plugins\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 Music\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 Pictures\n<SNIP>\n    \u2502           \u2502   \u251c\u2500\u2500 .ssh\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 authorized_keys\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 id_ed25519\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 id_ed25519.pub\n    \u2502           \u2502   \u2502   \u251c\u2500\u2500 known_hosts\n    \u2502           \u2502   \u2502   \u2514\u2500\u2500 known_hosts.old\n    \u2502           \u2502   \u251c\u2500\u2500 .tcshrc\n    \u2502           \u2502   \u251c\u2500\u2500 toprc\n    \u2502           \u2502   \u251c\u2500\u2500 updateGeoIPDatabases\n    \u2502           \u2502   \u251c\u2500\u2500 updateVBExtPack.sh\n    \u2502           \u2502   \u251c\u2500\u2500 UpgradeFedora.sh\n    \u2502           \u2502   \u251c\u2500\u2500 .viminfo\n    \u2502           \u2502   \u251c\u2500\u2500 xinitrc\n    \u2502           \u2502   \u2514\u2500\u2500 xinitrc.alt\n    \u2502           \u2514\u2500\u2500 TimeStamp\n    \u2514\u2500\u2500 lost+found\n\n120 directories, 130 files<\/code><\/pre>\n\n\n\n
      Figure 1: The existence of these directories and files indicate that the backup was successfully created.<\/em><\/p>\n\n\n\n
      Since this backup was performed on a test VM, I have no personal data files in my home directory, just configuration files for the most part.<\/p>\n\n\n\n
      Backup Strategy<\/h2>\n\n\n\n
      I use a backup strategy in which I employ the capabilities of rsync command in my script. I designed it to create a date-sequence of backups for each host in my network. The backup drives end up with a structure similar to the one shown in the output from the tree command above. You can see a directory level sample of how this would look for multiple hosts over multiple days in Figure 2. <\/p>\n\n\n\n
      This makes it easy for both SysAdmins and users to locate specific files that might need to be restored.<\/p>\n\n\n\n
      \/-\n |\n \/-path to backup media\n           |\n            \/Backups\n               |\n               |--\/host1\n               |   |--\/2018-01-01\n               |   |      |--\/etc\n               |   |      |--\/home\n               |   |      |--\/var\n               |   |      |--\/usr\/local\n               |   V      V\n               |   |--2018-01-02\n               |   |      |--\/etc\n               |   |      |--\/home\n               |   |      |--\/var\n               |   |      |--\/usr\/local\n               |   V      V\n               |   |--2018-01-03\n               |   |      |--\/etc\n               |   |      |--\/home\n               |   |      |--\/var\n               |   |      |--\/usr\/local\n               |   V      V\n               |--host2\n               |   |--2018-01-01\n               |   |      |--\/etc\n               |   |      |--\/home\n               |   |      |    |\n               |   |      |    |--\/student\n               |   |      |    |     |\n               |   |      |    |     |--\/file1.txt (Unchanged) \n               |   |      |    |     |\n               |   |      |    V     V\n               |   |      |--\/var\n               |   |      |--\/usr\/local\n               |   |      V\n               |   |--2018-01-02\n               |   |      |--\/etc\n               |   |      |--\/home\n               |   |      |    |\n               |   |      |    |--\/student\n               |   |      |    |     |\n               |   |      |    |     |--\/file1.txt (Unchanged)\n               |   |      |    V     V\n               |   |      |--\/var\n               |   |      |--\/usr\/local\n               |   |      |\n               |   |      V\n               |   |      \n               |   |--2018-01-03\n               |   |      |--\/etc\n               |   |      |--\/home\n               |   |      |    |\n               |   |      |    |--\/student\n               |   |      |    |     |\n               |   |      |    |     |--\/file1.txt  (Changed)\n               |   |      |    V     V\n               |   |      |--\/var\n               |   |      |--\/usr\/local\n               |   |      |\n               V   V      V<\/code><\/pre>\n\n\n\n
      Figure 2. The directory structure for my backup data storage disks.<\/em><\/p>\n\n\n\n
      Starting with an empty disk on January 1, the rsbu script makes a complete backup for each host of all the files and directories that I have specified in the configuration file. This first backup can take several hours if you have a lot of data like I do.<\/p>\n\n\n\n
      On January 2, the rsync command uses the \u2013link-dest= option to create a complete new directory structure identical to that of January 1, then it looks for files that have changed in the source directories. If any have changed, A copy of the original file from January 1 is made in the January 2 directory and then the parts of the file that have been altered are updated from the original.<\/p>\n\n\n\n
      After the first backup onto an empty drive, the backups take very little time because the hard links are created first, and then only the files that have been changed since the previous backup need any further work. The resulting backups will look similar to that in Figure 1.<\/p>\n\n\n\n
      Figure 2 also shows a bit more detail for the host2 series of backups for one file, \/home\/student\/file1.txt, on the dates January 1, 2, and 3. On January 2 the file has not changed since January 1. In this case, the rsync backup does not copy the original data from January 1. It simply creates a directory entry with a hard link in the January 2 directory to the January 1 directory which is a very fast procedure. We now have two two directory entries pointing to the same data on the hard drive. On January 3, the file has been changed. In this case, the data for ..\/2018-01-02\/home\/student\/file1.txt is copied to the new directory, ..\/2018-01-03\/home\/student\/file1.txt and any data blocks that have changed are then copied to the backup file for January 3. These strategies, that are implemented using features of the rsync program, allow backing up huge amounts of data while saving disk space and much of the time that would otherwise be required to copy data files that are identical.<\/p>\n\n\n\n
      One of my procedures is to run the backup script twice each day from a single systemd timer. The first iteration performs a backup to an internal 4TB hard drive. This is the backup that is always available and always at the most recent version of all my data. If something happens and I need to recover one file or all of them, the most I could possibly lose is a few hours worth of work.<\/p>\n\n\n\n
      The second backup is made to one of a rotating series of 4TB external USB hard drive. I take the most recent drive to my safe deposit box at the bank at least once per week. If my home office is destroyed and the backups I maintain there are destroyed along with it, I just have to get the external hard drive from the bank and I have lost at most a single week of data. That type of loss is easily recovered.<\/p>\n\n\n\n
      The drives I am using for backups, not just the internal hard drive but also the external USB storage devices that I rotate weekly, never fill up. This is because the rsbu script I wrote checks the ages in days of the backups on each drive before a new backup is made. If there are any backups on the drive that are older than the specified number of days, they are deleted. The script uses the find command to locate these backups. The number of days is specified in the rsbu.conf configuration file.<\/p>\n\n\n\n
      Of course after a complete disaster, I would first have to find a new place to live with office space for my wife and I, purchase parts and build new computers, restore from the remaining backups, and then recreate any lost data.<\/p>\n\n\n\n
      Tip: I have looked at a number of expensive commercial backup programs over the years. None of them are as easy to use as my script using rsync and some of them are actually just commercial front-ends for an rsync back-end.<\/p>\n\n\n\n
      Recovery Testing<\/h2>\n\n\n\n
      No backup regimen would be complete without testing. You should regularly test recovery of random files or entire directory structures to ensure not only that the backups are working, but that the data in the backups can be recovered for use after a disaster. I have seen too many instances where a backup could not be restored for one reason or another and valuable data was lost because the lack of testing prevented discovery of the problem.<\/p>\n\n\n\n
      Just select a file or directory to test and restore it to a test location such as \/tmp so that you won\u2019t overwrite a file that may have been updated since the backup was performed. Verify that the files\u2019 contents are as you expect them to be. Restoring files from a backup made using the rsync commands above simply a matter of finding the file you want to restore from the backup and then copying it to the location to which you want to restore it.<\/p>\n\n\n\n
      I have had a few circumstances where I have had to restore individual files and, occasionally, a complete directory structure. I have had to restore the entire contents of a hard drive on a couple occasions. Most of the time this has been self-inflicted when I accidentally deleted a file or directory. At least a few times it has been due to a crashed hard drive. So those backups do come in handy.<\/p>\n\n\n\n
      Configuring hosts and directories to backup<\/h2>\n\n\n\n
      Now that you’ve completed a successful test backup, you need to look at the rsbu.conf file in order to understand how to configure it for the specifics of your environment. Although the rsbu.conf file shown in Figure 3 is fairly simple and, I would like to think, self-explanatory, these things rarely are. Get ready for a new revelation about how cool is Bash!<\/p>\n\n\n\n
      ################################################################################\n#                               rsbu.conf                                      #\n################################################################################\n################################################################################\n################################################################################\n#                                                                              #\n#  Copyright (C) 2021 David Both                                               #\n#  LinuxGeek46@both.org                                                        #\n#                                                                              #\n#  This program is free software; you can redistribute it and\/or modify        #\n#  it under the terms of the GNU General Public License as published by        #\n#  the Free Software Foundation; either version 3 of the License, or           #\n#  (at your option) any later version.                                         #\n#                                                                              #\n#  This program is distributed in the hope that it will be useful,             #\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of              #\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #\n#  GNU General Public License for more details.                                #\n#                                                                              #\n#  You should have received a copy of the GNU General Public License along     #\n#  with this program; if not, If not, see <https:\/\/www.gnu.org\/licenses\/>.     #\n#                                                                              #\n################################################################################\n################################################################################\n################################################################################\n# NOTE: We now handle multiple backup devices using arrays for the various     #\n# data variables.                                                              #\n################################################################################\n# Configuration file for rsbu, David's RSYNC BackUp program. \n################################################################################\n################################################################################\n#                               rsbu.conf                                      #\n################################################################################\n################################################################################\n################################################################################\n#                                                                              #\n#  Copyright (C) 2021 David Both                                               #\n#  LinuxGeek46@both.org                                                        #\n#                                                                              #\n#  This program is free software; you can redistribute it and\/or modify        #\n#  it under the terms of the GNU General Public License as published by        #\n#  the Free Software Foundation; either version 3 of the License, or           #\n#  (at your option) any later version.                                         #\n#                                                                              #\n#  This program is distributed in the hope that it will be useful,             #\n#  but WITHOUT ANY WARRANTY; without even the implied warranty of              #\n#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the               #\n#  GNU General Public License for more details.                                #\n#                                                                              #\n#  You should have received a copy of the GNU General Public License along     #\n#  with this program; if not, If not, see <https:\/\/www.gnu.org\/licenses\/>.     #\n#                                                                              #\n################################################################################\n################################################################################\n################################################################################\n# NOTE: We now handle multiple backup devices using arrays for the various     #\n# data variables.                                                              #\n################################################################################\n# Configuration file for rsbu, David's RSYNC BackUp program. \n################################################################################\n# Always unmount the backup medium. This is for data safety \nunmount=1\n#\n################################################################################\n# The Backup Section contains arrays for defining the backups retention ages   #\n# and definitions of the backup media and their mount points.                  #\n################################################################################\n\n# Set the oldest age in days of the backup files to retain.\n# Files older than this number of days will be deleted.\n# Use a 1 day retention to only have one backup with no dates.\nAge[1]=60\n# Age[2]=90\n#\n\n################################################################################\n# The path to the media mount point. The mount point must already be           #\n# configured in \/etc\/fstab or this program will not work.                      #\n################################################################################\nMountPoint[1]=\"\/media\/MyBackups\"\n# MountPoint[2]=\"\/media\/MyBackups2\"\n################################################################################\n# The path to the backup directory which may be different from the mount-point #\n# especially in cases of NFS or SAMBA mounts                                   #\n################################################################################\nBasePath[1]=\"\/media\/MyBackups\/Backups\"\n# BasePath[2]=\"\/media\/MyBackups2\/Backups\"\n################################################################################\n# The device name is human readable text name                                  #\n################################################################################\nDeviceName[1]=\"External USB hard drive\"\n# DeviceName[2]=\"External HDD\"\n\n################################################################################\n# The hosts section contains the hostnames and a list of directories to be     #\n# backed up. Each is a semicolon separated line with the hostname, a list      #\n# of directories to back up and a list of exclusion patterns.                  #\n# The pattern of these lines is as follows:                                    #\n# hostname;excludes;list of directories to back up                             #\n# Numbering must be sequential as this creates an array used by rsbu.          #\n################################################################################\n\nBackup[1]=\"$ThisHost;--exclude .gvfs --exclude Cache ;:\/root :\/home :\/usr\/local\"\n# Backup[2]=\"mycomputer2;--exclude Cache --exclude yumdb --exclude .gvfs  ;:\/root :\/etc :\/home :\/usr\/local :\/var\"\n################################################################################\n# The Backup Section contains arrays for defining the backups retention ages   #\n# and definitions of the backup media and their mount points.                  #\n################################################################################\n\n# Set the oldest age in days of the backup files to retain.\n# Files older than this number of days will be deleted.\n# Use a 1 day retention to only have one backup with no dates.\nAge[1]=60\n# Age[2]=90\n#\n################################################################################\n# The path to the media mount point. The mount point must already be           #\n# configured in \/etc\/fstab or this program will not work.                      #\n################################################################################\nMountPoint[1]=\"\/media\/MyBackups\"\n# MountPoint[2]=\"\/media\/MyBackups2\"\n################################################################################\n# The path to the backup directory which may be different from the mount-point #\n# especially in cases of NFS or SAMBA mounts                                   #\n################################################################################\nBasePath[1]=\"\/media\/MyBackups\/Backups\"\n# BasePath[2]=\"\/media\/MyBackups2\/Backups\"\n################################################################################\n# The device name is human readable text name                                  #\n################################################################################\nDeviceName[1]=\"External USB hard drive\"\n# DeviceName[2]=\"External HDD\"\n\n################################################################################\n# The hosts section contains the hostnames and a list of directories to be     #\n# backed up. Each is a semicolon separated line with the hostname, a list      #\n# of directories to back up and a list of exclusion patterns.                  #\n# The pattern of these lines is as follows:                                    #\n# hostname;excludes;list of directories to back up                             #\n# Numbering must be sequential as this creates an array used by rsbu.          #\n################################################################################\n\nBackup[1]=\"$ThisHost;--exclude .gvfs --exclude Cache ;:\/root :\/home :\/usr\/local\"\n# Backup[2]=\"mycomputer2;--exclude Cache --exclude yumdb --exclude .gvfs  ;:\/root :\/etc :\/home :\/usr\/local :\/var\"<\/code><\/pre>\n\n\n\n
      Figure 3: The rsbu.conf file defines the hosts and directories on each that are to be backed up.<\/em><\/p>\n\n\n\n
      The rsbu.conf file contains five one-dimensional indexed arrays<\/a>. I didn’t realize that Bash supported arrays until I wrote this code.3<\/a><\/sup> I can use them even if they’re unidimensional. Many languages I’ve used support multidimensional arrays but we can do everything we need with Bash and its one-dimensional ones. Bash arrays are zero-based so that the first item in the array has a subscript of zero such as Item[1]<\/code>.<\/p>\n\n\n\n
      There are five arrays defined in this configuration file. <\/p>\n\n\n\n
        \n
      1. Age[1]=60<\/strong>  — The oldest age in days of the backup files to retain — 60 days in this case. Files older than this will be deleted. The [1] indicates that this is the first item in the array and thus, the first backup device. The second item is commented out, but would be used if a second type of backup were used, such as an internal hard drive or a different external hard drive. The age value can only be set in the rsbu.config file.<\/li>\n\n\n\n
      2. MountPoint[1]=”\/media\/MyBackups”<\/strong> — Defines the mountpoint for the first backup device.<\/li>\n\n\n\n
      3. BasePath[1]=”\/media\/MyBackups\/Backups”<\/strong> — Defines the fully qualified base path for the first backup device. You can see this in Figure 1.<\/li>\n\n\n\n
      4. DeviceName[1]=”External USB hard drive”<\/strong> — The device name is human readable text name that has no effect on the actual backup process. It’s just a good way to identify the device for SysAdmins.<\/li>\n\n\n\n
      5. Backup[1]=”$ThisHost;–exclude .gvfs –exclude Cache ;:\/root :\/home :\/usr\/local”<\/strong> — This array is used to define the list of hosts and the directories to be backed up for each host. It can also be used to define specific patterns for files and directories that can be excluded from the backup. Usually, insteqad of using the variable $ThisHost, the actuasl hostname is given. Note that everything in this line that comes after the hostname is options and arguments for the rsync command, and is passed to rsync at the time each host is backed up. Multiple hosts can be defined as shown in Figure 3, with different directories specified to be backed up for eah host.<\/li>\n<\/ol>\n\n\n\n
        Preparing additional backup media<\/h3>\n\n\n\n
        You can prepare additional backup media. The instructions are the same as in the first part of the preparation as shown above.<\/p>\n\n\n\n
        root@testvm1:~# rsbu-setup -p<\/strong><\/code><\/pre>\n\n\n\n
        Multiple backup devices, i.e., more than a single external hard drive can be used for a given mountpoint so long as it has the proper extended filesystem name applied to it. The device preparation does this. This allows multiple devices to be used in a rotation so that the newest backup can be stored off-site for additional security in case the main site is destroyed.<\/p>\n\n\n\n
        Do It Yourself (DIY)<\/h2>\n\n\n\n
        Since this backup tool is already working for you — or at least it should be if you’ve followed along — the next step is for you to make a copy of the existing, working rsbu.conf file. Then make any changes necessary to the Backup[1] array item to create the backups you need for the current host. Be sure to specify the hostname instead of using the variable. Test this before going any further.<\/p>\n\n\n\n
        Next, copy the public key from this host to the next host you want to perform backups for. The ssh-copy-id<\/code> command is how you’ll do this. <\/p>\n\n\n\n
        # ssh-copy-id hostname<\/strong><\/code><\/pre>\n\n\n\n
        Then respond to the prompts as you did for this section of the initial preparation. <\/p>\n\n\n\n
        Add a new line to the Backup array. Specify the hostname and the directories to be backed up. Then test to ensure that both hosts are properly backed up. <\/p>\n\n\n\n
        Repeat for additional hosts.<\/p>\n\n\n\n
        Automation<\/h2>\n\n\n\n
        Running the rsbu<\/code> program is easy. But what if you need to do it at oh-dark-thirty when no one is using the computers? Well — that’s another article. <\/p>\n\n\n\n
        Summary<\/h2>\n\n\n\n
        Backups are an incredibly important part of our jobs as SysAdmins. I have experienced many instances where backups have enabled rapid operational recovery for places I have worked as well as for my own business and personal data.<\/p>\n\n\n\n
        We use the rsync command as the basis for making backups that save both time and storage space. These backups can also be directly accessible to regular users. The scripts I have provided for download can get you started using this advanced and powerful backup method.<\/p>\n\n\n\n
        Like everything else, backups are all about what you need. Whatever you do \u2013 do something! Figure out how much pain you would have if you lost everything \u2013 data, computers, hard copy records \u2013 everything. The pain includes the cost of replacing the hardware and the cost of the time required to restore data that was backed up and to recover data that was not backed up. Then plan and implement your backup systems and procedures accordingly.<\/p>\n\n\n\n
        There are many options for performing and maintaining data backups. I do what works for me and have never had a situation where I lost more than a few hours worth of data.<\/p>\n\n\n\n
        \n\n\n\n
        Resources<\/h2>\n\n\n\n