Files synchronization monitoring (FSM)

Files synchronization monitoring (FSM)

These days one-way file synchronization or mirroring is a widely used process. It is a process where files from one remote “source” location are copied/updated to another remote “target” location, but never in the reverse direction. This process is mostly used during backup or update procedures. This paper describes the Monitis custom monitor approach to online monitoring for one-way file synchronization in Unix based systems.

Usage approach

The logic of the FSM monitor consists of two parts: watching and processing.

The watching part fetches the monitoring folders files list with corresponding checksums and accumulates necessary information. The processing part periodically reads the accumulated information, executes necessary comparisons (by file name and checksum) and calculations and sends them to Monitis’ main server. So, you can simply see the FSM status online on your Monitis account dashboard. This approach allows you to monitor as many folders as you need. If more than one folder is monitored, the program will compare the succeeding folders defined on Host A with the same defined on Host B. To illustrate, if you declare:

declare -r DIR_A="/home/folder1.//./home/folder2"

declare -r DIR_B="/opt/folder1.//./opt/folder2"

then the content of the folder “/home/folder1” on Host A will be compared with the content of the “/opt/folder1” folder on Host B.

Consecutively, the content of the folder “/home/folder2” on Host A will be compared with the content of the “/opt/folder2” folder on Host B.

Note that the final results will consist of the accumulated results of all compared files.

file synchronization monitoring-1 

FSM monitor calculates and shows the following metrics: 

  1. The Monitored Files Count – The total number of “source” folder(s) files which are supposed to be copied/updated.
  2. The percent of desynch – The percentage of files that are not synchronized yet.

FSM monitor customization

The two main scripts of the monitor are: the watching and accumulating script (called and the processing script (called which periodically requests data, processes it and sends it to the Monitis server.

Note: since the runs on remote machines you need to have ssh access to remote servers which have folders that are monitored.

Also, there is the Monitis OpenAPI wrapper script (called and a few additional serviced scripts. Thus, as you can see, it was fully implemented by using Bash scripts.

To use it you should tune it by replacing some parameters (constants) such as APIKey, ServiceKey, Host A IP (“source” folder’s location), Host B IP (“target” folder’s location), and so on with the ones you own. For detailed instructions you can look through the sourcecode.

FSM monitor test

The simplest configuration was chosen for testing the one way file synchronization.

In two different hosts one-way file synchronization was configured and run. As a result we got the following monitoring table in our Monitis account:


And the graph view of the same information:

As you can see in the graph, at some point the desynch percent is the highest possible, which is quite possible since some files were added to the “source” folder and had not yet been copied/updated to the “target” folder.

You can find the necessary files in our Git.