synqa: file synch via SSH

synqa is a Ruby gem which can be used to script the uploading of files to a remote server via SSH. Typically you would use it for maintaining a static website.

Installation

gem install synqa

Links

• Github repository for synqa
• A sample rakefile using synqa to upload files to a remove server
• The official synqa gem page

Alternatives

I wrote synqa as a simple alternative to rsync, which I couldn't get to work with Cygwin rsync running on Windows Vista talking to my website server on nearlyfreespeech.net.

I also wrote it partly as a way of gaining experience with SSH, putty, Ruby, rake and gems (which is my excuse for not trying really, really hard to solve my original problem with rsync).

How It Works

synqa works by constructing content trees for the local and remote directories which are to be synchronized. Each content tree consists of a list of all sub-directories of the base directory, all files within the base directory, and for each file, the hexadecimal cryptographic hash, and, also the local time that the calculation of the relevant content tree started.

The remote content tree is calculated by calling a hashing utility on the remote system, such as sha256sum (or sha256 -r for some systems), and using find and xargs to loop through all the files.

Files are copied using a specified SCP client (I recommend pscp, but scp should work). Remote commands are executed using a specified SSH client (I use plink, but ssh will work, as long as you don't have a setup where interactive password entry is required). Remote files are deleted by remotely executing the command rm via the SSH client.

A possible future optimisation would be to use psftp for all copying and remote deletion, which would allow all file update operations to occur within a single SSH session, thus minimising the overhead due to SSH session initiation.

Optimisations

To avoid having to calculate the hashes of all local and remote files every time, synqa has the option to specify that the local and remote content trees can be saved to local files.

Given the existence of either of these cached content trees, synqa can then do the following:

• For the local directory, synqa still enumerates the actual directories and files, however, for any file whose last modification time is earlier than the timestamp on the cached content tree, it reuses the cached hash value.
• For the remote directory, synqa assumes that the cached content tree is up-to-date, i.e. that the remote directory has not been updated by any other software other than the instance of synqa that created the same cached content tree file.
• When updating the remote directory, synqa deletes the cached remote content tree file, and, if the update completes successfully, it copies the local content tree file to the remote content tree file. (And if the update is interrupted, or fails for some reason, the cached remote content tree file has been deleted – so next time synqa will be forced to recalculate the remote content tree.)

If a synqa script has been configured to use cached content trees, and you want to force synqa to fully recalculate either content tree, simply delete the relevant content tree first. (The sample rakefile includes a clean target which deletes both content tree files.)