Filesharing Syncthing...........................
Keep different devices in sync on your LAN or over the internet
Neil Bothwick loves to share, in fact he just won’t stop sharing everything. Please! Someone make him stop! His latest tool of torture is Syncthing.
Do you need to keep some important files synchronised between different devices? Neil Bothwick shows you an easy way to do that.
Acouple of months ago, we looked at quick and simple ways to transfer files between computers, without using a commercial service or going through the complexities of setting up OwnCloud [Tutorials, p74 LXF201] and as good as they were, there was some work involved in getting the process up and running. Since then we’ve come across a simple file synchronisation program that automatically keeps different devices in sync, using local connections when they are on the same network and secure Internet transfers when they are not. This program is called Syncthing and uses a peer-to-peer protocol.
Computers running Syncthing are called nodes and each is identified by a certificate that is created when you first install the software. Communication between nodes is encrypted using TLS and the data is only stored on the individual nodes and not on a central server anywhere. The central server is only needed to enable the nodes to find one another and even that is not necessary if the computers are on the same LAN. Using certificates like this means that when you move your computer to a different network with a different address, it can still sync with your other nodes.
The first step is install the software. Debian and Ubuntu users can add their repository with these commands: $ curl -s https://syncthing.net/release-key.txt | sudo apt-key add - $ echo deb http://apt.syncthing.net/ syncthing release | sudo tee /etc/apt/sources.list.d/syncthing-release.list then install syncthing package with: $ sudo apt-get update $ sudo apt-get install syncthing
Users of other distros can grab the latest package from https://syncthing.net. This contains the syncthing binary that you can copy somewhere in your PATH.
First run
When you run Syncthing from a terminal, as a normal user, it will generate a default configuration along with your identity keys and open its administrative interface in a web browser. The left side of the window shows the directories you are sharing and one is created by default. The right side shows some information on your current computer and a list of the computers with which you are syncing. Of course, this list is currently empty, so you’ll need to repeat the process on another computer – if you don’t have two, why are you reading a tutorial on file synchronisation?
By default, the web interface is only accessible from the same computer. If you want to administer Syncthing on a remote computer, you need to edit ~/.config/syncthing/ config.xml. Find the section starting <gui and change the address from 127.0.0.1:8384 to 0.0.0.0:8384 to enable access from any IP address. If you are able to load the local browser interface, you can also change the GUI Listen Address from the Settings menu option. If you are going to enable access to the web interface from outside your router, you should set a username and password here for your protection. If access is only going to be from your LAN, you don’t need it.
Now let’s get the two computers talking. On the first computer, select ‘Show ID’ from the Action menu. While you can give each computer a name in the Settings panel, it usually defaults to the hostname. Syncthing uses unique ID strings to distinguish between computers. Copy the ID to your clipboard then open the other computer’s admin window in another tab, click on ‘Add Device’ and paste in the ID. You can give it a name if you wish, but the other options can be left alone for now. When you save, Syncthing will ask if it can restart itself; it does this in response to just about all configuration changes. Repeat the process on the other computer and they should show up on each other’s web interface. Because they create a default sync directory, with an identical folder ID, they will already be syncing that directory. Copy some files into the directory on one computer
and hit the ‘Rescan’ button and those files should now be on both computers.
That’s the basics working but, unlike Dropbox, you can sync more than one directory and you can control what does and doesn’t get copied. Click on ‘Add Folder’ and fill in the details. There are a number of important options here and the most critical is the folder ID. This must be the same on both computers for them to sync; it is the ID that matches up the directories not their paths on the disk. You can sync between different paths, which is useful, eg if you are syncing from your home directory to a backup folder on a file server.
Let’s look at some of the other options that can be set when creating or editing a folder for syncing. The Rescan interval is pretty obvious, the default is sixty seconds, which seems rather frequent for many uses, but it’s your choice.
Syncthing treats all copies of a synced folder as equal and changes to any one are propagated to the others (a folder can be synced with several computers). If a file is changed on more than one computer between scans, both copies are saved with conflict extensions. You’ll have to work out which one you want as the computer cannot know. The folder master option changes this behaviour, so that only changes on the computer with this flag set will propagate changes. You should apply this option if you are using Syncthing to maintain a backup copy. You can select which of your connected devices to share with, in which case a requester will pop up on the other computer’s web interface to confirm the action. Running all computers’ admin pages as tabs in the same browser makes setup much easier.
The file versioning option allows you to keep multiple copies of a file, in various ways. The Help link explains the choices in full, but the simple file versioning choice keeps a configurable number of recent copies of each file in a directory called .stversion. As new copies are added, older ones are deleted. Other settings give more or less control over the version control. You may not want to sync all the files in a directory and Syncthing provides a way to do this with a file called .stignore. This file, in the root of the directory you are sharing, contains a number of lines with file patters. If a file matches one of those lines, it’s ignored and not synced, eg: **/*.crdownload /.cache /.emacs.d/auto-save-list
The first is a pattern, ** which matches any number of directories, including none. This pattern matches the temporary files that Chromium creates for downloads, files that will change during the sync. The second matches a complete directory, the / anchors it to the folder path, not the root of the filesystem, the third matches a single file. You can also prefix a pattern with ! , which causes any match to be included, even if a subsequent pattern would have excluded it. You can use this to specify only those paths you want synced, which may be simpler than creating a separate folder for each of them. For example you could sync your
home directory and put this in .stignore to only include three directories. !/Dropbox !/Documents !/Music
You can edit ignore patterns in your favourite text editor or from a button in the folder settings window. Note that the
.stignore file itself is not synced to the other computer(s). If you want the contents to be synced, put them in a different file, I use .stglobalignore, and put this in . stignore #include /.stglobalignore.
Syncing on the move
Syncing between your computers is great, but Syncthing also has an Android app and a beta iOS app. The interface is rather different, but you will quickly see how to add devices and folders. Remember when you opened the window to show your computer’s Device ID and it contained a QR code? You can scan that with your phone’s camera to add your computer to your camera.
Syncthing uses a peer-to-peer protocol. Each computer has a unique ID and the central server simply puts them in touch with one another. If you are syncing between computers on the same network, you can use local discovery, in the Settings panel, to avoid the need for any Internet traffic. The whole thing is open source so if you want to communicate across the internet without sending any information to a third party, you can build and run your own discovery server. There’s a section on the comprehensive online documentation, linked from the bottom of the web admin page, that covers installing and running your own discovery server for total privacy.
The online documentation contains much more than we have room for here. This is enough to get you started, but hit the ‘Documentation’ button at the bottom of the web page if you need to know more.