ownCloud Delta Sync to Accelerate Synchronization of Updated Files

Launching Delta Sync, ownCloud GmbH offers a technology that accelerates the synchronization of updated files.

Diagram shows synchronization process compared
to previous ‘full’ synchronization
Click to enlarge

This means that in case of a file change, only the modified parts will be uploaded or downloaded instead of the whole file. Delta Sync reduces the volume of data to be transmitted. The feature was developed by the ownCloud community and will be included in the next major version of ownCloud.

Delta Sync allows the company’s users to reduce the file sizes that have to be uploaded or downloaded from the server after changing a file. Before transferring a file, the software compares its new version with the previous one, and then only replaces the part of the file that has been modified.

For example, if an uncompressed large video file (e.g. 50GB) had to be synchronized with the firm’s service and edited on the server in the past (e.g. by adding a cover image), it would have been necessary to replace the entire 50GB. Delta Sync now allows you to upload and download essentially only the difference between these two files (the delta) – in this case the amount of 10MB that represents the cover image. The technology is particularly effective for downloads, since only the changes are synchronized here and files are often shared with several users – the potentially saved data volume thus increases parallel to the number of recipients.

Delta Sync is currently in the testing phase and works for uncompressed formats. It is particularly useful for files that have been changed only slightly. The feature is licensed on the server side under AGPLv3 and the client side under GPLv2, and can therefore be freely used and further developed by the community.

Developed by community
The demand for the implementation of Delta Sync already existed for several years within the developer community. The interest is accordingly high. On the developer platform, Bountysource, the project was endowed with the highest developer bonus ever offered among all ownCloud projects. Delta Sync was developed by a community member with the support of the firm’s development team and has been further supported by numerous contributors, some of whom contributed three-figure bounty fees.

The main implementation was the integration of the ZSync algorithm into the ownCloud server and the desktop client. It is mainly used in Linux distributions where rapidly changing beta ISO image files are distributed. It uses the HTTP protocol and .zsync files with precalculated rolling hashes to minimize server load while still enabling diff-transfer transmission for network optimization.

Delta Sync supports following file formats:

• Crypto Container (e.g. VeraCrypt)

• Disc images (.iso /.img)

• Audio (raw audio formats /.wav /.aiff)

• Images (.svg /.tiff /.bmp /.dng)

• Video (raw video formats, e.g. uncompressed.avi)

• VM Images (.vdi /.ova /.img /RAW)

• Microsoft PST Files (emails, calendar entries, etc.)

• Text Files

• Tar Archives

Testing
The official release of Delta Sync for the company’s server and desktop client will take place with the next major release later this year. The testing phase will then begin, in which the tech community will be able to try out the feature and provide feedback. This step is followed by stabilization of the new sync algorithm and further testing, performance measurements and performance optimization, until the feature is finally available to all firm’s users.