Introduction

<aside> 💡 NOTE: KySync v1.1 is released, but if you are new to KySync read this document first.

</aside>

KySync is an efficient way to distribute new data which makes use of older (but similar) data that you may already have present locally. KySync supports HTTP v1.1, but can easily be extended to support any server protocol which supports range queries.

KySync is released under the MIT Open Source License (see COPYING in root of repository).

KySync is a full rewrite of Zsync in modern C++. While no code was reused from Zsync, the awesome Zsync technical paper was the major resource used for the implementation of KySync.

The value proposition of KySync over Zsync is that it takes advantage of modern architecture features (multi-core multi-CPU systems as well as exceedingly fast IO subsystems, e.g. NVMe SSDs). KySync is 3x-10x (or more) faster than Zsync on such commonly available modern hardware. We have not spent much time optimizing KySync single-thread performance, so there are cases where with sufficiently high similarity, Zsync is faster when less then 4 threads are used in KySync.

During the development of KySync, two side projects emerged:

• KyDeps — a simple package manager for CMake/C++ which grew organically as I was refactoring more and more of my CMake hacks to handle the needed dependencies;
• KyFarm — a simple distributed orchestrator that allows you to provision AWS instances, run experiments on them, collect the results through DynamoDB, and finally analyze them.

Both of these could be expanded and offered separately, but in this document I will focus only on the bare minimum necessary to build KySync, use it, and reproduce the experimental results presented.

We also have a presentation (WIP) explaining Rsync/Zsync basics, which eventually will cover KySync implementation details as well.

Some frequently asked questions will be addressed in our FAQ

Performance

In this section we compare the performance of KySync and Zsync along several dimensions on synthetically generated data. While we believe that the experiments are reasonably representative, we welcome feedback from the community on other experiments that may represent their scenarios.

Dimensions

The following are the dimensions along which we compare performance:

• Data Size — the size of remote source data that needs to be synced locally. We tested data sizes of 128 MiB, 512 MiB, 2 GiB, and 8 GiB.
• Similarity — a measure of how similar is the synced data to the data present locally in %. For example, a similarity of 95% means that we have 95% of the remote data present locally in the data we use as a seed. We tested similarities of 0%, 50%, 75%, 90%, 95%, and 100%.
• Threads — number of threads used in KySync for parallel execution. This is irrelevant for Zsync. We tested 1, 2, 4, 8, 16, 32, and 64 threads.