TVL depot development (mail to depot@tvl.su)
 help / color / mirror / code / Atom feed
* here is how to write efficient castore
@ 2024-05-22 12:52 Askar Safin
  0 siblings, 0 replies; only message in thread
From: Askar Safin @ 2024-05-22 12:52 UTC (permalink / raw)
  To: depot

Hi. I spent a lot of time writing my own simple (but very efficient)
castore in Rust. It is parallel. It deduplicates data using chunks
(but not CDC). And it compresses. And it has 200 lines of Rust code
only!!! It has many important optimizations, which are absent in
alternative castores. Consider taking ideas from my castore and
comparing efficiency of my castore and yours. Here is link to github
comment with my castore (small and self-contained) implementation:
https://github.com/borgbackup/borg/issues/7674#issuecomment-1654175985
(it is called azwyon). (linked version of code is slightly old, it
uses "rayon" instead of "pariter" [which is used by current version of
azwyon], but this doesn't affect performance of the code, so don't
worry; if you need current version of the code, i can send it)

here are benchmarks, which compare azwyon to many other alternative
castores: https://github.com/borgbackup/borg/issues/7674#issuecomment-1656787394

Also see the whole thread (
https://github.com/borgbackup/borg/issues/7674 ) for discussion of
various optimizations made in azwyon.

Also see this summary, which describes what is wrong with existing castores:
https://lobste.rs/s/0itosu/look_at_rapidcdc_quickcdc#c_ygqxsl

Assume my code is public domain.

Also I want to note that I'm slightly cheating: I use fixed sized
chunking instead of CDC. This is what I exactly need for my problem
domain (i. e. for storing VM images). Your problem domain probably
need CDC. So, if you want to take my code, you will probably have to
switch it to CDC. This will decrease its speed, but will allow more
compression.

Okay, so how did I achieve this? Why my azwyon is so efficient? Well,
here are reasons for success:

- I used Rust as opposed to slower languages, such as Go
- I used Rust, and this enabled me to have "fearless concurrency"
- I used this library for parallelization
https://dpc.pw/posts/adding-parallelism-to-your-rust-iterators/ and it
is absolutely great (note: version linked above uses rayon instead of
pariter, but don't worry, this doesn't really matter)
- I "cheated" by using fixed sized chunking instead of slower CDC
(fortunately, fixed sized chunking is exactly what I need for my
particular kind of data)
- I used zstd as opposed to less efficient compression methods
- I used blake3 as opposed to slower hash algorithms

-- 
Askar Safin

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2024-05-22 12:52 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-05-22 12:52 here is how to write efficient castore Askar Safin

Code repositories for project(s) associated with this public inbox

	https://code.tvl.fyi

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).