Performance of 🦙 with C++ Nix and Lix

Performance of 🦙 with C++ Nix and Lix

[sterni]

Hello everyone,

I performed some benchmarks of various (versions of) Nix implementations 
calculating the .drv path of //ops/pipelines/depot on nevsky. All IFD 
store paths were prepopulated in the store and the Nix daemon was the 
same in each case.

Hyperfine did 2 warmup runs and followed by 5 actual benchmark runs of

   nix-instantiate -I depot=. -I store=/nix/store --show-trace \
     --option restrict-eval true --option allowed-uris https:// \
     -A ops.pipelines.depot

on depot r/9761. The results are as follows.

//3p/cppnix (r/9761) (~ C++ Nix 2.3)
   Time (mean ± σ):     310.226 s ±  6.187 s    [User: 257.286 s, 
System: 42.878 s]
   Range (min … max):   300.664 s … 316.956 s    5 runs

C++ Nix 2.31.2
   Time (mean ± σ):     197.645 s ±  7.784 s    [User: 212.113 s, 
System: 36.639 s]
   Range (min … max):   189.609 s … 208.111 s    5 runs

C++ Nix 2.32.3
   Time (mean ± σ):     204.543 s ±  1.365 s    [User: 192.059 s, 
System: 39.068 s]
   Range (min … max):   203.058 s … 206.042 s    5 runs

Lix 2.93.3
   Time (mean ± σ):     209.785 s ± 15.477 s    [User: 159.669 s, 
System: 38.836 s]
   Range (min … max):   188.076 s … 231.488 s    5 runs


As you can see, performance has improved a lot since Nix 2.3.18, at 
least for our workload. Switching to Lix would probably make the most 
sense if we were to switch as their goals are most aligned with our desires.

In general, performance between Lix and C++ Nix >= 2.31 is pretty much a 
wash (at least it's not really possible to conclude much with just 5 
runs). C++ Nix 2.31 outperforming 2.32 is sort of expected at the moment 
(though according to Raito, we can expect this to be resolved in the 
future). 2.32 should in theory use less memory which I haven't measured 
at all, but would also be interesting to us as we recently (r/9688) had 
to reduce pipeline eval parallelism due to memory demands.

~sterni