Re: [tvix] string contexts vs. reference scanning

[Vincent Ambo <tazjin@tvl.su>]

Okay, there's some (for me) new information on string contexts that will 
require some careful handling if we want to proceed with reference scanning.

Let me preface by saying that despite this problem, reference-scanning 
for inputs yields perfectly functional, *equivalent* but not *identical* 
derivations to Nix. We do currently consider it a problem because we 
want to be fully hash-equal with C++ Nix, both to prove that our 
implementation is correct and to make use of Hydra's cache.

Now, for the problem:

There are some real-life cases, for example during nixpkgs 
bootstrapping, where multiple different fixed-output derivations are 
written to produce the same hash.

For example, bootstrap sources that are downloaded early are fetched 
using a special "builder hack", in which the `builder` field of the 
derivation is populated with the magic string `builtins:fetchurl` and 
the builder itself will perform a fetch, with everything looking like a 
normal derivation to the user.

These bootstrap sources are later on defined *again*, once `curl` is 
available, to be downloaded using the standard pkgs.fetchtarball 
mechanism, but yielding the *same* outputs (as the same files are being 
fetched).

In our reference scanning implementation, this output scanning of FOD 
will yield whatever the *first* derivation was that produced the given 
path as the drv to be stored in the `inputDrvs` field of the derivation.

There's an orthogonal problem which made this confusing to understand, 
where C++ Nix has some special logic for how it hashes derivations that 
use fixed-output paths, which we haven't fully replicated yet. This led 
to hash differences which masked this underlying problem (the 
differences still exist, and are a separate issue).

I discussed this with Adam yesterday and he suggested an approach 
similar to `builtins.placeholder`, which would only be in effect for 
fixed-output derivations. I think this is feasible but haven't sketched 
anything yet.

Either way, for me this also raises the thought of whether we should 
decouple Tvix's internal representation of a derivation from that of C++ 
Nix and only "materialise" C++ Nix derivations (and accompanying hashes) 
where needed. Something to think about ...

//V