Re: [tvix] string contexts vs. reference scanning

[Vincent Ambo <mail@tazj•in>]

sterni <sternenseemann@systemli•org> writes:
> One thing I'd want to see answered is how to handle import from
> derivation.

Files enter an evaluation through the `EvalIO` interface, which is
passed the path(s) to read and returns the contents or store path.

In the actual implementation of `EvalIO` that supports store
interactions, store path access is detected and blocked/suspended until
build completion.

At the time that this calculation occurs, we already called
builtins.derivation or equivalent on the thing yielding that path, so
the build is already running.

> There are also similar occasions where things get realised while
> evaluating (interactively?) except reading / importing, but I don't have
> a very good handle on those yet.

I'm not sure what those would be, but there's nothing the evaluator can
do to read from disk that wouldn't go through EvalIO.

> 1. You can discard some using builtins.unsafeDiscardOutputDependency.

This doesn't really discard the context, it changes the _type_ of the
context entry. See cl/7807.

I believe that this is unnecessary, all uses of it in nixpkgs are a hack.

> My _unconfirmed_ theory is that this was a quick and easy workaround
> that was implemented without considering the underlying problem. In my
> view, there is no reason why `drvPath` should incur a reference to all
> outputs of the derivation as well as the derivation file itself (I
> think this is thanks to the reference scanner the store runs after the
> fact which determines if the derivation and/or any of its outputs are
> actually referenced).

Well, the drv file brings everything inside of it into scope (that
includes its output paths, which are part of the serialised
representation). Otherwise referencing a drv file would make "dead
paths" available in the builder.

It's conceptionally reasonable that this closure is yielded, I think. It
seems odd to me that it involves store database queries, though. All the
information should be *in* the drv file.

In Tvix, if a drv path is referenced then that drv was also part of the
evaluation, so we *should* already know all of that information without
running any queries.

> - To drop wrongfully retained string context. All string
>   operations retain string context, even though some actually
>   destroy any reference that was present in the string.

This is basically a workaround for how contexts work in C++ Nix. It's
not a problem if contexts are replaced by scanning.

> - As an escape hatch from references to the derivations in question.
> We use this in //nix/buildkite:

Yes, but we use it to work around a problem that we only have because of
the model of C++ Nix. In Tvix this whole thing would work differently
(the evalution itself drives the creation of the build targets).

We'd essentially have a buildkite-driver that uses Tvix to evaluate and
yields the build targets as necessary, using a store (local or remote)
as the synchronisation point.

The other uses are covered in https://cl.tvl.fyi/7807 and I think fairly
uncontroversial.

My gut feeling is that we can get away with not implementing any of the
destructive string context operations, and if we _really_ end up needing
unsafeDiscardStringContext, it's not terribly difficult to implement
(could be a `Value` variant containing another Value that just acts as a
marker).

//V