[tvix] Value Location, Value Documentation

[tvix] Value Location, Value Documentation

[sternenseemann]

Hi all,

I want to feel you out on a specific issue that is mostly a question of 
//tvix/eval, but still far-ish away.

Currently we can relate specific VM instructions back to a source 
location that caused them to be emitted by the compiler. This seems to 
work well enough for traces stemming from runtime errors.

We may also need to or want to relate values back to source locations in 
the following ways:

* builtins.unsafeGetAttrPos demands that we are able to return the
   source location where an attribute (given by its name) of a specific
   attribute set (given as a value) is defined. Note that the location
   where the attribute and its attribute value were defined may differ,
   e.g.:

     let val = 42; set = { attrName = val; }; in
     builtins.unsafeGetAttrPos "attrName" set

   val itself would trace back to a different location compared to the
   unsafeGetAttrPos expression.

* Additionally, I have the inclination that it would be useful to be
   able to retrieve the definition location of arbitrary values at
   runtime, but this is currently not strictly required.

   This feature would not (need to) be exposed in the language via a
   builtin, but instead used in the following ways:

   1. To enhance runtime traces. Similarly to how C++ Nix is able to
      display the source locations of lambdas during evaluation.

   2. To retrieve documentation: This would then work similarly to
      what Robert Hensing proposed for C++ Nix: The already tracked
      value location would be used to extract the documentation
      comment associated with it from the still available source
      file. This has the advantage that it has no runtime overhead
      unless used (assuming you are already tracking value source
      locations). See <https://github.com/NixOS/nix/pull/1652>.
      Robert's PR has become stale since, but would become relevant
      again if RFC0145 (<https://github.com/NixOS/rfcs/pull/145>)
      succeeds.

      An alternative was proposed by Eelco to support documentation
      via an extra attribute in a functor, but I don't think too
      many like this solution (<https://github.com/NixOS/nix/pull/5527>).
      andi- floated the idea of having a builtin to attach documentation
      to values in the same thread 
(<https://github.com/NixOS/nix/pull/5527#issuecomment-967236780>).
      Both ideas have the disadvantage that they cause runtime overhead
      even if the documentation is not used. Additionally, they make it
      hard to impossible to statically generate documentation à la
      nixdoc.

      hsjobeki has proposed a CL (<https://cl.tvl.fyi/8530>) for Tvix
      that implements a weird mix of both approaches: Documentation
      comments' contents are attached to lambda values (and lambda
      values only) at compile time, so that they can be retrieved
      at runtime (e.g. via the repl). This of course incurs an
      increased memory overhead regardless of whether the documentation
      is used or not.

Given this outline, I'd be interested to know:

- How can we implement source location tracking of values? How far are
   we willing/able to go (i.e. do we want to limit ourselves to
   unsafeGetAttrPos)?

- Or more explicitly: Do we want to stay away from mapping values to
   source locations as much as possible, as it is an inherently
   treacherous business in Nix?

- How do we want documentation to work? What are our requirements? This
   is especially interesting for RFC0145—as I am one of the shepherds,
   I can provisionally enter Tvix's perspective into the discussion.

-sterni

Re: [tvix] Value Location, Value Documentation

[Adam Joseph]

I know this message is a ~month old, but...

Quoting sternenseemann (2023-06-02 14:49:16)
> Currently we can relate specific VM instructions back to a source
> location that caused them to be emitted by the compiler.
> ...
> We may also need to or want to relate values back to source locations

It's important to keep in mind that adding metadata to values is much, much,
more expensive than adding metadata to instructions/opcodes.

> * builtins.unsafeGetAttrPos demands that we are able to return the
>    source location where an attribute (given by its name) of a specific
>    attribute set (given as a value) is defined.

Yes, unfortunately.  Hopefully when tvix will allow disabling it at compile
time, so there is no memory overhead.  I bet it's worth it for Hydra or
(especially) OfBorg, who have no need for `nix develop`.

>    1. To enhance runtime traces.   Similarly to how C++ Nix is able to
>       display the source locations of lambdas during evaluation.

Hrm, does this require adding metadata to *values*?  You should be able to
recover a stack trace just by looking at VM::frames and extracting from those a
Chunk, which has a SourceSpan.

>    2. To retrieve documentation:

These seem low-cost if they're opt-in.

>       hsjobeki has proposed a CL (<https://cl.tvl.fyi/8530>) for Tvix
>       that implements a weird mix of both approaches:

I've been having trouble understanding what's going on in that CL; there doesn't
seem to be a "here's the big picture" overview in the commit message.

> Documentation comments' contents are attached to lambda values

Why not attach them to a Chunk instead?  Every Lambda knows what Chunk it came
from, but Chunks are created only at compile time, not at runtime.

Since doc-comments can't be mutated at compile time it seems weird for them to
have a runtime representation.  Especially when attached to functions, which
(unlike the other variants of Value) can already be associated back to a
SourceSpan.

  - a