Make the Derived Path family of types inductive for dynamic derivations

We want to be able to write down `foo.drv^bar.drv^baz`:
`foo.drv^bar.drv` is the dynamic derivation (since it is itself a
derivation output, `bar.drv` from `foo.drv`).

To that end, we create `Single{Derivation,BuiltPath}` types, that are
very similar except instead of having multiple outputs (in a set or
map), they have a single one. This is for everything to the left of the
rightmost `^`.

`NixStringContextElem` has an analogous change, and now can reuse
`SingleDerivedPath` at the top level. In fact, if we ever get rid of
`DrvDeep`, `NixStringContextElem` could be replaced with
`SingleDerivedPath` entirely!

Important note: some JSON formats have changed.

We already can *produce* dynamic derivations, but we can't refer to them
directly. Today, we can merely express building or example at the top
imperatively over time by building `foo.drv^bar.drv`, and then with a
second nix invocation doing `<result-from-first>^baz`, but this is not
declarative. The ethos of Nix of being able to write down the full plan
everything you want to do, and then execute than plan with a single
command, and for that we need the new inductive form of these types.

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

1 files changed, 190 insertions, 12 deletions

diff --git a/src/libstore/derived-path.hh b/src/libstore/derived-path.hh
index 7a4261ce0..ec30dac61 100644
--- a/src/libstore/derived-path.hh
+++ b/src/libstore/derived-path.hh
@@ -24,15 +24,135 @@ class Store;
 struct DerivedPathOpaque {
     StorePath path;
 
-    nlohmann::json toJSON(ref<Store> store) const;
     std::string to_string(const Store & store) const;
     static DerivedPathOpaque parse(const Store & store, std::string_view);
+    nlohmann::json toJSON(const Store & store) const;
 
     GENERATE_CMP(DerivedPathOpaque, me->path);
 };
 
+struct SingleDerivedPath;
+
+/**
+ * A single derived path that is built from a derivation
+ *
+ * Built derived paths are pair of a derivation and an output name. They are
+ * evaluated by building the derivation, and then taking the resulting output
+ * path of the given output name.
+ */
+struct SingleDerivedPathBuilt {
+    ref<SingleDerivedPath> drvPath;
+    std::string output;
+
+    /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
+    /**
+     * Uses `^` as the separator
+     */
+    std::string to_string(const Store & store) const;
+    /**
+     * Uses `!` as the separator
+     */
+    std::string to_string_legacy(const Store & store) const;
+    /**
+     * The caller splits on the separator, so it works for both variants.
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static SingleDerivedPathBuilt parse(
+        const Store & store, ref<SingleDerivedPath> drvPath,
+        std::string_view outputs,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    nlohmann::json toJSON(Store & store) const;
+
+    DECLARE_CMP(SingleDerivedPathBuilt);
+};
+
+using _SingleDerivedPathRaw = std::variant<
+    DerivedPathOpaque,
+    SingleDerivedPathBuilt
+>;
+
 /**
- * A derived path that is built from a derivation
+ * A "derived path" is a very simple sort of expression (not a Nix
+ * language expression! But an expression in a the general sense) that
+ * evaluates to (concrete) store path. It is either:
+ *
+ * - opaque, in which case it is just a concrete store path with
+ *   possibly no known derivation
+ *
+ * - built, in which case it is a pair of a derivation path and an
+ *   output name.
+ */
+struct SingleDerivedPath : _SingleDerivedPathRaw {
+    using Raw = _SingleDerivedPathRaw;
+    using Raw::Raw;
+
+    using Opaque = DerivedPathOpaque;
+    using Built = SingleDerivedPathBuilt;
+
+    inline const Raw & raw() const {
+        return static_cast<const Raw &>(*this);
+    }
+
+    /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
+    /**
+     * Uses `^` as the separator
+     */
+    std::string to_string(const Store & store) const;
+    /**
+     * Uses `!` as the separator
+     */
+    std::string to_string_legacy(const Store & store) const;
+    /**
+     * Uses `^` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static SingleDerivedPath parse(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    /**
+     * Uses `!` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
+     */
+    static SingleDerivedPath parseLegacy(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    nlohmann::json toJSON(Store & store) const;
+};
+
+static inline ref<SingleDerivedPath> makeConstantStorePathRef(StorePath drvPath)
+{
+    return make_ref<SingleDerivedPath>(SingleDerivedPath::Opaque { drvPath });
+}
+
+/**
+ * A set of derived paths that are built from a derivation
  *
  * Built derived paths are pair of a derivation and some output names.
  * They are evaluated by building the derivation, and then replacing the
@@ -44,10 +164,22 @@ struct DerivedPathOpaque {
  * output name.
  */
 struct DerivedPathBuilt {
-    StorePath drvPath;
+    ref<SingleDerivedPath> drvPath;
     OutputsSpec outputs;
 
     /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
+    /**
      * Uses `^` as the separator
      */
     std::string to_string(const Store & store) const;
@@ -57,11 +189,16 @@ struct DerivedPathBuilt {
     std::string to_string_legacy(const Store & store) const;
     /**
      * The caller splits on the separator, so it works for both variants.
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
-    static DerivedPathBuilt parse(const Store & store, std::string_view drvPath, std::string_view outputs);
-    nlohmann::json toJSON(ref<Store> store) const;
+    static DerivedPathBuilt parse(
+        const Store & store, ref<SingleDerivedPath>,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+    nlohmann::json toJSON(Store & store) const;
 
-    GENERATE_CMP(DerivedPathBuilt, me->drvPath, me->outputs);
+    DECLARE_CMP(DerivedPathBuilt);
 };
 
 using _DerivedPathRaw = std::variant<
@@ -71,13 +208,13 @@ using _DerivedPathRaw = std::variant<
 
 /**
  * A "derived path" is a very simple sort of expression that evaluates
- * to (concrete) store path. It is either:
+ * to one or more (concrete) store paths. It is either:
  *
- * - opaque, in which case it is just a concrete store path with
+ * - opaque, in which case it is just a single concrete store path with
  *   possibly no known derivation
  *
- * - built, in which case it is a pair of a derivation path and an
- *   output name.
+ * - built, in which case it is a pair of a derivation path and some
+ *   output names.
  */
 struct DerivedPath : _DerivedPathRaw {
     using Raw = _DerivedPathRaw;
@@ -91,6 +228,18 @@ struct DerivedPath : _DerivedPathRaw {
     }
 
     /**
+     * Get the store path this is ultimately derived from (by realising
+     * and projecting outputs).
+     *
+     * Note that this is *not* a property of the store object being
+     * referred to, but just of this path --- how we happened to be
+     * referring to that store object. In other words, this means this
+     * function breaks "referential transparency". It should therefore
+     * be used only with great care.
+     */
+    const StorePath & getBaseStorePath() const;
+
+    /**
      * Uses `^` as the separator
      */
     std::string to_string(const Store & store) const;
@@ -100,14 +249,43 @@ struct DerivedPath : _DerivedPathRaw {
     std::string to_string_legacy(const Store & store) const;
     /**
      * Uses `^` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
-    static DerivedPath parse(const Store & store, std::string_view);
+    static DerivedPath parse(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
     /**
      * Uses `!` as the separator
+     *
+     * @param xpSettings Stop-gap to avoid globals during unit tests.
      */
-    static DerivedPath parseLegacy(const Store & store, std::string_view);
+    static DerivedPath parseLegacy(
+        const Store & store,
+        std::string_view,
+        const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
+
+    /**
+     * Convert a `SingleDerivedPath` to a `DerivedPath`.
+     */
+    static DerivedPath fromSingle(const SingleDerivedPath &);
+
+    nlohmann::json toJSON(Store & store) const;
 };
 
 typedef std::vector<DerivedPath> DerivedPaths;
 
+/**
+ * Used by various parser functions to require experimental features as
+ * needed.
+ *
+ * Somewhat unfortunate this cannot just be an implementation detail for
+ * this module.
+ *
+ * @param xpSettings Stop-gap to avoid globals during unit tests.
+ */
+void drvRequireExperiment(
+    const SingleDerivedPath & drv,
+    const ExperimentalFeatureSettings & xpSettings = experimentalFeatureSettings);
 }