Merge branch 'master' into valid_deriver_2

6 files changed, 215 insertions, 176 deletions

diff --git a/doc/manual/src/advanced-topics/advanced-topics.md b/doc/manual/src/advanced-topics/advanced-topics.md
index 8b1378917..9a4d12a33 100644
--- a/doc/manual/src/advanced-topics/advanced-topics.md
+++ b/doc/manual/src/advanced-topics/advanced-topics.md
@@ -1 +1 @@
-
+This section lists advanced topics related to builds and builds performance
diff --git a/doc/manual/src/command-ref/nix-prefetch-url.md b/doc/manual/src/command-ref/nix-prefetch-url.md
index 3bcd209e2..45ef01e02 100644
--- a/doc/manual/src/command-ref/nix-prefetch-url.md
+++ b/doc/manual/src/command-ref/nix-prefetch-url.md
@@ -31,15 +31,18 @@ store already contains a file with the same hash and base name.
 Otherwise, the file is downloaded, and an error is signaled if the
 actual hash of the file does not match the specified hash.
 
-This command prints the hash on standard output. Additionally, if the
-option `--print-path` is used, the path of the downloaded file in the
-Nix store is also printed.
+This command prints the hash on standard output.
+The hash is printed using base-32 unless `--type md5` is specified,
+in which case it's printed using base-16.
+Additionally, if the option `--print-path` is used,
+the path of the downloaded file in the Nix store is also printed.
 
 # Options
 
   - `--type` *hashAlgo*\
-    Use the specified cryptographic hash algorithm, which can be one of
-    `md5`, `sha1`, `sha256`, and `sha512`.
+    Use the specified cryptographic hash algorithm,
+    which can be one of `md5`, `sha1`, `sha256`, and `sha512`.
+    The default is `sha256`.
 
   - `--print-path`\
     Print the store path of the downloaded file on standard output.
diff --git a/doc/manual/src/contributing/testing.md b/doc/manual/src/contributing/testing.md
index c3c82e3c0..cd94d5cfb 100644
--- a/doc/manual/src/contributing/testing.md
+++ b/doc/manual/src/contributing/testing.md
@@ -7,7 +7,8 @@ under `src/{library_name}/tests` using the
 [googletest](https://google.github.io/googletest/) and
 [rapidcheck](https://github.com/emil-e/rapidcheck) frameworks.
 
-You can run the whole testsuite with `make check`, or the tests for a specific component with `make libfoo-tests_RUN`. Finer-grained filtering is also possible using the [--gtest_filter](https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests) command-line option.
+You can run the whole testsuite with `make check`, or the tests for a specific component with `make libfoo-tests_RUN`.
+Finer-grained filtering is also possible using the [--gtest_filter](https://google.github.io/googletest/advanced.html#running-a-subset-of-the-tests) command-line option, or the `GTEST_FILTER` environment variable.
 
 ## Functional tests
 
diff --git a/doc/manual/src/glossary.md b/doc/manual/src/glossary.md
index ac0bb3c2f..d950a4ca8 100644
--- a/doc/manual/src/glossary.md
+++ b/doc/manual/src/glossary.md
@@ -1,236 +1,269 @@
 # Glossary
 
-  - [derivation]{#gloss-derivation}\
-    A description of a build task. The result of a derivation is a
-    store object. Derivations are typically specified in Nix expressions
-    using the [`derivation` primitive](./language/derivations.md). These are
-    translated into low-level *store derivations* (implicitly by
-    `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
+- [derivation]{#gloss-derivation}
 
-    [derivation]: #gloss-derivation
+  A description of a build task. The result of a derivation is a
+  store object. Derivations are typically specified in Nix expressions
+  using the [`derivation` primitive](./language/derivations.md). These are
+  translated into low-level *store derivations* (implicitly by
+  `nix-env` and `nix-build`, or explicitly by `nix-instantiate`).
 
-  - [store derivation]{#gloss-store-derivation}\
-    A [derivation] represented as a `.drv` file in the [store].
-    It has a [store path], like any [store object].
+  [derivation]: #gloss-derivation
 
-    Example: `/nix/store/g946hcz4c8mdvq2g8vxx42z51qb71rvp-git-2.38.1.drv`
+- [store derivation]{#gloss-store-derivation}
 
-    See [`nix derivation show`](./command-ref/new-cli/nix3-derivation-show.md) (experimental) for displaying the contents of store derivations.
+  A [derivation] represented as a `.drv` file in the [store].
+  It has a [store path], like any [store object].
 
-    [store derivation]: #gloss-store-derivation
+  Example: `/nix/store/g946hcz4c8mdvq2g8vxx42z51qb71rvp-git-2.38.1.drv`
 
-  - [instantiate]{#gloss-instantiate}, instantiation\
-    Translate a [derivation] into a [store derivation].
+  See [`nix derivation show`](./command-ref/new-cli/nix3-derivation-show.md) (experimental) for displaying the contents of store derivations.
 
-    See [`nix-instantiate`](./command-ref/nix-instantiate.md).
+  [store derivation]: #gloss-store-derivation
 
-    [instantiate]: #gloss-instantiate
+- [instantiate]{#gloss-instantiate}, instantiation
 
-  - [realise]{#gloss-realise}, realisation\
-    Ensure a [store path] is [valid][validity].
+  Translate a [derivation] into a [store derivation].
 
-    This means either running the `builder` executable as specified in the corresponding [derivation] or fetching a pre-built [store object] from a [substituter].
+  See [`nix-instantiate`](./command-ref/nix-instantiate.md).
 
-    See [`nix-build`](./command-ref/nix-build.md) and [`nix-store --realise`](@docroot@/command-ref/nix-store/realise.md).
+  [instantiate]: #gloss-instantiate
 
-    See [`nix build`](./command-ref/new-cli/nix3-build.md) (experimental).
+- [realise]{#gloss-realise}, realisation
 
-    [realise]: #gloss-realise
+  Ensure a [store path] is [valid][validity].
 
-  - [content-addressed derivation]{#gloss-content-addressed-derivation}\
-    A derivation which has the
-    [`__contentAddressed`](./language/advanced-attributes.md#adv-attr-__contentAddressed)
-    attribute set to `true`.
+  This means either running the `builder` executable as specified in the corresponding [derivation] or fetching a pre-built [store object] from a [substituter].
 
-  - [fixed-output derivation]{#gloss-fixed-output-derivation}\
-    A derivation which includes the
-    [`outputHash`](./language/advanced-attributes.md#adv-attr-outputHash) attribute.
+  See [`nix-build`](./command-ref/nix-build.md) and [`nix-store --realise`](@docroot@/command-ref/nix-store/realise.md).
 
-  - [store]{#gloss-store}\
-    The location in the file system where store objects live. Typically
-    `/nix/store`.
+  See [`nix build`](./command-ref/new-cli/nix3-build.md) (experimental).
 
-    From   the  perspective   of   the  location   where  Nix   is
-    invoked, the  Nix store can be  referred to
-    as a "_local_" or a "_remote_" one:
+  [realise]: #gloss-realise
 
-    + A [local store]{#gloss-local-store} exists on the filesystem of
-      the machine where Nix is  invoked. You can use other
-      local stores  by passing  the `--store` flag  to the
-      `nix` command.  Local stores can be used for building derivations.
+- [content-addressed derivation]{#gloss-content-addressed-derivation}
 
-    + A  *remote store*  exists  anywhere  other than  the
-      local  filesystem. One  example is  the `/nix/store`
-      directory on another machine,  accessed via `ssh` or
-      served by the `nix-serve` Perl script.
+  A derivation which has the
+  [`__contentAddressed`](./language/advanced-attributes.md#adv-attr-__contentAddressed)
+  attribute set to `true`.
 
-    [store]: #gloss-store
-    [local store]: #gloss-local-store
+- [fixed-output derivation]{#gloss-fixed-output-derivation}
 
-  - [chroot store]{#gloss-chroot-store}\
-    A [local store] whose canonical path is anything other than `/nix/store`.
+  A derivation which includes the
+  [`outputHash`](./language/advanced-attributes.md#adv-attr-outputHash) attribute.
 
-  - [binary cache]{#gloss-binary-cache}\
-    A *binary cache* is a Nix store which uses a different format: its
-    metadata and signatures are kept in `.narinfo` files rather than in a
-    [Nix database]. This different format simplifies serving store objects
-    over the network, but cannot host builds. Examples of binary caches
-    include S3 buckets and the [NixOS binary cache](https://cache.nixos.org).
+- [store]{#gloss-store}
 
-  - [store path]{#gloss-store-path}\
-    The location of a [store object] in the file system, i.e., an
-    immediate child of the Nix store directory.
+  The location in the file system where store objects live. Typically
+  `/nix/store`.
 
-    Example: `/nix/store/a040m110amc4h71lds2jmr8qrkj2jhxd-git-2.38.1`
+  From   the  perspective   of   the  location   where  Nix   is
+  invoked, the  Nix store can be  referred to
+  as a "_local_" or a "_remote_" one:
 
-    [store path]: #gloss-store-path
+  + A [local store]{#gloss-local-store} exists on the filesystem of
+    the machine where Nix is  invoked. You can use other
+    local stores  by passing  the `--store` flag  to the
+    `nix` command.  Local stores can be used for building derivations.
 
-  - [file system object]{#gloss-store-object}\
-    The Nix data model for representing simplified file system data.
+  + A  *remote store*  exists  anywhere  other than  the
+    local  filesystem. One  example is  the `/nix/store`
+    directory on another machine,  accessed via `ssh` or
+    served by the `nix-serve` Perl script.
 
-    See [File System Object](@docroot@/architecture/file-system-object.md) for details.
+  [store]: #gloss-store
+  [local store]: #gloss-local-store
 
-    [file system object]: #gloss-file-system-object
+- [chroot store]{#gloss-chroot-store}
 
-  - [store object]{#gloss-store-object}\
+  A [local store] whose canonical path is anything other than `/nix/store`.
 
-    A store object consists of a [file system object], [reference]s to other store objects, and other metadata.
-    It can be referred to by a [store path].
+- [binary cache]{#gloss-binary-cache}
 
-    [store object]: #gloss-store-object
+  A *binary cache* is a Nix store which uses a different format: its
+  metadata and signatures are kept in `.narinfo` files rather than in a
+  [Nix database]. This different format simplifies serving store objects
+  over the network, but cannot host builds. Examples of binary caches
+  include S3 buckets and the [NixOS binary cache](https://cache.nixos.org).
 
-  - [input-addressed store object]{#gloss-input-addressed-store-object}\
-    A store object produced by building a
-    non-[content-addressed](#gloss-content-addressed-derivation),
-    non-[fixed-output](#gloss-fixed-output-derivation)
-    derivation.
+- [store path]{#gloss-store-path}
 
-  - [output-addressed store object]{#gloss-output-addressed-store-object}\
-    A [store object] whose [store path] is determined by its contents.
-    This includes derivations, the outputs of [content-addressed derivations](#gloss-content-addressed-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
+  The location of a [store object] in the file system, i.e., an
+  immediate child of the Nix store directory.
 
-  - [substitute]{#gloss-substitute}\
-    A substitute is a command invocation stored in the [Nix database] that
-    describes how to build a store object, bypassing the normal build
-    mechanism (i.e., derivations). Typically, the substitute builds the
-    store object by downloading a pre-built version of the store object
-    from some server.
+  Example: `/nix/store/a040m110amc4h71lds2jmr8qrkj2jhxd-git-2.38.1`
 
-  - [substituter]{#gloss-substituter}\
-    An additional [store]{#gloss-store} from which Nix can obtain store objects instead of building them.
-    Often the substituter is a [binary cache](#gloss-binary-cache), but any store can serve as substituter.
+  [store path]: #gloss-store-path
 
-    See the [`substituters` configuration option](./command-ref/conf-file.md#conf-substituters) for details.
+- [file system object]{#gloss-store-object}
 
-    [substituter]: #gloss-substituter
+  The Nix data model for representing simplified file system data.
 
-  - [purity]{#gloss-purity}\
-    The assumption that equal Nix derivations when run always produce
-    the same output. This cannot be guaranteed in general (e.g., a
-    builder can rely on external inputs such as the network or the
-    system time) but the Nix model assumes it.
+  See [File System Object](@docroot@/architecture/file-system-object.md) for details.
 
-  - [Nix database]{#gloss-nix-database}\
-    An SQlite database to track [reference]s between [store object]s.
-    This is an implementation detail of the [local store].
+  [file system object]: #gloss-file-system-object
 
-    Default location: `/nix/var/nix/db`.
+- [store object]{#gloss-store-object}
 
-    [Nix database]: #gloss-nix-database
 
-  - [Nix expression]{#gloss-nix-expression}\
-    A high-level description of software packages and compositions
-    thereof. Deploying software using Nix entails writing Nix
-    expressions for your packages. Nix expressions are translated to
-    derivations that are stored in the Nix store. These derivations can
-    then be built.
+  A store object consists of a [file system object], [reference]s to other store objects, and other metadata.
+  It can be referred to by a [store path].
 
-  - [reference]{#gloss-reference}\
-    A [store object] `O` is said to have a *reference* to a store object `P` if a [store path] to `P` appears in the contents of `O`.
+  [store object]: #gloss-store-object
 
-    Store objects can refer to both other store objects and themselves.
-    References from a store object to itself are called *self-references*.
-    References other than a self-reference must not form a cycle.
+- [input-addressed store object]{#gloss-input-addressed-store-object}
 
-    [reference]: #gloss-reference
+  A store object produced by building a
+  non-[content-addressed](#gloss-content-addressed-derivation),
+  non-[fixed-output](#gloss-fixed-output-derivation)
+  derivation.
 
-  - [reachable]{#gloss-reachable}\
-    A store path `Q` is reachable from another store path `P` if `Q`
-    is in the *closure* of the *references* relation.
+- [output-addressed store object]{#gloss-output-addressed-store-object}
 
-  - [closure]{#gloss-closure}\
-    The closure of a store path is the set of store paths that are
-    directly or indirectly “reachable” from that store path; that is,
-    it’s the closure of the path under the *references* relation. For
-    a package, the closure of its derivation is equivalent to the
-    build-time dependencies, while the closure of its output path is
-    equivalent to its runtime dependencies. For correct deployment it
-    is necessary to deploy whole closures, since otherwise at runtime
-    files could be missing. The command `nix-store --query --requisites ` prints out
-    closures of store paths.
+  A [store object] whose [store path] is determined by its contents.
+  This includes derivations, the outputs of [content-addressed derivations](#gloss-content-addressed-derivation), and the outputs of [fixed-output derivations](#gloss-fixed-output-derivation).
 
-    As an example, if the [store object] at path `P` contains a [reference]
-    to a store object at path `Q`, then `Q` is in the closure of `P`. Further, if `Q`
-    references `R` then `R` is also in the closure of `P`.
+- [substitute]{#gloss-substitute}
 
-    [closure]: #gloss-closure
+  A substitute is a command invocation stored in the [Nix database] that
+  describes how to build a store object, bypassing the normal build
+  mechanism (i.e., derivations). Typically, the substitute builds the
+  store object by downloading a pre-built version of the store object
+  from some server.
 
-  - [output path]{#gloss-output-path}\
-    A [store path] produced by a [derivation].
+- [substituter]{#gloss-substituter}
 
-    [output path]: #gloss-output-path
+  An additional [store]{#gloss-store} from which Nix can obtain store objects instead of building them.
+  Often the substituter is a [binary cache](#gloss-binary-cache), but any store can serve as substituter.
 
-  - [deriver]{#gloss-deriver}\
-    The [store derivation] that produced an [output path].
+  See the [`substituters` configuration option](./command-ref/conf-file.md#conf-substituters) for details.
 
-  - [validity]{#gloss-validity}\
-    A store path is valid if all [store object]s in its [closure] can be read from the [store].
+  [substituter]: #gloss-substituter
 
-    For a [local store], this means:
-    - The store path leads to an existing [store object] in that [store].
-    - The store path is listed in the [Nix database] as being valid.
-    - All paths in the store path's [closure] are valid.
+- [purity]{#gloss-purity}
 
-    [validity]: #gloss-validity
+  The assumption that equal Nix derivations when run always produce
+  the same output. This cannot be guaranteed in general (e.g., a
+  builder can rely on external inputs such as the network or the
+  system time) but the Nix model assumes it.
 
-  - [user environment]{#gloss-user-env}\
-    An automatically generated store object that consists of a set of
-    symlinks to “active” applications, i.e., other store paths. These
-    are generated automatically by
-    [`nix-env`](./command-ref/nix-env.md). See *profiles*.
+- [Nix database]{#gloss-nix-database}
 
-  - [profile]{#gloss-profile}\
-    A symlink to the current *user environment* of a user, e.g.,
-    `/nix/var/nix/profiles/default`.
+  An SQlite database to track [reference]s between [store object]s.
+  This is an implementation detail of the [local store].
 
-  - [installable]{#gloss-installable}\
-    Something that can be realised in the Nix store.
+  Default location: `/nix/var/nix/db`.
 
-    See [installables](./command-ref/new-cli/nix.md#installables) for [`nix` commands](./command-ref/new-cli/nix.md) (experimental) for details.
+  [Nix database]: #gloss-nix-database
 
-  - [NAR]{#gloss-nar}\
-    A *N*ix *AR*chive. This is a serialisation of a path in the Nix
-    store. It can contain regular files, directories and symbolic
-    links.  NARs are generated and unpacked using `nix-store --dump`
-    and `nix-store --restore`.
+- [Nix expression]{#gloss-nix-expression}
 
-  - [`∅`]{#gloss-emtpy-set}\
-    The empty set symbol. In the context of profile history, this denotes a package is not present in a particular version of the profile.
+  A high-level description of software packages and compositions
+  thereof. Deploying software using Nix entails writing Nix
+  expressions for your packages. Nix expressions are translated to
+  derivations that are stored in the Nix store. These derivations can
+  then be built.
 
-  - [`ε`]{#gloss-epsilon}\
-    The epsilon symbol. In the context of a package, this means the version is empty. More precisely, the derivation does not have a version attribute.
+- [reference]{#gloss-reference}
 
-  - [string interpolation]{#gloss-string-interpolation}\
-    Expanding expressions enclosed in `${ }` within a [string], [path], or [attribute name].
+  A [store object] `O` is said to have a *reference* to a store object `P` if a [store path] to `P` appears in the contents of `O`.
 
-    See [String interpolation](./language/string-interpolation.md) for details.
+  Store objects can refer to both other store objects and themselves.
+  References from a store object to itself are called *self-references*.
+  References other than a self-reference must not form a cycle.
 
-    [string]: ./language/values.md#type-string
-    [path]: ./language/values.md#type-path
-    [attribute name]: ./language/values.md#attribute-set
+  [reference]: #gloss-reference
 
-  - [experimental feature]{#gloss-experimental-feature}\
-    Not yet stabilized functionality guarded by named experimental feature flags.
-    These flags are enabled or disabled with the [`experimental-features`](./command-ref/conf-file.html#conf-experimental-features) setting.
+- [reachable]{#gloss-reachable}
 
-    See the contribution guide on the [purpose and lifecycle of experimental feaures](@docroot@/contributing/experimental-features.md).
+  A store path `Q` is reachable from another store path `P` if `Q`
+  is in the *closure* of the *references* relation.
+
+- [closure]{#gloss-closure}
+
+  The closure of a store path is the set of store paths that are
+  directly or indirectly “reachable” from that store path; that is,
+  it’s the closure of the path under the *references* relation. For
+  a package, the closure of its derivation is equivalent to the
+  build-time dependencies, while the closure of its output path is
+  equivalent to its runtime dependencies. For correct deployment it
+  is necessary to deploy whole closures, since otherwise at runtime
+  files could be missing. The command `nix-store --query --requisites ` prints out
+  closures of store paths.
+
+  As an example, if the [store object] at path `P` contains a [reference]
+  to a store object at path `Q`, then `Q` is in the closure of `P`. Further, if `Q`
+  references `R` then `R` is also in the closure of `P`.
+
+  [closure]: #gloss-closure
+
+- [output path]{#gloss-output-path}
+
+  A [store path] produced by a [derivation].
+
+  [output path]: #gloss-output-path
+
+- [deriver]{#gloss-deriver}
+
+  The [store derivation] that produced an [output path].
+
+- [validity]{#gloss-validity}
+
+  A store path is valid if all [store object]s in its [closure] can be read from the [store].
+
+  For a [local store], this means:
+  - The store path leads to an existing [store object] in that [store].
+  - The store path is listed in the [Nix database] as being valid.
+  - All paths in the store path's [closure] are valid.
+
+  [validity]: #gloss-validity
+
+- [user environment]{#gloss-user-env}
+
+  An automatically generated store object that consists of a set of
+  symlinks to “active” applications, i.e., other store paths. These
+  are generated automatically by
+  [`nix-env`](./command-ref/nix-env.md). See *profiles*.
+
+- [profile]{#gloss-profile}
+
+  A symlink to the current *user environment* of a user, e.g.,
+  `/nix/var/nix/profiles/default`.
+
+- [installable]{#gloss-installable}
+
+  Something that can be realised in the Nix store.
+
+  See [installables](./command-ref/new-cli/nix.md#installables) for [`nix` commands](./command-ref/new-cli/nix.md) (experimental) for details.
+
+- [NAR]{#gloss-nar}
+
+  A *N*ix *AR*chive. This is a serialisation of a path in the Nix
+  store. It can contain regular files, directories and symbolic
+  links.  NARs are generated and unpacked using `nix-store --dump`
+  and `nix-store --restore`.
+
+- [`∅`]{#gloss-emtpy-set}
+
+  The empty set symbol. In the context of profile history, this denotes a package is not present in a particular version of the profile.
+
+- [`ε`]{#gloss-epsilon}
+
+  The epsilon symbol. In the context of a package, this means the version is empty. More precisely, the derivation does not have a version attribute.
+
+- [string interpolation]{#gloss-string-interpolation}
+
+  Expanding expressions enclosed in `${ }` within a [string], [path], or [attribute name].
+
+  See [String interpolation](./language/string-interpolation.md) for details.
+
+  [string]: ./language/values.md#type-string
+  [path]: ./language/values.md#type-path
+  [attribute name]: ./language/values.md#attribute-set
+
+- [experimental feature]{#gloss-experimental-feature}
+
+  Not yet stabilized functionality guarded by named experimental feature flags.
+  These flags are enabled or disabled with the [`experimental-features`](./command-ref/conf-file.html#conf-experimental-features) setting.
+
+  See the contribution guide on the [purpose and lifecycle of experimental feaures](@docroot@/contributing/experimental-features.md).
diff --git a/doc/manual/src/protocols/tarball-fetcher.md b/doc/manual/src/protocols/tarball-fetcher.md
index 0d3212303..274fa6d63 100644
--- a/doc/manual/src/protocols/tarball-fetcher.md
+++ b/doc/manual/src/protocols/tarball-fetcher.md
@@ -20,8 +20,8 @@ Link: <flakeref>; rel="immutable"
 
 (Note the required `<` and `>` characters around *flakeref*.)
 
-*flakeref* must be a tarball flakeref. It can contain flake attributes
-such as `narHash`, `rev` and `revCount`. If `narHash` is included, its
+*flakeref* must be a tarball flakeref. It can contain the tarball flake attributes
+`narHash`, `rev`, `revCount` and `lastModified`. If `narHash` is included, its
 value must be the NAR hash of the unpacked tarball (as computed via
 `nix hash path`). Nix checks the contents of the returned tarball
 against the `narHash` attribute. The `rev` and `revCount` attributes
diff --git a/doc/manual/src/release-notes/rl-next.md b/doc/manual/src/release-notes/rl-next.md
index 66fe0c427..02ad0a661 100644
--- a/doc/manual/src/release-notes/rl-next.md
+++ b/doc/manual/src/release-notes/rl-next.md
@@ -23,6 +23,8 @@
 - Introduce a new [`outputOf`](@docroot@/language/builtins.md#builtins-outputOf) builtin.
   It is part of the [`dynamic-derivations`](@docroot@/contributing/experimental-features.md#xp-feature-dynamic-derivations) experimental feature.
 
+- Flake follow paths at depths greater than 2 are now handled correctly, preventing "follows a non-existent input" errors.
+
 - [`nix-store --query`](@docroot@/command-ref/nix-store/query.md) gained a new type of query: `--valid-derivers`. It returns all `.drv` files in the local store that *can be* used to build the output passed in argument.
 This is in contrast to `--deriver`, which returns the single `.drv` file that *was actually* used to build the output passed in argument. In case the output was substituted from a binary cache,
 this `.drv` file may only exist on said binary cache and not locally.