aboutsummaryrefslogtreecommitdiff
path: root/doc/manual/src/architecture
diff options
context:
space:
mode:
Diffstat (limited to 'doc/manual/src/architecture')
-rw-r--r--doc/manual/src/architecture/architecture.md146
-rw-r--r--doc/manual/src/architecture/store/fso.md69
-rw-r--r--doc/manual/src/architecture/store/path.md105
-rw-r--r--doc/manual/src/architecture/store/store.md151
-rw-r--r--doc/manual/src/architecture/store/store/build-system-terminology.md32
-rw-r--r--doc/manual/src/architecture/store/store/closure.md29
6 files changed, 91 insertions, 441 deletions
diff --git a/doc/manual/src/architecture/architecture.md b/doc/manual/src/architecture/architecture.md
index 41deb07af..e51958052 100644
--- a/doc/manual/src/architecture/architecture.md
+++ b/doc/manual/src/architecture/architecture.md
@@ -1,79 +1,115 @@
# Architecture
-*(This chapter is unstable and a work in progress. Incoming links may rot.)*
-
This chapter describes how Nix works.
It should help users understand why Nix behaves as it does, and it should help developers understand how to modify Nix and how to write similar tools.
## Overview
-Nix consists of [hierarchical layers][layer-architecture].
+Nix consists of [hierarchical layers].
+
+[hierarchical layers]: https://en.m.wikipedia.org/wiki/Multitier_architecture#Layers
+
+The following [concept map] shows its main components (rectangles), the objects they operate on (rounded rectangles), and their interactions (connecting phrases):
+
+[concept map]: https://en.m.wikipedia.org/wiki/Concept_map
```
-+-----------------------------------------------------------------+
-| Nix |
-| [ commmand line interface ]------, |
-| | | |
-| evaluates | |
-| | manages |
-| V | |
-| [ configuration language ] | |
-| | | |
-| +-----------------------------|-------------------V-----------+ |
-| | store evaluates to | |
-| | | | |
-| | referenced by V builds | |
-| | [ build input ] ---> [ build plan ] ---> [ build result ] | |
-| | | |
-| +-------------------------------------------------------------+ |
-+-----------------------------------------------------------------+
+
+ .----------------.
+ | Nix expression |----------.
+ '----------------' |
+ | passed to
+ | |
++----------|-------------------|--------------------------------+
+| Nix | V |
+| | +-------------------------+ |
+| | | commmand line interface |------. |
+| | +-------------------------+ | |
+| | | | |
+| evaluated by calls manages |
+| | | | |
+| | V | |
+| | +--------------------+ | |
+| '-------->| language evaluator | | |
+| +--------------------+ | |
+| | | |
+| produces | |
+| | V |
+| +----------------------------|------------------------------+ |
+| | store | | |
+| | referenced by V builds | |
+| | .-------------. .------------. .--------------. | |
+| | | build input |----->| build plan |----->| build result | | |
+| | '-------------' '------------' '--------------' | |
+| +-------------------------------------------------|---------+ |
++---------------------------------------------------|-----------+
+ |
+ represented as
+ |
+ V
+ .---------------.
+ | file |
+ '---------------'
```
-At the top is the [command line interface](../command-ref/command-ref.md), translating from invocations of Nix executables to interactions with the underlying layers.
+At the top is the [command line interface](../command-ref/command-ref.md) that drives the underlying layers.
-Below that is the [Nix expression language](../expressions/expression-language.md), a [purely functional][purely-functional-programming] configuration language.
-It is used to compose expressions which ultimately evaluate to self-contained *build plans*, used to derive *build results* from referenced *build inputs*.
+The [Nix language](../language/index.md) evaluator transforms Nix expressions into self-contained *build plans*, which are used to derive *build results* from referenced *build inputs*.
-The command line and Nix language are what users interact with most.
+The command line interface and Nix expressions are what users deal with most.
> **Note**
> The Nix language itself does not have a notion of *packages* or *configurations*.
> As far as we are concerned here, the inputs and results of a build plan are just data.
-Underlying these is the [Nix store](./store/store.md), a mechanism to keep track of build plans, data, and references between them.
-It can also execute build plans to produce new data.
+Underlying the command line interface and the Nix language evaluator is the [Nix store](../glossary.md#gloss-store), a mechanism to keep track of build plans, data, and references between them.
+It can also execute build plans to produce new data, which are made available to the operating system as files.
-A build plan is a series of *build tasks*.
-Each build task has a special build input which is used as *build instructions*.
+A build plan itself is a series of *build tasks*, together with their build inputs.
+
+> **Important**
+> A build task in Nix is called [derivation](../glossary.md#gloss-derivation).
+
+Each build task has a special build input executed as *build instructions* in order to perform the build.
The result of a build task can be input to another build task.
+The following [data flow diagram] shows a build plan for illustration.
+Build inputs used as instructions to a build task are marked accordingly:
+
+[data flow diagram]: https://en.m.wikipedia.org/wiki/Data-flow_diagram
+
```
-+-----------------------------------------------------------------------------------------+
-| store |
-| ................................................. |
-| : build plan : |
-| : : |
-| [ build input ]-----instructions-, : |
-| : | : |
-| : v : |
-| [ build input ]----------->[ build task ]--instructions-, : |
-| : | : |
-| : | : |
-| : v : |
-| : [ build task ]----->[ build result ] |
-| [ build input ]-----instructions-, ^ : |
-| : | | : |
-| : v | : |
-| [ build input ]----------->[ build task ]---------------' : |
-| : ^ : |
-| : | : |
-| [ build input ]------------------' : |
-| : : |
-| : : |
-| :...............................................: |
-| |
-+-----------------------------------------------------------------------------------------+
++--------------------------------------------------------------------+
+| build plan |
+| |
+| .-------------. |
+| | build input |---------. |
+| '-------------' | |
+| instructions |
+| | |
+| v |
+| .-------------. .----------. |
+| | build input |-->( build task )-------. |
+| '-------------' '----------' | |
+| instructions |
+| | |
+| v |
+| .-------------. .----------. .--------------. |
+| | build input |---------. ( build task )--->| build result | |
+| '-------------' | '----------' '--------------' |
+| instructions ^ |
+| | | |
+| v | |
+| .-------------. .----------. | |
+| | build input |-->( build task )-------' |
+| '-------------' '----------' |
+| ^ |
+| | |
+| | |
+| .-------------. | |
+| | build input |---------' |
+| '-------------' |
+| |
++--------------------------------------------------------------------+
```
-[layer-architecture]: https://en.m.wikipedia.org/wiki/Multitier_architecture#Layers
-[purely-functional-programming]: https://en.m.wikipedia.org/wiki/Purely_functional_programming
diff --git a/doc/manual/src/architecture/store/fso.md b/doc/manual/src/architecture/store/fso.md
deleted file mode 100644
index e0eb69f60..000000000
--- a/doc/manual/src/architecture/store/fso.md
+++ /dev/null
@@ -1,69 +0,0 @@
-# File System Object
-
-The Nix store uses a simple file system model for the data it holds in [store objects](store.md#store-object).
-
-Every file system object is one of the following:
-
- - File: an executable flag, and arbitrary data for contents
- - Directory: mapping of names to child file system objects
- - [Symbolic link][symlink]: may point anywhere.
-
-We call a store object's outermost file system object the *root*.
-
- data FileSystemObject
- = File { isExecutable :: Bool, contents :: Bytes }
- | Directory { entries :: Map FileName FileSystemObject }
- | SymLink { target :: Path }
-
-Examples:
-
-- a directory with contents
-
- /nix/store/<hash>-hello-2.10
- ├── bin
- │   └── hello
- └── share
- ├── info
- │   └── hello.info
- └── man
- └── man1
- └── hello.1.gz
-
-- a directory with relative symlink and other contents
-
- /nix/store/<hash>-go-1.16.9
- ├── bin -> share/go/bin
- ├── nix-support/
- └── share/
-
-- a directory with absolute symlink
-
- /nix/store/d3k...-nodejs
- └── nix_node -> /nix/store/f20...-nodejs-10.24.
-
-A bare file or symlink can be a root file system object.
-Examples:
-
- /nix/store/<hash>-hello-2.10.tar.gz
-
- /nix/store/4j5...-pkg-config-wrapper-0.29.2-doc -> /nix/store/i99...-pkg-config-0.29.2-doc
-
-Symlinks pointing outside of their own root or to a store object without a matching reference are allowed, but might not function as intended.
-Examples:
-
-- an arbitrarily symlinked file may change or not exist at all
-
- /nix/store/<hash>-foo
- └── foo -> /home/foo
-
-- if a symlink to a store path was not automatically created by Nix, it may be invalid or get invalidated when the store object is deleted
-
- /nix/store/<hash>-bar
- └── bar -> /nix/store/abc...-foo
-
-Nix file system objects do not support [hard links][hardlink]:
-each file system object which is not the root has exactly one parent and one name.
-However, as store objects are immutable, an underlying file system can use hard links for optimization.
-
-[symlink]: https://en.m.wikipedia.org/wiki/Symbolic_link
-[hardlink]: https://en.m.wikipedia.org/wiki/Hard_link
diff --git a/doc/manual/src/architecture/store/path.md b/doc/manual/src/architecture/store/path.md
deleted file mode 100644
index 663f04f46..000000000
--- a/doc/manual/src/architecture/store/path.md
+++ /dev/null
@@ -1,105 +0,0 @@
-# Store Path
-
-Nix implements [references](store.md#reference) to [store objects](store.md#store-object) as *store paths*.
-
-Store paths are pairs of
-
-- a 20-byte [digest](#digest) for identification
-- a symbolic name for people to read.
-
-Example:
-
-- digest: `b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z`
-- name: `firefox-33.1`
-
-It is rendered to a file system path as the concatenation of
-
- - [store directory](#store-directory)
- - path-separator (`/`)
- - [digest](#digest) rendered in a custom variant of [base-32](https://en.m.wikipedia.org/wiki/Base32) (20 arbitrary bytes become 32 ASCII characters)
- - hyphen (`-`)
- - name
-
-Example:
-
- /nix/store/b6gvzjyb2pg0kjfwrjmg1vfhh54ad73z-firefox-33.1
- |--------| |------------------------------| |----------|
- store directory digest name
-
-## Store Directory
-
-Every [store](./store.md) has a store directory.
-
-If the store has a [file system representation](./store.md#files-and-processes), this directory contains the store’s [file system objects](#file-system-object), which can be addressed by [store paths](#store-path).
-
-This means a store path is not just derived from the referenced store object itself, but depends on the store the store object is in.
-
-> **Note**
-> The store directory defaults to `/nix/store`, but is in principle arbitrary.
-
-It is important which store a given store object belongs to:
-Files in the store object can contain store paths, and processes may read these paths.
-Nix can only guarantee [referential integrity](store/closure.md) if store paths do not cross store boundaries.
-
-Therefore one can only copy store objects to a different store if
-
-- the source and target stores' directories match
-
- or
-
-- the store object in question has no references, that is, contains no store paths.
-
-One cannot copy a store object to a store with a different store directory.
-Instead, it has to be rebuilt, together with all its dependencies.
-It is in general not enough to replace the store directory string in file contents, as this may render executables unusable by invalidating their internal offsets or checksums.
-
-# Digest
-
-In a [store path](#store-path), the [digest][digest] is the output of a [cryptographic hash function][hash] of either all *inputs* involved in building the referenced store object or its actual *contents*.
-
-Store objects are therefore said to be either [input-addressed](#input-addressing) or [content-addressed](#content-addressing).
-
-> **Historical Note**
-> The 20 byte restriction is because originally digests were [SHA-1][sha-1] hashes.
-> Nix now uses [SHA-256][sha-256], and longer hashes are still reduced to 20 bytes for compatibility.
-
-[digest]: https://en.m.wiktionary.org/wiki/digest#Noun
-[hash]: https://en.m.wikipedia.org/wiki/Cryptographic_hash_function
-[sha-1]: https://en.m.wikipedia.org/wiki/SHA-1
-[sha-256]: https://en.m.wikipedia.org/wiki/SHA-256
-
-### Reference scanning
-
-When a new store object is built, Nix scans its file contents for store paths to construct its set of references.
-
-The special format of a store path's [digest](#digest) allows reliably detecting it among arbitrary data.
-Nix uses the [closure](store.md#closure) of build inputs to derive the list of allowed store paths, to avoid false positives.
-
-This way, scanning files captures run time dependencies without the user having to declare them explicitly.
-Doing it at build time and persisting references in the store object avoids repeating this time-consuming operation.
-
-> **Note**
-> In practice, it is sometimes still necessary for users to declare certain dependencies explicitly, if they are to be preserved in the build result's closure.
-This depends on the specifics of the software to build and run.
->
-> For example, Java programs are compressed after compilation, which obfuscates any store paths they may refer to and prevents Nix from automatically detecting them.
-
-## Input Addressing
-
-Input addressing means that the digest derives from how the store object was produced, namely its build inputs and build plan.
-
-To compute the hash of a store object one needs a deterministic serialisation, i.e., a binary string representation which only changes if the store object changes.
-
-Nix has a custom serialisation format called Nix Archive (NAR)
-
-Store object references of this sort can *not* be validated from the content of the store object.
-Rather, a cryptographic signature has to be used to indicate that someone is vouching for the store object really being produced from a build plan with that digest.
-
-## Content Addressing
-
-Content addressing means that the digest derives from the store object's contents, namely its file system objects and references.
-If one knows content addressing was used, one can recalculate the reference and thus verify the store object.
-
-Content addressing is currently only used for the special cases of source files and "fixed-output derivations", where the contents of a store object are known in advance.
-Content addressing of build results is still an [experimental feature subject to some restrictions](https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md).
-
diff --git a/doc/manual/src/architecture/store/store.md b/doc/manual/src/architecture/store/store.md
deleted file mode 100644
index 08b6701d5..000000000
--- a/doc/manual/src/architecture/store/store.md
+++ /dev/null
@@ -1,151 +0,0 @@
-# Store
-
-A Nix store is a collection of *store objects* with references between them.
-It supports operations to manipulate that collection.
-
-The following concept map is a graphical outline of this chapter.
-Arrows indicate suggested reading order.
-
-```
- ,--------------[ store ]----------------,
- | | |
- v v v
- [ store object ] [ closure ]--, [ operations ]
- | | | | | |
- v | | v v |
- [ files and processes ] | | [ garbage collection ] |
- / \ | | |
- v v | v v
-[ file system object ] [ store path ] | [ derivation ]--->[ building ]
- | ^ | | |
- v | v v |
- [ digest ]----' [ reference scanning ]<------------'
- / \
- v v
-[ input addressing ] [ content addressing ]
-```
-
-## Store Object
-
-A store object can hold
-
-- arbitrary *data*
-- *references* to other store objects.
-
-Store objects can be build inputs, build results, or build tasks.
-
-Store objects are [immutable][immutable-object]: once created, they do not change until they are deleted.
-
-## Reference
-
-A store object reference is an [opaque][opaque-data-type], [unique identifier][unique-identifier]:
-The only way to obtain references is by adding or building store objects.
-A reference will always point to exactly one store object.
-
-## Operations
-
-A Nix store can *add*, *retrieve*, and *delete* store objects.
-
- [ data ]
- |
- V
- [ store ] ---> add ----> [ store' ]
- |
- V
- [ reference ]
-
-<!-- -->
-
- [ reference ]
- |
- V
- [ store ] ---> get
- |
- V
- [ store object ]
-
-<!-- -->
-
- [ reference ]
- |
- V
- [ store ] --> delete --> [ store' ]
-
-
-It can *perform builds*, that is, create new store objects by transforming build inputs into build outputs, using instructions from the build tasks.
-
-
- [ reference ]
- |
- V
- [ store ] --> build --(maybe)--> [ store' ]
- |
- V
- [ reference ]
-
-
-As it keeps track of references, it can [garbage-collect][garbage-collection] unused store objects.
-
-
- [ store ] --> collect garbage --> [ store' ]
-
-## Files and Processes
-
-Nix maps between its store model and the [Unix paradigm][unix-paradigm] of [files and processes][file-descriptor], by encoding immutable store objects and opaque identifiers as file system primitives: files and directories, and paths.
-That allows processes to resolve references contained in files and thus access the contents of store objects.
-
-Store objects are therefore implemented as the pair of
-
- - a [file system object](fso.md) for data
- - a set of [store paths](path.md) for references.
-
-[unix-paradigm]: https://en.m.wikipedia.org/wiki/Everything_is_a_file
-[file-descriptor]: https://en.m.wikipedia.org/wiki/File_descriptor
-
-The following diagram shows a radical simplification of how Nix interacts with the operating system:
-It uses files as build inputs, and build outputs are files again.
-On the operating system, files can be run as processes, which in turn operate on files.
-A build function also amounts to an operating system process (not depicted).
-
-```
-+-----------------------------------------------------------------+
-| Nix |
-| [ commmand line interface ]------, |
-| | | |
-| evaluates | |
-| | manages |
-| V | |
-| [ configuration language ] | |
-| | | |
-| +-----------------------------|-------------------V-----------+ |
-| | store evaluates to | |
-| | | | |
-| | referenced by V builds | |
-| | [ build input ] ---> [ build plan ] ---> [ build result ] | |
-| | ^ | | |
-| +---------|----------------------------------------|----------+ |
-+-----------|----------------------------------------|------------+
- | |
- file system object store path
- | |
-+-----------|----------------------------------------|------------+
-| operating system +------------+ | |
-| '------------ | | <-----------' |
-| | file | |
-| ,-- | | <-, |
-| | +------------+ | |
-| execute as | | read, write, execute |
-| | +------------+ | |
-| '-> | process | --' |
-| +------------+ |
-+-----------------------------------------------------------------+
-```
-
-There exist different types of stores, which all follow this model.
-Examples:
-- store on the local file system
-- remote store accessible via SSH
-- binary cache store accessible via HTTP
-
-To make store objects accessible to processes, stores ultimately have to expose store objects through the file system.
-
diff --git a/doc/manual/src/architecture/store/store/build-system-terminology.md b/doc/manual/src/architecture/store/store/build-system-terminology.md
deleted file mode 100644
index eefbaa630..000000000
--- a/doc/manual/src/architecture/store/store/build-system-terminology.md
+++ /dev/null
@@ -1,32 +0,0 @@
-# A [Rosetta stone][rosetta-stone] for build system terminology
-
-The Nix store's design is comparable to other build systems.
-Usage of terms is, for historic reasons, not entirely consistent within the Nix ecosystem, and still subject to slow change.
-
-The following translation table points out similarities and equivalent terms, to help clarify their meaning and inform consistent use in the future.
-
-| generic build system | Nix | [Bazel][bazel] | [Build Systems à la Carte][bsalc] | programming language |
-| -------------------------------- | ---------------- | -------------------------------------------------------------------- | --------------------------------- | ------------------------ |
-| data (build input, build result) | store object | [artifact][bazel-artifact] | value | value |
-| build instructions | builder | ([depends on action type][bazel-actions]) | function | function |
-| build task | derivation | [action][bazel-action] | `Task` | [thunk][thunk] |
-| build plan | derivation graph | [action graph][bazel-action-graph], [build graph][bazel-build-graph] | `Tasks` | [call graph][call-graph] |
-| build | build | build | application of `Build` | evaluation |
-| persistence layer | store | [action cache][bazel-action-cache] | `Store` | heap |
-
-All of these systems share features of [declarative programming][declarative-programming] languages, a key insight first put forward by Eelco Dolstra et al. in [Imposing a Memory Management Discipline on Software Deployment][immdsd] (2004), elaborated in his PhD thesis [The Purely Functional Software Deployment Model][phd-thesis] (2006), and further refined by Andrey Mokhov et al. in [Build Systems à la Carte][bsalc] (2018).
-
-[rosetta-stone]: https://en.m.wikipedia.org/wiki/Rosetta_Stone
-[bazel]: https://bazel.build/start/bazel-intro
-[bazel-artifact]: https://bazel.build/reference/glossary#artifact
-[bazel-actions]: https://docs.bazel.build/versions/main/skylark/lib/actions.html
-[bazel-action]: https://bazel.build/reference/glossary#action
-[bazel-action-graph]: https://bazel.build/reference/glossary#action-graph
-[bazel-build-graph]: https://bazel.build/reference/glossary#build-graph
-[bazel-action-cache]: https://bazel.build/reference/glossary#action-cache
-[thunk]: https://en.m.wikipedia.org/wiki/Thunk
-[call-graph]: https://en.m.wikipedia.org/wiki/Call_graph
-[declarative-programming]: https://en.m.wikipedia.org/wiki/Declarative_programming
-[immdsd]: https://edolstra.github.io/pubs/immdsd-icse2004-final.pdf
-[phd-thesis]: https://edolstra.github.io/pubs/phd-thesis.pdf
-[bsalc]: https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf
diff --git a/doc/manual/src/architecture/store/store/closure.md b/doc/manual/src/architecture/store/store/closure.md
deleted file mode 100644
index 065b95ffc..000000000
--- a/doc/manual/src/architecture/store/store/closure.md
+++ /dev/null
@@ -1,29 +0,0 @@
-# Closure
-
-Nix stores ensure [referential integrity][referential-integrity]: for each store object in the store, all the store objects it references must also be in the store.
-
-The set of all store objects reachable by following references from a given initial set of store objects is called a *closure*.
-
-Adding, building, copying and deleting store objects must be done in a way that preserves referential integrity:
-
-- A newly added store object cannot have references, unless it is a build task.
-
-- Build results must only refer to store objects in the closure of the build inputs.
-
- Building a store object will add appropriate references, according to the build task.
-
-- Store objects being copied must refer to objects already in the destination store.
-
- Recursive copying must either proceed in dependency order or be atomic.
-
-- We can only safely delete store objects which are not reachable from any reference still in use.
-
- <!-- more details in section on garbage collection, link to it once it exists -->
-
-[referential-integrity]: https://en.m.wikipedia.org/wiki/Referential_integrity
-[garbage-collection]: https://en.m.wikipedia.org/wiki/Garbage_collection_(computer_science)
-[immutable-object]: https://en.m.wikipedia.org/wiki/Immutable_object
-[opaque-data-type]: https://en.m.wikipedia.org/wiki/Opaque_data_type
-[unique-identifier]: https://en.m.wikipedia.org/wiki/Unique_identifier
-
-
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-23 23:38:48 +0000