aboutsummaryrefslogtreecommitdiff
path: root/src/libstore
AgeCommit message (Collapse)Author
2017-10-24Remove the builder-files optionEelco Dolstra
You can now include files via the "builders" option, using the syntax "@<filename>". Having only one option makes it easier to override builders completely. For backward compatibility, the default is "@/etc/nix/machines", or "@<filename>" for each file name in NIX_REMOTE_SYSTEMS.
2017-10-23Pass all settings to build-remoteEelco Dolstra
This ensures that command line flags such as --builders get passed correctly.
2017-10-23Turn $NIX_REMOTE into a configuration optionEelco Dolstra
2017-10-20Merge pull request #1616 from copumpkin/separate-darwin-sandboxEelco Dolstra
Shift Darwin sandbox to separate installed files
2017-10-19Revert "Let's not populate the NAR cache from hydra-queue-runner for now"Eelco Dolstra
This reverts commit 908590dc6cfcca3a98755b194d93b2da39aee95c. Since hydra-server can have a different store URI from hydra-queue-runner now, we don't really need this.
2017-10-18Let's not populate the NAR cache from hydra-queue-runner for nowEelco Dolstra
2017-10-18Errors writing to the NAR cache should not be fatalEelco Dolstra
2017-10-18Suppress "copying 0 paths" messageEelco Dolstra
2017-10-17Shift Darwin sandbox to separate installed filesDan Peebles
This makes it slightly more manageable to see at a glance what in a build's sandbox profile is unique to the build and what is standard. Also a first step to factoring more of our Darwin logic into scheme functions that will allow us a bit more flexibility. And of course less of that nasty codegen in C++! 😀
2017-10-17BinaryCacheStore::addToStore(): Add NARs to the local cacheEelco Dolstra
2017-10-17BinaryCacheStore: Support local caching of NARsEelco Dolstra
This speeds up commands like "nix cat-store". For example: $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf real 0m4.336s $ time nix cat-store --store https://cache.nixos.org?local-nar-cache=/tmp/nar-cache /nix/store/i60yncmq6w9dyv37zd2k454g0fkl3arl-systemd-234/etc/udev/udev.conf real 0m0.045s The primary motivation is to allow hydra-server to serve files from S3 binary caches. Previously Hydra had a hack to do "nix-store -r <path>", but that fetches the entire closure so is prohibitively expensive. There is no garbage collection of the NAR cache yet. Also, the entire NAR is read when accessing a single member file. We could generate the NAR listing to provide random access. Note: the NAR cache is indexed by the store path hash, not the content hash, so NAR caches should not be shared between binary caches, unless you're sure that all your builds are binary-reproducible.
2017-10-17Remove BinaryCacheStoreAccessorEelco Dolstra
Probably as a result of a bad merge in 4b8f1b0ec066a5b994747b1afd050f5f62d857f6, we had both a BinaryCacheStoreAccessor and a RemoteFSAccessor. BinaryCacheStore::getFSAccessor() returned the latter, but BinaryCacheStore::addToStore() checked for the former. This probably caused hydra-queue-runner to download paths that it just uploaded.
2017-10-12Add option to disable the seccomp filterEelco Dolstra
I needed this to test ACL/xattr removal in canonicalisePathMetaData(). Might also be useful if you need to build old Nixpkgs that doesn't have the required patches to remove setuid/setgid creation.
2017-10-02Reverse retry logic to retry in all but a few casesDan Peebles
It was getting too much like whac-a-mole listing all the retriable error conditions, so we now retry by default and list the cases where retrying is almost certainly hopeless.
2017-09-18Don't remove Content-Encoding in fetchurl / nix-prefetch-urlEelco Dolstra
Fixes #1568.
2017-09-14RemoteStore: Add option to drop old connections from the poolEelco Dolstra
This is a hack to make hydra-queue-runner free its temproots periodically, thereby ensuring that garbage collection of the corresponding paths is not blocked until the queue runner is restarted. It would be better if temproots could be released earlier than at process exit. I started working on a RAII object returned by functions like addToStore() that releases temproots. However, this would be a pretty massive change so I gave up on it for now.
2017-09-14Show the pid of temporary rootsEelco Dolstra
2017-09-14nix-store -q --roots / --gc --print-roots: Print temporary / in-memory rootsEelco Dolstra
For example, $ nix-store -q --roots /nix/store/7phd2sav7068nivgvmj2vpm3v47fd27l-patchelf-0.8pre845_0315148 {temp:1} denotes that the path is only being kept alive by a temporary root (i.e. /nix/var/nix/temproots/). Similarly, $ nix-store --gc --print-roots ... {memory:9} -> /nix/store/094gpjn9f15ip17wzxhma4r51nvsj17p-curl-7.53.1 shows that curl is being used by some process.
2017-09-08LegacySSHStore: Include signatures etc.Eelco Dolstra
2017-09-08copyStorePath(): Fill in missing narHash regardless of checkSigsEelco Dolstra
I don't remember what the reasoning was here, but security is provided by the signatures, not by whether the hash is provided by the other store.
2017-09-08Communicate the structured log FD to buildersEelco Dolstra
Since we may use a dedicated file descriptor in the future, this allows us to change it. So builders can do if [[ -n $NIX_LOG_FD ]]; then echo "@nix { message... }" >&$NIX_LOG_FD fi
2017-09-06Fix abort when the GC thread gets an exceptionEelco Dolstra
2017-09-05Add automatic garbage collectionEelco Dolstra
Nix can now automatically run the garbage collector during builds or while adding paths to the store. The option "min-free = <bytes>" specifies that Nix should run the garbage collector whenever free space in the Nix store drops below <bytes>. It will then delete garbage until "max-free" bytes are available. Garbage collection during builds is asynchronous; running builds are not paused and new builds are not blocked. However, there also is a synchronous GC run prior to the first build/substitution. Currently, no old GC roots are deleted (as in "nix-collect-garbage -d").
2017-09-05GC: Don't delete own temproots fileEelco Dolstra
Since file locks are per-process rather than per-file-descriptor, the garbage collector would always acquire a lock on its own temproots file and conclude that it's stale.
2017-09-01Abort curl downloads if there is no progress for 5 minutesEelco Dolstra
Maybe this will fix the curl hangs on macOS. (We could also use CURLOPT_TIMEOUT but that seems more of a sledgehammer.)
2017-08-31Call queryMissing() prior to buildingEelco Dolstra
Without this, substitute info is fetched sequentially, which is superslow. In the old UI (e.g. nix-build), we call printMissing(), which calls queryMissing(), thereby preheating the binary cache cache. But the new UI doesn't do that.
2017-08-31Add an activity for binary cache queriesEelco Dolstra
2017-08-31Rename a few configuration optionsEelco Dolstra
In particular, drop the "build-" and "gc-" prefixes which are pointless. So now you can say nix build --no-sandbox instead of nix build --no-build-use-sandbox
2017-08-28Give activities a verbosity level againEelco Dolstra
And print them (separately from the progress bar) given sufficient -v flags.
2017-08-28Don't send progress messages to older clientsEelco Dolstra
2017-08-28Tunnel progress messages from the daemon to the clientEelco Dolstra
This makes the progress bar work for non-root users.
2017-08-25SimplifyEelco Dolstra
2017-08-25Allow derivations to update the build phaseEelco Dolstra
So the progress bar can show [1/0/1 built, 0.0 MiB DL] building hello-2.10 (configuring): checking whether pread is declared without a macro... yes
2017-08-25Allow activities to be nestedEelco Dolstra
In particular, this allows more relevant activities ("substituting X") to supersede inferior ones ("downloading X").
2017-08-25Restore activity metadataEelco Dolstra
This allows the progress bar to display "building perl-5.22.3" instead of "building /nix/store/<hash>-perl-5.22.3.drv".
2017-08-21Clean up JSON constructionEelco Dolstra
2017-08-21Allow builders to create activitiesEelco Dolstra
Actually, currently they can only create download activities. Thus, downloads by builtins.fetchurl show up in the progress bar.
2017-08-16nix optimise-store: Show how much space has been freedEelco Dolstra
2017-08-16nix optimise-store: AddEelco Dolstra
This replaces "nix-store --optimise". Main difference is that it has a progress indicator.
2017-08-16Progress indicator: CleanupEelco Dolstra
2017-08-16Progress indicator: More improvementsEelco Dolstra
2017-08-16Progress indicator: Show number of active itemsEelco Dolstra
2017-08-16Progress indicator: Unify "copying" and "substituting"Eelco Dolstra
They're the same thing after all. Example: $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 49/98 copied, 16.3/92.8 MiB DL, 55.8/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/0pl9li1jigcj2dany47hpmn0r3r48wc4nz48v5mqhh426lgz3bz6.nar.xz'
2017-08-16Improve substitution progress indicatorEelco Dolstra
E.g. $ nix build --store local?root=/tmp/nix nixpkgs.firefox-unwrapped [0/1 built, 1/97/98 fetched, 65.8/92.8 MiB DL, 203.2/309.2 MiB copied] downloading 'https://cache.nixos.org/nar/1czm9fk0svacy4h6a3fzkpafi4f7a9gml36kk8cq1igaghbspg3k.nar.xz'
2017-08-16nix copy: Improve progress indicatorEelco Dolstra
It now shows the amount of data copied: [8/1038 copied, 160.4/1590.9 MiB copied] copying path '...'
2017-08-16nix copy: Revive progress barEelco Dolstra
2017-08-16Disallow SSH host names starting with a dashEelco Dolstra
2017-08-10Style fixEelco Dolstra
2017-08-09Use /proc/self/fd to efficiently close all FDs on LinuxEelco Dolstra
Issue #1506.
2017-07-31Merge branch 'macOS' of https://github.com/davidak/nixEelco Dolstra