aboutsummaryrefslogtreecommitdiff
path: root/src/libstore
AgeCommit message (Collapse)Author
2016-12-06Use a steady clock for timeoutsEelco Dolstra
Fixes #1146.
2016-12-06nix-store --serve: Suppress log output on stderr when repeating a buildEelco Dolstra
2016-11-26Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25Get rid of unicode quotes (#1140)Guillaume Maudoux
2016-11-17Update darwin build for optional sandbox pathsShea Levy
Fixes #1132
2016-11-16S3BinaryCacheStore:: Eliminate a string copy while uploadingEelco Dolstra
This cuts hydra-queue-runner's peak memory usage by about a third.
2016-11-14buildPaths(): Handle ecIncompleteClosureEelco Dolstra
buildPaths() on a non-derivation would incorrectly not throw an error if the path didn't have a substitute.
2016-11-14Don't rely on %mEelco Dolstra
2016-11-14daemon: Do not error out when deduplication fails due to ENOSPC.Ludovic Courtès
This solves a problem whereby if /gnu/store/.links had enough entries, ext4's directory index would be full, leading to link(2) returning ENOSPC. * nix/libstore/optimise-store.cc (LocalStore::optimisePath_): Upon ENOSPC from link(2), print a message and return instead of throwing a 'SysError'.
2016-11-10Store::computeFSClosure(): Support a set of pathsEelco Dolstra
This way, callers can exploits the parallelism of computeFSClosure() when they have multiple paths that they need the (combined) closure of.
2016-11-09copyClosure() / copyStorePath(): Expose dontCheckSigsEelco Dolstra
Needed by Hydra.
2016-11-09Merge branch 'ssh-store' of https://github.com/shlevy/nixEelco Dolstra
2016-11-09Implement backwards-compatible RemoteStore::addToStore()Eelco Dolstra
The SSHStore PR adds this functionality to the daemon, but we have to handle the case where the Nix daemon is 1.11. Also, don't require signatures for trusted users. This restores 1.11 behaviour. Fixes https://github.com/NixOS/hydra/issues/398.
2016-11-08TypoEelco Dolstra
2016-10-31Support optional sandbox pathsEelco Dolstra
For example, you can now set build-sandbox-paths = /dev/nvidiactl? to specify that /dev/nvidiactl should only be mounted in the sandbox if it exists in the host filesystem. This is useful e.g. for EC2 images that should support both CUDA and non-CUDA instances.
2016-10-27Add nix.conf options for -k and -KShea Levy
Fixes #1084
2016-10-26Fix build log output in nix-store --serveEelco Dolstra
2016-10-26Restore the CachedFailure status codeEelco Dolstra
The removal of CachedFailure caused the value of TimedOut to change, which broke timed-out handling in Hydra (so timed-out builds would show up as "aborted" and would be retried, e.g. at http://hydra.nixos.org/build/42537427).
2016-10-21Remove addPathToAccessorEelco Dolstra
2016-10-21BinaryCacheStore: Optionally write a NAR listingEelco Dolstra
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-19Merge branch 'priorityqueue' of https://github.com/groxxda/nixEelco Dolstra
2016-10-19Shut up clang warningEelco Dolstra
2016-10-19Fix uninitialised variableEelco Dolstra
2016-10-18downloader: use priority_queueAlexander Ried
2016-10-14Fix /dev/ptmx in sandboxesEelco Dolstra
This was broken since ff0c0b645cc1448959126185bb2fafe41cf0bddf. Since I can't figure out how to mount a devpts instance in the sandbox, let's just bind-mount the host devpts.
2016-10-14CURL_AT_LEAST_VERSION -> LIBCURL_VERSION_NUMEelco Dolstra
http://hydra.nixos.org/build/42025230
2016-10-13Fix HTTP/2 supportEelco Dolstra
Commit 86e8c67efc33cf756500a1dec7fd6313658f2664 broke it, because CURL_* are not actually #defines.
2016-10-13SSL_CERT_FILE -> NIX_SSL_CERT_FILEEelco Dolstra
This prevents collisions with the "native" OpenSSL, in particular on OS X. Fixes #921.
2016-10-12Shut up some warningsEelco Dolstra
2016-10-07querySubstitutablePaths(): Don't query paths for which we already have a ↵Eelco Dolstra
substituter
2016-10-07LocalStore::querySubstitutablePaths(): Implement using queryValidPaths()Eelco Dolstra
2016-10-07Store::queryValidPaths(): Use async queryPathInfo()Eelco Dolstra
This allows the binary cache substituter to pipeline requests.
2016-10-07Implement generic Store::queryValidPaths()Eelco Dolstra
2016-10-07Add copyClosure utility function for HydraEelco Dolstra
2016-10-07importPaths(): Fix accessor support for HydraEelco Dolstra
2016-10-06Fix getS3Stats()Eelco Dolstra
2016-10-05Use std::random_deviceEelco Dolstra
2016-10-04Fix build with older versions of libcurlEelco Dolstra
2016-09-22Handle the case where signed-binary-caches consists of whitespaceEelco Dolstra
2016-09-21Add sandbox-dev-shm-size optionEelco Dolstra
Fixes #1069.
2016-09-21printMsg(lvlError, ...) -> printError(...) etc.Eelco Dolstra
2016-09-21Some notational convenience for formatting stringsEelco Dolstra
We can now write throw Error("file '%s' not found", path); instead of throw Error(format("file '%s' not found") % path); and similarly printError("file '%s' not found", path); instead of printMsg(lvlError, format("file '%s' not found") % path);
2016-09-20TweakEelco Dolstra
2016-09-20Improve robustnessEelco Dolstra
2016-09-20Fix "Promise already satisfied" errorEelco Dolstra
2016-09-16armv5tel can be built by armv6l and armv7l (#1063)Eric Litak
2016-09-16Make computeFSClosure() single-threaded againEelco Dolstra
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-14Revive binary-caches-parallel-connectionsEelco Dolstra
It's a slight misnomer now because it actually limits *all* downloads, not just binary cache lookups. Also add a "enable-http2" option to allow disabling use of HTTP/2 (enabled by default).
2016-09-14Enable HTTP/2 supportEelco Dolstra
The binary cache store can now use HTTP/2 to do lookups. This is much more efficient than HTTP/1.1 due to multiplexing: we can issue many requests in parallel over a single TCP connection. Thus it's no longer necessary to use a bunch of concurrent TCP connections (25 by default). For example, downloading 802 .narinfo files from https://cache.nixos.org/, using a single TCP connection, takes 11.8s with HTTP/1.1, but only 0.61s with HTTP/2. This did require a fairly substantial rewrite of the Downloader class to use the curl multi interface, because otherwise curl wouldn't be able to do multiplexing for us. As a bonus, we get connection reuse even with HTTP/1.1. All downloads are now handled by a single worker thread. Clients call Downloader::enqueueDownload() to tell the worker thread to start the download, getting a std::future to the result.
2016-09-14Move some .drv parsing functions out of utilEelco Dolstra