aboutsummaryrefslogtreecommitdiff
path: root/src/libstore/binary-cache-store.cc
AgeCommit message (Collapse)Author
2016-11-26Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25Get rid of unicode quotes (#1140)Guillaume Maudoux
2016-11-09Merge branch 'ssh-store' of https://github.com/shlevy/nixEelco Dolstra
2016-10-21Remove addPathToAccessorEelco Dolstra
2016-10-21BinaryCacheStore: Optionally write a NAR listingEelco Dolstra
The store parameter "write-nar-listing=1" will cause BinaryCacheStore to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’ added to the binary cache. This file contains an XZ-compressed JSON file describing the contents of the NAR, excluding the contents of regular files. E.g. { "version": 1, "root": { "type": "directory", "entries": { "lib": { "type": "directory", "entries": { "Mcrt1.o": { "type": "regular", "size": 1288 }, "Scrt1.o": { "type": "regular", "size": 3920 }, } } } ... } } (The actual file has no indentation.) This is intended to speed up the NixOS channels programs index generator [1], since fetching gazillions of large NARs from cache.nixos.org is currently a bottleneck for updating the regular (non-small) channel. [1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-07importPaths(): Fix accessor support for HydraEelco Dolstra
2016-09-16Make computeFSClosure() single-threaded againEelco Dolstra
The fact that queryPathInfo() is synchronous meant that we needed a thread for every concurrent binary cache lookup, even though they end up being handled by the same download thread. Requiring hundreds of threads is not a good idea. So now there is an asynchronous version of queryPathInfo() that takes a callback function to process the result. Similarly, enqueueDownload() now takes a callback rather than returning a future. Thus, a command like nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5 that returns 4941 paths now takes 1.87s using only 2 threads (the main thread and the downloader thread). (This is with a prewarmed CloudFront.)
2016-09-02Factor a general remote FS accessor out of BinaryCacheStoreShea Levy
2016-08-10FSAccessor: Throw InvalidPathEelco Dolstra
2016-07-26makeFixedOutputPath(): Drop superfluous HashType argumentEelco Dolstra
2016-06-01Make the store directory a member variable of StoreEelco Dolstra
2016-05-31nix-copy-closure / build-remote.pl: Disable signature checkingEelco Dolstra
This restores the Nix 1.11 behaviour.
2016-05-30Re-implement the WantMassQuery property of binary cachesEelco Dolstra
2016-05-30BinaryCacheStore: Remove buildPaths() / ensurePath()Eelco Dolstra
2016-05-04Add a Store::addToStore() variant that accepts a NAREelco Dolstra
As a side effect, this ensures that signatures are propagated when copying paths between stores. Also refactored import/export to make use of this.
2016-05-04Remove OpenSSL-based signingEelco Dolstra
2016-04-29BinaryCacheStore: Support bzip2 compressionEelco Dolstra
2016-04-29Better error messageEelco Dolstra
2016-04-29BinaryCacheStore: Support "none" compression methodEelco Dolstra
2016-04-29BinaryCacheStore: Make the signing key a parameterEelco Dolstra
2016-04-29BinaryCacheStore::queryPathInfo: Don't check signaturesEelco Dolstra
Other stores don't do this either. It's up to the caller to check signatures.
2016-04-28Fix error messageEelco Dolstra
2016-04-22BinaryCacheStore: When adding a path, ensure the references are validEelco Dolstra
This prevents copying a partial closure to a binary cache.
2016-04-21Implement S3BinaryCacheStore::queryAllValidPaths()Eelco Dolstra
This allows commands like "nix verify --all" or "nix path-info --all" to work on S3 caches. Unfortunately, this requires some ugly hackery: when querying the contents of the bucket, we don't want to have to read every .narinfo file. But the S3 bucket keys only include the hash part of each store path, not the name part. So as a special exception queryAllValidPaths() can now return store paths *without* the name part, and queryPathInfo() accepts such store paths (returning a ValidPathInfo object containing the full name).
2016-04-20BinaryCacheStore: Insert new paths into the disk cacheEelco Dolstra
2016-04-20Cache path info lookups in SQLiteEelco Dolstra
This re-implements the binary cache database in C++, allowing it to be used by other Store backends, in particular the S3 backend.
2016-04-19Move path info caching from BinaryCacheStore to StoreEelco Dolstra
Caching path info is generally useful. For instance, it speeds up "nix path-info -rS /run/current-system" (i.e. showing the closure sizes of all paths in the closure of the current system) from 5.6s to 0.15s. This also eliminates some APIs like Store::queryDeriver() and Store::queryReferences().
2016-04-15BinaryCacheStore: Do negative caching of .narinfo lookupsEelco Dolstra
2016-04-15BinaryCacheStore::readFile(): Return a shared_ptr to a stringEelco Dolstra
This allows readFile() to indicate that a file doesn't exist, and might eliminate some large string copying.
2016-03-22Don't overload dumpPath()Eelco Dolstra
2016-03-21Move signatures from NarInfo to ValidPathInfoEelco Dolstra
This allows queryPathInfo() to return signatures.
2016-03-21Add Store::dumpPath() methodEelco Dolstra
This allows applying nix-store --verify-path to binary cache stores: NIX_REMOTE=https://cache.nixos.org nix-store --verify-path /nix/store/s5c7...
2016-03-14BinaryCacheStore::isValidPath(): Use .narinfo cacheEelco Dolstra
If a path is in the .narinfo cache, obviously it's valid.
2016-03-04BinaryCacheStore: Remove publicKeyFile argumentEelco Dolstra
The public key can be derived from the secret key, so there's no need for the user to supply it separately.
2016-03-04Eliminate some large string copyingEelco Dolstra
2016-02-26importPaths(): Optionally add NARs to binary cache accessorEelco Dolstra
This enables an optimisation in hydra-queue-runner, preventing a download of a NAR it just uploaded to the cache when reading files like hydra-build-products.
2016-02-25Add NAR / Store accessor abstractionEelco Dolstra
This is primary to allow hydra-queue-runner to extract files like "nix-support/hydra-build-products" from NARs in binary caches.
2016-02-24BinaryCacheStore: Implement addToStore()Eelco Dolstra
So now you can do $ NIX_REMOTE=file:///tmp/binary-cache nix-instantiate '<nixpkgs>' -A hello and lots of other operations.
2016-02-24Show a proper error message for unimplemented binary cache operationsEelco Dolstra
2016-02-24Move BinaryCacheStore / LocalBinaryCacheStore from HydraEelco Dolstra
So you can now do: $ NIX_REMOTE=file:///tmp/binary-cache nix-store -qR /nix/store/...