aboutsummaryrefslogtreecommitdiff
path: root/src/libstore
AgeCommit message (Collapse)Author
2017-03-21Move istringstream_nocopy to a separate fileEelco Dolstra
2017-03-21Honor $NIX_SSHOPTS againEelco Dolstra
NixOps needs this.
2017-03-20Merge branch 'darwin-s3-binary-cache-store' of https://github.com/shlevy/nixEelco Dolstra
2017-03-16ssh:// -> ssh-ng://, legacy-ssh:// -> ssh://Eelco Dolstra
2017-03-16copyPaths(): Use queryValidPaths() to reduce SSH latencyEelco Dolstra
2017-03-16copyPaths(): Don't query path info for a path the target already hasEelco Dolstra
For example, this cuts "nix-copy-closure --from" on a NixOS system closure from 15.9s to 0.5s.
2017-03-16LegacySSHStore: Provide a faster implementation of computeFSClosure()Eelco Dolstra
This avoids the latency of the standard implementation, which can make a huge difference (e.g. 16.5s -> 0.5s on a NixOS system closure).
2017-03-16Fix nix-copy-closure --toEelco Dolstra
2017-03-16Remove "killing process <pid>" messagesEelco Dolstra
They convey no useful information.
2017-03-15More precise compression settingsEelco Dolstra
2017-03-15runProgram(): Distinguish between empty input and no inputEelco Dolstra
For example, if we call brotli with an empty input, it shouldn't read from the caller's stdin.
2017-03-15Compress NAR listings using the "text-compression" methodEelco Dolstra
So if "text-compression=br", the .ls file in S3 will get a Content-Encoding of "br". Brotli appears to compress better than xz for this kind of file and is natively supported by browsers.
2017-03-15S3BinaryCacheStore: Set Content-TypeEelco Dolstra
This is necessary for serving log files to browsers.
2017-03-15S3BinaryCacheStore: Support compression of narinfo and log filesEelco Dolstra
You can now set the store parameter "text-compression=br" to compress textual files in the binary cache (i.e. narinfo and logs) using Brotli. This sets the Content-Encoding header; the extension of compressed files is unchanged. You can separately specify the compression of log files using "log-compression=br". This is useful when you don't want to compress narinfo files for backward compatibility.
2017-03-15Add support for brotli compressionEelco Dolstra
Build logs on cache.nixos.org are compressed using Brotli (since this allows them to be decompressed automatically by Chrome and Firefox), so it's handy if "nix log" can decompress them.
2017-03-15BinaryCacheStore: Implement getBuildLog()Eelco Dolstra
We assume that build logs are stored under log/<drv>, e.g. /nix/store/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv maps to https://cache.nixos.org/log/q7ab198v13p0f8x8wgnd75dva7d5mip6-friday-devil-0.1.1.1.drv
2017-03-15Store: Add a method for getting build logsEelco Dolstra
This allows various Store implementations to provide different ways to get build logs. For example, BinaryCacheStore can get the build logs from the binary cache. Also, remove the log-servers option since we can use substituters for this.
2017-03-12Merge branch 'configurable-aws-region'Shea Levy
2017-03-08Add option to disable import-from-derivation completely, even if the drv is ↵Shea Levy
already realized
2017-03-06Properly set the caFile for aws-sdk-cpp s3Shea Levy
2017-03-06istringstream_nocopy: Implement in a standards-compliant way.Shea Levy
Fixes the problem mentioned in e6a61b8da788efbbbb0eb690c49434b6b5fc9741 See #1135
2017-03-03Add aws-region param to S3 store URLsShea Levy
2017-03-03Cache connection failuresEelco Dolstra
2017-03-03SSHMaster: Make thread-safeEelco Dolstra
2017-03-03build-remote: Don't use a SSH masterEelco Dolstra
This is unnecessary because we make only one connection.
2017-03-03Improve SSH handlingEelco Dolstra
* Unify SSH code in SSHStore and LegacySSHStore. * Fix a race starting the SSH master. We now wait synchronously for the SSH master to finish starting. This prevents the SSH clients from starting their own connections. * Don't use a master if max-connections == 1. * Add a "max-connections" store parameter. * Add a "compress" store parameter.
2017-03-03Fix fatal "broken pipe" error when $NIX_BUILD_HOOK is missingEelco Dolstra
2017-03-01TeeSink: Pre-reserve string spaceEelco Dolstra
When receiving a very large file, this can prevent the string from having tobe copied, which temporarily doubles memory consumption.
2017-03-01RemoteStore::addToStore(): Send NAR rather than string containing NAREelco Dolstra
This allows the NAR to be streamed in the future (though we're not doing that yet).
2017-03-01Tweak messageEelco Dolstra
2017-03-01Handle importing NARs containing files greater than 4 GiBEelco Dolstra
Also templatize readInt() to work for various integer types.
2017-02-28Fix building against older curl versionsEelco Dolstra
http://hydra.nixos.org/build/49490928
2017-02-28Improve SQLite busy handlingEelco Dolstra
2017-02-28NarInfoDiskCache: Handle SQLite busy errorsEelco Dolstra
2017-02-28In SQLite errors, include the database pathEelco Dolstra
This is necessary because we have multiple SQLite databases (e.g. the binary cache cache).
2017-02-28Support auto-configuration of build-max-jobsEelco Dolstra
"build-max-jobs" and the "-j" option can now be set to "auto" to use the number of CPUs in the system. (Unlike build-cores, it doesn't use 0 to imply auto-configuration, because a) magic values are a bad idea in general; b) 0 is a legitimate value used to disable local building.) Fixes #1198.
2017-02-27_SC_NPROCESSORS_ONLN -> std::thread::hardware_concurrency()Eelco Dolstra
2017-02-27Retry downloads on HTTP/2 stream errorsEelco Dolstra
Issue #1254.
2017-02-24Register content-addressability assertion for fixed outputsEelco Dolstra
2017-02-24Verify content-addressability assertions at registration timeEelco Dolstra
2017-02-22RemoteStore::addToStore(): Pass content-addressability assertionEelco Dolstra
... and use this in Downloader::downloadCached(). This fixes $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-21Handle CURLE_RECV_ERROR as a transient errorEelco Dolstra
This fixes unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21Fix building without S3 supportEelco Dolstra
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21Log AWS retriesEelco Dolstra
2017-02-16Support netrc in <nix/fetchurl.nix>Eelco Dolstra
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial repositories, e.g. import <nix/fetchurl.nix> { url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2; sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7"; } where /etc/nix/netrc contains: machine bitbucket.org login edolstra password blabla... This works even when sandboxing is enabled. To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16Move netrcFile to SettingsEelco Dolstra
Also get rid of Settings::processEnvironment(), it appears to be useless.
2017-02-16Include "curl" in the User-Agent headerEelco Dolstra
Some sites (e.g. BitBucket) give a helpful 401 error when trying to download a private archive if the User-Agent contains "curl", but give a redirect to a login page otherwise (so for instance "nix-prefetch-url" will succeed but produce useless output).
2017-02-16Merge pull request #1215 from k0001/netrc-1Eelco Dolstra
Add netrc-file support
2017-02-14Add support for s3:// URIsEelco Dolstra
This adds support for s3:// URIs in all places where Nix allows URIs, e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix> and NIX_PATH. It allows fetching resources from private S3 buckets, using credentials obtained from the standard places (i.e. AWS_* environment variables, ~/.aws/credentials and the EC2 metadata server). This may not be super-useful in general, but since we already depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-09Documentation.Renzo Carbonara