aboutsummaryrefslogtreecommitdiff
path: root/src/libstore
AgeCommit message (Collapse)Author
2017-03-01TeeSink: Pre-reserve string spaceEelco Dolstra
When receiving a very large file, this can prevent the string from having tobe copied, which temporarily doubles memory consumption.
2017-03-01RemoteStore::addToStore(): Send NAR rather than string containing NAREelco Dolstra
This allows the NAR to be streamed in the future (though we're not doing that yet).
2017-03-01Tweak messageEelco Dolstra
2017-03-01Handle importing NARs containing files greater than 4 GiBEelco Dolstra
Also templatize readInt() to work for various integer types.
2017-02-28Fix building against older curl versionsEelco Dolstra
http://hydra.nixos.org/build/49490928
2017-02-28Improve SQLite busy handlingEelco Dolstra
2017-02-28NarInfoDiskCache: Handle SQLite busy errorsEelco Dolstra
2017-02-28In SQLite errors, include the database pathEelco Dolstra
This is necessary because we have multiple SQLite databases (e.g. the binary cache cache).
2017-02-28Support auto-configuration of build-max-jobsEelco Dolstra
"build-max-jobs" and the "-j" option can now be set to "auto" to use the number of CPUs in the system. (Unlike build-cores, it doesn't use 0 to imply auto-configuration, because a) magic values are a bad idea in general; b) 0 is a legitimate value used to disable local building.) Fixes #1198.
2017-02-27_SC_NPROCESSORS_ONLN -> std::thread::hardware_concurrency()Eelco Dolstra
2017-02-27Retry downloads on HTTP/2 stream errorsEelco Dolstra
Issue #1254.
2017-02-24Register content-addressability assertion for fixed outputsEelco Dolstra
2017-02-24Verify content-addressability assertions at registration timeEelco Dolstra
2017-02-22RemoteStore::addToStore(): Pass content-addressability assertionEelco Dolstra
... and use this in Downloader::downloadCached(). This fixes $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-21Handle CURLE_RECV_ERROR as a transient errorEelco Dolstra
This fixes unable to download ‘https://cache.nixos.org/nar/077h8ji74y9b0qx7rjk71xd80vjqp6q5gy137r553jlvdlxdcdlk.nar.xz’: HTTP error 200 (curl error: Failure when receiving data from the peer)
2017-02-21Fix building without S3 supportEelco Dolstra
http://hydra.nixos.org/build/49031196/nixlog/2/raw
2017-02-21Log AWS retriesEelco Dolstra
2017-02-16Support netrc in <nix/fetchurl.nix>Eelco Dolstra
This allows <nix/fetchurl.nix> to fetch private Git/Mercurial repositories, e.g. import <nix/fetchurl.nix> { url = https://edolstra@bitbucket.org/edolstra/my-private-repo/get/80a14018daed.tar.bz2; sha256 = "1mgqzn7biqkq3hf2697b0jc4wabkqhmzq2srdymjfa6sb9zb6qs7"; } where /etc/nix/netrc contains: machine bitbucket.org login edolstra password blabla... This works even when sandboxing is enabled. To do: add unpacking support (i.e. fetchzip functionality).
2017-02-16Move netrcFile to SettingsEelco Dolstra
Also get rid of Settings::processEnvironment(), it appears to be useless.
2017-02-16Include "curl" in the User-Agent headerEelco Dolstra
Some sites (e.g. BitBucket) give a helpful 401 error when trying to download a private archive if the User-Agent contains "curl", but give a redirect to a login page otherwise (so for instance "nix-prefetch-url" will succeed but produce useless output).
2017-02-16Merge pull request #1215 from k0001/netrc-1Eelco Dolstra
Add netrc-file support
2017-02-14Add support for s3:// URIsEelco Dolstra
This adds support for s3:// URIs in all places where Nix allows URIs, e.g. in builtins.fetchurl, builtins.fetchTarball, <nix/fetchurl.nix> and NIX_PATH. It allows fetching resources from private S3 buckets, using credentials obtained from the standard places (i.e. AWS_* environment variables, ~/.aws/credentials and the EC2 metadata server). This may not be super-useful in general, but since we already depend on aws-sdk-cpp, it's a cheap feature to add.
2017-02-09Documentation.Renzo Carbonara
2017-02-08Include config.h implicitly with '-include config.h' in CFLAGSTuomas Tynkkynen
Because config.h can #define things like _FILE_OFFSET_BITS=64 and not every compilation unit includes config.h, we currently compile half of Nix with _FILE_OFFSET_BITS=64 and other half with _FILE_OFFSET_BITS unset. This causes major havoc with the Settings class on e.g. 32-bit ARM, where different compilation units disagree with the struct layout. E.g.: diff --git a/src/libstore/globals.cc b/src/libstore/globals.cc @@ -166,6 +166,8 @@ void Settings::update() _get(useSubstitutes, "build-use-substitutes"); + fprintf(stderr, "at Settings::update(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes); _get(buildUsersGroup, "build-users-group"); diff --git a/src/libstore/remote-store.cc b/src/libstore/remote-store.cc +++ b/src/libstore/remote-store.cc @@ -138,6 +138,8 @@ void RemoteStore::initConnection(Connection & conn) void RemoteStore::setOptions(Connection & conn) { + fprintf(stderr, "at RemoteStore::setOptions(): &useSubstitutes = %p\n", &nix::settings.useSubstitutes); conn.to << wopSetOptions Gave me: at Settings::update(): &useSubstitutes = 0xb6e5c5cb at RemoteStore::setOptions(): &useSubstitutes = 0xb6e5c5c7 That was not a fun one to debug!
2017-02-07Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nixEelco Dolstra
2017-02-07Add a LegacySSHStore that uses nix-store --serveEelco Dolstra
This is useful for nix-copy-closure.
2017-02-07copyStorePath(): Don't require signatures for "trusted" storesEelco Dolstra
For example, SSH stores could be trusted.
2017-02-07Provide default implementations for a couple of Store methodsEelco Dolstra
2017-02-07SSHStore: uri -> hostEelco Dolstra
2017-02-07CleanupEelco Dolstra
2017-02-02Add netrc-file supportRenzo Carbonara
2017-02-02exportReferencesGraph: Only export in JSON format when in structured modeEelco Dolstra
This prevents breaking compatibility with builders that read "closure.*", since they would accidentally pick up the new JSON files.
2017-02-01Restore default signal handling in child processesEelco Dolstra
In particular, this fixes Ctrl-C in nix-shell sessions.
2017-01-27Periodically purge binary-cache.sqliteEelco Dolstra
2017-01-27Implement TTL for binary cache lookupsEelco Dolstra
2017-01-27Remove unused NARExistence tableEelco Dolstra
2017-01-26exportReferencesGraph: Export more complete info in JSON formatEelco Dolstra
This writes info about every path in the closure in the same format as ‘nix path-info --json’. Thus it also includes NAR hashes and sizes. Example: [ { "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl", "narSize": 197648, "references": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24" ], "closureSize": 20939776 }, { "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24", "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3", "narSize": 20742128, "references": [ "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24" ], "closureSize": 20742128 } ] Fixes #1134.
2017-01-26Add support for passing structured data to buildersEelco Dolstra
Previously, all derivation attributes had to be coerced into strings so that they could be passed via the environment. This is lossy (e.g. lists get flattened, necessitating configureFlags vs. configureFlagsArray, of which the latter cannot be specified as an attribute), doesn't support attribute sets at all, and has size limitations (necessitating hacks like passAsFile). This patch adds a new mode for passing attributes to builders, namely encoded as a JSON file ".attrs.json" in the current directory of the builder. This mode is activated via the special attribute __structuredAttrs = true; (The idea is that one day we can set this in stdenv.mkDerivation.) For example, stdenv.mkDerivation { __structuredAttrs = true; name = "foo"; buildInputs = [ pkgs.hello pkgs.cowsay ]; doCheck = true; hardening.format = false; } results in a ".attrs.json" file containing (sans the indentation): { "buildInputs": [], "builder": "/nix/store/ygl61ycpr2vjqrx775l1r2mw1g2rb754-bash-4.3-p48/bin/bash", "configureFlags": [ "--with-foo", "--with-bar=1 2" ], "doCheck": true, "hardening": { "format": false }, "name": "foo", "nativeBuildInputs": [ "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10", "/nix/store/4jnvjin0r6wp6cv1hdm5jbkx3vinlcvk-cowsay-3.03" ], "propagatedBuildInputs": [], "propagatedNativeBuildInputs": [], "stdenv": "/nix/store/f3hw3p8armnzy6xhd4h8s7anfjrs15n2-stdenv", "system": "x86_64-linux" } "passAsFile" is ignored in this mode because it's not needed - large strings are included directly in the JSON representation. It is up to the builder to do something with the JSON representation. For example, in bash-based builders, lists/attrsets of string values could be mapped to bash (associative) arrays.
2017-01-26Fix assertion failure when a path is lockedEelco Dolstra
Fixes: nix-store: src/libstore/build.cc:3649: void nix::Worker::run(const Goals&): Assertion `!awake.empty()' failed.
2017-01-26UserLock: Fix multi-threaded access to a global variableEelco Dolstra
2017-01-26openLockFile: Return an AutoCloseFDEelco Dolstra
2017-01-26UserLock: Make more RAII-ishEelco Dolstra
2017-01-26Moving more code out of DerivationGoal::startBuilder()Eelco Dolstra
2017-01-26On HTTP errors, also show the curl errorEelco Dolstra
This is a hopefully temporary measure to diagnose the intermittent "HTTP error 200" failures.
2017-01-26Move exportReferencesGraph into a separate methodEelco Dolstra
startBuilder() is getting rather obese.
2017-01-24Enable verbose curl outputEelco Dolstra
Closes #1182.
2017-01-20nix-copy-closure: Implement in C++.Shea Levy
Tests fail currently because the database is not given proper hashes in the VM
2017-01-19Merge pull request #981 from shlevy/build-remote-c++Eelco Dolstra
build-remote: Implement in C++
2017-01-19Kill builds when we get EOF on the log FDEelco Dolstra
This closes a long-time bug that allowed builds to hang Nix indefinitely (regardless of timeouts) simply by doing exec > /dev/null 2>&1; while true; do true; done Now, on EOF, we just send SIGKILL to the child to make sure it's really gone.
2017-01-19Use std::unique_ptr for HookInstanceEelco Dolstra