aboutsummaryrefslogtreecommitdiff
path: root/src/libstore/s3-binary-cache-store.cc
AgeCommit message (Collapse)Author
2020-12-20Overhaul store subclassingJohn Ericson
We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-11treat s3 permission errors as file-not-foundMichael Bishop
Signed-off-by: Jonathan Ringer <jonringer117@gmail.com>
2020-12-04Fix compatibility with newer AWS SDKsStéphan Kochen
Tested against AWS SDK 1.8.99. Fixes #3201.
2020-12-02Sink: Use std::string_viewEelco Dolstra
2020-10-06Remove static variable name clashesEelco Dolstra
This was useful for an experiment with building Nix as a single compilation unit. It's not very useful otherwise but also doesn't hurt...
2020-09-16Fix the s3 storeregnat
Add some necessary casts in the initialisation of the store's config
2020-09-16Fix build issues with gccregnat
2020-09-16Add a name to the storesregnat
So that it can be printed by `nix describe-stores`
2020-09-16Document the new store hierarchyregnat
2020-09-16Properly filter the stores according to their declared uriSchemesregnat
When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-16Correctly call all the parent contructors of the storesregnat
Using virtual inheritance means that only the default constructors of the parent classes will be called, which isn't what we want
2020-09-16Separate store configs from the implemsregnat
Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16Separate the instantiation and initialisation of the storesregnat
Add a new `init()` method to the `Store` class that is supposed to handle all the effectful initialisation needed to set-up the store. The constructor should remain side-effect free and just initialize the c++ data structure. The goal behind that is that we can create “dummy” instances of each store to query static properties about it (the parameters it accepts for example)
2020-09-16Fix the registration of storesregnat
2020-09-16Make the store plugins more introspectableregnat
Directly register the store classes rather than a function to build an instance of them. This gives the possibility to introspect static members of the class or choose different ways of instantiating them.
2020-08-04S3BinaryCacheStore: Fix size determinationEelco Dolstra
2020-08-04Revert "Remove putBytes"Eelco Dolstra
This reverts commit b8eea7e81af53905be7845dffc6d0a83ea8edc97.
2020-07-28Remove putBytesEelco Dolstra
istream->tellg() returns -1 so we can't get the number of bytes written. Fixes 'uploaded 's3://nix-cache/nar/00819r9lp5kajr6baxfw5dhhc0cx8ndxaz43qmd2f0gn1hk1ynlp.nar.xz' (-1 bytes) in 11620 ms' messages.
2020-07-13Make 'nix copy' to s3:// binary caches run in constant memoryEelco Dolstra
2020-07-13Remove istringstream_nocopyEelco Dolstra
2020-07-13Make 'nix copy' to file:// binary caches run in constant memoryEelco Dolstra
2020-07-13Fix 'nix verify --all' on a binary cache and add a testEelco Dolstra
2020-05-11Merge branch 'master' into errors-phase-2Ben Burdette
2020-05-03convert some printError calls to logErrorBen Burdette
2020-04-21remove 'format' from Error constructor callsBen Burdette
2020-04-08datatransfer.{cc,hh} -> filetransfer.{cc,hh}Nikola Knezevic
2020-04-08DataTransfer -> FileTransferNikola Knezevic
2020-04-08DownloadResult -> DataTransferResultNikola Knezevic
2020-04-08Rename src/lib/download.* to src/lib/datatransfer.*Nikola Knezevic
2019-12-17Add priority setting to storesEelco Dolstra
This allows overriding the priority of substituters, e.g. $ nix-store --store ~/my-nix/ -r /nix/store/df3m4da96d84ljzxx4mygfshm1p0r2n3-geeqie-1.4 \ --substituters 'http://cache.nixos.org?priority=100 daemon?priority=10' Fixes #3264.
2019-12-10Make the Store API more type-safeEelco Dolstra
Most functions now take a StorePath argument rather than a Path (which is just an alias for std::string). The StorePath constructor ensures that the path is syntactically correct (i.e. it looks like <store-dir>/<base32-hash>-<name>). Similarly, functions like buildPaths() now take a StorePathWithOutputs, rather than abusing Path by adding a '!<outputs>' suffix. Note that the StorePath type is implemented in Rust. This involves some hackery to allow Rust values to be used directly in C++, via a helper type whose destructor calls the Rust type's drop() function. The main issue is the dynamic nature of C++ move semantics: after we have moved a Rust value, we should not call the drop function on the original value. So when we move a value, we set the original value to bitwise zero, and the destructor only calls drop() if the value is not bitwise zero. This should be sufficient for most types. Also lots of minor cleanups to the C++ API to make it more modern (e.g. using std::optional and std::string_view in some places).
2019-03-15Add a 5 seconds of timeouts to connect to S3.Chaker Benhamed
The default is 1000ms, but we can hit it a lot of we don't have direct link to AWS (e.g. using VPN).
2018-12-07s3: make scheme configurableDaiderd Jordan
This enables using for http for S3 request for debugging or implementations that don't have https configured. This is not a problem for binary caches since they should not contain sensitive information. Both package signatures and AWS auth already protect against tampering.
2018-11-01Disable the S3 content-encoding patchEelco Dolstra
Since we're not using multi-part uploads at the moment, we can drop this patch.
2018-10-30Remove redundant messageEelco Dolstra
2018-10-30S3BinaryCacheStore: Allow disabling multipart uploadsEelco Dolstra
The use of TransferManager has several issues, including that it doesn't allow setting a Content-Encoding without a patch, and it doesn't handle exceptions in worker threads (causing termination on memory allocation failure). Fixes #2493.
2018-10-30Fix broken uploadProgressCallback closureEelco Dolstra
Since the callback is global we can't refer to 'path' in it. This could cause a segfault or printing of arbitrary data.
2018-09-27S3BinaryCacheStore: Don't create bucketsEelco Dolstra
This meant that making a typo in an s3:// URI would cause a bucket to be created. Also it didn't handle eventual consistency very well. Now it's up to the user to create the bucket.
2018-08-09S3BinaryCacheStore: Share TransferManagerEelco Dolstra
TransferManager allocates a lot of memory (50 MiB by default), and it might leak but I'm not sure about that. In any case it was causing OOMs in hydra-queue-runner. So allocate only one TransferManager per S3BinaryCacheStore. Hopefully fixes https://github.com/NixOS/hydra/issues/586.
2018-08-08S3BinaryCacheStore: Don't use the transfer status callbackEelco Dolstra
This callback is executed on a different thread, so exceptions thrown from the callback are not caught: Aug 08 16:25:48 chef hydra-queue-runner[11967]: terminate called after throwing an instance of 'nix::Error' Aug 08 16:25:48 chef hydra-queue-runner[11967]: what(): AWS error: failed to upload 's3://nix-cache/19dbddlfb0vp68g68y19p9fswrgl0bg7.ls' Therefore, just check the transfer status after it completes. Also include the S3 error message in the exception.
2018-08-06Fix setting Content-Encoding in S3 uploadsEelco Dolstra
Fixes https://github.com/NixOS/nix/issues/2333 and https://github.com/NixOS/nixpkgs/issues/44337.
2018-08-06Fix Brotli decompression in 'nix log'Eelco Dolstra
This didn't work anymore since decompression was only done in the non-coroutine case. Decompressors are now sinks, just like compressors. Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK rather than BZ_OK), which we didn't notice because there was a missing 'throw': if (ret != BZ_OK) CompressionError("error while compressing bzip2 file");
2018-08-02s3 binary cache: support specifying an endpointGraham Christensen
Works for uploading and not downloading.
2018-06-08use debug instead of printTalkative to avoid spamming hydra queueAmineChikhaoui
runner logs.
2018-05-30Make LocalBinaryCacheStore::narFromPath() run in constant memoryEelco Dolstra
This reduces memory consumption of nix copy --from file://... --to ~/my-nix /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 514 MiB to 18 MiB for an uncompressed binary cache, and from 192 MiB to 53 MiB for a bzipped binary cache. It may also be faster because fetching can happen concurrently with decompression/writing. Continuation of 48662d151bdf4a38670897beacea9d1bd750376a. Issue https://github.com/NixOS/nix/issues/1681.
2018-05-30Simplify the callback mechanismEelco Dolstra
2018-05-10Remove mention of default since we already know the defaultEelco Dolstra
2018-05-07share the executor between multiple copy threads.AmineChikhaoui
2018-05-07add a FIXME note to find a way to abort the multipart uploads inAmineChikhaoui
case the nix command is interrupted.
2018-05-07Support multi-part uploads for large NARs that exceed the size ofAmineChikhaoui
5Gb.
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-23 19:18:15 +0000