aboutsummaryrefslogtreecommitdiff
path: root/src/libutil/serialise.cc
AgeCommit message (Collapse)Author
2022-04-08Remove unused Error.name fieldEelco Dolstra
2022-02-25Remove std::string alias (for real this time)Eelco Dolstra
Also use std::string_view in a few more places.
2022-01-18Get rid of std::shared_ptr<std::string> and ref<std::string>Eelco Dolstra
These were needed back in the pre-C++11 era because we didn't have move semantics. But now we do.
2022-01-17serialise.hh: Use std::string_viewEelco Dolstra
2021-09-14Fix clang warningEelco Dolstra
2021-03-10Use libarchive for all compressionYorick van Pelt
2021-01-21Improve error formattingEelco Dolstra
Changes: * The divider lines are gone. These were in practice a bit confusing, in particular with --show-trace or --keep-going, since then there were multiple lines, suggesting a start/end which wasn't the case. * Instead, multi-line error messages are now indented to align with the prefix (e.g. "error: "). * The 'description' field is gone since we weren't really using it. * 'hint' is renamed to 'msg' since it really wasn't a hint. * The error is now printed *before* the location info. * The 'name' field is no longer printed since most of the time it wasn't very useful since it was just the name of the exception (like EvalError). Ideally in the future this would be a unique, easily googleable error ID (like rustc). * "trace:" is now just "…". This assumes error contexts start with something like "while doing X". Example before: error: --- AssertionError ---------------------------------------------------------------------------------------- nix at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| assertion 'false' failed ----------------------------------------------------- show-trace ----------------------------------------------------- trace: while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) { Example after: error: assertion 'false' failed at: (7:7) in file: /home/eelco/Dev/nixpkgs/pkgs/applications/misc/hello/default.nix 6| 7| x = assert false; 1; | ^ 8| … while evaluating the attribute 'x' of the derivation 'hello-2.10' at: (192:11) in file: /home/eelco/Dev/nixpkgs/pkgs/stdenv/generic/make-derivation.nix 191| // (lib.optionalAttrs (!(attrs ? name) && attrs ? pname && attrs ? version)) { 192| name = "${attrs.pname}-${attrs.version}"; | ^ 193| } // (lib.optionalAttrs (stdenv.hostPlatform != stdenv.buildPlatform && !dontAddHostSuffix && (attrs ? name || (attrs ? pname && attrs ? version)))) {
2020-12-02read(): Use char * instead of unsigned char *Eelco Dolstra
This gets rid of some pointless casts.
2020-12-02Sink: Use std::string_viewEelco Dolstra
2020-11-10Fix stack overflow introduced in #4206Ricardo M. Correia
2020-10-30Fix memory corruption caused by GC-invisible coroutine stacksRobert Hensing
Crucially this introduces BoehmGCStackAllocator, but it also adds a bunch of wiring to avoid making libutil depend on bdw-gc. Part of the solutions for #4178, #4200
2020-10-07Serialize exceptions from the sandbox process to the parentEelco Dolstra
Fixes #4118.
2020-09-17Add Source.drainInto(Sink)Robert Hensing
2020-07-15Revert "LocalStore::addToStoreFromDump copy in chunks"John Ericson
This reverts commit 592851fb67cd15807109d6f65fb81f6af89af966. We don't need this extra feature anymore
2020-07-15Get rid of `LocalStore::addToStoreCommon`John Ericson
I got it to just become `LocalStore::addToStoreFromDump`, cleanly taking a store and then doing nothing too fancy with it. `LocalStore::addToStore(...Path...)` is now just a simple wrapper with a bare-bones sinkToSource of the right dump command.
2020-07-14LocalStore::addToStoreFromDump copy in chunksJohn Ericson
Rather than copying byte-by-byte, we let the coroutine know how much data we would like it to send back to us.
2020-06-15Get rid of explicit ErrorInfo constructorsEelco Dolstra
2020-05-11switch from printError warnings to logWarningsBen Burdette
2019-03-14experimental/optional -> optionalEelco Dolstra
2018-09-26sinkToSource(): Start the coroutine lazilyEelco Dolstra
In particular this causes copyStorePath() from HttpBinaryCacheStore to only start a download if needed. E.g. if the destination LocalStore goes to sleep waiting for the path lock and another process creates the path, then LocalStore::addToStore() will never read from the source so we don't have to do the download.
2018-09-26Make NAR header check more robustEelco Dolstra
Changes std::bad_alloc into bad archive: input doesn't look like a Nix archive
2018-08-21Improve 'coroutine has finished' error messageEelco Dolstra
2018-07-31Add a check for broken Boost versionsEelco Dolstra
In some Boost versions, coroutines don't propagate exceptions properly, causing Nix to fail with the exception 'coroutine has finished'.
2018-05-21serialise: fix buffer size used, hide method for internal use onlyWill Dietz
Fixes #2169.
2018-03-19serialise.cc: remove pessimising moveWill Dietz
from clang6: src/libutil/serialise.cc:189:23: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
2018-03-16Reduce substitution memory consumptionEelco Dolstra
copyStorePath() now pipes the output of srcStore->narFromPath() directly into dstStore->addToStore(). The sink used by the former is converted into a source usable by the latter using boost::coroutine2. This is based on [1]. This reduces the maximum resident size of $ nix build --store ~/my-nix/ /nix/store/b0zlxla7dmy1iwc3g459rjznx59797xy-binutils-2.28.1 --substituters file:///tmp/binary-cache-xz/ --no-require-sigs from 418592 KiB to 53416 KiB. (The previous commit also reduced the runtime from ~4.2s to ~3.4s, not sure why.) A further improvement will be to download files into a Sink. [1] https://github.com/NixOS/nix/compare/master...Mathnerd314:dump-fix-coroutine#diff-dcbcac55a634031f9cc73707da6e4b18 Issue #1969.
2018-02-13Fix #1762Linus Heckemann
nix-store --export, nix-store --dump, and nix dump-path would previously fail silently if writing the data out failed, because a) FdSink::write ignored exceptions, and b) the commands relied on FdSink's destructor, which ignores exceptions, to flush the data out. This could cause rather opaque issues with installing nixos, because nix-store --export would happily proceed even if it couldn't write its data out (e.g. if nix-store --import on the other side of the pipe failed). This commit adds tests that expose these issues in the nix-store commands, and fixes them for all three.
2017-03-01readString(): Read directly into std::stringEelco Dolstra
When reading a huge string, this halves memory consumption. (Strictly speaking, this appears only valid in C++17, but who cares...)
2017-03-01Handle importing NARs containing files greater than 4 GiBEelco Dolstra
Also templatize readInt() to work for various integer types.
2017-01-16AutoDeleteArray -> std::unique_ptrEelco Dolstra
Also, switch to C++14 for std::make_unique.
2016-09-21printMsg(lvlError, ...) -> printError(...) etc.Eelco Dolstra
2016-07-13Make Buffered{Source,Sink} move-safeShea Levy
2016-03-04Eliminate some large string copyingEelco Dolstra
2016-02-26FdSource: track number of bytes readEelco Dolstra
2016-02-24Remove bad daemon connections from the poolEelco Dolstra
This is necessary for long-running processes like hydra-queue-runner: if a nix-daemon worker is killed, we need to stop reusing that connection.
2015-09-03Implement buildDerivation() via the daemonEelco Dolstra
2015-07-20More cleanupEelco Dolstra
2015-07-17OCD: foreach -> C++11 ranged forEelco Dolstra
2015-07-17Allow remote builds without sending the derivation closureEelco Dolstra
Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2014-06-10Print a warning when loading a large path into memoryEelco Dolstra
I.e. if you have a derivation with src = ./huge-directory; you'll get a warning that this is not a good idea.
2014-03-12Remove unnecessary null pointer checksEelco Dolstra
Fixes #225.
2013-06-07Process stderr from substituters while doing have/info queriesEelco Dolstra
2012-02-09Use data() instead of c_str() where appropriateEelco Dolstra
2011-12-16* importPath() -> importPaths(). Because of buffering of the inputEelco Dolstra
stream it's now necessary for the daemon to process the entire sequence of exported paths, rather than letting the client do it.
2011-12-16* Avoid expensive conversions from char arrays to STL strings.Eelco Dolstra
2011-12-16* Make the import operation through the daemon much more efficientEelco Dolstra
(way fewer roundtrips) by allowing the client to send data in bigger chunks. * Some refactoring.
2011-12-16* Clean up exception handling.Eelco Dolstra
2011-12-15* Refactoring: move sink/source buffering into separate classes.Eelco Dolstra
* Buffer the HashSink. This speeds up hashing a bit because it prevents lots of calls to the hash update functions (e.g. nix-hash went from 9.3s to 8.7s of user time on the closure of my /var/run/current-system).
2011-12-15* Buffer reads in FdSource. Together with write buffering, thisEelco Dolstra
significantly cuts down the number of syscalls (e.g., for "nix-store -qR /var/run/current-system" via the daemon, it reduced the number of syscalls in the client from 29134 to 4766 and in the daemon from 44266 to 20666).
2011-12-14* Buffer writes in FdSink. This significantly reduces the number ofEelco Dolstra
system calls / context switches when dumping a NAR and in the worker protocol.
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-23 02:10:43 +0000