aboutsummaryrefslogtreecommitdiff
path: root/src/libutil/compression.cc
AgeCommit message (Collapse)Author
2024-06-20BrotliDecompressionSource: don't bail out too earlyK900
If we've consumed the entire input, that doesn't actually mean we're done decompressing - there might be more output left. This worked (?) in most cases because the input and output sizes are pretty comparable, but sometimes they're not and then things get very funny. Change-Id: I73435a654a911b8ce25119f713b80706c5783c1b
2024-06-19libutil: add makeDecompressionSourceeldritch horrors
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
2024-06-19libutil: BrotliDecompression{Sink -> Source}eldritch horrors
Change-Id: I9579dd08f7bd0f927bde9d3128515b0cee15f320
2024-05-29util.hh: Delete remaining file and clean up headersTom Hubrecht
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-03-11util.hh: split out signals stuffJade Lovelace
Copies part of the changes of ac89bb064aeea85a62b82a6daf0ecca7190a28b7 Change-Id: I9ce601875cd6d4db5eb1132d7835c5bab9f126d8
2023-04-07Rename and protect `BufferedSink::write`Théophane Hufschmitt
The `write` name is ambiguous and could lead to some funny bugs like https://github.com/NixOS/nix/pull/8173#issuecomment-1500009480. So rename it to the more explicit `writeUnbuffered`. Besides, this method shouldn't be (and isn't) used outside of the class implementation, so mark it `protected`. This makes it more symetrical to `BufferedSource` which uses a `protected readUnbuffered` method.
2022-01-18Get rid of std::shared_ptr<std::string> and ref<std::string>Eelco Dolstra
These were needed back in the pre-C++11 era because we didn't have move semantics. But now we do.
2021-10-13Style tweaksEelco Dolstra
2021-10-12Add compression level for NARsTom Bereknyei
Based off on @dtzWill's #2276
2021-08-1015f4d4f follow upPamplemousse
* libstore: `bz2` should not be linked * libutil: `zlib.h` should not be included Signed-off-by: Pamplemousse <xav.maso@gmail.com>
2021-04-22Remove makeDecompressionSource()Eelco Dolstra
This function doesn't support all compression methods (i.e. 'none' and 'br') so it shouldn't be exposed. Also restore the original decompress() as a wrapper around makeDecompressionSink().
2021-04-22libutil: allow decompression with none/empty methodLuke Granger-Brown
The S3 store relies on the ability to be able to decompress things with an empty method, because it just passes the value of the Content-Encoding directly to decompress. If the file is not compressed, then this will cause the compression routine to get confused. This caused NixOS/nixpkgs#120120.
2021-04-20Fix brotli compression of files > 128 KiBEelco Dolstra
This has been broken since faa31f40846f7a4dbc2487d000b112a6aef69d1b.
2021-04-15Drop libbz2 / zlib / lzma dependency + style fixesEelco Dolstra
2021-03-10Use libarchive for all compressionYorick van Pelt
2020-12-02Sink: Use std::string_viewEelco Dolstra
2020-04-21remove 'format' from Error constructor callsBen Burdette
2019-12-13Improve gzip error messageEelco Dolstra
2019-12-13Initial gzip supportTom Bereknyei
Closes #3256
2018-09-17Fix build failure if parallel xz is not availableEelco Dolstra
2018-08-21Fix warnings in compression.ccEelco Dolstra
2018-08-06Fix Brotli decompression in 'nix log'Eelco Dolstra
This didn't work anymore since decompression was only done in the non-coroutine case. Decompressors are now sinks, just like compressors. Also fixed a bug in bzip2 API handling (we have to handle BZ_RUN_OK rather than BZ_OK), which we didn't notice because there was a missing 'throw': if (ret != BZ_OK) CompressionError("error while compressing bzip2 file");
2018-08-06Require libbrotliEelco Dolstra
2018-05-02Fix bzip2 compression of files > 4 GiBEelco Dolstra
Bzip2's 'avail_in' parameter is declared as an unsigned int, so assigning a size_t length to it led to silent truncation. Fixes #2111.
2018-03-22Fix buildEelco Dolstra
2018-03-16decompress(): Use a Source and SinkEelco Dolstra
This allows decompression to happen in O(1) memory.
2018-02-11compression: make parallel sink separate classWill Dietz
2018-02-11compression: print warning if parallel requested but not supportedWill Dietz
2018-02-09configure.ac: check if lzma has MT support, fix deb build/etc.Will Dietz
2018-02-07make multi threaded compression configurable and use single threadedAmineChikhaoui
by default.
2018-02-07set block size to 0 to let the lzma lib choose the right one, addAmineChikhaoui
some comments about possible improvements wrt memory usage/threading.
2018-02-06support multi threaded xz encoder, this might be particularly useful inAmineChikhaoui
the case of hydra where the overhead of single threaded encoding is more noticeable e.g most of the time spent in "Sending inputs"/"Receiving outputs" is due to compression while the actual upload to the binary cache seems to be negligible.
2017-12-30use libbrotli directly when availableWill Dietz
* Look for both 'brotli' and 'bro' as external command, since upstream has renamed it in newer versions. If neither are found, current runtime behavior is preserved: try to find 'bro' on PATH. * Limit amount handed to BrotliEncoderCompressStream to ensure interrupts are processed in a timely manner. Testing shows negligible performance impact. (Other compression sinks don't seem to require this)
2017-07-30Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-03-22Fix xz decompressionEelco Dolstra
Fixes #1285.
2017-03-21LocalFSStore::getBuildLog(): Handle corrupted logsEelco Dolstra
2017-03-21Don't hang in decompression if bzip2 data ends prematurelyEelco Dolstra
2017-03-15runProgram(): Distinguish between empty input and no inputEelco Dolstra
For example, if we call brotli with an empty input, it shouldn't read from the caller's stdin.
2017-03-15S3BinaryCacheStore: Support compression of narinfo and log filesEelco Dolstra
You can now set the store parameter "text-compression=br" to compress textual files in the binary cache (i.e. narinfo and logs) using Brotli. This sets the Content-Encoding header; the extension of compressed files is unchanged. You can separately specify the compression of log files using "log-compression=br". This is useful when you don't want to compress narinfo files for backward compatibility.
2017-03-15Fix assertion failure on SIGINTEelco Dolstra
nix: src/libutil/compression.cc:142: virtual nix::XzSink::~XzSink(): Assertion `finished' failed.
2017-03-15Add support for brotli compressionEelco Dolstra
Build logs on cache.nixos.org are compressed using Brotli (since this allows them to be decompressed automatically by Chrome and Firefox), so it's handy if "nix log" can decompress them.
2016-11-26Revert "Get rid of unicode quotes (#1140)"Eelco Dolstra
This reverts commit f78126bfd6b6c8477fcdbc09b2f98772dbe9a1e7. There really is no need for such a massive change...
2016-11-25Get rid of unicode quotes (#1140)Guillaume Maudoux
2016-05-04Do compression in a sinkEelco Dolstra
2016-05-04Add a Store::addToStore() variant that accepts a NAREelco Dolstra
As a side effect, this ensures that signatures are propagated when copying paths between stores. Also refactored import/export to make use of this.
2016-05-04Fix segfault in xz compression/decompressionEelco Dolstra
2016-04-29BinaryCacheStore: Support bzip2 compressionEelco Dolstra
2016-04-29BinaryCacheStore: Support "none" compression methodEelco Dolstra
2016-04-22Make compression interruptibleEelco Dolstra
2016-04-15BinaryCacheStore::readFile(): Return a shared_ptr to a stringEelco Dolstra
This allows readFile() to indicate that a file doesn't exist, and might eliminate some large string copying.