aboutsummaryrefslogtreecommitdiff
path: root/src/libstore
AgeCommit message (Collapse)Author
2024-04-25gc: Find roots using libproc on DarwinArtemis Tosini
Previously, the garbage collector found runtime roots on Darwin by shelling out to `lsof -n -w -F n` then parsing the result. However, this requires an lsof binary and can be extremely slow. The official Apple lsof returns in a reasonable amount of time, about 250ms in my tests, but the lsof packaged in nixpkgs is quite slow, taking about 40 seconds to run the command. Using libproc directly is about the same speed as Apple lsof, and allows us to reënable several tests that were disabled on Darwin. Change-Id: Ifa0adda7984e13c15535693baba835aae79a3577
2024-04-25filetransfer: drop errorSinkeldritch horrors
just accumulate error data into result.data as we would for successful transfers without a dataCallback. errorSink and data would contain the same data in error cases anyway, so splitting them is not very useful. Change-Id: I00e449866454389ac6a564ab411c903fd357dabf
2024-04-25filetransfer: restore http status line reportingeldritch horrors
this was broken in 75b62e52600a44b42693944b50638bf580a2c86e. Change-Id: If8583e802afbcde822623036bf41a9708fbc7c8d
2024-04-25filetransfer: remove decompress request parametereldritch horrors
this is never read. Change-Id: I4c46f140519843a21e452958900e81edd2f78be2
2024-04-24Merge "libstore: Create platform LocalStore subclasses" into mainArtemis Tosini
2024-04-23libstore: Create platform LocalStore subclassesArtemis Tosini
This creates new subclasses of LocalStore for each OS to include platform-specific functionality. Currently this just includes garbage collector roots but it could be extended to sandboxing as well. In order to make sure that the generic LocalStore is not accidentally constructed, its constructor is protected. A Fallback is provided which implements no functionality except constructors. Change-Id: I836a28e90b68309873f75afb83e0f1b2e2c89fb3
2024-04-23Merge changes Ia3e7b1e6,If09be814 into mainQyriad
* changes: meson: flip the switch!! meson: fix cross compilation
2024-04-22meson: fix cross compilationQyriad
This should fix cross compilation in the base case, but this is difficult to test as cross compilation is broken in many different places right now. This should bring Meson back up to cross parity with the Make buildsystem though. Change-Id: If09be8142d1fc975a82b994143ff35be1297dad8
2024-04-23libstore: use curl functions for reading headerseldritch horrors
don't reimplement header parsing. this was only really needed due to the ancient github bug we no longer care about, everything else we have done in custom code can also be done using curl itself. doing this also fixes possible sources of header smuggling (because the header function didn't unfold headers and we'd trim them before parsing, which would've made us read contents of one header as a fully formed header in itself). this is a slight behavior change because we now honor only the first instance of a given header where previous behavior was to honor either the last or a combination of all of them (accept-ranges was logical-or'd by accident). Change-Id: I93cb93ddb91ab98c8991f846014926f6ef039fdb
2024-04-23libstore: remove github etag workaroundeldritch horrors
this was a workaround for a *github* bug that happend *in 2015*. not only is github no longer buggy, it shouldn't have been nix's responsibility to work around these bugs like this to begin with while we're at it we'll also remove another workaround—again for github specifically and again for etag handling—from 2021 that's also not needed any more. future workarounds for serverside bugs should probably come with an expiration date that mutates into a build warning after a while, otherwise this *will* happen again. Change-Id: I74f739ae3e36d40350f78bebcb5869aa8cc9adcd
2024-04-22libstore: use curl_multi_{poll,wakeup}eldritch horrors
the previous solution to the wakeup problem (adding a pipe and passing it as an additional fd to curl_multi_wait) worked, but there have been builtin alternatives for this since 2020. not only do these save code, they're also a lot more likely to work natively on windows when needed Change-Id: Iab751b900997110a8d15de45ea3ab0c42f7e5973
2024-04-22libstore: remove ancient libcurl feature checkseldritch horrors
the oldest version checked for here is 7.47, which was released in 2016. it's probably safe to say that we do not need these any more Change-Id: I003411f6b2ce6d56f7ca337390df3ea86bd59a99
2024-04-21Fix exportReferencesGraph when given store subpathAlyssa Ross
With Nix 2.3, it was possible to pass a subpath of a store path to exportReferencesGraph: with import <nixpkgs> {}; let hello = writeShellScriptBin "hello" '' echo ${toString builtins.currentTime} ''; in writeClosure [ "${hello}/bin/hello" ] This regressed with Nix 2.4, with a very confusing error message, that presumably indicates it was unintentional: error: path '/nix/store/3gl7kgjr4pwf03f0x70dgx9ln3bhl7zc-hello/bin/hello' is not in the Nix store (cherry picked from commit 0774e8ba33c060f56bad3ff696796028249e915a) Change-Id: I00920fb33077b831a1bb4a1b68d515ba8c3c2a69
2024-04-18meson: correctly embed sandbox shell when askedQyriad
Change-Id: I2f6c0d42245204a516d2e424eea26a6391e975ad
2024-04-18meson: correctly set -DSANDBOX_SHELL if we have itQyriad
The statically embedded busybox is not required for Lix to work, but package.nix explicitly sets this, which was accidentally being ignored. Change-Id: Ieeff830ac7d1f5fabe84d1a6cfd82f13d79035bf
2024-04-16Merge "libstore/build: set NO_NEW_PRIVS for the sandbox" into mainIlya K
2024-04-15fix probable format bug in DerivationGoal::buildDoneQyriad
Either the contents of `line` could cause format errors, or this usage is Technically safe. However, I trust nothing, especially with boost::format. Change-Id: I07933b20bde3b305a6e5d61c2a7bab6ecb042ad9
2024-04-15libstore/build: set NO_NEW_PRIVS for the sandboxK900
Change-Id: I711f64e2b68495ed9c85c1a4bd5025405805e43a
2024-04-14don't throw an exception for the trivial case of isStorePath()...Qyriad
Previously if isStorePath() was called on anything other than a top-level /nix/store/some-path, it would throw a BadStorePath exception. This commit duplicates the absolutely trivial check, into maybeParseStorePath(), and leaves exception throwing to parseStorePath(), the function that assumes you're already giving a valid path instead of the one whose purpose is to check if its valid or not... Change-Id: I8dda548f0f88d14ca8c3ee927d64e0ec0681fc7b
2024-04-13libstore/build: just copy the magic /etc files into the sandboxK900
Saves us a bunch of thinking about how to handle symlinks, and prevents the DNS config from changing on the fly under the build, which may or may not be a good thing? Change-Id: I071e6ae7e220884690b788d94f480866f428db71
2024-04-11Merge "Merge pull request #10456 from NixOS/fixpermdeniedbind" into mainIlya K
2024-04-11Merge "Merge pull request #10362 from obsidiansystems/maybeLstat" into mainIlya K
2024-04-11Merge pull request #10456 from NixOS/fixpermdeniedbindThéophane Hufschmitt
Fix adding symlink to the sandbox paths (cherry-picked from commit da1e977bf48cff2a635034c85e7c13878e38efc2) Change-Id: I221c85a38180800ec6552d2e86a88df48398fad8
2024-04-11Merge pull request #10362 from obsidiansystems/maybeLstatJohn Ericson
Factor out `nix::maybeLstat` (cherry-picked from commit 9b88e5284608116b7db0dbd3d5dd7a33b90d52d7) Change-Id: Id890525e847c890fad6593c594772826ac4d1d50
2024-04-11libstore: fix glossary link in documentationeldritch horrors
this should be a link, not an anchor. it should also point to the `gloss-store` element, not the `#gloss-store` element. Change-Id: I1f2803093179549637e10f917ad73399a419131b
2024-04-09meson: correctly set NIX_CONF_DIR to $sysconfdir/nixQyriad
Instead of $sysconfdir. Fixes #231, but there's more to do in following commits to make Meson-built Lix actually look in /etc/nix. Change-Id: Ia8d627070f405843add46e05cff5134b76b8eb48
2024-04-08pragma once and ///@file everything missing itJade Lovelace
Change-Id: Ia1a72348336794b5fb9f2694dd750266089b904e
2024-04-05Revert "libutil: remove Pool::Handle::bad"eldritch horrors
This reverts commit 792844fb861ea7367ac2316c78fec055363f2f9e. Change-Id: I3ca208b62edfd5cd1199478f75cd2edf19a364f6
2024-04-05Revert "libstore: remove one Resource::good flag"eldritch horrors
This reverts commit 87249eb579bf57f4f09e9fca100588a4d6b90b4c. Change-Id: Ide4c6e00c4155216a17e46671ff47151d7bb85b4
2024-04-05Revert "libstore: using throwing finally in withFramedSink"eldritch horrors
This reverts commit 491caad6f62c21ffbcdebe662e63ec0f72e6f3a2. this is not actually legal for nix! throwing exceptions in destructors is fine, but the way nix is set up we'll end up throwing the exception we received from the remote *twice* in some cases, and such cases will cause an immediate terminate without active exception. Change-Id: I74c46b9f26fd791086e4193ec60eb1deb9a5bb2a
2024-04-05Revert "libutil: drop Fs{Source,Sink}::good"eldritch horrors
This reverts commit 1340807e30dba4b3972c31f02861bbaeaeb60e61. Change-Id: I34d2a80eb3c3e9d79cb02b92cd1189da32d18cb6
2024-03-31libutil: drop Fs{Source,Sink}::goodeldritch horrors
setting this only on exceptions caused by actual fd access is not sufficient to diagnose all errors (such as SerialisationError) in some cases. this usually does not have any negative effects since those errors will end up killing the process in another way. this is not a reliable assumption though and we should be using proper error handling (and closing connections more often, preferring to close over keeping something open that might be in a weird state) Change-Id: I1b792cd7ad8ba9ff0f6bd174945ab2575ff2208e
2024-03-31Merge changes Ib62d3d68,Ic3e7affe into mainjade
* changes: Make things that can throw not noexcept anymore Fix various clang-tidy lints
2024-03-31libstore: using throwing finally in withFramedSinkeldritch horrors
the duplication of exception handling was added without justification, so we can only assume that it was done like this because Finally could not throw exceptions safely. since this has now been rectified we will deduplicate this handler code again. Change-Id: I40721f3378c0fd9f34e2914a16d383f6e2713b40
2024-03-31libstore: remove one Resource::good flageldritch horrors
usage of this flag previously kept connections open much longer than necessary, and at the same time obscured that a connection was being dropped when it *was* set. new variable names clarify this somewhat. Change-Id: I11f6f08f37a5e4dc04ea6c6036ea589154b121c6
2024-03-31libutil: remove Pool::Handle::badeldritch horrors
it was used incorrectly (not swapped on handle move), only used in one place (that is now handled with exception handling detection in Handle itself), and if ever reintroduced should be replaced with a different, more understandable mechanism (like an explicit dropAsInvalid method). Change-Id: Ie3e5d5cfa81d335429cb2ee5c3ad85c74a9df17b
2024-03-31libutil: remove Pool::flushBadeldritch horrors
this was never actually used, and bad design in the first place—why should a bad resource be put back into the idle pool? just drop it. Change-Id: Idab8774bee19dadae0209d404c4fb86dd4aeba1e
2024-03-29Add `pre-commit` checksRebecca Turner
The big ones here are `trim-trailing-whitespace` and `end-of-file-fixer` (which makes sure that every file ends with exactly one newline character). Change-Id: Idca73b640883188f068f9903e013cf0d82aa1123
2024-03-29Make things that can throw not noexcept anymoreJade Lovelace
This does involve making a large number of destructors able to throw, because we had to change it high in the class hierarchy. Oh well. Change-Id: Ib62d3d6895b755f20322bb8acc9bf43daf0174b2
2024-03-29Fix various clang-tidy lintsJade Lovelace
* some things that can throw are marked noexcept yet the linter seems to think not. Maybe they can't throw in practice. I would rather not have the UB possibility in pretty obvious cold paths. * various default-case-missing complaints * a fair pile of casts from integer to character, which are in fact deliberate. * an instance of <https://clang.llvm.org/extra/clang-tidy/checks/bugprone/move-forwarding-reference.html> * bugprone-not-null-terminated-result on handing a string to curl in chunks of bytes. our usage is fine. * reassigning a unique_ptr by CRIMES instead of using release(), then using release() and ignoring the result. wild. let's use release() for its intended purpose. Change-Id: Ic3e7affef12383576213a8a7c8145c27e662513d
2024-03-29meson: fix state-dir default valueeldritch horrors
the autoconf build system defaults to /nix/var, not /nix/var/nix. the latter is only used in libstore, so we'll move the extra segment there. Change-Id: Idfbc988ee302355982abdcd51d6d7b5d5d661c0d
2024-03-29meson: add missing explicit dependency on nlohmann_jsonWinter
Without this, the Meson setup won't bail out if nlohmann_json is missing, leading to subpar DX (and maybe worse, but I'm not entirely sure). Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-29meson: install missing/generated headerseldritch horrors
one headers (args/root.hh) was simply missing, and the generated headers were not installed. not all of them *should* be installed either, only a select few (and sadly this needs a custom target for each one, it seems) Change-Id: I37b25517895d0e5e521abc1202fa65624de57ed1
2024-03-29meson: install pkg-config files for librarieseldritch horrors
Change-Id: I14b9d81d09f188eacfb9c68bcfb84751c18e3779
2024-03-29Merge "feat: unprivileged read-only open of SQLite DB" into mainRaito Bezarius
2024-03-27HOT SALE: 15% off your build times!Jade Lovelace
This was achieved by running maintainers/buildtime_report.sh on the build directory of a meson build, then asking "why the heck is json eating our build times", and strategically moving the json using bits out of widely included headers. It turns out that putting literally any metrics whatsoever into the build had immediate and predictable results. Results are 1382.5s frontend time -> 1175.4s frontend time, back end time approximately invariant. Related: https://git.lix.systems/lix-project/lix/issues/159 Change-Id: I7edea95c8536203325c8bb4dae5f32d727a21b2d
2024-03-26libstore/filetransfer: use Lix UA and unnix error messageRaito Bezarius
Once this commit lands, we are even more visible in analytics FWIW. Change-Id: Id7e0c162315d0f191edbea9cb5fb82ce363704b9 Signed-off-by: Raito Bezarius <raito@lix.systems>
2024-03-24libstore: despecialcase protocol version checkeldritch horrors
protocol versions are sent as u64. on the peer we read them as uint64, check that the upper half is 0, and throw an exception if not. we then read an arbitrary amount of data from the peer and dump it to the user terminal. this is a little bit ridiculous, can never happen in correct implementation, and is severly untested. let us just drop it entirely. Change-Id: Ibd2f53a765341ed6439d40d9d1eac11e79c6b5e3
2024-03-24libstore: un-inline copyNAR expansionseldritch horrors
these are copies of copyNAR with only some variables renamed. Change-Id: I98ddd7a98250fa5d304e18e1debf417e9f7768dd
2024-03-23feat: unprivileged read-only open of SQLite DBRaito Bezarius
If the state SQLite database is configured to use a write-ahead-log, it creates WAL files in the state directory. When the state SQLite database is closed by the `nix-daemon` after builds, those files are removed. When an unprivileged user would like to open _in read only_ that database, they cannot do so because they would need to create those WAL files and they do not have the permission to do so. For this, SQLite offers a "persistent WAL" feature [1] to leave the WAL files around, even after closing the database. This CL enable the persistent WAL mode. Fixes: https://github.com/NixOS/nix/issues/10300 [1]: https://www.sqlite.org/wal.html Change-Id: Id8ae534d7d2290457af28782e5215222ae051fe5 Signed-off-by: Raito Bezarius <raito@lix.systems>