aboutsummaryrefslogtreecommitdiff
path: root/src/libstore/build
AgeCommit message (Collapse)Author
2024-06-27store: guess the URL of failing fixed-output derivationsJade Lovelace
This is a shameless layering violation in favour of UX. It falls back trivially to "unknown", so it's purely a UX feature. Diagnostic sample: ``` error: hash mismatch in fixed-output derivation '/nix/store/sjfw324j4533lwnpmr5z4icpb85r63ai-x1.drv': likely URL: https://meow.puppy.forge/puppy.tar.gz specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= got: sha256-a1Qvp3FOOkWpL9kFHgugU1ok5UtRPSu+NwCZKbbaEro= ``` Change-Id: I873eedcf7984ab23f57a6754be00232b5cb5b02c
2024-06-24Add build-dir setting, clean up default TMPDIR handlingRobert Hensing
This is a squash of upstream PRs #10303, #10312 and #10883. fix: Treat empty TMPDIR as unset Fixes an instance of nix: src/libutil/util.cc:139: nix::Path nix::canonPath(PathView, bool): Assertion `path != ""' failed. ... which I've been getting in one of my shells for some reason. I have yet to find out why TMPDIR was empty, but it's no reason for Nix to break. (cherry picked from commit c3fb2aa1f9d1fa756dac38d3588c836c5a5395dc) fix: Treat empty XDG_RUNTIME_DIR as unset See preceding commit. Not observed in the wild, but is sensible and consistent with TMPDIR behavior. (cherry picked from commit b9e7f5aa2df3f0e223f5c44b8089cbf9b81be691) local-derivation-goal.cc: Reuse defaultTempDir() (cherry picked from commit fd31945742710984de22805ee8d97fbd83c3f8eb) fix: remove usage of XDG_RUNTIME_DIR for TMP (cherry picked from commit 1363f51bcb24ab9948b7b5093490a009947f7453) tests/functional: Add count() (cherry picked from commit 6221770c9de4d28137206bdcd1a67eea12e1e499) Remove uncalled for message (cherry picked from commit b1fe388d33530f0157dcf9f461348b61eda13228) Add build-dir setting (cherry picked from commit 8b16cced18925aa612049d08d5e78eccbf0530e4) Change-Id: Ic7b75ff0b6a3b19e50a4ac8ff2d70f15c683c16a
2024-06-23Merge "libstore: reduce loglevel of `waiting for a machine to build`" into mainMaximilian Bosch
2024-06-23libutil: return Pid from startProcess, not pid_teldritch horrors
Change-Id: Icc8a15090c77f54ea7d9220aadedcd4a19922814
2024-06-23libutil: give Pid proper resource semanticseldritch horrors
copy-constructing or assigning from pid_t can easily lead to duplicate Pid instances for the same process if a pid_t was used carelessly, and Pid itself was copy-constructible. both could cause surprising results such as killing processes twice (which could become very problemantic, but luckily modern systems don't reuse PIDs all that quickly), or more than one piece of the code believing it owns a process when neither do Change-Id: Ifea7445f84200b34c1a1d0acc2cdffe0f01e20c6
2024-06-23libutil: make Pid -> pid_t operations expliciteldritch horrors
Change-Id: I3137cc140590001fe7ba542844e735944a0a9255
2024-06-23libutil: remove sinkToSource eof callbackeldritch horrors
this is only used in one place, and only to set a nicer error message on EndOfFile. the only caller that actually *catches* this exception should provide an error message in that catch block rather than forcing support for setting error message so deep into the stack. copyStorePath is never called outside of PathSubstitutionGoal anyway, which catches everything. Change-Id: Ifbae8706d781c388737706faf4c8a8b7917ca278
2024-06-23libstore: Start creating LocalDerivationGoal subclassesArtemis Tosini
LocalDerivationGoal includes a large number of low-level sandboxing primitives for Darwin and Linux, intermingled with ifdefs. Start creating platform-specific classes to make it easier to add new platforms and review platform-specific code. This change only creates support infrastructure and moves two function, more functions will be moved in future changes. Change-Id: I9fc29fa2a7345107d4fc96c46fa90b4eabf6bb89
2024-06-22libstore: reduce loglevel of `waiting for a machine to build`Maximilian Bosch
This comes quite often when the available job slots on all remote builders are exhausted and this is pretty spammy. This isn't really an issue, but expected behavior. A better way to display this is a nom-like approach where all scheduled builds are shown in a tree and pending builds are being marked as such IMHO. Change-Id: I6bc14e6054f84e3eb0768127b490e263d8cdcf89
2024-06-19libutil: return a source from readFileeldritch horrors
don't consume a sink, return a source instead. the only reason to not do this is a very slight reduction in dynamic allocations, but since we are going to *at least* do disk io that will not be a lot of overhead anyway Change-Id: Iae2f879ec64c3c3ac1d5310eeb6a85e696d4614a
2024-06-18store: fix null reference from DerivationGoal::waiteeDoneJade Lovelace
This happened during a PathSubstitutionGoal of a .drv file: substitution of '/tmp/jade/nix-test/ca/eval-store/store/1lj7lsq5y0f25mfbnq6d3zd0bw5ay33n-dependencies-input-2.drv' What happened here is that since PathSubstitutionGoal is not a DerivationGoal, in production builds, the UB was not caught, since it would early-exit from failing a dynamic_cast to DerivationGoal * on the very next line, but before the null reference was ever used. This was nonetheless UB. The fix should be to just rearrange the two lines; I don't think there is a further bug there, since *substituting a .drv* **necessarily** means you cannot have the representation of the derivation as would be necessary for drv to not be null there. Test failure: ++(eval-store.sh:12) _RR_TRACE_DIR=/home/jade/.local/share/rr rr record -- nix build -f dependencies.nix --eval-store /tmp/jade/nix-test/ca/eval-store/eval-store -o /tmp/jade/nix-test/ca/eval-store/result don't know how to build these paths: /tmp/jade/nix-test/ca/eval-store/store/6y51mf0p57ggipgab6hdjabbvplzsicq-dependencies-top.drv copying 1 paths... copying path '/tmp/jade/nix-test/ca/eval-store/store/8027afyvqb87y1sf5xhdkqsflqn1ziy8-dependencies.builder0.sh' to 'local'... copying 1 paths... copying path '/tmp/jade/nix-test/ca/eval-store/store/7r5pqyncvfgrryf9gzy1z56z3xigi61x-builder-dependencies-input-0.sh' to 'local'... copying 1 paths... copying path '/tmp/jade/nix-test/ca/eval-store/store/nhmgm87zlqy3ks96dxrn7l37b72azi99-builder-dependencies-input-1.sh' to 'local'... copying 1 paths... copying path '/tmp/jade/nix-test/ca/eval-store/store/nq4qa2j6y8ajqazlfq6h46ck637my1n6-builder-dependencies-input-2.sh' to 'local'... copying 1 paths... copying path '/tmp/jade/nix-test/ca/eval-store/store/6vh0vna9l5afck01y7iaks3hm9ikwqyj-builder-fod-input.sh' to 'local'... building '/tmp/jade/nix-test/ca/eval-store/store/gy91pqymf2nc5v7ld1bad94xpwxdi25s-dependencies-input-0.drv'... building '/tmp/jade/nix-test/ca/eval-store/store/w7wlkjx97ivmnrymkac5av3nyp94hzvq-dependencies-input-1.drv'... ../src/libstore/build/derivation-goal.cc:1556:22: runtime error: reference binding to null pointer of type 'Derivation' 0 0x734ba59a6886 in nix::DerivationGoal::waiteeDone(std::shared_ptr<nix::Goal>, nix::Goal::ExitCode) /home/jade/lix/lix2/build/src/libstore/build/derivation-goal.cc:1556:12 1 0x734ba59c0962 in nix::Goal::amDone(nix::Goal::ExitCode, std::optional<nix::Error>) /home/jade/lix/lix2/build/src/libstore/build/goal.cc:95:25 2 0x734ba5a1c44a in nix::PathSubstitutionGoal::done(nix::Goal::ExitCode, nix::BuildResult::Status, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>) /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:38:5 3 0x734ba5a1b454 in nix::PathSubstitutionGoal::init() /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:56:9 4 0x734ba5a2a6c6 in nix::Worker::run(std::set<std::shared_ptr<nix::Goal>, nix::CompareGoalPtrs, std::allocator<std::shared_ptr<nix::Goal>>> const&) /home/jade/lix/lix2/build/src/libstore/build/worker.cc:320:23 5 0x734ba59b93d8 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath>> const&, nix::BuildMode, std::shared_ptr<nix::Store>) /home/jade/lix/lix2/build/src/libstore/build/entry-points.cc:60:12 6 0x734ba663c107 in nix::Installable::build2(nix::ref<nix::Store>, nix::ref<nix::Store>, nix::Realise, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable>>> const&, nix::BuildMode) /home/jade/lix/lix2/build/src/libcmd/installables.cc:637:36 Change-Id: Id0e651e480bebf6356733b01bc639e9bb59c7bd0
2024-05-31libstore/build: copy ca-certificates tooAlois Wohlschlager
In b469c6509ba616da6df8a27e4ccb205a877c66c9, the ca-certificates file was missed. It should be copied too so that we don't end up bind-mounting a broken symlink. Change-Id: Ic9b292d602eb94b0e78f77f2a27a19d24665783c
2024-05-29Remove 100s of CPU time (10%) from build times (1465s -> 1302s)Jade Lovelace
I saw that boost/lexical_cast was costing about 100s in CPU time on our compiles. We can fix this trivially by doing explicit template instantiation in exactly one place and eliminating all other includes of it, which is a code improvement anyway by hiding the boost. Before: ``` lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin Analyzing build trace from 'buildtimeold.bin'... **** Time summary: Compilation (551 times): Parsing (frontend): 1465.3 s Codegen & opts (backend): 1110.9 s <snip> **** Expensive headers: 178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via: 40x: command.hh 5x: command-installable-value.hh 3x: installable-flake.hh 2x: <direct include> 2x: installable-attr-path.hh 176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via: 36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 12x: globals.hh config.hh experimental-features.hh 11x: file-system.hh file-descriptor.hh 6x: serialise.hh strings.hh 6x: <direct include> 6x: archive.hh serialise.hh strings.hh ... 173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via: 55x: <direct include> 39x: command.hh installable-value.hh installables.hh 7x: libexpr.hh 4x: local-store.hh 4x: command-installable-value.hh installable-value.hh installables.hh 3x: binary-cache-store.hh ... 170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via: 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh 14x: store-api.hh nar-info.hh hash.hh 11x: <direct include> 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh 7x: libexpr.hh value.hh source-path.hh archive.hh 6x: fetchers.hh hash.hh ... 169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via: 40x: command.hh installable-value.hh 5x: command-installable-value.hh installable-value.hh 3x: installable-flake.hh installable-value.hh 2x: <direct include> 1x: installable-derived-path.hh 1x: installable-value.hh ... 159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via: 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh 19x: <direct include> 14x: store-api.hh nar-info.hh hash.hh serialise.hh 11x: serialise.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh 7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh ... 156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via: 42x: <direct include> 7x: command-installable-value.hh 2x: installable-attr-path.hh 150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh 11x: file-system.hh 10x: globals.hh 6x: fetchers.hh 6x: serialise.hh strings.hh error.hh 5x: archive.hh ... 133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via : 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh 19x: file-system.hh 11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh ... 132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 16x: gtest.h 11x: file-system.hh types.hh ref.hh 10x: globals.hh types.hh ref.hh 10x: json.hpp 6x: serialise.hh ... done in 0.6s. ``` After: ``` lix/lix2 » maintainers/buildtime_report.sh build Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'... done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it. Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'... **** Time summary: Compilation (551 times): Parsing (frontend): 1302.1 s Codegen & opts (backend): 956.3 s <snip> **** Expensive headers: 178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via: 36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 12x: globals.hh config.hh experimental-features.hh 11x: file-system.hh file-descriptor.hh 6x: <direct include> 6x: serialise.hh strings.hh 6x: fetchers.hh hash.hh serialise.hh strings.hh ... 154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via: 40x: command.hh 5x: command-installable-value.hh 3x: installable-flake.hh 2x: <direct include> 2x: installable-attr-path.hh 153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via: 55x: <direct include> 39x: command.hh installable-value.hh installables.hh 7x: libexpr.hh 4x: local-store.hh 4x: command-installable-value.hh installable-value.hh installables.hh 3x: binary-cache-store.hh ... 149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh 11x: file-system.hh 10x: globals.hh 6x: fetchers.hh 6x: serialise.hh strings.hh error.hh 5x: archive.hh ... 144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via: 40x: command.hh installable-value.hh 5x: command-installable-value.hh installable-value.hh 3x: installable-flake.hh installable-value.hh 2x: <direct include> 1x: installable-value.hh 1x: installable-derived-path.hh ... 136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via: 42x: <direct include> 7x: command-installable-value.hh 2x: installable-attr-path.hh 133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 16x: gtest.h 11x: file-system.hh types.hh ref.hh 10x: globals.hh types.hh ref.hh 10x: json.hpp 6x: serialise.hh ... 89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via: 37x: command.hh installable-value.hh installables.hh 25x: store-api.hh realisation.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh 6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh 6x: libexpr.hh value.hh context.hh 6x: shared.hh ... 87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via: 35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h 12x: regex sstream istream 10x: file-system.hh types.hh ref.hh memory unique_ptr.h 10x: gtest.h memory unique_ptr.h 10x: globals.hh types.hh ref.hh memory unique_ptr.h 6x: fetchers.hh types.hh ref.hh memory unique_ptr.h ... 85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via: 37x: command.hh installable-value.hh installables.hh derived-path.hh 20x: globals.hh 20x: logging.hh 16x: store-api.hh logging.hh 6x: <direct include> 6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh ... done in 0.5s. ``` Change-Id: I27f0a2d566db17832cd9be935f12efe7f95b92d0
2024-05-29util.hh: Delete remaining file and clean up headersTom Hubrecht
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29util.hh: Move nativeSystem to local-derivation-goal.ccTom Hubrecht
Change-Id: I74565fbfd3aeedef8f50465808fac712b84e47ad
2024-05-29util.{hh,cc}: Split out unix-domain-socket.{hh,cc}Tom Hubrecht
Change-Id: I3f9a628e0f8998b6146f5caa8ae9842361a66b8b
2024-05-29util.{hh,cc}: Split out child.{hh,cc}Tom Hubrecht
Change-Id: Iec4824e071f537b17dd62dbb8c01b8eec14e9783
2024-05-29util.{hh,cc}: Split out current-process.{hh,cc}Tom Hubrecht
Change-Id: I77095b9d37e85310075bada7a076ccd482c28e47
2024-05-29util.{hh,cc}: Split out processes.{hh,cc}Tom Hubrecht
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-29util.{hh,cc}: Split out file-system.{hh,cc}Tom Hubrecht
Change-Id: Ifa89a529e7e34e7291eca87d802d2f569cf2493e
2024-05-26worker: process timeouts first, and exclusivelyeldritch horrors
check goals for timeouts first, and their activity fds only if no timeout has occurred. checking for timeouts *after* activity sets us up for assertion failures by running multiple build completion notifiers, the first of which will kill/reap the the goal process and consuming the Pid instance. when the second notifier attempts to do the same it will core dump with an assertion failure in Pid and take down not only the single goal, but the entire daemon and all goals it was building. luckily this is rare in practice since it requires a build to both finish and time out at the same time. writing a test for this is not feasible due to how much it relies on scheduling to actually trigger the underlying bug, but on idle machines it can usually be triggered by running multiple sleeping builds with timeout set to the sleep duration and `--keep-going`: nix-build --timeout 10 --builders '' --keep-going -E ' with import <nixpkgs> {}; builtins.genList (i: runCommand "foo-${toString i}" {} "sleep 10") 100 ' Change-Id: I394d36b2e5ffb909cf8a19977d569bbdb71cb67b
2024-05-25libstore: remove duplicate builder variableArtemis Tosini
The `builder` local variable and duplicate `args.push_back` are no longer required since the Darwin sandbox stopped using `sandbox-exec`. The `drv->isBuiltin` check is not required either, as args are not accessed when the builder is builtin. Change-Id: I80b939bbd6f727b01793809921810ff09b579d54
2024-05-24libstore/build: always enable seccomp filtering and no-new-privilegesAlois Wohlschlager
Seccomp filtering and the no-new-privileges functionality improve the security of the sandbox, and have been enabled by default for a long time. In https://git.lix.systems/lix-project/lix/issues/265 it was decided that they should be enabled unconditionally. Accordingly, remove the allow-new-privileges (which had weird behavior anyway) and filter-syscall settings, and force the security features on. Syscall filtering can still be enabled at build time to support building on architectures libseccomp doesn't support. Change-Id: Iedbfa18d720ae557dee07a24f69b2520f30119cb
2024-05-19libutil: Create chmodPath functionArtemis Tosini
Move the identical static `chmod_` functions in libstore to libutil. the function is called `chmodPath` instead of `chmod` as otherwise it will shadow the standard library chmod in the nix namespace, which is somewhat confusing. Change-Id: I7b5ce379c6c602e3d3a1bbc49dbb70b1ae8f7bad
2024-05-16Allow enabling core dumps from builds for nix & child processesmidnightveil
Fixes https://git.lix.systems/lix-project/lix/issues/268 Change-Id: I3f1b0ddf064f891cca8b53229c5c31c74cea3d9f
2024-05-12Merge "libstore: Fix sandbox=relaxed" into mainArtemis Tosini
2024-05-10libutil: remove callback.hheldritch horrors
it's no longer used. it really shouldn't have existed this long since it was just a mashup of both std::promise and std::packaged_task in a shape that makes composition unnecessarily difficult. all but a single case of Callback pattern calls were fully synchronous anyway, and even this sole outlier was by far not important enough to justify the extra complexity. Change-Id: I208aec4572bf2501cdbd0f331f27d505fca3a62f
2024-05-09libstore: un-callback-ify Store::queryRealisationUncachedeldritch horrors
Change-Id: I4a328f46eaac3bb8b19ddc091306de83348be9cf
2024-05-09libstore: de-callback-ify Store::queryPathInfoUncachedeldritch horrors
Change-Id: I23a156aaff5328f67ca16ccd85c0ea1711b21e35
2024-05-09libstore: de-callback-ify CA realisation substitutioneldritch horrors
this is the *only* real user of file transfer download completion callbacks, and a pretty spurious user at that (seeing how nothing here is even turned on by default and indeed a dependency of path substitution which *isn't* async, and concurrency-limited). it'll be a real pain to keep this around, and realistically it would be a lot better to overhaul substitution in general to be *actually* async. that requires a proper async framework footing though, and we don't have anything of the sort, but it's also blocking *that* Change-Id: I1bf671f217c654a67377087607bf608728cbfc83
2024-05-08libstore: Fix sandbox=relaxedThéophane Hufschmitt
The fix for the Darwin vulnerability in ecdbc3b207eaec1a2cafd2a0d494bcbabdd60a11 also broke setting `__sandboxProfile` when `sandbox=relaxed` or `sandbox=false`. This cppnix change fixes `sandbox=relaxed` and adds a suitable test. Co-Authored-By: Artemis Tosini <lix@artem.ist> Co-Authored-By: Eelco Dolstra <edolstra@gmail.com> Change-Id: I40190f44f3e1d61846df1c7b89677c20a1488522
2024-05-07libstore: limit CA realisation info substitution concurrencyeldritch horrors
this seems to be an oversight, considering that regular substitutions are concurrency-limited. while not particularly necessary at present, once we've removed the `Callback` based interfaces it will be needed. Change-Id: Ide2d08169fcc24752cbd07a1d33fb8482f7034f5
2024-05-04Fix /etc/group having desynced IDs from the actual UID in the sandboxJade Lovelace
This was found when `logrotate.conf` failed to build in a NixOS system with: /nix/store/26zdl4pyw5qazppj8if5lm8bjzxlc07l-coreutils-9.3/bin/id: cannot find name for group ID 30000 This was surprising because it seemed to mean that /etc/group was busted in the sandbox. Indeed it was: root:x:0: nixbld:!:100: nogroup:x:65534: We diagnosed this to sandboxUid() being called before usingUserNamespace() was called, in setting up /etc/group inside the sandbox. This code desperately needs refactoring. We also moved the /etc/group code to be with the /etc/passwd code, but honestly this code is all spaghetti'd all over the place and needs some more serious tidying than we did here. We also moved some checks to be earlier to improve locality with where the things they are checking come from. Change-Id: Ie29798771f3593c46ec313a32960fa955054aceb
2024-05-04Merge "libstore/local-derivation-goal: prohibit creating setuid/setgid ↵Maximilian Bosch
binaries" into main
2024-05-03libstore/local-derivation-goal: prohibit creating setuid/setgid binariesMaximilian Bosch
With Linux kernel >=6.6 & glibc 2.39 a `fchmodat2(2)` is available that isn't filtered away by the libseccomp sandbox. Being able to use this to bypass that restriction has surprising results for some builds such as lxc[1]: > With kernel ≥6.6 and glibc 2.39, lxc's install phase uses fchmodat2, > which slips through https://github.com/NixOS/nix/blob/9b88e5284608116b7db0dbd3d5dd7a33b90d52d7/src/libstore/build/local-derivation-goal.cc#L1650-L1663. > The fixupPhase then uses fchmodat, which fails. > With older kernel or glibc, setting the suid bit fails in the > install phase, which is not treated as fatal, and then the > fixup phase does not try to set it again. Please note that there are still ways to bypass this sandbox[2] and this is mostly a fix for the breaking builds. This change works by creating a syscall filter for the `fchmodat2` syscall (number 452 on most systems). The problem is that glibc 2.39 is needed to have the correct syscall number available via `__NR_fchmodat2` / `__SNR_fchmodat2`, but this flake is still on nixpkgs 23.11. To have this change everywhere and not dependent on the glibc this package is built against, I added a header "fchmodat2-compat.hh" that sets the syscall number based on the architecture. On most platforms its 452 according to glibc with a few exceptions: $ rg --pcre2 'define __NR_fchmodat2 (?!452)' sysdeps/unix/sysv/linux/x86_64/x32/arch-syscall.h 58:#define __NR_fchmodat2 1073742276 sysdeps/unix/sysv/linux/mips/mips64/n32/arch-syscall.h 67:#define __NR_fchmodat2 6452 sysdeps/unix/sysv/linux/mips/mips64/n64/arch-syscall.h 62:#define __NR_fchmodat2 5452 sysdeps/unix/sysv/linux/mips/mips32/arch-syscall.h 70:#define __NR_fchmodat2 4452 sysdeps/unix/sysv/linux/alpha/arch-syscall.h 59:#define __NR_fchmodat2 562 I added a small regression-test to the setuid integration-test that attempts to set the suid bit on a file using the fchmodat2 syscall. I confirmed that the test fails without the change in local-derivation-goal. Additionally, we require libseccomp 2.5.5 or greater now: as it turns out, libseccomp maintains an internal syscall table and validates each rule against it. This means that when using libseccomp 2.5.4 or older, one may pass `452` as syscall number against it, but since it doesn't exist in the internal structure, `libseccomp` will refuse to create a filter for that. This happens with nixpkgs-23.11, i.e. on stable NixOS and when building Lix against the project's flake. To work around that * a backport of libseccomp 2.5.5 on upstream nixpkgs has been scheduled[3]. * the package now uses libseccomp 2.5.5 on its own already. This is to provide a quick fix since the correct fix for 23.11 is still a staging cycle away. We still need the compat header though since `SCMP_SYS(fchmodat2)` internally transforms this into `__SNR_fchmodat2` which points to `__NR_fchmodat2` from glibc 2.39, so it wouldn't build on glibc 2.38. The updated syscall table from libseccomp 2.5.5 is NOT used for that step, but used later, so we need both, our compat header and their syscall table 🤷 Relevant PRs in CppNix: * https://github.com/NixOS/nix/pull/10591 * https://github.com/NixOS/nix/pull/10501 [1] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2031073804 [2] https://github.com/NixOS/nixpkgs/issues/300635#issuecomment-2030844251 [3] https://github.com/NixOS/nixpkgs/pull/306070 (cherry picked from commit ba6804518772e6afb403dd55478365d4b863c854) Change-Id: I6921ab5a363188c6bff617750d00bb517276b7fe
2024-05-03libstore: check additionalSandboxProfileArtemis Tosini
Currently LocalDerivationGoal allows setting `__sandboxProfile` to add sandbox parameters on Darwin when `sandbox=true`. This was only supposed to have an effect when `sandbox=relaxed` Change-Id: Ide44ee82d7e4d6b545285eab26547e7014817d3f
2024-04-27Run all derivation builders inside the sandbox on macOSPuck Meerburg
This replaces the external sandbox-exec call with direct calls into libsandbox. This API is technically deprecated and is missing some prototypes, but all major browsers depend on it, so it is unlikely to materially change without warning. This commit also ensures the netrc file is only written if the derivation is in fact meant to be able to access the internet. This change commits a sin of not actually actively declaring its dependency on macOS's libsandbox.dylib; this is due to the dylib cache in macOS making that explicit dependency unnecessary. In the future this might become a problem, so this commit marks our sins. Co-authored-by: Artemis Tosini <lix@artem.ist> Co-authored-by: Lunaphied <lunaphied@lunaphied.me> Change-Id: Ia302141a53ce7b0327c1aad86a117b6645fe1189
2024-04-16Merge "libstore/build: set NO_NEW_PRIVS for the sandbox" into mainIlya K
2024-04-15fix probable format bug in DerivationGoal::buildDoneQyriad
Either the contents of `line` could cause format errors, or this usage is Technically safe. However, I trust nothing, especially with boost::format. Change-Id: I07933b20bde3b305a6e5d61c2a7bab6ecb042ad9
2024-04-15libstore/build: set NO_NEW_PRIVS for the sandboxK900
Change-Id: I711f64e2b68495ed9c85c1a4bd5025405805e43a
2024-04-13libstore/build: just copy the magic /etc files into the sandboxK900
Saves us a bunch of thinking about how to handle symlinks, and prevents the DNS config from changing on the fly under the build, which may or may not be a good thing? Change-Id: I071e6ae7e220884690b788d94f480866f428db71
2024-04-11Merge pull request #10456 from NixOS/fixpermdeniedbindThéophane Hufschmitt
Fix adding symlink to the sandbox paths (cherry-picked from commit da1e977bf48cff2a635034c85e7c13878e38efc2) Change-Id: I221c85a38180800ec6552d2e86a88df48398fad8
2024-04-11Merge pull request #10362 from obsidiansystems/maybeLstatJohn Ericson
Factor out `nix::maybeLstat` (cherry-picked from commit 9b88e5284608116b7db0dbd3d5dd7a33b90d52d7) Change-Id: Id890525e847c890fad6593c594772826ac4d1d50
2024-03-31Merge changes Ib62d3d68,Ic3e7affe into mainjade
* changes: Make things that can throw not noexcept anymore Fix various clang-tidy lints
2024-03-29Add `pre-commit` checksRebecca Turner
The big ones here are `trim-trailing-whitespace` and `end-of-file-fixer` (which makes sure that every file ends with exactly one newline character). Change-Id: Idca73b640883188f068f9903e013cf0d82aa1123
2024-03-29Make things that can throw not noexcept anymoreJade Lovelace
This does involve making a large number of destructors able to throw, because we had to change it high in the class hierarchy. Oh well. Change-Id: Ib62d3d6895b755f20322bb8acc9bf43daf0174b2
2024-03-27HOT SALE: 15% off your build times!Jade Lovelace
This was achieved by running maintainers/buildtime_report.sh on the build directory of a meson build, then asking "why the heck is json eating our build times", and strategically moving the json using bits out of widely included headers. It turns out that putting literally any metrics whatsoever into the build had immediate and predictable results. Results are 1382.5s frontend time -> 1175.4s frontend time, back end time approximately invariant. Related: https://git.lix.systems/lix-project/lix/issues/159 Change-Id: I7edea95c8536203325c8bb4dae5f32d727a21b2d
2024-03-18libutil: make AutoCloseFD a better resourceeldritch horrors
add a reset() method to close the wrapped fd instead of assigning magic constants. also make the from-fd constructor explicit so you can't accidentally assign the *wrong* magic constant, or even an unrelated integer that also just happens to be an fd by pure chance. Change-Id: I51311b0f6e040240886b5103d39d1794a6acc325
2024-03-17Delete hasPrefix and hasSuffix from the codebaseJade Lovelace
These now have equivalents in the standard lib in C++20. This change was performed with a custom clang-tidy check which I will submit later. Executed like so: ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../tests | tee -a clang-tidy-result Change-Id: I62679e315ff9e7ce72a40b91b79c3e9fc01b27e9
2024-03-11util.hh: split out signals stuffJade Lovelace
Copies part of the changes of ac89bb064aeea85a62b82a6daf0ecca7190a28b7 Change-Id: I9ce601875cd6d4db5eb1132d7835c5bab9f126d8