aboutsummaryrefslogtreecommitdiff
path: root/src/libutil
AgeCommit message (Collapse)Author
2024-05-30Merge "Remove 100s of CPU time (10%) from build times (1465s -> 1302s)" into ↵jade
main
2024-05-30Merge "unix-domain-socket.cc: add comment explaining why ↵jade
bindConnectProcHelper" into main
2024-05-29Remove 100s of CPU time (10%) from build times (1465s -> 1302s)Jade Lovelace
I saw that boost/lexical_cast was costing about 100s in CPU time on our compiles. We can fix this trivially by doing explicit template instantiation in exactly one place and eliminating all other includes of it, which is a code improvement anyway by hiding the boost. Before: ``` lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin Analyzing build trace from 'buildtimeold.bin'... **** Time summary: Compilation (551 times): Parsing (frontend): 1465.3 s Codegen & opts (backend): 1110.9 s <snip> **** Expensive headers: 178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via: 40x: command.hh 5x: command-installable-value.hh 3x: installable-flake.hh 2x: <direct include> 2x: installable-attr-path.hh 176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via: 36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 12x: globals.hh config.hh experimental-features.hh 11x: file-system.hh file-descriptor.hh 6x: serialise.hh strings.hh 6x: <direct include> 6x: archive.hh serialise.hh strings.hh ... 173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via: 55x: <direct include> 39x: command.hh installable-value.hh installables.hh 7x: libexpr.hh 4x: local-store.hh 4x: command-installable-value.hh installable-value.hh installables.hh 3x: binary-cache-store.hh ... 170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via: 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh 14x: store-api.hh nar-info.hh hash.hh 11x: <direct include> 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh 7x: libexpr.hh value.hh source-path.hh archive.hh 6x: fetchers.hh hash.hh ... 169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via: 40x: command.hh installable-value.hh 5x: command-installable-value.hh installable-value.hh 3x: installable-flake.hh installable-value.hh 2x: <direct include> 1x: installable-derived-path.hh 1x: installable-value.hh ... 159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via: 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh 19x: <direct include> 14x: store-api.hh nar-info.hh hash.hh serialise.hh 11x: serialise.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh 7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh ... 156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via: 42x: <direct include> 7x: command-installable-value.hh 2x: installable-attr-path.hh 150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh 11x: file-system.hh 10x: globals.hh 6x: fetchers.hh 6x: serialise.hh strings.hh error.hh 5x: archive.hh ... 133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via : 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh 19x: file-system.hh 11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh ... 132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 16x: gtest.h 11x: file-system.hh types.hh ref.hh 10x: globals.hh types.hh ref.hh 10x: json.hpp 6x: serialise.hh ... done in 0.6s. ``` After: ``` lix/lix2 » maintainers/buildtime_report.sh build Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'... done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it. Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'... **** Time summary: Compilation (551 times): Parsing (frontend): 1302.1 s Codegen & opts (backend): 956.3 s <snip> **** Expensive headers: 178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via: 36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 12x: globals.hh config.hh experimental-features.hh 11x: file-system.hh file-descriptor.hh 6x: <direct include> 6x: serialise.hh strings.hh 6x: fetchers.hh hash.hh serialise.hh strings.hh ... 154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via: 40x: command.hh 5x: command-installable-value.hh 3x: installable-flake.hh 2x: <direct include> 2x: installable-attr-path.hh 153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via: 55x: <direct include> 39x: command.hh installable-value.hh installables.hh 7x: libexpr.hh 4x: local-store.hh 4x: command-installable-value.hh installable-value.hh installables.hh 3x: binary-cache-store.hh ... 149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh 11x: file-system.hh 10x: globals.hh 6x: fetchers.hh 6x: serialise.hh strings.hh error.hh 5x: archive.hh ... 144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via: 40x: command.hh installable-value.hh 5x: command-installable-value.hh installable-value.hh 3x: installable-flake.hh installable-value.hh 2x: <direct include> 1x: installable-value.hh 1x: installable-derived-path.hh ... 136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via: 42x: <direct include> 7x: command-installable-value.hh 2x: installable-attr-path.hh 133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 16x: gtest.h 11x: file-system.hh types.hh ref.hh 10x: globals.hh types.hh ref.hh 10x: json.hpp 6x: serialise.hh ... 89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via: 37x: command.hh installable-value.hh installables.hh 25x: store-api.hh realisation.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh 6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh 6x: libexpr.hh value.hh context.hh 6x: shared.hh ... 87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via: 35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h 12x: regex sstream istream 10x: file-system.hh types.hh ref.hh memory unique_ptr.h 10x: gtest.h memory unique_ptr.h 10x: globals.hh types.hh ref.hh memory unique_ptr.h 6x: fetchers.hh types.hh ref.hh memory unique_ptr.h ... 85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via: 37x: command.hh installable-value.hh installables.hh derived-path.hh 20x: globals.hh 20x: logging.hh 16x: store-api.hh logging.hh 6x: <direct include> 6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh ... done in 0.5s. ``` Change-Id: I27f0a2d566db17832cd9be935f12efe7f95b92d0
2024-05-30libutil/args: warn on unknown settings after parsing all flagsCole Helbling
Upstream change: https://github.com/NixOS/nix/pull/10701 Change-Id: Icf271df57ec529dd8c64667d1ef9f6dbf02d33d3
2024-05-29unix-domain-socket.cc: add comment explaining why bindConnectProcHelperJade Lovelace
We reviewed this code a while ago, and we neglected to get a comment in saying why it's Like This at the time. Let's fix that, since it is code that looks very absurd at first glance. Change-Id: Ib67b49605ef9ef1c84ecda1db16be74fc9105398
2024-05-29util.hh: Delete remaining file and clean up headersTom Hubrecht
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29util.hh: Move nativeSystem to local-derivation-goal.ccTom Hubrecht
Change-Id: I74565fbfd3aeedef8f50465808fac712b84e47ad
2024-05-29util.hh: Move stuff to types.hhTom Hubrecht
Change-Id: Ia852306a4b8aac6856dc42bc69e4b58b53a0d67c
2024-05-29util.cc: Delete remaining fileTom Hubrecht
Change-Id: I2b47848904f2ce7bd78b83738e99a4c9da627751
2024-05-29util.{hh,cc}: Move ignoreException to error.{hh,cc}Tom Hubrecht
Change-Id: Iae6464217a55c313a983e5c651b26a4a1e446706
2024-05-29util.{hh,cc}: Split out namespaces.{hh,cc}Tom Hubrecht
Change-Id: I8fd3f3b50c15ede29d489066b4e8d99c2c4636a6
2024-05-29util.{hh,cc}: Split out users.{hh,cc}Tom Hubrecht
Change-Id: I1bd92479a2cb7e5c2c2e1541b80474adb05ea0df
2024-05-29util.{hh,cc}: Split out strings.{hh,cc}Tom Hubrecht
Change-Id: I4f642d1046d56b5db26f1b0296ee16a0e02d444a
2024-05-29util.{hh,cc}: Split out unix-domain-socket.{hh,cc}Tom Hubrecht
Change-Id: I3f9a628e0f8998b6146f5caa8ae9842361a66b8b
2024-05-29util.{hh,cc}: Split out child.{hh,cc}Tom Hubrecht
Change-Id: Iec4824e071f537b17dd62dbb8c01b8eec14e9783
2024-05-29util.{hh,cc}: Split out current-process.{hh,cc}Tom Hubrecht
Change-Id: I77095b9d37e85310075bada7a076ccd482c28e47
2024-05-29util.{hh,cc}: Split out processes.{hh,cc}Tom Hubrecht
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-29util.{hh,cc}: Split out file-descriptor.{hh,cc}Tom Hubrecht
Change-Id: I0dd0f9a9c2003fb887e076127e7f825fd3289c76
2024-05-29util.{hh,cc}: Split out file-system.{hh,cc}Tom Hubrecht
Change-Id: Ifa89a529e7e34e7291eca87d802d2f569cf2493e
2024-05-29util.{hh,cc}: Split out terminal.{hh,cc}Tom Hubrecht
Change-Id: I9de2296b4012d50f540124001d54d6ca3be4c6da
2024-05-28util.{hh,cc}: Split out environment-variables.{hh,cc}Tom Hubrecht
Change-Id: Icff0aa33fda5147bd5dbe256a0b9d6a6c8a2c3f6
2024-05-23Merge "packaging: rename nixexpr -> lixexpr and so on" into mainjade
2024-05-23packaging: rename nixexpr -> lixexpr and so onJade Lovelace
This breaks downstreams linking to us on purpose to make sure that if someone is linking to Lix they're doing it on purpose and crucially not mixing up Nix and Lix versions in compatibility code. We still need to fix the internal includes to follow the same schema so we can drop the single-level include system entirely. However, this requires a little more effort. This adds pkg-config for libfetchers and config.h. Migration path: expr.hh -> lix/libexpr/expr.hh nix/config.h -> lix/config.h To apply this migration automatically, remove all `<nix/>` from includes, so: `#include <nix/expr.hh>` -> `#include <expr.hh>`. Then, the correct paths will be resolved from the tangled mess, and the clang-tidy automated fix will work. Then run the following for out of tree projects: ``` lix_root=$HOME/lix (cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build') run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -fix src ``` Related: https://git.lix.systems/lix-project/nix-eval-jobs/pulls/5 Fixes: https://git.lix.systems/lix-project/lix/issues/279 Change-Id: I7498e903afa6850a731ef8ce77a70da6b2b46966
2024-05-22libstore: remove unused copyPath functioneldritch horrors
Change-Id: Ibfb0848d8b337bd38947a745c240838865cea401
2024-05-20Merge "libutil: Create chmodPath function" into mainArtemis Tosini
2024-05-19libutil: Create chmodPath functionArtemis Tosini
Move the identical static `chmod_` functions in libstore to libutil. the function is called `chmodPath` instead of `chmod` as otherwise it will shadow the standard library chmod in the nix namespace, which is somewhat confusing. Change-Id: I7b5ce379c6c602e3d3a1bbc49dbb70b1ae8f7bad
2024-05-19fix -Wdeprecated-copy on clang (BaseError copy assignment)Qyriad
2bbe3efd1¹ added the -Wdeprecated-copy warning, and fixed the instances of it which GCC warned about, in HintFmt and ref<T>. However, when building with Clang, there is an additional deprecated-copy warning in BaseError. This commit explicitly defaults the copy assignment operator for BaseError and silences this warning. 1: 2bbe3efd169534f538184ff788eecb398ead70a4 Change-Id: I50aa4a7ab1a7aae5d7b31f765994abd3db06379d
2024-05-17Merge "add clarifying doc-comments on {get,create}NixStateDir()" into mainQyriad
2024-05-13libutil: remove the interrupt-blocking codeAlois Wohlschlager
The interrupt-blocking code was originally introduced 20 years ago so that trying to log an error message does not result in an interrupt exception being thrown and then going unhandled (c8d3882cdc8f9e22c58af285c1996265c1af75d5). However, the logging code does not check for interrupts any more (054be5025762c5e1c7e853c4fa5d7eed8da1727f), so this reasoning is no longer applicable. Delete this code so that later interrupts are unblocked again, for example in the next line entered into the repl. Closes: https://git.lix.systems/lix-project/lix/issues/296 Change-Id: I48253f5f4272e75001148c13046e709ef5427fbd
2024-05-11add clarifying doc-comments on {get,create}NixStateDir()Qyriad
NixStateDir() != NIX_STATE_DIR. These functions should honestly probably be renamed. Change-Id: I00f54b742bba6188bbc7f2410956d956780b99d3
2024-05-10libutil: remove callback.hheldritch horrors
it's no longer used. it really shouldn't have existed this long since it was just a mashup of both std::promise and std::packaged_task in a shape that makes composition unnecessarily difficult. all but a single case of Callback pattern calls were fully synchronous anyway, and even this sole outlier was by far not important enough to justify the extra complexity. Change-Id: I208aec4572bf2501cdbd0f331f27d505fca3a62f
2024-05-09add and fix -Wdeprecated-copyQyriad
*so* many warnings, from only two definitions Change-Id: If2561cd500c05a1e33cce984faf9f3e42a8a95ac
2024-05-07remove the autoconf+Make buildsystemQyriad
We're not using it anymore. Any leftover bugs in the Meson buildsystem are now just bugs. Closes #249. Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-05-07libutil: de-callback-ify computeClosureeldritch horrors
only two users of this function exist. only one used it in a way that even bears resemblance to asynchronicity, and even that one didn't do it right. fully async and parallel computation would have only worked if any getEdgesAsync never calls the continuation it receives itself, only from more derived callbacks running on other threads. calling it directly would cause the decoupling promise to be awaited immediately *on the original thread*, completely negating all nice async effects. Change-Id: I0aa640950cf327533a32dee410105efdabb448df
2024-05-07libutil: throw EndOfFile at sourceToSink endeldritch horrors
... how has this never broken anything before Change-Id: If3789c02028e8f929481514f63d76b0b46bfc182
2024-05-05Actually try making a userns before assuming they don't workJade Lovelace
If unprivileged userns are *believed* to be disabled (such as with "kernel.unprivileged_userns_clone = 0"), Lix would previously *give up* on trying to use a user namespace before actually trying it, even if, in cases such as unprivileged_userns_clone, it would actually be allowed since Nix has CAP_SYS_ADMIN when running as daemon. (see, e.g. https://github.com/archlinux/linux/commit/25d4709a4fc0e4f3b432c24b60dd508fb84f0cb2) We changed it to actually try it first, and then diagnose possible causes, and also to be more loud about the whole thing, using warnings instead of debugs. These warnings will only print on the first build run by the daemon, which is, tbh, eh, shrug. This is what led to us realizing that no-userns was a poorly exercised condition. Change-Id: I8e4f21afc89c574020dc7e89a560cc740ce6573a
2024-05-03libutil: make rewriteStrings soundeldritch horrors
this is used in CA rewriting, replacement of placeholders in derivations, generating scripts for devShells, and some more places. in all of these transitive replacements are unsound, and overlapping replacements would be as well. there even is a test that transitive replacements do not happen (in the CA RewriteSink suite), but none for overlapping replacements. a minimally surprising binary rewriter surely would not do any of these replacements, the only reason we have not seen this break yet is probably that rewriteStrings is only called for store paths and things that look like store paths (and those should never overlap nor admit such transitive replacements) Change-Id: I6fc29f939d5061d9f56c752624a823ece8437c07
2024-05-02getNameFromURL(): Support uppercase characters in attribute namesEelco Dolstra
In particular, this makes it handle 'legacyPackages' correctly. (cherry picked from commit 936a3642264ac159f3f9093710be3465b70e0e89) Upstream-PR: https://github.com/NixOS/nix/pull/9657 Change-Id: Icc4efe02f7f8e90a2970589f72fd3d3cd4418d95
2024-04-30implement parsing human-readable names from URLsQyriad
Based off of commit 257b768436a0e8ab7887f9b790c5b92a7fe51ef5 Upstream-PR: https://github.com/NixOS/nix/pull/8678 Co-authored-by: Felix Uhl <felix.uhl@outlook.com> Change-Id: Idcb7f6191ca3310ef9dc854197f7798260c3f71d
2024-04-25gc: Find roots using libproc on DarwinArtemis Tosini
Previously, the garbage collector found runtime roots on Darwin by shelling out to `lsof -n -w -F n` then parsing the result. However, this requires an lsof binary and can be extremely slow. The official Apple lsof returns in a reasonable amount of time, about 250ms in my tests, but the lsof packaged in nixpkgs is quite slow, taking about 40 seconds to run the command. Using libproc directly is about the same speed as Apple lsof, and allows us to reënable several tests that were disabled on Darwin. Change-Id: Ifa0adda7984e13c15535693baba835aae79a3577
2024-04-15abort with a descriptive message on bad HintFmt usageQyriad
Change-Id: Ic2f05572042343a8160fd971394372f5f2706fc4
2024-04-13libstore/build: just copy the magic /etc files into the sandboxK900
Saves us a bunch of thinking about how to handle symlinks, and prevents the DNS config from changing on the fly under the build, which may or may not be a good thing? Change-Id: I071e6ae7e220884690b788d94f480866f428db71
2024-04-11Merge changes Iedf46484,I76b51eac,I6a084827,I60193f9f into maineldritch horrors
* changes: meson: fix log-dir manual: build docs with dummy envs libcmd: install generated headers as well docs: redo content generation for mdbook and manual
2024-04-11docs: redo content generation for mdbook and manualeldritch horrors
manpages can be rendered using the markdown output of mdbook, the rest of the manual can generated out of the main doc/manual source tree. we still use lowdown to actually render manpages instead of eg mdbook-man because lowdown does generate reasonably good manpages (though that is also somewhat debatable, but they're a lot better than mdbook-man). doing this not only lets us drastically simplify the lowdown pipeline, but also remove all custom {{#include}} handling since now mdbook does all of it, even for the manpage builds. even the lowdown wrapper isn't entirely necessary because lowdown can take all wrapper arguments with command line flags rather than bits of input file content. This also implements running mdbook in Meson, in order to generate the manpages. The mdbook outputs are also installed in the usual location. Co-authored-by: Qyriad <qyriad@qyriad.me> Change-Id: I60193f9fd0f15d48872f071af35855cda2a0f40b
2024-04-11Merge pull request #10362 from obsidiansystems/maybeLstatJohn Ericson
Factor out `nix::maybeLstat` (cherry-picked from commit 9b88e5284608116b7db0dbd3d5dd7a33b90d52d7) Change-Id: Id890525e847c890fad6593c594772826ac4d1d50
2024-04-08pragma once and ///@file everything missing itJade Lovelace
Change-Id: Ia1a72348336794b5fb9f2694dd750266089b904e
2024-04-07Add `PathsSetting`Rebecca Turner
Change-Id: I1165f6ef033a5f757ca3716d3f8008ba36b01fd0
2024-04-05Revert "libutil: drop Pool resources on exceptional free"eldritch horrors
This reverts commit de2884b82b376d10de5c400d8e73bc7d98f195d2. Change-Id: I1fa301149d7c2ed3d266a40c15b2d010e12e44e6
2024-04-05Revert "libutil: remove Pool::Handle::bad"eldritch horrors
This reverts commit 792844fb861ea7367ac2316c78fec055363f2f9e. Change-Id: I3ca208b62edfd5cd1199478f75cd2edf19a364f6
2024-04-05Revert "libutil: allow graceful dropping of Pool::Handle"eldritch horrors
This reverts commit 8075541d82d05347321d35b9934ccee5f82142f4. Change-Id: I05fa6a9de1308a4827a6557cf2807eb47ca64da6
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-23 02:07:29 +0000