aboutsummaryrefslogtreecommitdiff
path: root/src/libstore/machines.cc
AgeCommit message (Collapse)Author
2024-10-14ssh-ng: better way to keep SSH errors visibleMaximilian Bosch
A better fix than in 104448e75d87d03d2fb0b4ac96d4da72e1dae50d, hence a revert + the fix. It turns out that this commit has the side-effect that when having e.g. `StrictHostKeyChecking=accept-new` for a remote builder, the warnings à la Warning: Permanently added 'builder' (ED25519) to the list of known hosts. actually end up in the derivation's log whereas hostkey verification errors don't, but only in the stderr of the `nix-build` invocation (which was the motivation for the patch). This change writes the stderr from the build-hook to * the daemon's stderr, so that the SSH errors appear in the journal (which was the case before 104448e75d87d03d2fb0b4ac96d4da72e1dae50d) * the client's stderr, as a log message * NOT to the drv log (this is handled via `handleJSONLogMessage`) I tried to fix the issue for legacy-ssh as well, but failed and ultimately decided to not bother. I know that we'll sooner or later replace the entire component, however this is the part of the patch I have working for a while, so I figured I might still submit it for the time being. Change-Id: I21ca1aa0d8ae281d2eacddf26e0aa825272707e5
2024-08-04tree-wide: fix various lint warningsJade Lovelace
Change-Id: I0fc80718eb7e02d84cc4b5d5deec4c0f41116134
2024-05-29Remove 100s of CPU time (10%) from build times (1465s -> 1302s)Jade Lovelace
I saw that boost/lexical_cast was costing about 100s in CPU time on our compiles. We can fix this trivially by doing explicit template instantiation in exactly one place and eliminating all other includes of it, which is a code improvement anyway by hiding the boost. Before: ``` lix/lix2 » ClangBuildAnalyzer --analyze buildtimeold.bin Analyzing build trace from 'buildtimeold.bin'... **** Time summary: Compilation (551 times): Parsing (frontend): 1465.3 s Codegen & opts (backend): 1110.9 s <snip> **** Expensive headers: 178153 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 3426 ms), included via: 40x: command.hh 5x: command-installable-value.hh 3x: installable-flake.hh 2x: <direct include> 2x: installable-attr-path.hh 176217 ms: ../src/libutil/error.hh (included 246 times, avg 716 ms), included via: 36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 12x: globals.hh config.hh experimental-features.hh 11x: file-system.hh file-descriptor.hh 6x: serialise.hh strings.hh 6x: <direct include> 6x: archive.hh serialise.hh strings.hh ... 173243 ms: ../src/libstore/store-api.hh (included 152 times, avg 1139 ms), included via: 55x: <direct include> 39x: command.hh installable-value.hh installables.hh 7x: libexpr.hh 4x: local-store.hh 4x: command-installable-value.hh installable-value.hh installables.hh 3x: binary-cache-store.hh ... 170482 ms: ../src/libutil/serialise.hh (included 201 times, avg 848 ms), included via: 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh 14x: store-api.hh nar-info.hh hash.hh 11x: <direct include> 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh 7x: libexpr.hh value.hh source-path.hh archive.hh 6x: fetchers.hh hash.hh ... 169397 ms: ../src/libcmd/installables.hh (included 53 times, avg 3196 ms), included via: 40x: command.hh installable-value.hh 5x: command-installable-value.hh installable-value.hh 3x: installable-flake.hh installable-value.hh 2x: <direct include> 1x: installable-derived-path.hh 1x: installable-value.hh ... 159740 ms: ../src/libutil/strings.hh (included 221 times, avg 722 ms), included via: 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh 19x: <direct include> 14x: store-api.hh nar-info.hh hash.hh serialise.hh 11x: serialise.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh 7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh ... 156796 ms: ../src/libcmd/command.hh (included 51 times, avg 3074 ms), included via: 42x: <direct include> 7x: command-installable-value.hh 2x: installable-attr-path.hh 150392 ms: ../src/libutil/types.hh (included 251 times, avg 599 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh 11x: file-system.hh 10x: globals.hh 6x: fetchers.hh 6x: serialise.hh strings.hh error.hh 5x: archive.hh ... 133101 ms: /nix/store/644b90j1vms44nr18yw3520pzkrg4dd1-boost-1.81.0-dev/include/boost/lexical_cast.hpp (included 226 times, avg 588 ms), included via : 37x: command.hh installable-value.hh installables.hh built-path.hh realisation.hh hash.hh serialise.hh strings.hh 19x: file-system.hh 11x: store-api.hh nar-info.hh hash.hh serialise.hh strings.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 7x: libexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh 6x: eval.hh attr-set.hh nixexpr.hh value.hh source-path.hh archive.hh serialise.hh strings.hh ... 132887 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 507 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 16x: gtest.h 11x: file-system.hh types.hh ref.hh 10x: globals.hh types.hh ref.hh 10x: json.hpp 6x: serialise.hh ... done in 0.6s. ``` After: ``` lix/lix2 » maintainers/buildtime_report.sh build Processing all files and saving to '/home/jade/lix/lix2/maintainers/../buildtime.bin'... done in 0.6s. Run 'ClangBuildAnalyzer --analyze /home/jade/lix/lix2/maintainers/../buildtime.bin' to analyze it. Analyzing build trace from '/home/jade/lix/lix2/maintainers/../buildtime.bin'... **** Time summary: Compilation (551 times): Parsing (frontend): 1302.1 s Codegen & opts (backend): 956.3 s <snip> **** Expensive headers: 178145 ms: ../src/libutil/error.hh (included 246 times, avg 724 ms), included via: 36x: command.hh installable-value.hh installables.hh derived-path.hh config.hh experimental-features.hh 12x: globals.hh config.hh experimental-features.hh 11x: file-system.hh file-descriptor.hh 6x: <direct include> 6x: serialise.hh strings.hh 6x: fetchers.hh hash.hh serialise.hh strings.hh ... 154043 ms: ../src/libcmd/installable-value.hh (included 52 times, avg 2962 ms), included via: 40x: command.hh 5x: command-installable-value.hh 3x: installable-flake.hh 2x: <direct include> 2x: installable-attr-path.hh 153593 ms: ../src/libstore/store-api.hh (included 152 times, avg 1010 ms), included via: 55x: <direct include> 39x: command.hh installable-value.hh installables.hh 7x: libexpr.hh 4x: local-store.hh 4x: command-installable-value.hh installable-value.hh installables.hh 3x: binary-cache-store.hh ... 149948 ms: ../src/libutil/types.hh (included 251 times, avg 597 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh 11x: file-system.hh 10x: globals.hh 6x: fetchers.hh 6x: serialise.hh strings.hh error.hh 5x: archive.hh ... 144560 ms: ../src/libcmd/installables.hh (included 53 times, avg 2727 ms), included via: 40x: command.hh installable-value.hh 5x: command-installable-value.hh installable-value.hh 3x: installable-flake.hh installable-value.hh 2x: <direct include> 1x: installable-value.hh 1x: installable-derived-path.hh ... 136585 ms: ../src/libcmd/command.hh (included 51 times, avg 2678 ms), included via: 42x: <direct include> 7x: command-installable-value.hh 2x: installable-attr-path.hh 133394 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/memory (included 262 times, avg 509 ms), included via: 36x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh 16x: gtest.h 11x: file-system.hh types.hh ref.hh 10x: globals.hh types.hh ref.hh 10x: json.hpp 6x: serialise.hh ... 89315 ms: ../src/libstore/derived-path.hh (included 178 times, avg 501 ms), included via: 37x: command.hh installable-value.hh installables.hh 25x: store-api.hh realisation.hh 7x: primops.hh eval.hh attr-set.hh nixexpr.hh value.hh context.hh 6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh 6x: libexpr.hh value.hh context.hh 6x: shared.hh ... 87347 ms: /nix/store/h2abv2l8irqj942i5rq9wbrj42kbsh5y-gcc-12.3.0/include/c++/12.3.0/ostream (included 273 times, avg 319 ms), included via: 35x: command.hh installable-value.hh installables.hh path.hh types.hh ref.hh memory unique_ptr.h 12x: regex sstream istream 10x: file-system.hh types.hh ref.hh memory unique_ptr.h 10x: gtest.h memory unique_ptr.h 10x: globals.hh types.hh ref.hh memory unique_ptr.h 6x: fetchers.hh types.hh ref.hh memory unique_ptr.h ... 85249 ms: ../src/libutil/config.hh (included 213 times, avg 400 ms), included via: 37x: command.hh installable-value.hh installables.hh derived-path.hh 20x: globals.hh 20x: logging.hh 16x: store-api.hh logging.hh 6x: <direct include> 6x: eval.hh attr-set.hh nixexpr.hh value.hh context.hh derived-path.hh ... done in 0.5s. ``` Change-Id: I27f0a2d566db17832cd9be935f12efe7f95b92d0
2024-05-29util.{hh,cc}: Split out strings.{hh,cc}Tom Hubrecht
Change-Id: I4f642d1046d56b5db26f1b0296ee16a0e02d444a
2024-04-26ssh-ng: Set log-fd for ssh to `4` by defaultMaximilian Bosch
That's expected by `build-remote` and makes sure that errors are correctly forwarded to the user. For instance, let's say that the host-key of `example.org` is unknown and nix-build ../nixpkgs -A hello -j0 --builders 'ssh-ng://example.org' is issued, then you get the following output: cannot build on 'ssh-ng://example.org?&': error: failed to start SSH connection to 'example.org' Failed to find a machine for remote build! derivation: yh46gakxq3kchrbihwxvpn5bmadcw90b-hello-2.12.1.drv required (system, features): (x86_64-linux, []) 2 available machines: [...] The relevant information (`Host key verification failed`) ends up in the daemon's log, but that's not very obvious considering that the daemon isn't very chatty normally. This can be fixed - the same way as its done for legacy-ssh - by passing fd 4 to the SSH wrapper. Now you'd get the following error: cannot build on 'ssh-ng://example.org': error: failed to start SSH connection to 'example.org': Host key verification failed. Failed to find a machine for remote build! [...] ...and now it's clear what's wrong. Please note that this is won't end up in the derivation's log. For previous discussion about this change see https://github.com/NixOS/nix/pull/7659. Change-Id: I5790856dbf58e53ea3e63238b015ea06c347cf92
2024-03-17Delete hasPrefix and hasSuffix from the codebaseJade Lovelace
These now have equivalents in the standard lib in C++20. This change was performed with a custom clang-tidy check which I will submit later. Executed like so: ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../tests | tee -a clang-tidy-result Change-Id: I62679e315ff9e7ce72a40b91b79c3e9fc01b27e9
2024-03-04Merge pull request #9841 from obsidiansystems/float-speed-factoreldritch horrors
Convert `Machine::speedFactor` from a non-neg int to a non-neg float (cherry picked from commit 69d0ae27e376e7c7c4f237716b0149223b8a805a) Change-Id: I2afb5cf9e4fe1384985c58353946135c3d102b42
2024-03-04Merge pull request #9838 from obsidiansystems/systemTypes-seteldritch horrors
Make `Machine::systemTypes` a set not vector (cherry picked from commit f1b030415376e81c5804647c055d71eaba4aa725) Change-Id: I6d4f5c0bfc226e9bd66c58c360cd99e3fac9a129
2022-02-25Remove std::string alias (for real this time)Eelco Dolstra
Also use std::string_view in a few more places.
2021-10-29StyleEelco Dolstra
2021-10-17Add error reporting to machine spec paserAlexey Novikov
Currently machine specification (`/etc/nix/machine`) parser fails with a vague exception if the file had incorrect format. This commit adds verbose exceptions and unit-tests for the parser.
2021-06-23Improve machine store URI parsingThomas Churchman
2021-02-25distributed builds: load remote builder host key from the machines fileGraham Christensen
This is already used by Hydra, and is very useful when materializing a remote builder list from service discovery. This allows the service discovery tool to only sync one file instead of two.
2020-08-12Make `system-features` a store settingJohn Ericson
This seems more correct. It also means one can specify the features a store should support with --store and remote-store=..., which is useful. I use this to clean up the build remotes test.
2019-04-01getMachines(): Cache resultEelco Dolstra
2017-10-24Allow shorter syntax for chroot storesEelco Dolstra
You can now say '--store /tmp/nix' instead of '--store local?root=/tmp/nix'.
2017-10-24Remove the builder-files optionEelco Dolstra
You can now include files via the "builders" option, using the syntax "@<filename>". Having only one option makes it easier to override builders completely. For backward compatibility, the default is "@/etc/nix/machines", or "@<filename>" for each file name in NIX_REMOTE_SYSTEMS.
2017-07-30Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-05-05Make the location of the build directory in the sandbox configurableEelco Dolstra
This is mostly for use in the sandbox tests, since if the Nix store is under /build, then we can't use /build as the build directory.
2017-05-02Replace $NIX_REMOTE_SYSTEMS with an option "builder-files"Eelco Dolstra
Also, to unify with hydra-queue-runner, allow it to be a list of files.
2017-05-02Add an option for specifying remote buildersEelco Dolstra
This is useful for one-off situations where you want to specify a builder on the command line instead of having to mess with nix.machines. E.g. $ nix-build -A hello --argstr system x86_64-darwin \ --option builders 'root@macstadium1 x86_64-darwin' will perform the specified build on "macstadium1". It also removes the need for a separate nix.machines file since you can specify builders in nix.conf directly. (In fact nix.machines is yet another hack that predates the general nix.conf configuration file, IIRC.) Note: this option is supported by the daemon for trusted users. The fact that this allows trusted users to specify paths to SSH keys to which they don't normally have access is maybe a bit too much trust...
2017-05-02Factor out machines.conf parsingEelco Dolstra
This allows hydra-queue-runner to use it.
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-22 05:52:22 +0000