aboutsummaryrefslogtreecommitdiff
path: root/src/libfetchers
AgeCommit message (Collapse)Author
2024-09-02libfetchers: serialise accept-flake-config properlyAlois Wohlschlager
The AcceptFlakeConfig type used was missing its JSON serialisation definition, so it was incorrectly serialised as an integer, ending up that way for example in the nix.conf manual page. Declare a proper serialisation. Change-Id: If8ec210f9d4dd42fe480c4e97d0a4920eb66a01e
2024-09-01Merge changes I5566a985,I88cf53d3 into mainrebecca “wiggles” turner
* changes: Support relative and `~/` paths in config settings Thread `ApplyConfigOptions` through config parsing
2024-08-28tree-wide: shuffle headers around for about 30s compile timeJade Lovelace
This didn't really feel so worth it afterwards, but I did untangle a bunch of stuff that should not have been tangled. The general gist of this change is that variant bullshit was causing a bunch of compile time, and it seems like the only way to deal with variant induced compile time is to keep variant types out of headers. Explicit template instantiation seems to do nothing for them. I also seem to have gotten some back-end time improvement from explicitly instantiating regex, but I don't know why. There is no corresponding front-end time improvement from it: regex is still at the top of the sinners list. **** Templates that took longest to instantiate: 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms) 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms) 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms) 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms) 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms) 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms) 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms) 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms) Back-end time before messing with the regex: **** Function sets that took longest to compile / optimize: 8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms) 4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms) 3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms) 2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms) 2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms) 2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms) 2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms) 2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms) 1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms) 1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms) 1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms) 1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms) 1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms) 1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms) 1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms) 1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms) 1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms) 1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms) 1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms) 1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms) 1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms) 1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms) 1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms) After messing with the regex (notice std::__detail::_Compiler vanishes here, but I don't know why): **** Function sets that took longest to compile / optimize: 8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms) 4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms) 3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms) 3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms) 2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms) 1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms) 1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms) 1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms) 1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms) 1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms) 1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms) 1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms) 1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms) 1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms) 1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms) 1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms) 1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms) 1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms) 1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms) 1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms) 1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms) 1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms) 1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms) 1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms) 1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms) 1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms) 1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms) 1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms) 1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms) 1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms) Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
2024-08-25Thread `ApplyConfigOptions` through config parsingRebecca Turner
This makes no changes to logic but makes the `ApplyConfigOptions` value available to consumers. Change-Id: I88cf53d38faac8472c556aee55c13d0acbd1e5db
2024-08-08refactor: make HashType and Base enum classes for type safetyJade Lovelace
Change-Id: I9fbd55a9d50464a56fe11cb42a06a206914150d8
2024-08-01Reapply "libfetchers: make attribute / URL query handling consistent"Maximilian Bosch
The original attempt at this introduced a regression; this commit reverts the revert and fixes the regression. This reverts commit 3e151d4d77b5296b9da8c3ad209932d1dfa44c68. Fix to the regression: flakeref: fix handling of `?dir=` param for flakes in subdirs As reported in #419[1], accessing a flake in a subdir of a Git repository fails with the previous commit[2] applied with the error error: unsupported Git input attribute 'dir' The problem is that the `dir`-param is inserted into the parsed URL if a flake is fetched from the subdir of a Git repository. However, for the fetching part this isn't even needed. The fix is to just pass `subdir` as second argument to `FlakeRef` (which needs a `basedir` that can be empty) and leave the parsedURL as-is. Added a regression test to make sure we don't run into this again. [1] https://git.lix.systems/lix-project/lix/issues/419 [2] e22172aaf6b6a366cecd3c025590e68fa2b91bcc, originally 3e151d4d77b5296b9da8c3ad209932d1dfa44c68 Change-Id: I2c72d5a32e406a7ca308e271730bd0af01c5d18b
2024-07-24fix building with Musl, fixing static buildsQyriad
Musl stdout macro expands¹ to something that isn't a valid identifier, so we get syntax errors when compiling usage of a method called stdout with Musl's stdio.h. [1]: https://git.musl-libc.org/cgit/musl/tree/include/stdio.h?id=ab31e9d6a0fa7c5c408856c89df2dfb12c344039#n67 Change-Id: I10e6f6a49504399bf8edd59c5d9e4e62449469e8
2024-07-06libutil: return sources from runProgram2eldritch horrors
this much more closely mimics what is actually happening: we're reading data from somewhere else, actively, rather than passively waiting. with the data flow matching the underlying system interactions better we can remove a few sinkToSource calls that merely exists to undo the mismatch caused by not treating subprocess output as a data source to begin with Change-Id: If4abfc2f8398fb5e88c9b91a8bdefd5504bb2d11
2024-07-05libutil: return a program handle from runProgram2eldritch horrors
this will let us also return a source for the program output later, which will in turn make sinkToSource unnecessary for program output processing. this may also reopen a path for provigin program input, but that still needs a proper async io framework to avoid problems. Change-Id: Iaf93f47db99c38cfaf134bd60ed6a804d7ddf688
2024-07-05libstore: convert dumpPath to a generatoreldritch horrors
Change-Id: Ic4cf5562504aa29130304469936f958c0426e5ef
2024-07-01tree-wide: unify progress bar inactive and paused statesAlois Wohlschlager
Previously, the progress bar had two subtly different states in which the bar would not actually render, both with their own shortcomings: inactive (which was irreversible) and paused (reversible, but swallowing logs). Furthermore, there was no way of resetting the statistics, so a very bad solution was implemented (243c0f18dae2a08ea0e46f7ff33277c63f7506d7) that would create a new logger for each line of the repl, leaking the previous one and discarding the value of printBuildLogs. Finally, if stderr was not attached to a TTY, the update thread was started even though the logger was not active, violating the invariant required by the destructor (which is not observed because the logger is leaked). In this commit, the two aforementioned states are unified into a single one, which can be exited again, correctly upholds the invariant that the update thread is only running while the progress bar is active, and does not swallow logs. The latter change in behavior is not expected to be a problems in the rare cases where the paused state was used before, since other loggers (like the simple one) don't exhibit it anyway. The startProgressBar/stopProgressBar API is removed due to being a footgun, and a new method for properly resetting the progress is added. Co-Authored-By: Qyriad <qyriad@qyriad.me> Change-Id: I2b7c3eb17d439cd0c16f7b896cfb61239ac7ff3a
2024-06-30libexpr/flake: allow automatic rejection of configuration options from flakesAlois Wohlschlager
The `allow-flake-configuration` option allows the user to control whether to accept configuration options supplied by flakes. Unfortunately, setting this to false really meant "ask each time" (with an option to remember the choice for each specific option encountered). Let no mean no, and introduce (and default to) a separate value for the "ask each time" behaviour. Co-Authored-By: Jade Lovelace <lix@jade.fyi> Change-Id: I7ccd67a95bfc92cffc1ebdc972d243f5191cc1b4
2024-06-25Merge "Revert "libfetchers: make attribute / URL query handling consistent"" ↵jade
into main
2024-06-24Revert "libfetchers: make attribute / URL query handling consistent"jade
This reverts commit 35eec921af1043fc6322edc0ad88c872d41623b8. Reason for revert: Regressed nix-eval-jobs, and it appears to be this change is buggy/missing a case. It just needs another pass. Code causing the problem in n-e-j, when invoked with `nix-eval-jobs --flake '.#hydraJobs'`: ``` n-e-j/tests/assets » ../../build/src/nix-eval-jobs --meta --workers 1 --flake .#hydraJobs warning: unknown setting 'trusted-users' warning: `--gc-roots-dir' not specified error: unsupported Git input attribute 'dir' error: worker error: error: unsupported Git input attribute 'dir' ``` ``` nix::Value *vRoot = [&]() { if (args.flake) { auto [flakeRef, fragment, outputSpec] = nix::parseFlakeRefWithFragmentAndExtendedOutputsSpec( args.releaseExpr, nix::absPath(".")); nix::InstallableFlake flake{ {}, state, std::move(flakeRef), fragment, outputSpec, {}, {}, args.lockFlags}; return flake.toValue(*state).first; } else { return releaseExprTopLevelValue(*state, autoArgs, args); } }(); ``` Inspecting the program behaviour reveals that `dir` was in fact set in the URL going into the fetcher. This is in turn because unlike in the case changed in this commit, it was not erased before handing it to libfetchers, which is probably just a mistake. ``` (rr) up 3 0x00007ffff60262ae in nix::fetchers::Input::fromURL (url=..., requireTree=requireTree@entry=true) at src/libfetchers/fetchers.cc:39 warning: Source file is more recent than executable. 39 auto res = inputScheme->inputFromURL(url, requireTree); (rr) p url $1 = (const nix::ParsedURL &) @0x7fffdc874190: {url = "git+file:///home/jade/lix/nix-eval-jobs", base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", authority = std::optional<std::string> = {[contained value] = ""}, path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = {["dir"] = "tests/assets"}, fragment = ""} (rr) up 4 0x00007ffff789d904 in nix::parseFlakeRefWithFragment (url=".#hydraJobs", baseDir=std::optional<std::string> = {...}, allowMissing=allowMissing@entry=false, isFlake=isFlake@entry=true) at src/libexpr/flake/flakeref.cc:179 warning: Source file is more recent than executable. 179 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")), (rr) p parsedURL $2 = {url = "git+file:///home/jade/lix/nix-eval-jobs", base = "git+file:///home/jade/lix/nix-eval-jobs", scheme = "git+file", authority = std::optional<std::string> = {[contained value] = ""}, path = "/home/jade/lix/nix-eval-jobs", query = std::map with 1 element = { ["dir"] = "tests/assets"}, fragment = ""} (rr) list 174 175 if (pathExists(flakeRoot + "/.git/shallow")) 176 parsedURL.query.insert_or_assign("shallow", "1"); 177 178 return std::make_pair( 179 FlakeRef(Input::fromURL(parsedURL, isFlake), getOr(parsedURL.query, "dir", "")), 180 fragment); 181 } ``` Change-Id: Ib55a882eaeb3e59228857761dc1e3b2e366b0f5e
2024-06-23libutil: remove runProgram2 stdin functionalityeldritch horrors
this was only used in one place, and that place has been rewritten to use a temporary file instead. keeping this around is not very helpful at this time, and in any case we'd be better off rewriting subprocess handling in rust where we not only have a much safer library for such things but also async frameworks necessary for this easily available. Change-Id: I6f8641b756857c84ae2602cdf41f74ee7a1fda02
2024-06-23libfetchers: write git commit message to tempfileeldritch horrors
we want to remove runProgram's ability to provide stdin to a process because the concurrency issues of handling both stdin and stdout are much more pronounced once runProgram returns not is collected output but a source. this is possible in the current c++ framework, however it isn't necessary in almost all cases (as demonstrated by only this single user existing) and in much better handled with a proper async concurrency model that lets the caller handle both at the same time. Change-Id: I29da1e1ad898d45e2e33a7320b246d5003e7712b
2024-06-22libfetchers: make attribute / URL query handling consistentMaximilian Bosch
The original idea was to fix lix#174, but for a user friendly solution, I figured that we'd need more consistency: * Invalid query params will cause an error, just like invalid attributes. This has the following two consequences: * The `?dir=`-param from flakes will be removed before the URL to be fetched is passed to libfetchers. * The tarball fetcher doesn't allow URLs with custom query params anymore. I think this was questionable anyways given that an arbitrary set of query params was silently removed from the URL you wanted to fetch. The correct way is to use an attribute-set with a key `url` that contains the tarball URL to fetch. * Same for the git & mercurial fetchers: in that case it doesn't even matter though: both fetchers added unused query params to the URL that's passed from the input scheme to the fetcher (`url2` in the code). It turns out that this was never used since the query parameters were erased again in `getActualUrl`. * Validation happens for both attributes and URLs. Previously, a lot of fetchers validated e.g. refs/revs only when specified in a URL and the validity of attribute names only in `inputFromAttrs`. Now, all the validation is done in `inputFromAttrs` and `inputFromURL` constructs attributes that will be passed to `inputFromAttrs`. * Accept all attributes as URL query parameters. That also includes lesser used ones such as `narHash`. And "output" attributes like `lastModified`: these could be declared already when declaring inputs as attribute rather than URL. Now the behavior is at least consistent. Personally, I think we should differentiate in the future between "fetched input" (basically the attr-set that ends up in the lock-file) and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints are probably OK for unfetched inputs, but for locked/fetched inputs a custom entrypoint should be used. Then, the current entrypoints wouldn't have to allow these attributes anymore. Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
2024-06-19Merge "libfetchers: represent unfetched submodules consistently" into mainalois31
2024-06-18filetransfer: {up,down}load -> transfereldritch horrors
even the transfer function is not all that necessary since there aren't that many users, but we'll keep it for now. we could've kept both names but we also kind of want to use `download` for something else very soon Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
2024-06-18libfetchers: represent unfetched submodules consistentlyAlois Wohlschlager
Unfetched submodules are included as empty directories in archives, so they end up as such in the store when fetched in clean mode. Make sure the same happens in dirty mode too. Fortunately, they are already correctly represented in the ls-files output, so we just need to make sure to include the empty directory in our filter. Fixes: https://github.com/NixOS/nix/issues/6247 Change-Id: I60d06ff360cfa305d081b920838c893c06da801c
2024-06-12doc: rewrite the multi-user documentation to actually talk about securityJade Lovelace
It's in the security section, and it was totally outdated anyway. I took the opportunity to write down the stuff we already believed. Change-Id: I73e62ae85a82dad13ef846e31f377c3efce13cb0
2024-06-06build: expose option to enable or disable precompiled std headersQyriad
They are enabled by default, and Meson will also prints whether or not they're enabled at the bottom at the end of configuration. Change-Id: I48db238510bf9e74340b86f243f4bbe360794281
2024-06-01Merge "build: fix static linking with a hack" into mainQyriad
2024-06-01Merge "libfetchers: allow fetching gitlab refs with >1 commit" into mainLinus Heckemann
2024-05-31build: fix static linking with a hackQyriad
This causes libstore, libexpr, libfetchers, and libutil to be linked with -Wl,--whole-archive to executables, when building statically. libstore for the store backends, libexpr for the primops, libfetchers for the fetcher backends I assume(?), and libutil for the nix::logger initializer (which notably shows in pre-main constructors when HOME is not owned by the user. cursed.). This workaround should be removed when #359 is fixed. Fixes #306. Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into mainjade
* changes: gitignore: delete 90% of it build-time: remove 20% more by PCH'ing C++ stdlib shellHook: make it actually run
2024-05-31libfetchers: allow fetching gitlab refs with >1 commitLinus Heckemann
Change-Id: I945c4c5512def9eff728bb67fe3c03ae17f99d6d
2024-05-30build-time: remove 20% more by PCH'ing C++ stdlibJade Lovelace
It seems like someone implemented precompiled headers a long time ago and then it never got ported to meson or maybe didn't work at all. This is, however, blessedly easy to simply implement. I went looking for `#define` that could affect the result of precompiling the headers, and as far as I can tell we aren't doing any of that, so this should truly just be free build time savings. Previous state: Compilation (551 times): Parsing (frontend): 1302.1 s Codegen & opts (backend): 956.3 s New state: **** Time summary: Compilation (567 times): Parsing (frontend): 1123.0 s Codegen & opts (backend): 1078.1 s I wonder if the "regression" in codegen time is just doing the PCH operation a few times, because meson does it per-target. Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30libfetchers: handle nonexistent refs in GitLab repos more gracefullyLinus Heckemann
Before: $ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent fetching git input 'git+file:///home/linus/projects/lix' fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent' error: [json.exception.type_error.302] type must be string, but is null After: $ outputs/out/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent fetching git input 'git+file:///home/linus/projects/lix' fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent' error: … while updating the lock file of flake 'git+file:///home/linus/projects/lix?ref=refs/heads/fix-gitlab-nonexistent&rev=915f16a619a36237a099b9aa9afed6d14ff613b4' … while updating the flake input 'nixpkgs' … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent' error: No commits returned by GitLab API -- does the ref really exist? Change-Id: Id9bc79d98348500e152ed519bb3ac79a3d15c38d
2024-05-29util.hh: Delete remaining file and clean up headersTom Hubrecht
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29util.{hh,cc}: Split out users.{hh,cc}Tom Hubrecht
Change-Id: I1bd92479a2cb7e5c2c2e1541b80474adb05ea0df
2024-05-29util.{hh,cc}: Split out processes.{hh,cc}Tom Hubrecht
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-27Merge "libfetchers: fix URL logging" into mainQyriad
2024-05-26Merge "git putFile: support flake maximalists" into mainMaximilian Bosch
2024-05-25libfetchers: fix URL loggingQyriad
8c06b7b43¹ made libfetchers log the URL being fetched just before the actual fetch, particularly in case something freezes. This used the base URL, to not include query parameters, as the Nixpkgs lib tests assume that stderr logs will be equal across shallow and non-shallow git fetches (and shallow fetches have the ?shallow=1 query parameter). 8c06b7b43 assumed that the `base` field of ParsedURL would be populated, as the comment simply says "URL without query/fragment"... but apparently it is not populated when the URL being fetched is *already* fetched, which caused libfetchers to log things like fetching gitlab input '' which is. silly. but you know, busted lix be busted. Anyway, with this commit we just remove the query params before printing instead, which seems to do the right thing [1]: 8c06b7b431d73431d8872c28d1ec5009cecd5d03 Change-Id: I9b9988992029aa6abef786f20b66e68c2ebb97d4
2024-05-24libfetchers: log fetches by URL just before they happenQyriad
Addresses but does not close #305, as we still need an indicator for frozen fetches. Change-Id: Iba34ad42dc1c8772f7da249b90fe794b041bbf73
2024-05-23Merge "packaging: rename nixexpr -> lixexpr and so on" into mainjade
2024-05-23packaging: rename nixexpr -> lixexpr and so onJade Lovelace
This breaks downstreams linking to us on purpose to make sure that if someone is linking to Lix they're doing it on purpose and crucially not mixing up Nix and Lix versions in compatibility code. We still need to fix the internal includes to follow the same schema so we can drop the single-level include system entirely. However, this requires a little more effort. This adds pkg-config for libfetchers and config.h. Migration path: expr.hh -> lix/libexpr/expr.hh nix/config.h -> lix/config.h To apply this migration automatically, remove all `<nix/>` from includes, so: `#include <nix/expr.hh>` -> `#include <expr.hh>`. Then, the correct paths will be resolved from the tangled mess, and the clang-tidy automated fix will work. Then run the following for out of tree projects: ``` lix_root=$HOME/lix (cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build') run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -fix src ``` Related: https://git.lix.systems/lix-project/nix-eval-jobs/pulls/5 Fixes: https://git.lix.systems/lix-project/lix/issues/279 Change-Id: I7498e903afa6850a731ef8ce77a70da6b2b46966
2024-05-21libfetchers: fallback to memory SQLite if fs IO failsQyriad
nix::fetchers::CacheImpl uses $XDG_CACHE_HOME, or its default based on $HOME, to store its SQLite database. If the current process can't write to that directory for whatever reason, though, any eval-time fetching would fail just initializing the cache. With this change, IO errors initializing the fetcher cache are logged but ignored, and nix::fetchers::CacheImpl falls back to an in-memory¹ database instead. Notably, this will fix any uses eval fetching while Lix itself is being run in a derivation builder (such as during tests), as the derivation builder does not set $XDG_CACHE_HOME, and sets $HOME to the non-existent directory /homeless-shelter. Before: $ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"' error: … while calling the 'fetchTarball' builtin at «string»:1:1: 1| fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz" | ^ error: creating directory '/homeless-shelter': Permission denied After: $ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"' warning: ignoring error initializing Lix fetcher cache: error: creating directory '/homeless-shelter': Permission denied "/nix/store/s9lxdnn0awp37n560bg4fgr497ah4hvw-source" ¹: https://www.sqlite.org/inmemorydb.html Change-Id: I15c38c9baaf215fc6e192b8a4c70b9692a69bc22
2024-05-20git putFile: support flake maximalistsGraham Christensen
Passing the commit message as an argument causes update failures on repositories with lots of flake inputs. In some cases, the commit message is over 250,000 bytes. Upstream PR: https://github.com/NixOS/nix/pull/10686 (cherry picked from commit 8b5e8f4fba5728f2b3e90fcd1ab15df77e3ea0e8) Change-Id: I2c196a21cc9bedc24d57a828a0c5b9467e072f76
2024-05-18Deprecate the online flake registries and vendor the default registryjulia
Fixes #183, #110, #116. The default flake-registry option becomes 'vendored', and refers to a vendored flake-registry.json file in the install path. Vendored copy of the flake-registry is from github:NixOS/flake-registry at commit 9c69f7bd2363e71fe5cd7f608113290c7614dcdd. Change-Id: I752b81c85ebeaab4e582ac01c239d69d65580f37
2024-05-09add and fix -Wignored-qualifiersQyriad
Change-Id: I4bffa766ae04dd80355f9b8c10e59700e4b406da
2024-05-07remove the autoconf+Make buildsystemQyriad
We're not using it anymore. Any leftover bugs in the Meson buildsystem are now just bugs. Closes #249. Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-04-21remove extraneous cache entry from github fetcherPuck Meerburg
This isn't necessary, as it's already covered by the tarball fetcher's cache. Change-Id: I85e35f5a61594f27b8f30d82145f92c5d6559e1f
2024-04-08pragma once and ///@file everything missing itJade Lovelace
Change-Id: Ia1a72348336794b5fb9f2694dd750266089b904e
2024-03-29meson: add missing explicit dependency on nlohmann_jsonWinter
Without this, the Meson setup won't bail out if nlohmann_json is missing, leading to subpar DX (and maybe worse, but I'm not entirely sure). Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-25Merge pull request #8817 from iFreilicht/flake-update-lock-overhaulThéophane Hufschmitt
Overhaul `nix flake update` and `nix flake lock` UX (cherry picked from commit 12a0ae73dbb37becefa5a442eb4532ff0de9ce65) Change-Id: Iff3b4f4235ebb1948ec612036b39ab29e4ca22b2
2024-03-25Input: Replace markFileChanged() by putFile()Eelco Dolstra
Committing a lock file using markFileChanged() required the input to be writable by the caller in the local filesystem (using the path returned by getSourcePath()). putFile() abstracts over this. (cherry picked from commit 95d657c8b3ae4282e24628ba7426edb90c8f3942) Change-Id: Ie081c5d9eb4e923b229191c5e23ece85145557ff
2024-03-22build: optionally build and install with mesonQyriad
This commit adds several meson.build, which successfully build and install Lix executables, libraries, and headers. Meson does not yet build docs, Perl bindings, or run tests, which will be added in following commits. As such, this commit does not remove the existing build system, or make it the default, and also as such, this commit has several FIXMEs and TODOs as notes for what should be done before the existing autoconf + make buildsystem can be removed and Meson made the default. This commit does not modify any source files. A Meson-enabled build is also added as a Hydra job, and to `nix flake check`. Change-Id: I667c8685b13b7bab91e281053f807a11616ae3d4
2024-03-17Delete hasPrefix and hasSuffix from the codebaseJade Lovelace
These now have equivalents in the standard lib in C++20. This change was performed with a custom clang-tidy check which I will submit later. Executed like so: ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../tests | tee -a clang-tidy-result Change-Id: I62679e315ff9e7ce72a40b91b79c3e9fc01b27e9