aboutsummaryrefslogtreecommitdiff
path: root/src/libfetchers
AgeCommit message (Collapse)Author
2024-06-12doc: rewrite the multi-user documentation to actually talk about securityJade Lovelace
It's in the security section, and it was totally outdated anyway. I took the opportunity to write down the stuff we already believed. Change-Id: I73e62ae85a82dad13ef846e31f377c3efce13cb0
2024-06-06build: expose option to enable or disable precompiled std headersQyriad
They are enabled by default, and Meson will also prints whether or not they're enabled at the bottom at the end of configuration. Change-Id: I48db238510bf9e74340b86f243f4bbe360794281
2024-06-01Merge "build: fix static linking with a hack" into mainQyriad
2024-06-01Merge "libfetchers: allow fetching gitlab refs with >1 commit" into mainLinus Heckemann
2024-05-31build: fix static linking with a hackQyriad
This causes libstore, libexpr, libfetchers, and libutil to be linked with -Wl,--whole-archive to executables, when building statically. libstore for the store backends, libexpr for the primops, libfetchers for the fetcher backends I assume(?), and libutil for the nix::logger initializer (which notably shows in pre-main constructors when HOME is not owned by the user. cursed.). This workaround should be removed when #359 is fixed. Fixes #306. Change-Id: Ie9ef0154e09a6ed97920ee8ab23810ca5e2de84c
2024-05-31Merge changes Ifcb0d310,I664366b8,Ibe7cf546 into mainjade
* changes: gitignore: delete 90% of it build-time: remove 20% more by PCH'ing C++ stdlib shellHook: make it actually run
2024-05-31libfetchers: allow fetching gitlab refs with >1 commitLinus Heckemann
Change-Id: I945c4c5512def9eff728bb67fe3c03ae17f99d6d
2024-05-30build-time: remove 20% more by PCH'ing C++ stdlibJade Lovelace
It seems like someone implemented precompiled headers a long time ago and then it never got ported to meson or maybe didn't work at all. This is, however, blessedly easy to simply implement. I went looking for `#define` that could affect the result of precompiling the headers, and as far as I can tell we aren't doing any of that, so this should truly just be free build time savings. Previous state: Compilation (551 times): Parsing (frontend): 1302.1 s Codegen & opts (backend): 956.3 s New state: **** Time summary: Compilation (567 times): Parsing (frontend): 1123.0 s Codegen & opts (backend): 1078.1 s I wonder if the "regression" in codegen time is just doing the PCH operation a few times, because meson does it per-target. Change-Id: I664366b8069bab4851308b3a7571bea97ac64022
2024-05-30libfetchers: handle nonexistent refs in GitLab repos more gracefullyLinus Heckemann
Before: $ nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent fetching git input 'git+file:///home/linus/projects/lix' fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent' error: [json.exception.type_error.302] type must be string, but is null After: $ outputs/out/bin/nix flake lock --override-input nixpkgs gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent fetching git input 'git+file:///home/linus/projects/lix' fetching gitlab input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent' error: … while updating the lock file of flake 'git+file:///home/linus/projects/lix?ref=refs/heads/fix-gitlab-nonexistent&rev=915f16a619a36237a099b9aa9afed6d14ff613b4' … while updating the flake input 'nixpkgs' … while fetching the input 'gitlab:simple-nixos-mailserver/nixos-mailserver/nonexistent' error: No commits returned by GitLab API -- does the ref really exist? Change-Id: Id9bc79d98348500e152ed519bb3ac79a3d15c38d
2024-05-29util.hh: Delete remaining file and clean up headersTom Hubrecht
Change-Id: Ic1f68e6af658e94ef7922841dd3ad4c69551ef56
2024-05-29util.{hh,cc}: Split out users.{hh,cc}Tom Hubrecht
Change-Id: I1bd92479a2cb7e5c2c2e1541b80474adb05ea0df
2024-05-29util.{hh,cc}: Split out processes.{hh,cc}Tom Hubrecht
Change-Id: I39280dc40ca3f7f9007bc6c898ffcf760e2238b7
2024-05-27Merge "libfetchers: fix URL logging" into mainQyriad
2024-05-26Merge "git putFile: support flake maximalists" into mainMaximilian Bosch
2024-05-25libfetchers: fix URL loggingQyriad
8c06b7b43¹ made libfetchers log the URL being fetched just before the actual fetch, particularly in case something freezes. This used the base URL, to not include query parameters, as the Nixpkgs lib tests assume that stderr logs will be equal across shallow and non-shallow git fetches (and shallow fetches have the ?shallow=1 query parameter). 8c06b7b43 assumed that the `base` field of ParsedURL would be populated, as the comment simply says "URL without query/fragment"... but apparently it is not populated when the URL being fetched is *already* fetched, which caused libfetchers to log things like fetching gitlab input '' which is. silly. but you know, busted lix be busted. Anyway, with this commit we just remove the query params before printing instead, which seems to do the right thing [1]: 8c06b7b431d73431d8872c28d1ec5009cecd5d03 Change-Id: I9b9988992029aa6abef786f20b66e68c2ebb97d4
2024-05-24libfetchers: log fetches by URL just before they happenQyriad
Addresses but does not close #305, as we still need an indicator for frozen fetches. Change-Id: Iba34ad42dc1c8772f7da249b90fe794b041bbf73
2024-05-23Merge "packaging: rename nixexpr -> lixexpr and so on" into mainjade
2024-05-23packaging: rename nixexpr -> lixexpr and so onJade Lovelace
This breaks downstreams linking to us on purpose to make sure that if someone is linking to Lix they're doing it on purpose and crucially not mixing up Nix and Lix versions in compatibility code. We still need to fix the internal includes to follow the same schema so we can drop the single-level include system entirely. However, this requires a little more effort. This adds pkg-config for libfetchers and config.h. Migration path: expr.hh -> lix/libexpr/expr.hh nix/config.h -> lix/config.h To apply this migration automatically, remove all `<nix/>` from includes, so: `#include <nix/expr.hh>` -> `#include <expr.hh>`. Then, the correct paths will be resolved from the tangled mess, and the clang-tidy automated fix will work. Then run the following for out of tree projects: ``` lix_root=$HOME/lix (cd $lix_root/clang-tidy && nix develop -c 'meson setup build && ninja -C build') run-clang-tidy -checks='-*,lix-fixincludes' -load=$lix_root/clang-tidy/build/liblix-clang-tidy.so -p build/ -fix src ``` Related: https://git.lix.systems/lix-project/nix-eval-jobs/pulls/5 Fixes: https://git.lix.systems/lix-project/lix/issues/279 Change-Id: I7498e903afa6850a731ef8ce77a70da6b2b46966
2024-05-21libfetchers: fallback to memory SQLite if fs IO failsQyriad
nix::fetchers::CacheImpl uses $XDG_CACHE_HOME, or its default based on $HOME, to store its SQLite database. If the current process can't write to that directory for whatever reason, though, any eval-time fetching would fail just initializing the cache. With this change, IO errors initializing the fetcher cache are logged but ignored, and nix::fetchers::CacheImpl falls back to an in-memory¹ database instead. Notably, this will fix any uses eval fetching while Lix itself is being run in a derivation builder (such as during tests), as the derivation builder does not set $XDG_CACHE_HOME, and sets $HOME to the non-existent directory /homeless-shelter. Before: $ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"' error: … while calling the 'fetchTarball' builtin at «string»:1:1: 1| fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz" | ^ error: creating directory '/homeless-shelter': Permission denied After: $ env -u XDG_CACHE_HOME HOME=/homeless-shelter nix -Lv eval --impure -E 'fetchTarball "https://git.lix.systems/lix-project/lix/archive/main.tar.gz"' warning: ignoring error initializing Lix fetcher cache: error: creating directory '/homeless-shelter': Permission denied "/nix/store/s9lxdnn0awp37n560bg4fgr497ah4hvw-source" ¹: https://www.sqlite.org/inmemorydb.html Change-Id: I15c38c9baaf215fc6e192b8a4c70b9692a69bc22
2024-05-20git putFile: support flake maximalistsGraham Christensen
Passing the commit message as an argument causes update failures on repositories with lots of flake inputs. In some cases, the commit message is over 250,000 bytes. Upstream PR: https://github.com/NixOS/nix/pull/10686 (cherry picked from commit 8b5e8f4fba5728f2b3e90fcd1ab15df77e3ea0e8) Change-Id: I2c196a21cc9bedc24d57a828a0c5b9467e072f76
2024-05-18Deprecate the online flake registries and vendor the default registryjulia
Fixes #183, #110, #116. The default flake-registry option becomes 'vendored', and refers to a vendored flake-registry.json file in the install path. Vendored copy of the flake-registry is from github:NixOS/flake-registry at commit 9c69f7bd2363e71fe5cd7f608113290c7614dcdd. Change-Id: I752b81c85ebeaab4e582ac01c239d69d65580f37
2024-05-09add and fix -Wignored-qualifiersQyriad
Change-Id: I4bffa766ae04dd80355f9b8c10e59700e4b406da
2024-05-07remove the autoconf+Make buildsystemQyriad
We're not using it anymore. Any leftover bugs in the Meson buildsystem are now just bugs. Closes #249. Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-04-21remove extraneous cache entry from github fetcherPuck Meerburg
This isn't necessary, as it's already covered by the tarball fetcher's cache. Change-Id: I85e35f5a61594f27b8f30d82145f92c5d6559e1f
2024-04-08pragma once and ///@file everything missing itJade Lovelace
Change-Id: Ia1a72348336794b5fb9f2694dd750266089b904e
2024-03-29meson: add missing explicit dependency on nlohmann_jsonWinter
Without this, the Meson setup won't bail out if nlohmann_json is missing, leading to subpar DX (and maybe worse, but I'm not entirely sure). Change-Id: I5913111060226b540dcf003257c99a08e84da0de
2024-03-25Merge pull request #8817 from iFreilicht/flake-update-lock-overhaulThéophane Hufschmitt
Overhaul `nix flake update` and `nix flake lock` UX (cherry picked from commit 12a0ae73dbb37becefa5a442eb4532ff0de9ce65) Change-Id: Iff3b4f4235ebb1948ec612036b39ab29e4ca22b2
2024-03-25Input: Replace markFileChanged() by putFile()Eelco Dolstra
Committing a lock file using markFileChanged() required the input to be writable by the caller in the local filesystem (using the path returned by getSourcePath()). putFile() abstracts over this. (cherry picked from commit 95d657c8b3ae4282e24628ba7426edb90c8f3942) Change-Id: Ie081c5d9eb4e923b229191c5e23ece85145557ff
2024-03-22build: optionally build and install with mesonQyriad
This commit adds several meson.build, which successfully build and install Lix executables, libraries, and headers. Meson does not yet build docs, Perl bindings, or run tests, which will be added in following commits. As such, this commit does not remove the existing build system, or make it the default, and also as such, this commit has several FIXMEs and TODOs as notes for what should be done before the existing autoconf + make buildsystem can be removed and Meson made the default. This commit does not modify any source files. A Meson-enabled build is also added as a Hydra job, and to `nix flake check`. Change-Id: I667c8685b13b7bab91e281053f807a11616ae3d4
2024-03-17Delete hasPrefix and hasSuffix from the codebaseJade Lovelace
These now have equivalents in the standard lib in C++20. This change was performed with a custom clang-tidy check which I will submit later. Executed like so: ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../tests | tee -a clang-tidy-result Change-Id: I62679e315ff9e7ce72a40b91b79c3e9fc01b27e9
2024-03-07Merge pull request #9324 from 9999years/fix-8854-take-2eldritch horrors
Don't attempt to `git add` ignored files (cherry picked from commit 359990dfdc713c80aabd7ea6f7e4528628fbe108) === also added a regression test that isn't upstream to be sure we're actually fixing the bug. Change-Id: I8267a3d0ece9909d8008b7435b90e7b3eee366f6
2024-03-05Merge pull request #9634 from 9999years/combine-abstract-pos-and-poseldritch horrors
Combine `AbstractPos`, `PosAdapter`, and `Pos` (cherry picked from commit 113499d16fc87d53b73fb62fe6242154909756ed) === this is a bit cursed because originally it was based on InputAccessor code that we don't have and moved/patched features we likewise don't have (fetchToStore caching, all the individual accessors, ContentAddressMethod). the commit is adjusted accordingly to match (remove caching, ignore accessors, use FileIngestionMethod). note that `state.rootPath . CanonPath == abs` and computeStorePathForPath works relative to cwd, so the slight rewrite in the moved fetchToStore is legal. Change-Id: I05fd340c273f0bcc8ffabfebdc4a88b98083bce5
2024-03-04Merge pull request #9445 from NixOS/allow-input-in-git-commiteldritch horrors
Allow user input in `git commit` (cherry picked from commit 43fb39ca29a0b054a4c18623520c496d27c57192) Change-Id: I13bd4c007234ee2133289c68ef8ab2eae4def78a
2023-08-22Tarball trees: Propagate lastModifiedEelco Dolstra
This makes them behave consistently with GitHub/GitLab flakes.
2023-08-01Don't require .tar/.zip extension for tarball flakerefsEelco Dolstra
Special-casing the file name is rather ugly, so we shouldn't do that. So now any {file,http,https} URL is handled by TarballInputScheme, except for non-flake inputs (i.e. inputs that have the attribute `flake = false`).
2023-07-07Simplify `ContentAddress`John Ericson
Whereas `ContentAddressWithReferences` is a sum type complex because different varieties support different notions of reference, and `ContentAddressMethod` is a nested enum to support that, `ContentAddress` can be a simple pair of a method and hash. `ContentAddress` does not need to be a sum type on the outside because the choice of method doesn't effect what type of hashes we can use. Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-06-24Merge pull request #5385 from Enzime/add/dirty-revThéophane Hufschmitt
Add `dirtyRev` and `dirtyShortRev` to `fetchGit`
2023-06-24Add `dirtyRev` and `dirtyShortRev` to `fetchGit`Michael Hoang
Fixes #4682
2023-06-13Allow tarball URLs to redirect to a lockable immutable URLEelco Dolstra
Previously, for tarball flakes, we recorded the original URL of the tarball flake, rather than the URL to which it ultimately redirects. Thus, a flake URL like http://example.org/patchelf-latest.tar that redirects to http://example.org/patchelf-<revision>.tar was not really usable. We couldn't record the redirected URL, because sites like GitHub redirect to CDN URLs that we can't rely on to be stable. So now we use the redirected URL only if the server returns the `x-nix-is-immutable` or `x-amz-meta-nix-is-immutable` headers in its response.
2023-06-13Add a generic check for rev attribute mismatchesEelco Dolstra
2023-06-06Fix SourcePath::resolveSymlinks()Eelco Dolstra
This fixes handling of symlinks that start with '..', and symlink targets that contain symlinks themselves.
2023-05-18Remove unusedKonstantin Vukolov
2023-05-18Add option isInteractiveKonstantin Vukolov
2023-05-17Ask for git credentials in fetcherKonstantin Vukolov
2023-05-03Fix hostRegex to accept hosts with a `-`figsoda
2023-04-24Add some more SourcePath docsEelco Dolstra
2023-04-24Document tMiscEelco Dolstra
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
2023-04-24Merge remote-tracking branch 'origin/master' into source-pathEelco Dolstra
2023-04-17Merge remote-tracking branch 'upstream/master' into source-pathRobert Hensing
2023-04-07Merge remote-tracking branch 'upstream/master' into path-infoJohn Ericson