| Age | Commit message (Collapse) | Author |
|---|
|
Better variable names, some comments, and a slight logic rearrange.
Change-Id: I9685ae252f83217aa85f06432234159c9ad19d1c
|
|
Change-Id: I63e79991a4bab93421266785e9258e0f5bb89b8f
|
|
This comes quite often when the available job slots on all remote
builders are exhausted and this is pretty spammy.
This isn't really an issue, but expected behavior.
A better way to display this is a nom-like approach where all scheduled
builds are shown in a tree and pending builds are being marked as such
IMHO.
Change-Id: I6bc14e6054f84e3eb0768127b490e263d8cdcf89
|
|
The original idea was to fix lix#174, but for a user friendly solution,
I figured that we'd need more consistency:
* Invalid query params will cause an error, just like invalid
attributes. This has the following two consequences:
* The `?dir=`-param from flakes will be removed before the URL to be
fetched is passed to libfetchers.
* The tarball fetcher doesn't allow URLs with custom query params
anymore. I think this was questionable anyways given that an
arbitrary set of query params was silently removed from the URL you
wanted to fetch. The correct way is to use an attribute-set
with a key `url` that contains the tarball URL to fetch.
* Same for the git & mercurial fetchers: in that case it doesn't even
matter though: both fetchers added unused query params to the URL
that's passed from the input scheme to the fetcher (`url2` in the code).
It turns out that this was never used since the query parameters were
erased again in `getActualUrl`.
* Validation happens for both attributes and URLs. Previously, a lot of
fetchers validated e.g. refs/revs only when specified in a URL and
the validity of attribute names only in `inputFromAttrs`.
Now, all the validation is done in `inputFromAttrs` and `inputFromURL`
constructs attributes that will be passed to `inputFromAttrs`.
* Accept all attributes as URL query parameters. That also includes
lesser used ones such as `narHash`.
And "output" attributes like `lastModified`: these could be declared
already when declaring inputs as attribute rather than URL. Now the
behavior is at least consistent.
Personally, I think we should differentiate in the future between
"fetched input" (basically the attr-set that ends up in the lock-file)
and "unfetched input" earlier: both inputFrom{Attrs,URL} entrypoints
are probably OK for unfetched inputs, but for locked/fetched inputs
a custom entrypoint should be used. Then, the current entrypoints
wouldn't have to allow these attributes anymore.
Change-Id: I1be1992249f7af8287cfc37891ab505ddaa2e8cd
|
|
Add the log-formats `multiline` and `multiline-with-logs` which offer
multiple current active building status lines.
Change-Id: Idd8afe62f8591b5d8b70e258c5cefa09be4cab03
|
|
|
|
clients" into main
|
|
Hooray for leaky abstraction allowing us to test this particular part of
the render pipeline.
Change-Id: Ie0f251ff874f63324e6a9c6388b84ec6507eeae2
|
|
Change-Id: Id12383f4741cba07159712700ebcfbe37e61560c
|
|
Change-Id: I125c8cac05c8e924b55e4eb1060496e35ea4e941
|
|
Change-Id: If85ea78954a45470b0b25c08dc7d40bfebd53610
|
|
Change-Id: Ief858c1488197211e2ee8b70aa40ed6c65743558
|
|
Change-Id: Icbac4ef927ddcaf0d2a74b376e5a77299529cd34
|
|
Change-Id: I08b1702310e863d15de26dc838eb0bcb62417c10
|
|
Change-Id: I621a455b1daba806fc498958aee7931fbfc55445
|
|
Change-Id: Ica9e2ec41d99eaa196a0d535501edf45c589b2b6
|
|
Binaries were identical before and after this commit on our machine
Change-Id: I6f8bfbe3298d6c5f43d5702c7a1e05cb180226cc
|
|
|
|
If we've consumed the entire input, that doesn't actually mean we're
done decompressing - there might be more output left. This worked (?)
in most cases because the input and output sizes are pretty comparable,
but sometimes they're not and then things get very funny.
Change-Id: I73435a654a911b8ce25119f713b80706c5783c1b
|
|
|
|
I did a whole bunch of `git log -S` to find out exactly when all these
things were obsoleted and found the commit in which their usage was
removed, which I have added in either the error message or a comment.
I've also made *some* of the version checks into static asserts for when
we update the minimum supported protocol version.
In the end this is not a lot of code we are deleting, but it's code that
we will never have to support into the future when we build a protocol
bridge, which is why I did it. It is not in the support baseline.
Change-Id: Iea3c80795c75ea74f328cf7ede7cbedf8c41926b
|
|
Share evaluation caches across installables
Before:
$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
'nixpkgs#hello' \
'nixpkgs#clang' \
'nixpkgs#cargo' \
'nixpkgs#rustup' \
'nixpkgs#bear' \
'nixpkgs#firefox' \
'nixpkgs#git-revise' \
'nixpkgs#hyperfine' \
'nixpkgs#curlie' \
'nixpkgs#xz' \
'nixpkgs#ripgrep'
0:03.61
After:
$ rm -rf ~/.cache/nix && time -f '%E' nix build --dry-run \
'nixpkgs#hello' \
'nixpkgs#clang' \
'nixpkgs#cargo' \
'nixpkgs#rustup' \
'nixpkgs#bear' \
'nixpkgs#firefox' \
'nixpkgs#git-revise' \
'nixpkgs#hyperfine' \
'nixpkgs#curlie' \
'nixpkgs#xz' \
'nixpkgs#ripgrep'
0:01.46
This could probably use a more proper benchmark...
Fixes #313
(cherry picked from commit de51e5c335865e3e0a8cccd283fec1a52cce243f)
Change-Id: I9350bebd462b6af12c51db5bf432321abfe84a16
|
|
without this we will not be able to get rid of makeDecompressionSink,
which in turn will be necessary to get rid of sourceToSink (since the
libarchive archive wrapper *must* be a Source due to api limitations)
Change-Id: Iccd3d333ba2cbcab49cb5a1d3125624de16bce27
|
|
don't consume a sink, return a source instead. the only reason to not do
this is a very slight reduction in dynamic allocations, but since we are
going to *at least* do disk io that will not be a lot of overhead anyway
Change-Id: Iae2f879ec64c3c3ac1d5310eeb6a85e696d4614a
|
|
Change-Id: Iac7f24d79e24417436b9b5cbefd6af051aeea0a6
|
|
if we want have getFile return a source instead of consuming a sink
we'll have to disambiguate this overload another way, eg like this.
Change-Id: Ia26de2020c309a37e7ccc3775c1ad1f32e0a778b
|
|
|
|
* changes:
store: fix null reference from DerivationGoal::waiteeDone
libmain: fix UB in verbosity assignment
build: make UBSan work :)
libexpr: fix accessing uninitialized values and fix pure-eval docs
|
|
|
|
This happened during a PathSubstitutionGoal of a .drv file:
substitution of '/tmp/jade/nix-test/ca/eval-store/store/1lj7lsq5y0f25mfbnq6d3zd0bw5ay33n-dependencies-input-2.drv'
What happened here is that since PathSubstitutionGoal is not a
DerivationGoal, in production builds, the UB was not caught, since it
would early-exit from failing a dynamic_cast to DerivationGoal * on the
very next line, but before the null reference was ever used.
This was nonetheless UB. The fix should be to just rearrange the two
lines; I don't think there is a further bug there, since *substituting a
.drv* **necessarily** means you cannot have the representation of
the derivation as would be necessary for drv to not be null there.
Test failure:
++(eval-store.sh:12) _RR_TRACE_DIR=/home/jade/.local/share/rr rr record -- nix build -f dependencies.nix --eval-store /tmp/jade/nix-test/ca/eval-store/eval-store -o /tmp/jade/nix-test/ca/eval-store/result
don't know how to build these paths:
/tmp/jade/nix-test/ca/eval-store/store/6y51mf0p57ggipgab6hdjabbvplzsicq-dependencies-top.drv
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/8027afyvqb87y1sf5xhdkqsflqn1ziy8-dependencies.builder0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/7r5pqyncvfgrryf9gzy1z56z3xigi61x-builder-dependencies-input-0.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nhmgm87zlqy3ks96dxrn7l37b72azi99-builder-dependencies-input-1.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/nq4qa2j6y8ajqazlfq6h46ck637my1n6-builder-dependencies-input-2.sh' to 'local'...
copying 1 paths...
copying path '/tmp/jade/nix-test/ca/eval-store/store/6vh0vna9l5afck01y7iaks3hm9ikwqyj-builder-fod-input.sh' to 'local'...
building '/tmp/jade/nix-test/ca/eval-store/store/gy91pqymf2nc5v7ld1bad94xpwxdi25s-dependencies-input-0.drv'...
building '/tmp/jade/nix-test/ca/eval-store/store/w7wlkjx97ivmnrymkac5av3nyp94hzvq-dependencies-input-1.drv'...
../src/libstore/build/derivation-goal.cc:1556:22: runtime error: reference binding to null pointer of type 'Derivation'
0 0x734ba59a6886 in nix::DerivationGoal::waiteeDone(std::shared_ptr<nix::Goal>, nix::Goal::ExitCode) /home/jade/lix/lix2/build/src/libstore/build/derivation-goal.cc:1556:12
1 0x734ba59c0962 in nix::Goal::amDone(nix::Goal::ExitCode, std::optional<nix::Error>) /home/jade/lix/lix2/build/src/libstore/build/goal.cc:95:25
2 0x734ba5a1c44a in nix::PathSubstitutionGoal::done(nix::Goal::ExitCode, nix::BuildResult::Status, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>) /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:38:5
3 0x734ba5a1b454 in nix::PathSubstitutionGoal::init() /home/jade/lix/lix2/build/src/libstore/build/substitution-goal.cc:56:9
4 0x734ba5a2a6c6 in nix::Worker::run(std::set<std::shared_ptr<nix::Goal>, nix::CompareGoalPtrs, std::allocator<std::shared_ptr<nix::Goal>>> const&) /home/jade/lix/lix2/build/src/libstore/build/worker.cc:320:23
5 0x734ba59b93d8 in nix::Store::buildPathsWithResults(std::vector<nix::DerivedPath, std::allocator<nix::DerivedPath>> const&, nix::BuildMode, std::shared_ptr<nix::Store>) /home/jade/lix/lix2/build/src/libstore/build/entry-points.cc:60:12
6 0x734ba663c107 in nix::Installable::build2(nix::ref<nix::Store>, nix::ref<nix::Store>, nix::Realise, std::vector<nix::ref<nix::Installable>, std::allocator<nix::ref<nix::Installable>>> const&, nix::BuildMode) /home/jade/lix/lix2/build/src/libcmd/installables.cc:637:36
Change-Id: Id0e651e480bebf6356733b01bc639e9bb59c7bd0
|
|
This was generating an out-of-range verbosity value. We should just
process it as an int and then convert to verbosity with a clamping
function, which trivially avoids any domain type violations.
Change-Id: I0ed20da8e1496a1225ff3008b76827d99265d404
|
|
We got UBSan working on Lix, so we of course immediately found a bug and
some definitely nonsense behaviour.
Accessing `pureEval` or `restrictEval` from a default setting value is
nonsense, since they would never be actually set by the time that value
is set so they are not going to do anything. The configuration is not
applied in an initializer (and even if it were, it's not going to be in
the right order).
After looking into *that*, we hunted down what actually was applying
these, since clearly this code did not do anything. The EvalState
constructor should have a "search path added and removed here :)" sign
on it, because that's where it is done. We added an explicit
initialization of the optional in there because it was otherwise unclear
why pureEval also has the search path to allowed paths setup code run.
We then realized that the `pureEval` documentation was *also* bogus, and
we rewrote it. In so doing, we realized that we forgot to file a bug to
make `builtins.storePath` work in pure eval mode, so we filed one of
those: https://git.lix.systems/lix-project/lix/issues/402
Yaks have been thoroughly shorn.
UBSan report:
../src/libexpr/eval-settings.cc:66:10: runtime error: member call on address 0x752fa9a13060 which does not point to an object of type 'nix::BaseSetting<b
ool>'
0x752fa9a13060: note: object has invalid vptr
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
^~~~~~~~~~~~~~~~~~~~~~~
invalid vptr
0 0x752fa95106a6 in nix::EvalSettings::getDefaultNixPath[abi:cxx11]() /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:66:10
1 0x752fa950e420 in nix::EvalSettings::EvalSettings() /home/jade/lix/lix2/build/src/libexpr/eval-settings.hh:36:15
2 0x752fa9469f1f in __cxx_global_var_init.50 /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc:98:14
3 0x752fa9469f1f in _GLOBAL__sub_I_eval_settings.cc /home/jade/lix/lix2/build/src/libexpr/eval-settings.cc
4 0x752fabbd308d in call_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x508d) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
5 0x752fabbd317b in _dl_init (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x517b) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
6 0x752fabbe9c2f in _dl_start_user (/nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2+0x1bc2f) (BuildId: a5b8228edc9f16078ac3c894af964eeb990ecb4c)
Change-Id: I5d8ffb7bfbe24b6584020ac74eed93d9f2e6d111
|
|
Change-Id: Iebffd5436109da270ee870670a20f5ee7db9a204
|
|
Change-Id: I9579dd08f7bd0f927bde9d3128515b0cee15f320
|
|
even the transfer function is not all that necessary since there aren't
that many users, but we'll keep it for now. we could've kept both names
but we also kind of want to use `download` for something else very soon
Change-Id: I005e403ee59de433e139e37aa2045c26a523ccbf
|
|
The lock usage was obviously wrong so it was entirely serialized. This
has the predicted speedups, the only question is whether it is sound
because it's exposing a bunch of new code to actual concurrency.
I did audit all the stores' queryPathInfoUncached implementations and
they all look *intended* to be thread safe, but whether that is actually
sound or not: lol lmao. I am highly confident in the s3 one because it
is calling s3 sdk methods that are thread safe and has no actual state.
Others are using Pool and look to be *supposed* to be thread safe, but
unsure if they actually are.
Change-Id: I0369152a510e878b5ac56c9ac956a98d48cd5fef
|
|
Found by looking for interesting asan reports from the test suite.
What happened here is that name got overwritten, but it was what
actually held the backing memory for the thing it got overwritten by,
which was a by-reference value coming out of std::regex.
Due to absurd reasons I cannot seem to use a string_view iterator here,
so I just copy the string with a longer lifetime instead. idk lol
==3796364==ERROR: AddressSanitizer: heap-use-after-free on address 0x503000014c61 at pc 0x74843523bf1d bp 0x7ffc68351330 sp 0x7ffc68350af0
READ of size 3 at 0x503000014c61 thread T0
0 0x74843523bf1c in __asan_memcpy (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x159f1c)
1 0x6403cf6cbff4 in std::char_traits<char>::copy(char*, char const*, unsigned long) /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/char_traits.h:445:33
<...>
7 0x6403cf6cbff4 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::str() const /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/regex.h:966:6
8 0x6403cf6cbff4 in std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>>::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>() const /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/regex.h:955:16
9 0x6403cf6cbff4 in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:37:26
10 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
11 0x6403cf873331 in CmdProfileDiffClosures::run(nix::ref<nix::Store>) /home/jade/lix/lix2/build/src/nix/profile.cc:479:17
<...>
0x503000014c61 is located 17 bytes inside of 21-byte region [0x503000014c50,0x503000014c65)
freed by thread T0 here:
0 0x748435250470 in operator delete(void*) (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x16e470)
<...>
6 0x6403cf6cbda2 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::~basic_string() /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/basic_string.h:792:9
7 0x6403cf6cbda2 in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:36:13
8 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
<...>
previously allocated by thread T0 here:
0 0x74843524fa38 in operator new(unsigned long) (/nix/store/mzhqknx2mc94jdz4n320hn1lml86398y-clang-wrapper-17.0.6/resource-root/lib/linux/libclang_rt.asan-x86_64.so+0x16da38)
<...>
9 0x6403cf6cb68c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::basic_string<std::basic_string_view<char, std::char_traits<char>>, void>(std::basic_string_view<char, std::char_traits<char>> const&, std::allocator<char> const&) /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/include/c++/13.2.0/bits/basic_string.h:784:4
10 0x6403cf6cb68c in nix::getClosureInfo[abi:cxx11](nix::ref<nix::Store>, nix::StorePath const&) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:33:21
11 0x6403cf6cd70c in nix::printClosureDiff(nix::ref<nix::Store>, nix::StorePath const&, nix::StorePath const&, std::basic_string_view<char, std::char_traits<char>>) /home/jade/lix/lix2/build/src/nix/diff-closures.cc:54:25
12 0x6403cf873331 in CmdProfileDiffClosures::run(nix::ref<nix::Store>) /home/jade/lix/lix2/build/src/nix/profile.cc:479:17
<...>
Change-Id: I9c408cf2a3d3155f9f7b2ad4848ee6c741331db0
|
|
This seems to fix a use of stack after return.
Change-Id: If690a6defb9a3225684685132cf78b227e271447
|
|
aws-sdk-cpp spams logs about sending TLS data in the otherwise rather
helpful debug logs. I've filed a PR upstream to stop it, but for now we
can just fix their verbosity ourselves.
Upstream-PR: https://github.com/aws/aws-sdk-cpp/pull/3003
Change-Id: I0c41a50d5f5958106836d6345843f4b05b9c8981
|
|
The versions checked for are so old that we can just drop support.
Change-Id: Ib9cf136d1cb9a4a91a6613102c4fd15e1190363b
|
|
There was a previously-unused move constructor that just called abort,
which makes no sense since it ought to just be `= delete` if you don't
want one (commit history says it was Eelco doing an optimistic
performance optimisation in 2016, so it probably would not pass review
today).
However, a Lock has some great reasons to be moved! You might need to
unlock it early, for instance, or give it to someone else. So we change
the move constructor to instead hollow out the moved-from object and
make it unusable.
Change-Id: Iff2a4c2f7ebd0a558c4866d4dfe526bc8558bed7
|
|
Unfetched submodules are included as empty directories in archives, so they end
up as such in the store when fetched in clean mode. Make sure the same happens
in dirty mode too. Fortunately, they are already correctly represented in the
ls-files output, so we just need to make sure to include the empty directory in
our filter.
Fixes: https://github.com/NixOS/nix/issues/6247
Change-Id: I60d06ff360cfa305d081b920838c893c06da801c
|
|
* changes:
libstore client: remove remaining dead code
libstore: refuse to serialise ancient protocols
libstore client: remove support for <2.3 clients
libstore daemon: remove very old protocol support (<2.3)
Delete old ValidPathInfo test, fix UnkeyedValidPathInfo
Set up minimum protocol version
|
|
with the prepatory work done this mostly means turning plain pointers
into unique_ptrs, with all the associated churn that necessitates. we
might want to change some of these to box_ptrs at some point as well,
but that would be a semantic change that isn't fully appropriate yet.
Change-Id: I0c238c118617420650432f4ed45569baa3e3f413
|
|
almost all places where Exprs are passed as pointers expect the pointers
to be non-null. pass them as references to encode this constraint in the
type system as well (and also communicate that Exprs must not be freed).
Change-Id: Ia98f166fec3c23151f906e13acb4a0954a5980a2
|
|
storing a pointer only adds an unnecessary indirection at runtime.
Change-Id: If06dd05effdf1ccb0df0873580f50c775608925d
|
|
now that destructors are hooked up we want to give the C skeleton every
real chance to actually run them. since bison does not call destructors
on values that have been passed to semantic actions even when an action
causes an abort we will also have to delete some things manually still.
Change-Id: Ia22bdaa9e969b74e17a6c496e35e6c2d86b7d750
|
|
this doesn't help much yet since the state objects themselves also leak
all memory they are given, but it is a first necessary step to properly
managing parser memory. notably we have to clear $$ when returning from
the parser since even the start symbol is subject to automatic deletion
by the bison-generated parser before returning control to the call site
Change-Id: I80245b0c747308e80923e7f18ce4e1a4898f93b0
|
|
Change-Id: I08d39c4ad5b967de526c0d5c5e6299256c7967f3
|
|
This uses a minor hack in which we check the rl_line_buffer global
variable to workaround editline not including the colon in its
completion callback.
Fixes #361
Change-Id: Id159d209c537443ef5e37a975982e8e12ce1f486
|
