aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2024-07-15libstore: rewrite narFromPath as generatoreldritch horrors
Change-Id: Ifa783c2c65c06ddd1d0212016d5bfd07666ea91c
2024-07-15Merge "use clangStdenv for the default devShell, so we get clangd by ↵lunaphied
default" into main
2024-07-14Merge "libexpr/eval.cc: remove unnecessary C string conversion" into mainnan-git
2024-07-14libexpr/eval.cc: remove unnecessary C string conversionNaN
Change-Id: I5b7c21df84ff8ff64cf6a1e261fc3729a06bd4f6
2024-07-13nix-support/binary-tarball.nix: Pass through root pathsArtemis Tosini
Passing through root paths allows external programs to see which nix and cacert are in a binary tarball, e.g. to recreate it from substituters Change-Id: I27431134df53bbc6623484f8a0822004b51f7c87
2024-07-13docs: update to define integer overflowJade Lovelace
Change-Id: Ie8a1b31035f2d27a220e5df2e9e178ec3b39ee68
2024-07-13language: cleanly ban integer overflowsJade Lovelace
This also bans various sneaking of negative numbers from the language into unsuspecting builtins as was exposed while auditing the consequences of changing the Nix language integer type to a newtype. It's unlikely that this change comprehensively ensures correctness when passing integers out of the Nix language and we should probably add a checked-narrowing function or something similar, but that's out of scope for the immediate change. During the development of this I found a few fun facts about the language: - You could overflow integers by converting from unsigned JSON values. - You could overflow unsigned integers by converting negative numbers into them when going into Nix config, into fetchTree, and into flake inputs. The flake inputs and Nix config cannot actually be tested properly since they both ban thunks, however, we put in checks anyway because it's possible these could somehow be used to do such shenanigans some other way. Note that Lix has banned Nix language integer overflows since the very first public beta, but threw a SIGILL about them because we run with -fsanitize=signed-overflow -fsanitize-undefined-trap-on-error in production builds. Since the Nix language uses signed integers, overflow was simply undefined behaviour, and since we defined that to trap, it did. Trapping on it was a bad UX, but we didn't even entirely notice that we had done this at all until it was reported as a bug a couple of months later (which is, to be fair, that flag working as intended), and it's got enough production time that, aside from code that is IMHO buggy (and which is, in any case, not in nixpkgs) such as https://git.lix.systems/lix-project/lix/issues/445, we don't think anyone doing anything reasonable actually depends on wrapping overflow. Even for weird use cases such as doing funny bit crimes, it doesn't make sense IMO to have wrapping behaviour, since two's complement arithmetic overflow behaviour is so *aggressively* not what you want for *any* kind of mathematics/algorithms. The Nix language exists for package management, a domain where bit crimes are already only dubiously in scope to begin with, and it makes a lot more sense for that domain for the integers to never lose precision, either by throwing errors if they would, or by being arbitrary-precision. This change will be ported to CppNix as well, to maintain language consistency. Fixes: https://git.lix.systems/lix-project/lix/issues/423 Change-Id: I51f253840c4af2ea5422b8a420aa5fafbf8fae75
2024-07-13libutil: add checked arithmetic toolsJade Lovelace
This is in preparation for adding checked arithmetic to the evaluator. Change-Id: I6e115ce8f5411feda1706624977a4dcd5efd4d13
2024-07-12use clangStdenv for the default devShell, so we get clangd by defaultLunaphied
The default-stdenv-devShell can always be used with `.#native-stdenvPackages`. Change-Id: I9b3e72210ba5219b6b65c71a2818110769623904
2024-07-12Use std::strong_ordering for version comparisonJade Lovelace
The actual motive here is the avoidance of integer overflow if we were to make these use checked NixInts and retain the subtraction. However, the actual *intent* of this code is a three-way comparison, which can be done with operator<=>, so we should just do *that* instead. Change-Id: I7f9a7da1f3176424b528af6d1b4f1591e4ab26bf
2024-07-11libstore: remove upcast_goaleldritch horrors
upcast_goal was only ever needed to break circular includes, but the same solution that gave us upcast_goal also lets us fully remove it: just upcast goals without a wrapper function, but only in .cc files. Change-Id: I9c71654b2535121459ba7dcfd6c5da5606904032
2024-07-11libstore: turn copyNAR into a generatoreldritch horrors
Change-Id: Id452f6a03faa1037ff13af0f63e32883966ff40d
2024-07-11libstore: turn the NAR parser into a passthrough generatoreldritch horrors
this will let us turn copyNAR into a generator as well, which in turn is necessary to turn the users of copyNAR into generators without resorting to sinkToSource coroutines. currently this uses the SerializingTransform in all cases, even for copyNAR where it is not necessary. should this be a performance problem we can easily swap out the transform for one which does not produce any bytes of its own, but that should not be necessary. Change-Id: I7e685879318fcbb78d8b88abfddd7752360eb0ce
2024-07-11libutil: remove makeDecompressionSinkeldritch horrors
the sole remaining user of this function can use makeDecompressionSource instead, while making the sinkToSource in the caller unnecessary as well Change-Id: I4258227b5dbbb735a75b477d8a57007bfca305e9
2024-07-11libstore: make BinaryCacheStore::getFile return a sourceeldritch horrors
this lets us remove the last true remaining uses of makeDecompressionSink. Change-Id: I146ca2bbe1a9ae9a367117a7b8a304b23a63e5e2
2024-07-11libutil: rewrite RewritingSink as sourceeldritch horrors
the rewriting sink was just broken. when given a rewrite set that contained a key that is also a proper infix of another key it was possible to produce an incorrectly rewritten result if the writer used the wrong block size. fixing this duplicates rewriteStrings, to avoid this we'll rewrite rewriteStrings to use RewritingSource in a new mode that'll allow rewrites we had previously forbidden. Change-Id: I57fa0a9a994e654e11d07172b8e31d15f0b7e8c0
2024-07-10Merge "lix-doc: update dependencies and refactor" into mainlunaphied
2024-07-10Merge changes I81552018,Ieb65c133 into maineldritch horrors
* changes: releng: add releaseTests flake output, test script add aarch64-linux as a cross-build target
2024-07-10devShell: Fix errors with environment variablespiegames
Previous code assumed that $MANPATH always exists Change-Id: I5a4d012045ba6ff9086373b3f46a75d82285d393
2024-07-10Merge "libmain: clear display attributes in the multiline progress bar" into ↵alois31
main
2024-07-10lix-doc: update dependencies and refactorLunaphied
This updates the version of rnix used and refactors the code generally to be more precise and capable in it's identification of both lambdas and determining which documentation comments are attached. Change-Id: Ib0dddabd71f772c95077f9d7654023b37a7a1fd2
2024-07-09releng: add releaseTests flake output, test scripteldritch horrors
this is supposed to be a set of outputs we want to always succeed for releases. sadly we can't add nixos installer tests using lix to these because the nixos test framework does not allow overriding nix in the installer test suites due to unfortunate oversights in the framework. Change-Id: I815520181ccca70a47205d38ba27e73529347f04
2024-07-09add aarch64-linux as a cross-build targeteldritch horrors
we want to be sure we can cross-build to aarch64 for releases, add a target to our crossSystems list to make those cheacks easier to run. Change-Id: Ieb65c1333a5232641ace0ba4d122fc7d528ebc04
2024-07-09Fix dry-run flag for nix-collect-garbageQuantum Jump
`nix-collect-garbage --dry-run` previously elided the entire garbage collection check, meaning that it would just exit the script without printing anything. This change makes the dry run flag instead set the GC action to `gcReturnDead` rather than `gcDeleteDead`, and then continue with the script. So if you set `--dry-run`, it will print the paths it *would* have garbage collected, but not actually delete them. I filed a bug for this: https://git.lix.systems/lix-project/lix/issues/432 but then realised I could give fixing it a go myself. Change-Id: I062dbf1a80bbab192b5fd0b3a453a0b555ad16f2
2024-07-08libexpr: stop lying about DrvInfo's constnessQyriad
DrvInfo's query methods all use mutable fields to cache, but like. that's basically the entire interface for DrvInfo. Not only that, but these formerly-const-marked functions can even throw due to eval errors! Changing this only required removing some `const` markers in nix-env, and changing a single inline `queryInstalled()` call to be an lvalue instead. Change-Id: I796807118f3b35b0e93668b5e28210d9e521b2ae
2024-07-08libmain: clear display attributes in the multiline progress barAlois Wohlschlager
Activities can set display attributes in their log output using the "Select Graphics Rendition" functionality. To prevent interfering with subsequent text displayed, these should be reset after writing the log line. The multiline progress bar neglected to do this, resulting for example in a colorised "building …" header in the next line. Reset the attributes properly, like the standard progress bar already does. Change-Id: I1dc69f4a1d747a76b83e8721a72d9bb0e5554488
2024-07-07libstore: make LocalDerivationGoal::needsHashRewrite virtualArtemis Tosini
This rather simple function existed just to check some flags, but the response varies by platform. This is a perfect case for our subclasses. Change-Id: Ieb1732a8d024019236e0d0028ad843a24ec3dc59
2024-07-06libutil: remove RewritingSink match/size trackingeldritch horrors
size tracking can be done with a LengthSink and a tee. match tracking was defeated by never having done any match tracking, all users would see the same (empty) set of matches at all times. match tracking with bytes offsets alone would not be sufficient in the general case, only because computeHashModulo uses a single rewrite could it have worked. Change-Id: Idb214b5222e0ea24f450f5505712a342b63d7570
2024-07-06libutil: turn HashModuloSink into a free functioneldritch horrors
Change-Id: I5878007502fa68c2816a0f4c61f7d0e60bdde702
2024-07-06libutil: return sources from runProgram2eldritch horrors
this much more closely mimics what is actually happening: we're reading data from somewhere else, actively, rather than passively waiting. with the data flow matching the underlying system interactions better we can remove a few sinkToSource calls that merely exists to undo the mismatch caused by not treating subprocess output as a data source to begin with Change-Id: If4abfc2f8398fb5e88c9b91a8bdefd5504bb2d11
2024-07-05libutil: return a program handle from runProgram2eldritch horrors
this will let us also return a source for the program output later, which will in turn make sinkToSource unnecessary for program output processing. this may also reopen a path for provigin program input, but that still needs a proper async io framework to avoid problems. Change-Id: Iaf93f47db99c38cfaf134bd60ed6a804d7ddf688
2024-07-05libutil: convert readFileSource to a generatoreldritch horrors
Change-Id: I5f92b15fd367d46eb047d74ab6e317b4f51a46d3
2024-07-05libstore: convert dumpPath to a generatoreldritch horrors
Change-Id: Ic4cf5562504aa29130304469936f958c0426e5ef
2024-07-05libutil: allow construction of sources from generatorseldritch horrors
Change-Id: I78ff8d0720f06bce731e26d5e1c53b1382bbd589
2024-07-04mildly cleanup libexpr/eval.hhQyriad
Change-Id: I40d01a8f8b7fb101279c6f88ebdf1f0969d9d7f0
2024-07-04distinguish between throws & errors during throwQyriad
Turns errors like this: let throwMsg = a: throw (a + " invalid bar"); in throwMsg "bullshit" error: … from call site at «string»:3:4: 2| throwMsg = a: throw (a + " invalid bar"); 3| in throwMsg "bullshit" | ^ … while calling 'throwMsg' at «string»:2:14: 1| let 2| throwMsg = a: throw (a + " invalid bar"); | ^ 3| in throwMsg "bullshit" … while calling the 'throw' builtin at «string»:2:17: 1| let 2| throwMsg = a: throw (a + " invalid bar"); | ^ 3| in throwMsg "bullshit" error: bullshit invalid bar into errors like this: let throwMsg = a: throw (a + " invalid bar"); in throwMsg "bullshit" error: … from call site at «string»:3:4: 2| throwMsg = a: throw (a + " invalid bar"); 3| in throwMsg "bullshit" | ^ … while calling 'throwMsg' at «string»:2:14: 1| let 2| throwMsg = a: throw (a + " invalid bar"); | ^ 3| in throwMsg "bullshit" … caused by explicit throw at «string»:2:17: 1| let 2| throwMsg = a: throw (a + " invalid bar"); | ^ 3| in throwMsg "bullshit" error: bullshit invalid bar Change-Id: I593688928ece20f97999d1bf03b2b46d9ac338cb
2024-07-04trace which part of `foo.bar.baz` errorsQyriad
Turns errors like: let somepkg.src = throw "invalid foobar"; in somepkg.src.meta error: … while evaluating the attribute 'src.meta' at «string»:2:3: 1| let 2| somepkg.src = throw "invalid foobar"; | ^ 3| in somepkg.src.meta … while calling the 'throw' builtin at «string»:2:17: 1| let 2| somepkg.src = throw "invalid foobar"; | ^ 3| in somepkg.src.meta error: invalid foobar into errors like: let somepkg.src = throw "invalid foobar"; in somepkg.src.meta error: … while evaluating the attribute 'src.meta' at «string»:2:3: 1| let 2| somepkg.src = throw "invalid foobar"; | ^ 3| in somepkg.src.meta … while evaluating 'somepkg.src' to select 'meta' on it at «string»:3:4: 2| somepkg.src = throw "invalid foobar"; 3| in somepkg.src.meta | ^ … while calling the 'throw' builtin at «string»:2:17: 1| let 2| somepkg.src = throw "invalid foobar"; | ^ 3| in somepkg.src.meta error: invalid foobar And for type errors, from: let somepkg.src = "I'm not an attrset"; in somepkg.src.meta error: … while evaluating the attribute 'src.meta' at «string»:2:3: 1| let 2| somepkg.src = "I'm not an attrset"; | ^ 3| in somepkg.src.meta … while selecting an attribute at «string»:3:4: 2| somepkg.src = "I'm not an attrset"; 3| in somepkg.src.meta | ^ error: expected a set but found a string: "I'm not an attrset" into: let somepkg.src = "I'm not an attrset"; in somepkg.src.meta error: … while evaluating the attribute 'src.meta' at «string»:2:3: 1| let 2| somepkg.src = "I'm not an attrset"; | ^ 3| in somepkg.src.meta … while selecting 'meta' on 'somepkg.src' at «string»:3:4: 2| somepkg.src = "I'm not an attrset"; 3| in somepkg.src.meta | ^ error: expected a set but found a string: "I'm not an attrset" For the low price of an enumerate() and a lambda you too can have the incorrect line of code actually show up in the trace! Change-Id: Ic1491c86e33c167891bdac9adad6224784760bd6
2024-07-04trace when the `foo` part of `foo.bar.baz` errorsQyriad
Turns errors like: let errpkg = throw "invalid foobar"; in errpkg.meta error: … while calling the 'throw' builtin at «string»:2:12: 1| let 2| errpkg = throw "invalid foobar"; | ^ 3| in errpkg.meta error: invalid foobar into errors like: let errpkg = throw "invalid foobar"; in errpkg.meta error: … while evaluating 'errpkg' to select 'meta' on it at «string»:3:4: 2| errpkg = throw "invalid foobar"; 3| in errpkg.meta | ^ … while calling the 'throw' builtin at «string»:2:12: 1| let 2| errpkg = throw "invalid foobar"; | ^ 3| in errpkg.meta error: invalid foobar For the low price of one try/catch, you too can have the incorrect line of code actually show up in the trace! Change-Id: If8d6200ec1567706669d405c34adcd7e2d2cd29d
2024-07-04add an ExprPrinter class, like ValuePrinterQyriad
To be used Shortly Change-Id: I9def7975aa55f251eb8486391677771f7352d7ce
2024-07-04add an impl of Expr::show for ExprInheritFrom that doesn't crashQyriad
ExprVar::show() assumes it has a name. dynamic inherits do not necessarily (ever?) have a name. Change-Id: If10893188e307431da17f0c1bd0787adc74f7141
2024-07-04give ExprInheritFrom a handle to what its standing in forQyriad
Change-Id: I12088e0b618407e5432523bbc97be63c8d6fce62
2024-07-03libstore: add LocalDerivationGoal startChild hookArtemis Tosini
Add a platform-specific function for starting sandboxed child. Generally this just means startProcess, but on Linux we use flags for clone to start a new namespace Change-Id: I41c8aba62676a162388bbe5ab8a7518904c7b058
2024-07-03libstore: Add LocalDerivationGoal prepareSandbox hookArtemis Tosini
Add a new OS-specific hook called `prepareSandbox`, run before forking On Darwin this is empty as nothing is required, on Linux this creates the chroot directory and adds basic files, and on platforms using a fallback this throws an exception Change-Id: Ie30c38c387f2e0e5844b2afa32fd4d33b1180dae
2024-07-03libutil: begin porting serialization to generatorseldritch horrors
generators are a better basis for serializers than streaming into sinks as we do currently for many reasons, such as being usable as sources if one wishes to (without requiring an intermediate sink to serialize full data sets into memory, or boost coroutines to turn sinks into sources), composing more naturally (as one can just yield a sub-generator instead of being forced to wrap entire substreams into clunky functions or even more clunky custom types to implement operator<< on), allowing wrappers to transform data with clear ownership semantics (removing the need for explicit memory allocations and Source wrappers), and many other things Change-Id: I361d89ff556354f6930d9204f55117565f2f7f20
2024-07-03libutil: convert drainFD to a Bytes generatoreldritch horrors
the `*Source` name is a slight misnomer since we do also have a Source type, but we can probably live with this for time being. Change-Id: I54eb2e59a4009014e324797f16b80b962759c7d3
2024-07-03libutil: allow draining Generator<Bytes> into sinkseldritch horrors
Change-Id: I442d03a5399096d4baca9a2618b4c4b64db36c4b
2024-07-03libutil: add Bytes typeeldritch horrors
not used anywhere yet, but we'll use this a lot soon for generators that return file contents, wire protocol fragments, or indeed any byte stream Change-Id: I01a46f9bf9d75aaf4a5d7662773b99f498862a28
2024-07-03libutil: generator type with on-yield value mappingeldritch horrors
this will be the basis of non-boost coroutines in lix. anything that is a boost coroutine *should* be representable with a Generator coroutine, and many things that are not currently boost coroutines but behave much like one (such as, notably, serializers) should be as well. this allows us to greatly simplify many things that look like iteration but aren't. Change-Id: I2cebcefa0148b631fb30df4c8cfa92167a407e34
2024-07-02Merge "Revert "Revert "bump the extra --version info from info to notice, ↵Qyriad
-vv -> -v""" into main
2024-07-02Revert "Revert "bump the extra --version info from info to notice, -vv -> -v""Qyriad
This reverts commit 9f16a20f3d35dbc13ba6c8830745e9fc40d408eb. Since c55e93ca2, the original reason for reverting d003dcd7f no longer applies. Change-Id: If88f8555a6060d0530dcfecdf55afaa40afd1ad9