| Age | Commit message (Collapse) | Author |
|---|
|
tl;dr: With this 1 line change I was able to get a speedup of 1.5x on 1Gbit/s
wan connections by enabling zstd compression in nginx.
Also nix already supported all common compression format for http
transfer, webservers usually only enable them if they are advertised
through the Accept-Encoding header.
This pull requests makes nix advertises content compression support for
zstd, br, gzip and deflate.
It's particular useful to add transparent compression for binary caches
that serve packages from the host nix store in particular nix-serve,
nix-serve-ng and harmonia.
I tried so far gzip, brotli and zstd, whereas only zstd was able to bring
me performance improvements for 1Gbit/s WAN connections.
The following nginx configuration was used in combination with the
[zstd module](https://github.com/tokers/zstd-nginx-module) and
[harmonia](https://github.com/nix-community/harmonia/)
```nix
{
services.nginx.virtualHosts."cache.yourhost.com" = {
locations."/".extraConfig = ''
proxy_pass http://127.0.0.1:5000;
proxy_set_header Host $host;
proxy_redirect http:// https://;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
zstd on;
zstd_types application/x-nix-archive;
'';
};
}
```
For testing I unpacked a linux kernel tarball to the nix store using
this command `nix-prefetch-url --unpack https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.1.8.tar.gz`.
Before:
```console
$ nix build && rm -rf /tmp/hello && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwbb88sw36fvah-linux-6.1.8.tar.gz'
warning: Git tree '/scratch/joerg/nix' is dirty
real 0m18,375s
user 0m2,889s
sys 0m1,558s
```
After:
```console
$ nix build && rm -rf /tmp/hello && time ./result/bin/nix copy --no-check-sigs --from https://cache.thalheim.io --to 'file:///tmp/hello?compression=none' '/nix/store/j42mahch5f0jvfmayhzwb
b88sw36fvah-linux-6.1.8.tar.gz'
real 0m11,884s
user 0m4,130s
sys 0m1,439s
```
Signed-off-by: Jörg Thalheim <joerg@thalheim.io>
Update src/libstore/filetransfer.cc
Co-authored-by: Théophane Hufschmitt <7226587+thufschmitt@users.noreply.github.com>
|
|
Co-authored-by: Théophane Hufschmitt <theophane.hufschmitt@tweag.io>
|
|
Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>
|
|
|
|
The property test in fact found a bug: we were excluding numbers!
|
|
|
|
Property tests are great!
Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
|
|
From @Ericson2314.
|
|
Fixes #7624.
|
|
f419ab48e6394838097f158265ac3cc531ee7958 was on the right track, but
there are a few more missing `raw()` calls to fix.
|
|
Failure: https://hydra.nixos.org/build/205357257/nixlog/1
The problem seems to be trying to `std::visit` a derived class of
`std::variant`. Per
https://stackoverflow.com/questions/63616709/incomplete-type-stdvariant-used-in-nested-name-specifier
certain C++ standard library implementations allow this, but others do
not.
The solution is simply to call the `raw` method, which upcasts the
reference back to the `std::variant`.
|
|
Originally there was no `path-info.*`, then there was `path-info.hh`,
then there was `path-info.cc`, but only for new things. Moving this
stuff over makes everything consistent.
|
|
Move the `getBuildLog` implementation to its own implementation file
|
|
`OutputSpec` for `DerivationGoal` and `DerivedPath`, today's `OutputSpec` -> `ExtendedOutputSpec`
|
|
Keep the header minimal and clean
|
|
Ca/fix nix log
|
|
|
|
|
|
|
|
Additionally get rid of the evil time we made an empty
`OutputSpec::Names()`.
|
|
|
|
|
|
This forces us to be explicit.
It also requires to rework how `from_json` works. A `JSON_IMPL` is added
to assist with this.
|
|
This should be a non-empty set, and so we don't want people doing this
by accident. We remove the zero-0 constructor with a little inheritance
trickery.
|
|
`DerivedPath::Built` and `DerivationGoal` were previously using a
regular set with the convention that the empty set means all outputs.
But it is easy to forget about this rule when processing those sets.
Using `OutputSpec` forces us to get it right.
|
|
Do this prior to making a new more limitted `OutputPath` we will use in
more places.
|
|
A few little changes preparing for the rest.
|
|
Parse string context elements properly
|
|
This way the links are clearly within the manual (ie not absolute paths),
while allowing snippets to reference the documentation root reliably,
regardless of at which base url they're included.
|
|
- Add a comment
- Put `OutputsSpec` in a different header (First part of #6815)
- Make a few stray uses of it in new code use `DerivedPath` instead.
|
|
libstore/derivation-goal: Elaborate a TODO for performance concern
|
|
On macOS with auto-uid-allocation and sandboxing, use the correct gid
|
|
Fix deadlock between auto-GC and addTempRoot()
|
|
Fix `nix why-depends --derivation`
|
|
macOS doesn't have user namespacing, so the gid of the builder needs
to be nixbld. The logic got "has sandboxing enabled" confused with
"has user namespaces".
Fixes #7529.
|
|
|
|
This basically reverts 6e5165b77370c76bfa39d4b55e9f83673f3bd466.
It fixes errors like
sandbox-exec: <internal init prelude>:292:47: unable to open sandbox-minimal.sb: not found
when trying to run a development Nix installed in a user's home
directory.
Also, we're trying to minimize the number of installed files
to make it possible to deploy Nix as a single statically-linked
binary.
|
|
Previously addTempRoot() acquired the LocalStore state lock and waited
for the garbage collector to reply. If the garbage collector is in the
same process (as it the case with auto-GC), this would deadlock as
soon as the garbage collector thread needs the LocalStore state lock.
So now addTempRoot() uses separate Syncs for the state that it
needs. As long at the auto-GC thread doesn't call addTempRoot() (which
it shouldn't), it shouldn't deadlock.
Fixes #3224.
|
|
This also moves the file handle into its own Sync object so we're not
holding the _state while acquiring the file lock. There was no real
deadlock risk here since locking a newly created file cannot block,
but it's still a bit nicer.
|
|
src/libstore: Print the reason opening the DB failed
|
|
This has the same goal as b13fd4c58e81b2b2b0d72caa5ce80de861622610,but
achieves it in a different way in order to not break
`nix why-depends --derivation`.
|
|
libstore: Make sure that initNix has been called
|
|
Release shared lock before acquiring exclusive lock
|
|
|
|
doc: sandbox-paths computes closures
|
|
|
|
In principle, this should avoid deadlocks where two instances of Nix are
holding a shared lock on big-lock and are both waiting to get an
exclusive lock.
However, it seems like `flock(2)` is supposed to do this automatically,
so it's not clear whether this is actually where the problem comes from.
|
|
Prevent bugs like https://github.com/cachix/cachix/pull/477
|
|
This makes 'nix develop' set the Linux personality in the same way
that the actual build does, allowing a command like 'nix develop
nix#devShells.i686-linux.default' on x86_64-linux to work correctly.
|
|
Without this, the error is lost, and it makes for a hard to debug
situation. Also remove some of the busyness inside the sqlite_open_v2
args.
The errcode returned is not the extended one. The only way to make open
return an extended code, would be to add SQLITE_OPEN_EXRESCODE to the
flags. In the future it might be worth making this change,
which would also simplify the existing SQLiteError code.
|
