diff options
author | eldritch horrors <pennae@lix.systems> | 2024-06-16 23:10:09 +0200 |
---|---|---|
committer | eldritch horrors <pennae@lix.systems> | 2024-06-25 12:24:58 +0000 |
commit | e6cd67591b44b4902bac73febcab3c4d96724aea (patch) | |
tree | 94c8ad90b8e756c5b00b8d68b2adf13c0f2febd9 /meson | |
parent | c097ebe66bf474da886ffa20d2f31bdb1d2196a8 (diff) |
libexpr: rewrite the parser with pegtl instead of flex/bison
this gives about 20% performance improvements on pure parsing. obviously
it will be less on full eval, but depending on how much parsing is to be
done (e.g. including hackage-packages.nix or not) it's more like 4%-10%.
this has been tested (with thousands of core hours of fuzzing) to ensure
that the ASTs produced by the new parser are exactly the same as the old
one would have produced. error messages will change (sometimes by a lot)
and are not yet perfect, but we would rather leave this as is for later.
test results for running only the parser (excluding the variable binding
code) in a tight loop with inputs and parameters as given are promising:
- 40% faster on lix's package.nix at 10000 iterations
- 1.3% faster on nixpkgs all-packages.nix at 1000 iterations
- equivalent on all of nixpkgs concatenated at 100 iterations
(excluding invalid files, each file surrounded with parens)
more realistic benchmarks are somewhere in between the extremes, parsing
once again getting the largest uplift. other realistic workloads improve
by a few percentage points as well, notably system builds are 4% faster.
Benchmarks summary (from ./bench/summarize.jq bench/bench-*.json)
old/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.408s ± 0.025s
user: 0.355s | system: 0.033s
median: 0.389s
range: 0.388s ... 0.442s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix
mean: 0.332s ± 0.024s
user: 0.279s | system: 0.033s
median: 0.314s
range: 0.313s ... 0.361s
relative: 0.814
---
old/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 6.133s ± 0.022s
user: 5.395s | system: 0.437s
median: 6.128s
range: 6.099s ... 6.183s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 5.925s ± 0.025s
user: 5.176s | system: 0.456s
median: 5.934s
range: 5.861s ... 5.943s
relative: 0.966
---
GC_INITIAL_HEAP_SIZE=10g old/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.503s ± 0.027s
user: 3.731s | system: 0.547s
median: 4.499s
range: 4.478s ... 4.541s
relative: 1
GC_INITIAL_HEAP_SIZE=10g new/bin/nix eval --extra-experimental-features 'nix-command flakes' --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
mean: 4.285s ± 0.031s
user: 3.504s | system: 0.571s
median: 4.281s
range: 4.221s ... 4.328s
relative: 0.951
---
old/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
mean: 16.475s ± 0.07s
user: 14.088s | system: 1.572s
median: 16.495s
range: 16.351s ... 16.536s
relative: 1
new/bin/nix --extra-experimental-features 'nix-command flakes' search --no-eval-cache github:nixos/nixpkgs/e1fa12d4f6c6fe19ccb59cac54b5b3f25e160870 hello
mean: 15.973s ± 0.013s
user: 13.558s | system: 1.615s
median: 15.973s
range: 15.946s ... 15.99s
relative: 0.97
---
Change-Id: Ie66ec2d045dec964632c6541e25f8f0797319ee2
Diffstat (limited to 'meson')
-rwxr-xr-x | meson/cleanup-install.bash | 50 |
1 files changed, 0 insertions, 50 deletions
diff --git a/meson/cleanup-install.bash b/meson/cleanup-install.bash deleted file mode 100755 index 928edc74a..000000000 --- a/meson/cleanup-install.bash +++ /dev/null @@ -1,50 +0,0 @@ -#!/usr/bin/env bash -# Meson will call this with an absolute path to Bash. -# The shebang is just for convenience. - -# The parser and lexer tab are generated via custom Meson targets in src/libexpr/meson.build, -# but Meson doesn't support marking only part of a target for install. The generation creates -# both headers (parser-tab.hh, lexer-tab.hh) and source files (parser-tab.cc, lexer-tab.cc), -# and we definitely want the former installed, but not the latter. This script is added to -# Meson's install steps to correct this, as the logic for it is just complex enough to -# warrant separate and careful handling, because both Meson's configured include directory -# may or may not be an absolute path, and DESTDIR may or may not be set at all, but can't be -# manipulated in Meson logic. - -set -euo pipefail - -echo "cleanup-install: removing Meson-placed C++ sources from dest includedir" - -if [[ "${1/--help/}" != "$1" ]]; then - echo "cleanup-install: this script should only be called from the Meson build system" - exit 1 -fi - -# Ensure the includedir was passed as the first argument -# (set -u will make this fail otherwise). -includedir="$1" -# And then ensure that first argument is a directory that exists. -if ! [[ -d "$1" ]]; then - echo "cleanup-install: this script should only be called from the Meson build system" - echo "argv[1] (${1@Q}) is not a directory" - exit 2 -fi - -# If DESTDIR environment variable is set, prepend it to the include dir. -# Unfortunately, we cannot do this on the Meson side. We do have an environment variable -# `MESON_INSTALL_DESTDIR_PREFIX`, but that will not refer to the include directory if -# includedir has been set separately, which Lix's split-output derivation does. -# We also cannot simply do an inline bash conditional like "${DESTDIR:=}" or similar, -# because we need to specifically *join* DESTDIR and includedir with a slash, and *not* -# have a slash if DESTDIR isn't set at all, since $includedir could be a relative directory. -# Finally, DESTDIR is only available to us as an environment variable in these install scripts, -# not in Meson logic. -# Therefore, our best option is to have Meson pass this script the configured includedir, -# and perform this dance with it and $DESTDIR. -if [[ -n "${DESTDIR:-}" ]]; then - includedir="$DESTDIR/$includedir" -fi - -# Intentionally not using -f. -# If these files don't exist then our assumptions have been violated and we should fail. -rm -v "$includedir/lix/libexpr/parser-tab.cc" "$includedir/lix/libexpr/lexer-tab.cc" |