aboutsummaryrefslogtreecommitdiff
path: root/src/libexpr/lexer.l
AgeCommit message (Collapse)Author
2020-04-22a few more 'format's rremovedBen Burdette
2018-10-27simplify handling of extra '}'Guillaume Maudoux
2018-08-29libexpr: Use int64_t for NixIntaszlig
Using a 64bit integer on 32bit systems will come with a bit of a performance overhead, but given that Nix doesn't use a lot of integers compared to other types, I think the overhead is negligible also considering that 32bit systems are in decline. The biggest advantage however is that when we use a consistent integer size across all platforms it's less likely that we miss things that we break due to that. One example would be: https://github.com/NixOS/nixpkgs/pull/44233 On Hydra it will evaluate, because the evaluator runs on a 64bit machine, but when evaluating the same on a 32bit machine it will fail, so using 64bit integers should make that consistent. While the change of the type in value.hh is rather easy to do, we have a few more options available for doing the conversion in the lexer: * Via an #ifdef on the architecture and using strtol() or strtoll() accordingly depending on which architecture we are. For the #ifdef we would need another AX_COMPILE_CHECK_SIZEOF in configure.ac. * Using istringstream, which would involve copying the value. * As we're already using boost, lexical_cast might be a good idea. Spoiler: I went for the latter, first of all because lexical_cast does have an overload for const char* and second of all, because it doesn't involve copying around the input string. Also, because istringstream seems to come with a bigger overhead than boost::lexical_cast: https://www.boost.org/doc/libs/release/doc/html/boost_lexical_cast/performance.html The first method (still using strtol/strtoll) also wasn't something I pursued further, because it is also locale-aware which I doubt is what we want, given that the regex for int is [0-9]+. Signed-off-by: aszlig <aszlig@nix.build> Fixes: #2339
2018-05-11Don't return negative numbers from the flex tokenizerEelco Dolstra
Fixes #1374. Closes #2129.
2018-05-11Revert "Throw a specific error for incomplete parse errors."Eelco Dolstra
This reverts commit 6498adb002bcf7e715afe46c23b8635d4592c156. We don't actually use IncompleteParseError in 'nix repl'.
2018-03-02libexpr: Recognize newline in more places in lexerTuomas Tynkkynen
Flex's regexes have an annoying feature: the dot matches everything except a newline. This causes problems for expressions like: "${0}\ " where the backslash-newline combination matches this rule instead of the intended one mentioned in the comment: <STRING>\$|\\|\$\\ { /* This can only occur when we reach EOF, otherwise the above (...|\$[^\{\"\\]|\\.|\$\\.)+ would have triggered. This is technically invalid, but we leave the problem to the parser who fails with exact location. */ return STR; } However, the parser actually accepts the resulting token sequence ('"' DOLLAR_CURLY 0 '}' STR '"'), which is a problem because the lexer rule didn't assign anything to yylval. Ultimately this leads to a crash when dereferencing a NULL pointer in ExprConcatStrings::bindVars(). The fix does change the syntax of the language in some corner cases but I think it's only turning previously invalid (or crashing) syntax to valid syntax. E.g. "a\ b" and ''a''\ b'' were previously syntax errors but now both result in "a\nb". Found by afl-fuzz.
2018-02-16libexpr: Pre-reserve space in string in unescapeStr()Tuomas Tynkkynen
Avoids some malloc() traffic.
2017-11-14Revert "Don't parse "x:x" as a URI"Eelco Dolstra
This reverts commit f90f660b243866b8860eeb24cc4a345d32cc7ce7. This broke Hydra's release.nix, which contained preCheck = ''export LOGNAME=${LOGNAME:-foo}'';
2017-10-30Don't parse "x:x" as a URIEelco Dolstra
URIs now have to contain "://" or start with "channel:".
2017-07-30Replace Unicode quotes in user-facing strings by ASCIIJörg Thalheim
Relevant RFC: NixOS/rfcs#4 $ ag -l | xargs sed -i -e "/\"/s/’/'/g;/\"/s/‘/'/g"
2017-05-01lexer: remove catch-all rules hiding real errorsGuillaume Maudoux
With catch-all rules, we hide potential errors. It turns out that a4744254 made one cath-all useless. Flex detected that is was impossible to reach. The other is more subtle, as it can only trigger on unfinished escapes in unfinished strings, which only occurs at EOF.
2017-05-01Fix lexer to support `$'` in multiline strings.Guillaume Maudoux
2016-12-06Tweak error messageEelco Dolstra
2016-11-27Improve error message on trailing path slashesGuillaume Maudoux
2016-11-13Fix comments parsingGuillaume Maudoux
Fixed the parsing of multiline strings ending with an even number of stars, like /** this **/. Added test cases for comments.
2016-02-24Throw a specific error for incomplete parse errors.Scott Olson
`nix-repl` will use this for deciding whether to keep waiting for input or error out right away.
2016-02-12Merge pull request #762 from ctheune/ctheune-floatsEelco Dolstra
Implement floats
2016-01-20Revert "Revert "next try for "don't abort when given unmatched '}' with ↵Eelco Dolstra
'start-condition stack underflow'. This fixes #751""" This reverts commit b669d3d2e83d3c50238751b57cff3ed0ca39bc8a.
2016-01-20Revert "next try for "don't abort when given unmatched '}' with ↵Eelco Dolstra
'start-condition stack underflow'. This fixes #751"" This reverts commit ed23c8568e10d15196bb4ff2b79fc14191d28109. Let's merge this *after* the 1.11.1 release.
2016-01-19next try for "don't abort when given unmatched '}' with 'start-condition ↵Fabian Schmitthenner
stack underflow'. This fixes #751" This reverts commit 8120b6fb8a4924f8ae717bba9bbda4a2f89e2141 and fixes the regression introduced in 8d22b26448a091c76ab972c0b0603daac5e255e4.
2016-01-19Revert "don't abort when given unmatched '}' with 'start-condition stack ↵Eelco Dolstra
underflow'. This fixes #751" This reverts commit 8d22b26448a091c76ab972c0b0603daac5e255e4. It breaks Nixpkgs: $ nix-env -qa error: syntax error, unexpected IND_STR, expecting '}', at /home/eelco/Dev/nixpkgs-stable/pkgs/top-level/python-packages.nix:7605:8
2016-01-12don't abort when given unmatched '}' with 'start-condition stack underflow'. ↵Fabian Schmitthenner
This fixes #751
2016-01-05Edge condition: parser did not pick up floats starting exactly with 0.Christian Theune
2016-01-05Fix up float parsing.Christian Theune
2016-01-05Try a simplified version of float lexing that didn't work.Christian Theune
The last one I tried was botchered anyway ...
2016-01-05First hit at providing support for floats in the language.Christian Theune
2015-07-03Fix the parsing of "$"'s in strings.Guillaume Maudoux
2015-07-03Fix the hack that resets the scanner state.Guillaume Maudoux
2015-02-19Allow the leading component of a path to be a ~Shea Levy
2014-08-20Use proper quotes everywhereEelco Dolstra
2014-01-14Allow "bare" dynamic attrsShea Levy
Now, in addition to a."${b}".c, you can write a.${b}.c (applicable wherever dynamic attributes are valid). Signed-off-by: Shea Levy <shea@shealevy.com>
2013-09-02Fix whitespaceEelco Dolstra
2013-08-19Store Nix integers as longsEelco Dolstra
So on 64-bit systems, integers are now 64-bit. Fixes #158.
2013-08-02Add comparison operators ‘<’, ‘<=’, ‘>’ and ‘>=’Eelco Dolstra
2013-03-14Fix building against Bison 2.6Eelco Dolstra
2012-09-27Allow dashes in identifiersEelco Dolstra
In Nixpkgs, the attribute in all-packages.nix corresponding to a package is usually equal to the package name. However, this doesn't work if the package contains a dash, which is fairly common. The convention is to replace the dash with an underscore (e.g. "dbus-lib" becomes "dbus_glib"), but that's annoying. So now dashes are valid in variable / attribute names, allowing you to write: dbus-glib = callPackage ../development/libraries/dbus-glib { }; and buildInputs = [ dbus-glib ]; Since we don't have a negation or subtraction operation in Nix, this is unambiguous.
2011-08-06* Add a Nix expression search path feature. Paths between angleEelco Dolstra
brackets, e.g. import <nixpkgs/pkgs/lib> are resolved by looking them up relative to the elements listed in the search path. This allows us to get rid of hacks like import "${builtins.getEnv "NIXPKGS_ALL"}/pkgs/lib" The search path can be specified through the ‘-I’ command-line flag and through the colon-separated ‘NIX_PATH’ environment variable, e.g., $ nix-build -I /etc/nixos ... If a file is not found in the search path, an error message is lazily thrown.
2011-07-13* Allow a default value in attribute selection by writingEelco Dolstra
x.y.z or default (as originally proposed in https://mail.cs.uu.nl/pipermail/nix-dev/2009-September/002989.html). For instance, an expression like stdenv.lib.attrByPath ["features" "ckSched"] false args can now be written as args.features.ckSched or false
2010-10-23* Optimise string constants by putting them in the symbol table.Eelco Dolstra
2010-05-06* Store attribute positions in the AST and report duplicate attributeEelco Dolstra
errors with position info. * For all positions, use the position of the first character of the first token, rather than the last character of the first token plus one.
2010-04-21* Fix the interpretation of ''\<character> in indented strings.Eelco Dolstra
2010-04-12* Indented strings.Eelco Dolstra
2010-04-12* More missing constructs.Eelco Dolstra
2010-04-12* Don't use ATerms for the abstract syntax trees anymore. NotEelco Dolstra
finished yet.
2010-04-12* Don't use ATerms to represent integers in the lexer.Eelco Dolstra
2008-08-14* Added an experimental feature suggested by Andres: ellipses ("...")Eelco Dolstra
in attribute set pattern matches. This allows defining a function that takes *at least* the listed attributes, while ignoring additional attributes. For instance, {stdenv, fetchurl, fuse, ...}: stdenv.mkDerivation { ... }; defines a function that requires an attribute set that contains the specified attributes but ignores others. The main advantage is that we can then write in all-packages.nix aefs = import ../bla/aefs pkgs; instead of aefs = import ../bla/aefs { inherit stdenv fetchurl fuse; }; This saves a lot of typing (not to mention not having to update all-packages.nix with purely mechanical changes). It saves as much typing as the "args: with args;" style, but has the advantage that the function arguments are properly declared (not implicit in what the body of the "with" uses).
2008-02-05* Fix the parsing ofEelco Dolstra
'' '${foo}' '' where the antiquote should work as expected, instead of giving the string "'${foo}'".
2007-12-06* Syntax to escape '', ${.Eelco Dolstra
2007-11-30* Added a new kind of multi-line string literal delimited by twoEelco Dolstra
single quotes. Example (from NixOS): job = '' start on network-interfaces start script rm -f /var/run/opengl-driver ${if videoDriver == "nvidia" then "ln -sf ${nvidiaDrivers} /var/run/opengl-driver" else if cfg.driSupport then "ln -sf ${mesa} /var/run/opengl-driver" else "" } rm -f /var/log/slim.log end script ''; This style has two big advantages: - \, ' and " aren't special, only '' and ${. So you get a lot less escaping in shell scripts / configuration files in Nixpkgs/NixOS. The delimiter '' is rare in scripts (and can usually be written as ""). ${ is also fairly rare. Other delimiters such as <<...>>, {{...}} and <|...|> were also considered but this one appears to have the fewest drawbacks (thanks Martin). - Indentation is intelligently stripped so that multi-line strings can follow the nesting structure of the containing Nix expression. E.g. in the example above 6 spaces are stripped from the start of each line. This prevents unnecessary indentation in generated files (which sometimes even breaks things). See tests/lang/eval-okay-ind-string.nix for some examples.
2006-10-16* Big cleanup of the semantics of paths, strings, contexts, stringEelco Dolstra
concatenation and string coercion. This was a big mess (see e.g. NIX-67). Contexts are now folded into strings, so that they don't cause evaluation errors when they're not expected. The semantics of paths has been clarified (see nixexpr-ast.def). toString() and coerceToString() have been merged. Semantic change: paths are now copied to the store when they're in a concatenation (and in most other situations - that's the formalisation of the meaning of a path). So "foo " + ./bla evaluates to "foo /nix/store/hash...-bla", not "foo /path/to/current-dir/bla". This prevents accidental impurities, and is more consistent with the treatment of derivation outputs, e.g., `"foo " + bla' where `bla' is a derivation. (Here `bla' would be replaced by the output path of `bla'.)