aboutsummaryrefslogtreecommitdiff
path: root/src/libstore/content-address.cc
AgeCommit message (Collapse)Author
2024-08-28tree-wide: shuffle headers around for about 30s compile timeJade Lovelace
This didn't really feel so worth it afterwards, but I did untangle a bunch of stuff that should not have been tangled. The general gist of this change is that variant bullshit was causing a bunch of compile time, and it seems like the only way to deal with variant induced compile time is to keep variant types out of headers. Explicit template instantiation seems to do nothing for them. I also seem to have gotten some back-end time improvement from explicitly instantiating regex, but I don't know why. There is no corresponding front-end time improvement from it: regex is still at the top of the sinners list. **** Templates that took longest to instantiate: 15231 ms: std::basic_regex<char>::_M_compile (28 times, avg 543 ms) 15066 ms: std::__detail::_Compiler<std::regex_traits<char>>::_Compiler (28 times, avg 538 ms) 12571 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_disjunction (28 times, avg 448 ms) 12454 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_alternative (28 times, avg 444 ms) 12225 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_term (28 times, avg 436 ms) 11363 ms: nlohmann::basic_json<>::parse<const char *> (21 times, avg 541 ms) 10628 ms: nlohmann::basic_json<>::basic_json (109 times, avg 97 ms) 10134 ms: std::__detail::_Compiler<std::regex_traits<char>>::_M_atom (28 times, avg 361 ms) Back-end time before messing with the regex: **** Function sets that took longest to compile / optimize: 8076 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms) 4382 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1247 times, avg 3 ms) 3137 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 22 ms) 2896 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 16 ms) 2304 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (210 times, avg 10 ms) 2116 ms: bool std::__detail::_Compiler<$>::_M_expression_term<$>(std::__detai... (112 times, avg 18 ms) 2051 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (244 times, avg 8 ms) 2037 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 21 ms) 1928 ms: std::__detail::_Compiler<$>::_M_quantifier() (28 times, avg 68 ms) 1859 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (41 times, avg 45 ms) 1824 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (973 times, avg 1 ms) 1810 ms: std::__detail::_BracketMatcher<$>::_BracketMatcher(std::__detail::_B... (112 times, avg 16 ms) 1793 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1793 ms) 1759 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (281 times, avg 6 ms) 1722 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 90 ms) 1677 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms) 1674 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (249 times, avg 6 ms) 1660 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (304 times, avg 5 ms) 1599 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (19 times, avg 84 ms) 1568 ms: void std::__detail::_Compiler<$>::_M_insert_bracket_matcher<$>(bool) (112 times, avg 14 ms) 1541 ms: std::__shared_ptr<$>::~__shared_ptr() (531 times, avg 2 ms) 1539 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (41 times, avg 37 ms) 1471 ms: void std::__detail::_Compiler<$>::_M_insert_character_class_matcher<... (112 times, avg 13 ms) After messing with the regex (notice std::__detail::_Compiler vanishes here, but I don't know why): **** Function sets that took longest to compile / optimize: 8054 ms: void boost::io::detail::put<$>(boost::io::detail::put_holder<$> cons... (177 times, avg 45 ms) 4313 ms: std::_Rb_tree<$>::_M_erase(std::_Rb_tree_node<$>*) (1217 times, avg 3 ms) 3259 ms: boost::stacktrace::detail::to_string_impl_base<boost::stacktrace::de... (137 times, avg 23 ms) 3045 ms: void boost::io::detail::mk_str<$>(std::__cxx11::basic_string<$>&, ch... (177 times, avg 17 ms) 2314 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (207 times, avg 11 ms) 1923 ms: std::_Rb_tree_iterator<$> std::_Rb_tree<$>::_M_emplace_hint_unique<$... (216 times, avg 8 ms) 1817 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 100 ms) 1816 ms: toml::result<$> toml::detail::sequence<$>::invoke<$>(toml::detail::l... (93 times, avg 19 ms) 1788 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump(nlohmann::js... (40 times, avg 44 ms) 1749 ms: std::_Rb_tree<$>::_M_get_insert_unique_pos(std::__cxx11::basic_strin... (278 times, avg 6 ms) 1724 ms: std::__cxx11::basic_string<$>::_M_mutate(unsigned long, unsigned lon... (248 times, avg 6 ms) 1697 ms: boost::io::basic_altstringbuf<$>::overflow(int) (194 times, avg 8 ms) 1684 ms: nix::fetchers::GitInputScheme::fetch(nix::ref<$>, nix::fetchers::Inp... (1 times, avg 1684 ms) 1680 ms: std::_Rb_tree_node<$>* std::_Rb_tree<$>::_M_copy<$>(std::_Rb_tree_no... (303 times, avg 5 ms) 1589 ms: bool nlohmann::json_abi_v3_11_3::detail::parser<$>::sax_parse_intern... (18 times, avg 88 ms) 1483 ms: non-virtual thunk to boost::wrapexcept<$>::~wrapexcept() (181 times, avg 8 ms) 1447 ms: nlohmann::json_abi_v3_11_3::detail::serializer<$>::dump_escaped(std:... (40 times, avg 36 ms) 1441 ms: std::__shared_ptr<$>::~__shared_ptr() (496 times, avg 2 ms) 1420 ms: boost::stacktrace::basic_stacktrace<$>::init(unsigned long, unsigned... (137 times, avg 10 ms) 1396 ms: boost::basic_format<$>::~basic_format() (194 times, avg 7 ms) 1290 ms: std::__cxx11::basic_string<$>::_M_replace_cold(char*, unsigned long,... (231 times, avg 5 ms) 1258 ms: std::vector<$>::~vector() (354 times, avg 3 ms) 1222 ms: std::__cxx11::basic_string<$>::_M_replace(unsigned long, unsigned lo... (231 times, avg 5 ms) 1194 ms: std::_Rb_tree<$>::_M_get_insert_hint_unique_pos(std::_Rb_tree_const_... (49 times, avg 24 ms) 1186 ms: bool tao::pegtl::internal::sor<$>::match<$>(std::integer_sequence<$>... (1 times, avg 1186 ms) 1149 ms: std::__detail::_Executor<$>::_M_dfs(std::__detail::_Executor<$>::_Ma... (70 times, avg 16 ms) 1123 ms: toml::detail::sequence<$>::invoke(toml::detail::location&) (69 times, avg 16 ms) 1110 ms: nlohmann::json_abi_v3_11_3::basic_json<$>::json_value::destroy(nlohm... (55 times, avg 20 ms) 1079 ms: std::_Function_handler<$>::_M_manager(std::_Any_data&, std::_Any_dat... (541 times, avg 1 ms) 1033 ms: nlohmann::json_abi_v3_11_3::detail::lexer<$>::scan_number() (20 times, avg 51 ms) Change-Id: I10af282bcd4fc39c2d3caae3453e599e4639c70b
2024-08-08refactor: make HashType and Base enum classes for type safetyJade Lovelace
Change-Id: I9fbd55a9d50464a56fe11cb42a06a206914150d8
2024-03-29Fix various clang-tidy lintsJade Lovelace
* some things that can throw are marked noexcept yet the linter seems to think not. Maybe they can't throw in practice. I would rather not have the UB possibility in pretty obvious cold paths. * various default-case-missing complaints * a fair pile of casts from integer to character, which are in fact deliberate. * an instance of <https://clang.llvm.org/extra/clang-tidy/checks/bugprone/move-forwarding-reference.html> * bugprone-not-null-terminated-result on handing a string to curl in chunks of bytes. our usage is fine. * reassigning a unique_ptr by CRIMES instead of using release(), then using release() and ignoring the result. wild. let's use release() for its intended purpose. Change-Id: Ic3e7affef12383576213a8a7c8145c27e662513d
2024-03-04Merge pull request #9229 from tfc/small-improvementseldritch horrors
Remove warnings, small improvements (cherry picked from commit 5ac87a75dd65c19c50976559a6f7810c315cd9d5) Change-Id: I88349b6e954398dde83c845f42d41a9dd89ba9e0
2024-03-04Merge pull request #9172 from tfc/bad-moveseldritch horrors
Fix/remove some bad std::moves (cherry picked from commit 8c049a9f044569ebda70231709f6f15d3073894a) Change-Id: I720273378d2506a13883acee28abd096d099b0d4
2023-08-18Fixing #7479John Ericson
Types converted: - `NixStringContextElem` - `OutputsSpec` - `ExtendedOutputsSpec` - `DerivationOutput` - `DerivationType` Existing ones mostly conforming the pattern cleaned up: - `ContentAddressMethod` - `ContentAddressWithReferences` The `DerivationGoal::derivationType` field had a bogus initialization, now caught, so I made it `std::optional`. I think #8829 can make it non-optional again because it will ensure we always have the derivation when we construct a `DerivationGoal`. See that issue (#7479) for details on the general goal. `git grep 'Raw::Raw'` indicates the two types I didn't yet convert `DerivedPath` and `BuiltPath` (and their `Single` variants) . This is because @roberth and I (can't find issue right now...) plan on reworking them somewhat, so I didn't want to churn them more just yet. Co-authored-by: Eelco Dolstra <edolstra@gmail.com>
2023-07-07Simplify `ContentAddress`John Ericson
Whereas `ContentAddressWithReferences` is a sum type complex because different varieties support different notions of reference, and `ContentAddressMethod` is a nested enum to support that, `ContentAddress` can be a simple pair of a method and hash. `ContentAddress` does not need to be a sum type on the outside because the choice of method doesn't effect what type of hashes we can use. Co-Authored-By: Cale Gibbard <cgibbard@gmail.com>
2023-05-09Improve error message for self reference with text hashingJohn Ericson
The `ContentAddressWithReferences` method is made total, with error handling now squarely the caller's job. This is better.
2023-05-09Cleanups to content address typesJohn Ericson
2023-04-19Remove references from fixed output derivation ab syntaxJohn Ericson
In other words, use a plain `ContentAddress` not `ContentAddressWithReferences` for `DerivationOutput::CAFixed`. Supporting fixed output derivations with (fixed) references would be a cool feature, but it is out of scope at this moment.
2023-04-19Add a few more content addressing methodsJohn Ericson
Good to round out the library interface.
2023-04-17`TextHashMethod` -> `TextIngestionMethod`, gate with XP featureJohn Ericson
I suppose we can use `dynamic-derivations` for the few things we neeed.
2023-04-01Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2023-04-01Merge commit 'aa99005004bccc9be506a2a2f162f78bad4bcb41' into ca-drv-exoticJohn Ericson
2023-03-30Use "raw pattern" for content address typesJohn Ericson
We weren't because this ancient PR predated it! This is actually a new version of the pattern which addresses some issues identified in #7479.
2023-03-30Merge remote-tracking branch 'upstream/master' into path-infoJohn Ericson
Also improve content-address.hh API docs.
2023-02-28Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2023-02-28No inheritance for `TextInfo` and `FixedOutputInfo`John Ericson
2023-02-28Revert "Remove some designated initializers"John Ericson
This reverts commit ee9eb83a842eb97d0180fd9d349d30ff27fdb485.
2023-02-01Remove some designated initializersJohn Ericson
With the switch to C++20, the rules became more strict, and we can no longer initialize base classes. Make them comments instead. (BTW https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p2287r1.html this offers some new syntax for this use-case. Hopefully this will be adopted and we can eventually use it.)
2023-01-30Shrink diff in two placesJohn Ericson
Stuff crept in there.
2023-01-14Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2023-01-14Make `ValidPathInfo` have plain `StorePathSet` references like beforeJohn Ericson
This change can wait for another PR.
2023-01-06Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2023-01-06Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2023-01-06Use named field initialization for referencesJohn Ericson
2022-04-19Slight cleanupsJohn Ericson
2022-03-10Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2022-03-10Merge remote-tracking branch 'upstream/master' into path-infoJohn Ericson
2021-11-23Fix use after free in content-address.ccAlex Shabalin
Inspired by https://github.com/NixOS/nix/pull/5599
2021-10-01Merge branch 'path-info' into ca-drv-exoticJohn Ericson
2021-10-01Merge remote-tracking branch 'upstream/master' into path-infoJohn Ericson
2021-09-30`std::visit` by referenceJohn Ericson
I had started the trend of doing `std::visit` by value (because a type error once mislead me into thinking that was the only form that existed). While the optomizer in principle should be able to deal with extra coppying or extra indirection once the lambdas inlined, sticking with by reference is the conventional default. I hope this might even improve performance.
2020-10-13Work around clang destructing + capturing bug againJohn Ericson
2020-10-13Derivations can output "text-hashed" dataJohn Ericson
In particular, this means that derivations can output derivations. But that ramification isn't (yet!) useful as we would want, since there is no way to have a dependent derivation that is itself a dependent derivation.
2020-10-07Use PathReferences more widelyJohn Ericson
2020-09-22Style fixesEelco Dolstra
2020-09-21parseContentAddressMethodPrefix: use string_viewRobert Hensing
Co-authored-by: John Ericson <git@JohnEricson.me>
2020-09-17Add ContentAddressMethod and parse/render itRobert Hensing
2020-08-05Merge branch 'misc-ca' of https://github.com/obsidiansystems/nixEelco Dolstra
2020-08-05Style fixEelco Dolstra
2020-08-05Merge remote-tracking branch 'upstream/master' into misc-caJohn Ericson
2020-07-27Merge remote-tracking branch 'upstream/master' into ca-derivation-data-typesJohn Ericson
2020-07-20parser.hh -> split.hhJohn Ericson
2020-07-13Merge branch 'hash-always-has-type' into better-ca-parse-errorsJohn Ericson
2020-07-13Merge remote-tracking branch 'upstream/master' into hash-always-has-typeJohn Ericson
2020-07-12Move C++17 "pattern matching" boilerplat to utils.hhJohn Ericson
2020-07-10Factor out commonality between nix-prefetch-url and nix-store --add-fixedEelco Dolstra
2020-07-02Have `splitPrefix` and `splitPrefixTo` parser helpersJohn Ericson
2020-07-02Tighten parsing for drv files and pathinfoCarlo Nucera