aboutsummaryrefslogtreecommitdiff
path: root/src/libstore/remote-store.cc
AgeCommit message (Collapse)Author
2021-07-22RemoteStore::buildPaths(): Handle evalStoreEelco Dolstra
2021-07-22buildPaths(): Add an evalStore argumentEelco Dolstra
With this, we don't have to copy the entire .drv closure to the destination store ahead of time (or at all). Instead, buildPaths() reads .drv files from the eval store and copies inputSrcs to the destination store if it needs to build a derivation. Issue #5025.
2021-07-22Hacky fast closure copying mechanismEelco Dolstra
2021-05-19Always send the realisations as JSONregnat
Align all the worker protocol with `buildDerivation` which inlines the realisations as one opaque json blob. That way we don’t have to bother changing the remote store protocol when the definition of `Realisation` changes, as long as we keep the json backwards-compatible
2021-05-03Bump version number for `DerivedPath` changesMaximilian Bosch
I guess I misunderstood John's initial explanation about why wildcards for outputs are sent to older stores[1]. My `nix-daemon` from 2021-03-26 also has version 1.29, but misses the wildcard[2]. So bumping seems to be the right call. [1] https://github.com/NixOS/nix/pull/4759#issuecomment-830812464 [2] 255d145ba7ac907d1cba8d088da556b591627756
2021-04-05Rename BuildableJohn Ericson
2021-04-05"newtype" BuildableReqJohn Ericson
This makes for better types errors and allows us to give it methods.
2021-04-05Use `BuildableReq` for `buildPaths` and `ensurePath`John Ericson
This avoids an ambiguity where the `StorePathWithOutputs { drvPath, {} }` could mean "build `brvPath`" or "substitute `drvPath`" depending on context. It also brings the internals closer in line to the new CLI, by generalizing the `Buildable` type is used there and makes that distinction already. In doing so, relegate `StorePathWithOutputs` to being a type just for backwards compatibility (CLI and RPC).
2021-03-22Fix typos in the last PR #4656John Ericson
2021-03-22Clean up serialization for `BuildResult`John Ericson
A few versioning mistakes were corrected: - In 27b5747ca7b5599768083dde5fa4d36bfbb0f66f, Daemon protocol had some version `>= 0xc` that should have been `>= 0x1c`, or `28` since the other conditions used decimal. - In a2b69660a9b326b95d48bd222993c5225bbd5b5f, legacy SSH gated new CAS info on version 6, but version 5 in the server. It is now 6 everywhere. Additionally, legacy ssh was sending over more metadata than the daemon one was. The daemon now sends that data too. CC @regnat Co-authored-by: Cole Helbling <cole.e.helbling@outlook.com>
2021-02-23RemoteStore: Send back the new realisationsregnat
To allow it to build ca derivations remotely
2021-02-23Make the build-hook work with ca derivationsregnat
- Pass it the name of the outputs rather than their output paths (as these don't exist for ca derivations) - Get the built output paths from the remote builder - Register the new received realisations
2020-12-20Overhaul store subclassingJohn Ericson
We embrace virtual the rest of the way, and get rid of the `assert(false)` 0-param constructors. We also list config base classes first, so the constructor order is always: 1. all the configs 2. all the stores Each in the same order
2020-12-11Store metadata about drv outputs realisationsregnat
For each known realisation, store: - its output - its output path This comes with a set of needed changes: - New `realisations` module declaring the types needed for describing these mappings - New `Store::registerDrvOutput` method registering all the needed informations about a derivation output (also replaces `LocalStore::linkDeriverToPath`) - new `Store::queryRealisation` method to retrieve the informations for a derivations This introcudes some redundancy on the remote-store side between `wopQueryDerivationOutputMap` and `wopQueryRealisation`. However we might need to keep both (regardless of backwards compat) because we sometimes need to get some infos for all the outputs of a derivation (where `wopQueryDerivationOutputMap` is handy), but all the stores can't implement it − because listing all the outputs of a derivation isn't really possible for binary caches where the server doesn't allow to list a directory.
2020-12-02read(): Use char * instead of unsigned char *Eelco Dolstra
This gets rid of some pointless casts.
2020-12-02Sink: Use std::string_viewEelco Dolstra
2020-11-17Merge pull request #4180 from Ma27/ssh-ng-substituteEelco Dolstra
Allow substituting paths when building remotely using `ssh-ng://`
2020-11-16Fix deadlock in nix-store when max-connections=1Jake Waksbaum
This fixes a bug I encountered where `nix-store -qR` will deadlock when the `--include-outputs` flag is passed and `max-connections=1`. The deadlock occurs because `RemoteStore::queryDerivationOutputs` takes the only connection from the connection pool and uses it to check the daemon version. If the version is new enough, it calls `Store::queryDerivationOutputs`, which eventually calls `RemoteStore::queryPartialDerivationOutputMap`, where we take another connection from the connection pool to check the version again. Because we still haven't released the connection from the caller, this waits for a connection to be available, causing a deadlock. This diff solves the issue by using `getProtocol` to check the protocol version in the caller `RemoteStore::queryDerivationOutputs`, which immediately frees the connection back to the pool before returning the protocol version. That way we've already freed the connection by the time we call `RemoteStore::queryPartialDerivationOutputMap`.
2020-11-05Allow substituting paths when building remotely using `ssh-ng://`Maximilian Bosch
Until now, it was not possible to substitute missing paths from e.g. `https://cache.nixos.org` on a remote server when building on it using the new `ssh-ng` protocol. This is because every store implementation except legacy `ssh://` ignores the substitution flag passed to `Store::queryValidPaths` while the `legacy-ssh-store` substitutes the remote store using `cmdQueryValidPaths` when the remote store is opened with `nix-store --serve`. This patch slightly modifies the daemon protocol to allow passing an integer value suggesting whether to substitute missing paths during `wopQueryValidPaths`. To implement this on the daemon-side, the substitution logic from `nix-store --serve` has been moved into a protected method named `Store::substitutePaths` which gets currently called from `LocalStore::queryValidPaths` and `Store::queryValidPaths` if `maybeSubstitute` is `true`. Fixes #2770
2020-11-02Merge pull request #4207 from hercules-ci/fix-RemoteStore-filterSource-deadlockEelco Dolstra
Fix RemoteStore pool deadlock in filterSource etc
2020-11-02Don't send eval-related settings to the daemonEelco Dolstra
2020-10-30Fix RemoteStore pool deadlock in filterSource etcRobert Hensing
2020-10-11Split out uds-remote-store.{cc.hh}John Ericson
2020-10-07Serialize exceptions from the daemon to the clientEelco Dolstra
2020-10-06Remove static variable name clashesEelco Dolstra
This was useful for an experiment with building Nix as a single compilation unit. It's not very useful otherwise but also doesn't hurt...
2020-09-30nix::worker_proto -> worker_protoJohn Ericson
2020-09-30Revert "Use template structs instead of phantoms"John Ericson
This reverts commit 9ab07e99f527d1fa3adfa02839da477a1528d64b.
2020-09-22Merge remote-tracking branch 'upstream/master' into templated-daemon-protocolJohn Ericson
2020-09-22RemoteStore::addCAToStore(): Don't hold connection while calling queryPathInfo()Eelco Dolstra
This leads to a deadlock if we're at the connection limit.
2020-09-22Style fixesEelco Dolstra
2020-09-22Merge branch 'add-ca-to-store' of https://github.com/hercules-ci/nixEelco Dolstra
2020-09-22Merge remote-tracking branch 'upstream/master' into templated-daemon-protocolJohn Ericson
2020-09-21Move Callback into its own headerEelco Dolstra
This gets rid of the inclusion of <future> in util.hh, cutting compilation time by ~20s (CPU time). Issue #4045.
2020-09-21wopAddToStore: return ValidPathInfoRobert Hensing
A ValidPathInfo is created anyway. By returning it we can save a roundtrip and we have a nicer interface.
2020-09-21wopAddToStore: add RepairFlagRobert Hensing
2020-09-21Move FramedSink next to FramedSourceRobert Hensing
2020-09-21Refactor wopAddToStore to make wopAddTextToStore obsoleteRobert Hensing
2020-09-21Overhaul wopAddToStoreRobert Hensing
2020-09-17Move FramedSource and FramedSink, extract withFramedSinkRobert Hensing
2020-09-16Merge remote-tracking branch 'upstream/master' into ↵John Ericson
remove-storetype-delegate-regStore
2020-09-16Fix build issues with gccregnat
2020-09-16Document the new store hierarchyregnat
2020-09-16Properly filter the stores according to their declared uriSchemesregnat
When opening a store, only try the stores whose `uriSchemes()` include the current one
2020-09-16Correctly call all the parent contructors of the storesregnat
Using virtual inheritance means that only the default constructors of the parent classes will be called, which isn't what we want
2020-09-16Separate store configs from the implemsregnat
Rework the `Store` hierarchy so that there's now one hierarchy for the store configs and one for the implementations (where each implementation extends the corresponding config). So a class hierarchy like ``` StoreConfig-------->Store | | v v SubStoreConfig----->SubStore | | v v SubSubStoreConfig-->SubSubStore ``` (with virtual inheritance to prevent DDD). The advantage of this architecture is that we can now introspect the configuration of a store without having to instantiate the store itself
2020-09-16Fix the registration of storesregnat
2020-09-16Make the store plugins more introspectableregnat
Directly register the store classes rather than a function to build an instance of them. This gives the possibility to introspect static members of the class or choose different ways of instantiating them.
2020-09-15Merge remote-tracking branch 'upstream/master' into ↵John Ericson
remove-storetype-delegate-regStore
2020-09-15gracefully handle old daemon versionsregnat
Add a fallback path in `queryPartialDerivationOutputMap` for daemons that don't support it. Also upstreams a couple methods from `SSHStore` to `RemoteStore` as this is needed to handle the fallback path.
2020-09-13Fix unspecified behaviour in readStorePathCAMapaszlig
When deploying a Hydra instance with current Nix master, most builds would not run because of errors like this: queue monitor: error: --- Error --- hydra-queue-runner error: --- UsageError --- nix-daemon not a content address because it is not in the form '<prefix>:<rest>': /nix/store/...-somedrv The last error message is from parseContentAddress, which expects a colon-separated string, however what we got here is a store path. Looking at the worker protocol, the following message sent to the Nix daemon caused the error above: 0x1E -> wopQuerySubstitutablePathInfos 0x01 -> Number of paths 0x16 -> Length of string "/nix/store/...-somedrv" 0x00 -> Length of string "" Looking at writeStorePathCAMap, the store path is indeed the first field that's transmitted. However, readStorePathCAMap expects it to be the *second* field *on my machine*, since expression evaluation order is a classic form of unspecified behaviour[1] in C++. This has been introduced in https://github.com/NixOS/nix/pull/3689, specifically in commit 66a62b3189c8c9b0965850e6b3c9b0fda0b50fd8. [1]: https://en.wikipedia.org/wiki/Unspecified_behavior#Order_of_evaluation_of_subexpressions Signed-off-by: aszlig <aszlig@nix.build>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-22 21:57:54 +0000