Merge pull request #2456 from grahamc/s3-substituters

Update docs to describe how s3:// URLS does in fact support endpoint, region, and profile for upload

1 files changed, 107 insertions, 95 deletions

diff --git a/doc/manual/packages/s3-substituter.xml b/doc/manual/packages/s3-substituter.xml
index bcd91cfdb..ea654392c 100644
--- a/doc/manual/packages/s3-substituter.xml
+++ b/doc/manual/packages/s3-substituter.xml
@@ -12,8 +12,49 @@ from Amazon S3 and S3 compatible services. This uses the same
 <emphasis>binary</emphasis> cache mechanism that Nix usually uses to
 fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
 
+<para>The following options can be specified as URL parameters to
+the S3 URL:</para>
+
+<variablelist>
+  <varlistentry><term><literal>profile</literal></term>
+  <listitem>
+    <para>
+      The name of the AWS configuration profile to use. By default
+      Nix will use the <literal>default</literal> profile.
+    </para>
+  </listitem>
+  </varlistentry>
+
+  <varlistentry><term><literal>region</literal></term>
+  <listitem>
+    <para>
+      The region of the S3 bucket. <literal>us–east-1</literal> by
+      default.
+    </para>
+
+    <para>
+      If your bucket is not in <literal>us–east-1</literal>, you
+      should always explicitly specify the region parameter.
+    </para>
+  </listitem>
+  </varlistentry>
+
+  <varlistentry><term><literal>endpoint</literal></term>
+  <listitem>
+    <para>
+      The URL to your S3-compatible service, for when not using
+      Amazon S3. Do not specify this value if you're using Amazon
+      S3.
+    </para>
+    <note><para>This endpoint must support HTTPS and will use
+    path-based addressing instead of virtual host based
+    addressing.</para></note>
+  </listitem>
+  </varlistentry>
+</variablelist>
+
 <para>In this example we will use the bucket named
-<literal>example-bucket</literal>.</para>
+<literal>example-nix-cache</literal>.</para>
 
 <section xml:id="ssec-s3-substituter-anonymous-reads">
   <title>Anonymous Reads to your S3-compatible binary cache</title>
@@ -24,65 +65,56 @@ fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
   cache.</para>
 
   <para>For AWS S3 the binary cache URL for example bucket will be
-  exactly <uri>https://example-bucket.s3.amazonaws.com</uri>. For S3
-  compatible binary caches ago have to consult your software's
-  documentation.</para>
+  exactly <uri>https://example-nix-cache.s3.amazonaws.com</uri> or
+  <uri>s3://example-nix-cache</uri>. For S3 compatible binary caches,
+  consult that cache's documentation.</para>
 
   <para>Your bucket will need the following bucket policy:</para>
 
-  <programlisting>
-<![CDATA[
+  <programlisting><![CDATA[
 {
-  "Id": "DirectReads",
-  "Version": "2012-10-17",
-  "Statement": [
-    {
-      "Sid": "AlowDirectReads",
-      "Action": [
-        "s3:GetObject"
-      ],
-      "Effect": "Allow",
-      "Resource": "arn:aws:s3:::example-bucket/*",
-      "Principal": "*"
-    }
-  ]
+    "Id": "DirectReads",
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "AlowDirectReads",
+            "Action": [
+                "s3:GetObject",
+                "s3:GetBucketLocation"
+            ],
+            "Effect": "Allow",
+            "Resource": [
+                "arn:aws:s3:::example-nix-cache",
+                "arn:aws:s3:::example-nix-cache/*"
+            ],
+            "Principal": "*"
+        }
+    ]
 }
-]]>
-</programlisting>
+]]></programlisting>
 </section>
 
 <section xml:id="ssec-s3-substituter-authenticated-reads">
   <title>Authenticated Reads to your S3 binary cache</title>
 
   <para>For AWS S3 the binary cache URL for example bucket will be
-  exactly <uri>s3://example-bucket</uri>.</para>
+  exactly <uri>s3://example-nix-cache</uri>.</para>
 
   <para>Nix will use the <link
   xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
   credential provider chain</link> for authenticating requests to
   Amazon S3.</para>
 
-  <para>Nix supports authenticated writes to S3 compatible binary
-  caches but only supports Authenticated reads from Amazon S3.
-  Additionally, the following limitations are in place for
-  authenticated reads:</para>
-
-  <itemizedlist>
-    <listitem><para>The bucket must actually be hosted by Amazon S3 and
-    <emphasis>not</emphasis> an S3 compatible
-    service.</para></listitem>
-
-    <listitem><para>The bucket must be within the
-    <literal>us-east-1</literal> region.</para></listitem>
-
-    <listitem><para>The Amazon credentials, if stored in a credential
-    profile, must be stored in the <literal>default</literal>
-    profile.</para></listitem>
-  </itemizedlist>
+  <para>Nix supports authenticated reads from Amazon S3 and S3
+  compatible binary caches.</para>
 
   <para>Your bucket will need a bucket policy allowing the desired
-  users to perform the <literal>s3:GetObject</literal> action on all
-  objects in the bucket.</para>
+  users to perform the <literal>s3:GetObject</literal> and
+  <literal>s3:GetBucketLocation</literal> action on all objects in the
+  bucket. The anonymous policy in <xref
+  linkend="ssec-s3-substituter-anonymous-reads" /> can be updated to
+  have a restricted <literal>Principal</literal> to support
+  this.</para>
 </section>
 
 
@@ -91,69 +123,49 @@ fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
 
   <para>Nix support fully supports writing to Amazon S3 and S3
   compatible buckets. The binary cache URL for our example bucket will
-  be <uri>s3://example-bucket</uri>.</para>
+  be <uri>s3://example-nix-cache</uri>.</para>
 
   <para>Nix will use the <link
   xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
   credential provider chain</link> for authenticating requests to
   Amazon S3.</para>
 
-  <para>The following options can be specified as URL parameters to
-  the S3 URL:</para>
-  <variablelist>
-    <varlistentry><term><literal>profile</literal></term>
-    <listitem>
-      <para>
-        The name of the AWS configuration profile to use. By default
-        Nix will use the <literal>default</literal> profile.
-      </para>
-    </listitem>
-    </varlistentry>
-
-    <varlistentry><term><literal>region</literal></term>
-    <listitem>
-      <para>
-        The region of the S3 bucket. <literal>us–east-1</literal> by
-        default.
-      </para>
-    </listitem>
-    </varlistentry>
-
-    <varlistentry><term><literal>endpoint</literal></term>
-    <listitem>
-      <para>
-        The URL to your S3-compatible service, for when not using
-        Amazon S3. Do not specify this value if you're using Amazon
-        S3.
-      </para>
-      <note><para>This endpoint must support HTTPS and will use
-      path-based addressing instead of virtual host based
-      addressing.</para></note>
-    </listitem>
-    </varlistentry>
-  </variablelist>
-
-  <example><title>Uploading with non-default credential profile for Amazon S3</title>
-    <para><command>nix copy --to ssh://machine nixpkgs.hello s3://example-bucket?profile=cache-upload</command></para>
+  <para>Your account will need the following IAM policy to
+  upload to the cache:</para>
+
+  <programlisting><![CDATA[
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "UploadToCache",
+      "Effect": "Allow",
+      "Action": [
+        "s3:AbortMultipartUpload",
+        "s3:GetBucketLocation",
+        "s3:GetObject",
+        "s3:ListBucket",
+        "s3:ListBucketMultipartUploads",
+        "s3:ListMultipartUploadParts",
+        "s3:ListObjects",
+        "s3:PutObject"
+      ],
+      "Resource": [
+        "arn:aws:s3:::example-nix-cache",
+        "arn:aws:s3:::example-nix-cache/*"
+      ]
+    }
+  ]
+}
+]]></programlisting>
+
+
+  <example><title>Uploading with a specific credential profile for Amazon S3</title>
+    <para><command>nix copy --to 's3://example-nix-cache?profile=cache-upload&amp;region=eu-west-2' nixpkgs.hello</command></para>
   </example>
 
   <example><title>Uploading to an S3-Compatible Binary Cache</title>
-    <para><command>nix copy --to ssh://machine nixpkgs.hello s3://example-bucket?profile=cache-upload&amp;endpoint=minio.example.com</command></para>
+    <para><command>nix copy --to 's3://example-nix-cache?profile=cache-upload&amp;endpoint=minio.example.com' nixpkgs.hello</command></para>
   </example>
-
-  <para>The user writing to the bucket will need to perform the
-  following actions against the bucket:</para>
-
-  <itemizedlist>
-    <listitem><para><literal>s3:ListBucket</literal></para></listitem>
-    <listitem><para><literal>s3:GetBucketLocation</literal></para></listitem>
-    <listitem><para><literal>s3:ListObjects</literal></para></listitem>
-    <listitem><para><literal>s3:GetObject</literal></para></listitem>
-    <listitem><para><literal>s3:PutObject</literal></para></listitem>
-    <listitem><para><literal>s3:ListBucketMultipartUploads</literal></para></listitem>
-    <listitem><para><literal>s3:CreateMultipartUpload</literal></para></listitem>
-    <listitem><para><literal>s3:ListMultipartUploadParts</literal></para></listitem>
-    <listitem><para><literal>s3:AbortMultipartUpload</literal></para></listitem>
-  </itemizedlist>
 </section>
 </section>