AWS v4 Signing with Custom Metadata Headers for GCS
Problem
When adding custom metadata headers (x-amz-meta-*) to GCS S3-compatible API requests,
the requests fail with signature mismatch errors. The headers must be properly included
in the AWS v4 signature calculation.
Context / Trigger Conditions
- Using GCS S3-compatible XML API (not the native JSON API)
- Adding
x-amz-meta-* headers for object metadata
- Error: "SignatureDoesNotMatch" or 403 Forbidden after adding headers
- Headers work with unsigned requests but fail with AWS v4 signing
- HMAC credentials configured for GCS interoperability
Solution
Custom headers must be included in both the canonical headers and the signed headers list,
sorted alphabetically:
1. Canonical Headers (alphabetically sorted)
let canonical_headers = format!(
"host:{}\nx-amz-content-sha256:{}\nx-amz-date:{}\nx-amz-meta-owner:{}\n",
host, payload_hash, amz_date, owner // Note: alphabetical order!
);
2. Signed Headers List
let signed_headers = "host;x-amz-content-sha256;x-amz-date;x-amz-meta-owner";
3. Complete Example (Rust)
fn sign_request_with_owner(
req: &mut Request,
config: &GCSConfig,
payload_hash: Option<String>,
owner: &str
) -> Result<()> {
// Set headers first
req.set_header("x-amz-date", &amz_date);
req.set_header("x-amz-content-sha256", &payload_hash);
req.set_header("x-amz-meta-owner", owner); // Custom metadata
// Canonical headers - MUST be alphabetically sorted
let signed_headers = "host;x-amz-content-sha256;x-amz-date;x-amz-meta-owner";
let canonical_headers = format!(
"host:{}\nx-amz-content-sha256:{}\nx-amz-date:{}\nx-amz-meta-owner:{}\n",
host, payload_hash, amz_date, owner
);
// Rest of AWS v4 signing...
let canonical_request = format!(
"{}\n{}\n{}\n{}\n{}\n{}",
method, uri, query, canonical_headers, signed_headers, payload_hash
);
// Sign and add Authorization header...
}
Verification
After uploading, verify metadata is present:
gsutil stat gs://bucket/object-name
# Should show:
# Metadata:
# owner: <value>
Or via S3 API:
aws s3api head-object --bucket bucket --key object-name --endpoint-url https://storage.googleapis.com
Example
Before (broken) - header not in signature:
let signed_headers = "host;x-amz-content-sha256;x-amz-date"; // Missing x-amz-meta-owner
req.set_header("x-amz-meta-owner", owner); // Header added but not signed
// Result: 403 SignatureDoesNotMatch
After (working) - header properly signed:
let signed_headers = "host;x-amz-content-sha256;x-amz-date;x-amz-meta-owner"; // Included!
req.set_header("x-amz-meta-owner", owner);
// Include in canonical_headers too
// Result: 200 OK, metadata visible in GCS
Notes
- Header names are case-insensitive but conventionally lowercase in canonical form
- The semicolon-separated signed headers list must match the newline-separated canonical headers
- For GCS, the metadata appears as
Metadata: key: value in gsutil stat output
- When using the native GCS JSON API, use the
metadata object field instead
- Multiple custom headers can be added - just ensure all are in canonical/signed lists
References