s3proxy

S3Proxy

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

• translation from S3 to Google Cloud, Microsoft Azure, OpenStack Swift, and SFTP
• testing without Amazon by using the local filesystem
• extension via middlewares
• embedding into Java applications

Usage with Docker

Docker Hub hosts a Docker image and has instructions on how to run it.

Usage with Kubernetes

Reference manifests show how to wire the Docker image into Kubernetes, including health probes, graceful shutdown, and Secret-based credentials. Helm users may prefer the third-party s3proxy-chart.

Usage without Docker

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 17 or newer to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>[email protected]</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Usage with Java

Maven Central hosts S3Proxy artifacts and the wiki has instructions on Java use.

Supported storage backends

• aws-s3
• azureblob
• filesystem (on-disk storage)
• google-cloud-storage
• openstack-swift (Keystone v3 only)
• sftp (SFTP storage via Apache MINA SSHD)
• transient (in-memory storage)

See the wiki for examples of configurations.

Backend credentials

jclouds.identity and jclouds.credential name the enduring half of a backend credential. Credentials that expire carry a third part, which jclouds.session-token names: an AWS session token, an Azure shared access signature, a Google OAuth 2.0 access token, or a Keystone token.

jclouds.provider=aws-s3
jclouds.identity=ASIAIOSFODNN7EXAMPLE
jclouds.credential=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
jclouds.session-token=IQoJb3JpZ2luX2VjEExample

A token written into the properties file serves no longer than the token itself does. A deployment that rotates them embeds S3Proxy and hands BlobStores.create a Supplier<Credentials> instead: a store asks it again for every request it signs rather than keeping the first answer, so an assumed role, a reissued signature, or a refreshed access token takes effect without the store being rebuilt.

Naming no credentials at all defers to the chain each backend’s own tools read – environment, profile, container, and instance role for aws-s3, DefaultAzureCredential for azureblob, and application default credentials for google-cloud-storage – every one of which renews what it finds.

A Microsoft Entra access token that the application obtained itself, for example from a sidecar or a token broker that DefaultAzureCredential cannot read, also goes in jclouds.session-token, with s3proxy.azureblob.auth=bearer saying that it is a bearer token rather than a shared access signature. The other values of s3proxy.azureblob.auth are sas, key and default. When the property is not set, the store uses sas if there is a session token, key if there are an identity and a credential, and default otherwise. S3Proxy asks the supplier for a bearer token again on every request, as it does for a signature, and does not renew it: the supplier must always return one that is still valid. The endpoint must be https, since the Azure client refuses to send a bearer token over http. Server-side copies sign their source with a user delegation key, so the token’s principal needs a role that includes generateUserDelegationKey, such as Storage Blob Delegator; without it reads and writes succeed but copies fail.

S3Proxy 3.x reached Microsoft Entra through jclouds, and its jclouds.azureblob.auth=azureAd is no longer read. By default it took a client id and secret in jclouds.identity and jclouds.credential and fetched and renewed tokens itself; that setup moves to default, where DefaultAzureCredential does the same from AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET. bearer replaces only the setup in which jclouds.oauth.credential-type was overridden so that the application supplied the access token itself.

Three credentials are read only when the store is built, because nothing about them expires: an Azure account key, which the client copies out when it is created; a Google service account key, which mints and renews access tokens of its own; and the sftp password, presented once when the connection opens. openstack-swift re-reads the supplier whenever the Keystone token it holds nears its expiry.

Assigning buckets to backends

S3Proxy can be configured to assign buckets to different backends with the same credentials. The configuration in the properties file is as follows:

s3proxy.bucket-locator.1=bucket
s3proxy.bucket-locator.2=another-bucket

In addition to the explicit names, glob syntax can be used to configure many buckets for a given backend.

A bucket (or a glob) cannot be assigned to multiple backends.

Middlewares

S3Proxy can modify its behavior based on middlewares:

• bucket aliasing
• bucket prefix scoping
• bucket locator
• eventual consistency modeling
• large object mocking
• latency
• read-only
• regex rename blobs
• sharded backend containers
• storage class override
• user metadata replacer
• no cache override

SSL Support

S3Proxy can listen on HTTPS by setting the secure-endpoint and configuring a keystore. You can read more about how configure S3Proxy for SSL Support in the dedicated wiki page with Docker, Kubernetes or simply Java.

Limitations

S3Proxy has broad compatibility with the S3 API, however, it does not support:

• ACLs other than private and public-read, including the x-amz-grant-* headers and any grant naming a specific grantee
• BitTorrent hosting
• bucket inventory, analytics, and metrics configuration
• bucket lifecycle configuration
• bucket logging
• bucket notification configuration
• bucket policies
• bucket policy status
• bucket replication
• CORS bucket operations like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).
• hosting static websites
• object lock, including legal hold and retention
• object ownership controls
• object tagging
• paginating ListParts with part-number-marker
• public access block
• reading a single part of a multipart object with partNumber, unless the object has only one part
• requester pays buckets
• restoring archived objects
• select object content
• transfer acceleration
• x-amz-expected-bucket-owner

S3Proxy emulates the following operations:

• multi-part upload on the filesystem and transient backends, which store a stub object to carry the metadata
• object and bucket owners, which are always the same synthetic user

Some limitations depend on the storage backend:

limitation	backends
no per-object ACLs, including public-read	azureblob, openstack-swift
ETag is not the object MD5	azureblob, google-cloud-storage
on a filesystem without user extended attributes, such as NFSv3: Content-Type and user metadata are not kept, and the ETag is derived from the file rather than the MD5, see #1210	filesystem
Cache-Control not preserved	google-cloud-storage, openstack-swift
Content-Encoding not preserved	google-cloud-storage
Content-Language not preserved	openstack-swift
Expires not preserved	azureblob
max-keys=0 not honored	azureblob
UploadPartCopy streams the data through S3Proxy instead of copying it on the backend	filesystem, transient, openstack-swift, sftp
conditional PUT refuses If-Match, honoring only If-None-Match: *	openstack-swift
no object versioning, see #1137	azureblob
no object versioning, see #1138	filesystem
no object versioning, for want of a service analog	openstack-swift, sftp
no server-side encryption, see #1134	google-cloud-storage
no server-side encryption, by design or for want of a service analog	filesystem, openstack-swift, sftp
SSE-S3 only: AES256 relayed from Azure’s own at-rest encryption; SSE-C (#1135) and SSE-KMS refused	azureblob

The backends that do version objects come by it differently. aws-s3 passes the requests through to a service that already versions, while google-cloud-storage translates S3 versioning onto GCS object generations, emulating delete markers as zero-byte generations; because GCS versioning is simply on or off, it refuses the Suspended state. transient implements versioning itself, keeping every version but the current one as a hidden object. filesystem shares that implementation and declines it: its versions would outlive the process, settling a layout on disk that S3Proxy would owe its users from then on.

Server-side encryption divides aws-s3 and transient from most of the rest, and divides the two nio2 stores on the same line. aws-s3 forwards the header families – SSE-S3, SSE-KMS and SSE-C alike – to a service that encrypts, while transient answers them itself: what it holds rests in a filesystem that dies with the process, so an object’s encryption is the headers and nothing besides, which is all S3 lets a caller observe of SSE-S3 anyway. SSE-C has an observable half beyond the headers – an object stored under a customer’s key answers only to that key – and transient enforces it, keeping the key’s MD5 and never the key. Unlike S3, S3Proxy accepts SSE-C over plain HTTP, its TLS commonly terminating elsewhere. filesystem writes to a real disk, where reporting AES256 over plaintext someone could go and read would be a claim S3Proxy has no business making, so it refuses every one of these headers instead. azureblob stands halfway in: Azure encrypts everything at rest and says so on each response, which the store relays as SSE-S3 – an object’s AES256 is Azure’s answer, not S3Proxy’s assertion – while SSE-C and SSE-KMS are refused rather than silently accepted.

Bucket default encryption, the ?encryption subresource, rides the same divide. aws-s3 passes the configuration through, transient keeps it and rests unadorned writes under it – a write naming its own encryption still wins – and every other backend answers NotImplemented. A bucket never given a configuration answers the 404 S3 spells ServerSideEncryptionConfigurationNotFoundError rather than the account-wide AES256 default modern AWS invents, which is also what the Ceph s3-tests expect of it.

The filesystem backend stores a key as a path, so on Windows a key holding a backslash is refused with 400 InvalidArgument. Windows reads that character as a directory separator and no file name can hold one, so the key back\slash.txt would come back from a listing as back/slash.txt and answer for whoever wrote that key instead – two S3 keys on one object. Keys are spelled with forward slashes on every host, whatever the filesystem underneath.

aws-s3, azureblob and google-cloud-storage copy a part on the server, in one request and without the bytes passing through S3Proxy. Only google-cloud-storage falls back to streaming it, for a range covering less than the whole object, which GCS cannot copy server-side.

Azure mints ETags like 0x8DD3F4A5F0B2C1E, which S3 SDKs decode as hex and abort the request when they cannot – the AWS SDK for .NET raises ArgumentOutOfRangeException (Parameter 'hex') and the one for Java reports Input is expected to be encoded in multiple of 2 bytes. azureblob therefore reports it under the -1 suffix S3 gives an object assembled from parts, which every client already treats as a value not to verify, and takes the suffix off again when a conditional request names one. The ETag remains opaque either way: it is not the object’s MD5 and nothing can check the object against it. Set s3proxy.azureblob.etag=native to report the bare Azure ETag, as releases before 4.0.0 did.

aws-s3, azureblob and google-cloud-storage perform a conditional PUT on the backend. The filesystem and transient backends resolve If-None-Match as they write and emulate If-Match within a single S3Proxy process, so it does not hold against another writer. Where a backend cannot do either, a conditional PUT is refused rather than emulated, since emulating it would mean a read followed by a write – which answers correctly only when nothing else is writing that key, and a conditional PUT is asked for precisely because something might be. Set s3proxy.aws-s3.conditional-writes=emulated for an S3-compatible endpoint that does not implement conditional writes itself.

aws-s3 also deletes conditionally on the backend – If-Match, x-amz-if-match-size, and x-amz-if-match-last-modified-time, on DeleteObject and DeleteObjects alike – and the backend decides what it honors: Amazon general purpose buckets accept only If-Match, directory buckets all three. The filesystem and transient backends compare all three against their own metadata, and sftp all but a condition naming an ETag, with the same single-process bound as their emulated If-Match put. A conditional delete of an absent object succeeds like an unconditional one, as Ceph RGW answers it. Other backends refuse conditional deletes.

S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties file (and corresponding ENV variables for Docker):

s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
s3proxy.cors-allow-credential=true

CORS cannot be configured per bucket. s3proxy.cors-allow-all=true will accept any origin and header. Actual CORS requests are supported for GET, PUT, POST, HEAD and DELETE methods. Cross-origin sharing is opt-in: a proxy with no CORS configuration, including one built through S3Proxy.Builder without corsRules, shares nothing. Responses that depend on the request Origin carry Vary: Origin so that shared caches do not serve one origin’s response to another.

The wiki collects compatibility notes for specific storage backends.

Support

• GitHub issues
• Stack Overflow
• commercial support

References

• Ceph s3-tests help maintain and improve compatibility with the S3 API
• fake-s3, gofakes3, minio, S3 ninja, and s3rver provide functionality similar to S3Proxy when using the filesystem backend
• GlacierProxy and SwiftProxy provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
• s3mock - Adobe’s s3 mock implementation
• s3proxy-chart - Helm chart for deploying S3Proxy
• sbt-s3 runs S3Proxy via the Scala Build Tool
• swift3 provides an S3 middleware for OpenStack Swift
• Zenko provide similar multi-cloud functionality

License

Copyright © 2014-2026 Andrew Gaul

Licensed under the Apache License, Version 2.0