S3-backed Nix binary cache with garbage collection

The idea is to have all reads be handled by the s3 cache (which itself can be high-available) and have a gc server that tracks all uploads to the cache and runs periodic garbage collection on s3 cache. Since writes to a binary cache are often not as critical as reads, we can vastly simplify the operational complexity of the GC server, i.e. only running one instance next to the CI infrastructure.

Architecture

Write path

flowchart LR
    niks3cli[niks3 CLI] -->|request upload| niks3[niks3 Server]
    niks3 -->|presigned S3 URLs| niks3cli
    niks3cli -->|PUT NAR + narinfo| s3[(S3 Bucket)]
    niks3 -->|track references| db[(PostgreSQL)]

The niks3 CLI requests an upload from the server, which returns pre-signed S3 URLs. The client uploads NAR files and narinfo directly to S3. The server tracks references in PostgreSQL for garbage collection.

Read path

flowchart LR
    nix[Nix Client] -->|read NAR + narinfo| s3[(S3 Bucket)]

Nix clients read directly from S3 (or a CDN in front of it) without going through niks3. This allows the read path to scale independently and remain highly available.

Read proxy (optional)

flowchart LR
    nix[Nix Client] -->|read request| niks3[niks3 Server]
    niks3 -->|fetch on behalf| s3[(S3 Bucket)]

For private S3 buckets, niks3 can proxy read requests from Nix clients to S3 using its own credentials. Enable with --enable-read-proxy. See the Private S3 Buckets wiki page.

With --read-redirect-ttl 15m NAR requests are answered with a 307 to a presigned S3 URL instead of being streamed, so NAR bytes bypass niks3. Narinfos and other metadata stay proxied. Note that anyone holding such a URL can fetch that object until it expires, regardless of read-proxy authentication.

Features

Binary Cache Protocol Support

niks3 implements the Nix binary cache specification with the following features:

• Cryptographic signing: NAR signatures using Ed25519 keys (compatible with nix key generate-secret)
• NAR files (nar/): Compressed with zstd, stored in S3
• Narinfo files (.narinfo): Metadata with cryptographic signatures
  • StorePath, URL, Compression, NarHash, NarSize
  • References, Deriver
  • Signatures (Sig fields)
  • CA field for content-addressed derivations
• Build logs (log/): Compressed build output storage
• Realisation files (realisations/*.doi): For content-addressed derivations
• Cache info (nix-cache-info): Automatic generation with WantMassQuery, Priority

Advanced Features

• Multipart uploads: Efficient handling of large NARs (>100MB)
• Transactional uploads: Atomic closure uploads with rollback on failure
• Garbage collection: Reference-tracking GC with configurable retention
• Parallel uploads: Client parallelizes NAR and metadata uploads
• Streaming push: niks3 push --stdin for long-running producers

Operational Features

• Authentication via API tokens (Bearer auth)
• OIDC authentication for CI/CD systems (GitHub Actions, GitLab CI)
• S3 credentials via static keys (--s3-access-key / --s3-secret-key) or IAM (--s3-use-iam for IRSA, EC2 instance profiles, ECS task roles)
• Automatic upload via post-build-hook with crash-safe SQLite queue

Choosing an S3 Provider

niks3 works with any S3-compatible storage provider. We recommend Cloudflare R2 for most users due to zero egress fees and excellent performance.

For detailed pricing comparison and alternative providers, see the S3 Provider Comparison wiki page.

Setup

For complete setup instructions, see the Setup Guide in the wiki.

Kubernetes

Chart: oci://ghcr.io/mic92/charts/niks3 (source in deploy/helm/niks3), image: ghcr.io/mic92/niks3. See the Kubernetes wiki page for Postgres/S3 wiring and letting pods push via their service account token.

Monitoring

Prometheus metrics on /metrics and a Grafana dashboard (deploy/helm/niks3/files/niks3.json, flake output dashboards.niks3). See the Monitoring wiki page.

OIDC Authentication (CI/CD)

niks3 supports OIDC authentication for CI/CD systems. See the wiki for details:

• OIDC Configuration
• GitHub Actions
• GitLab CI

Development

For development setup, database migrations, benchmarks, and contribution guidelines, see CONTRIBUTING.md.

Real-World Deployments

• Clan infra: Configuration | Instance
• Numtide: Instance
• TUM-DSE: Configuration | Instance

Need commercial support or customization?

For commercial support, please contact Mic92 at [email protected] or reach out to Numtide.