Skip to main content

Object Storage Configuration

Object storage is used for storing large binary files such as debugging symbols and other artifacts. zymtrace supports S3-compatible storage solutions including MinIO, AWS S3, and Google Cloud Storage.

Configuration Modes​

Connect to existing S3-compatible storage​

This mode connects to your existing MinIO, AWS S3, or Google Cloud Storage.

MinIO Configuration​

Artifact Hub backend

MinIO Configuration

MinIO is a high-performance, S3-compatible object storage solution that can be deployed on-premises or in the cloud. Configure your existing MinIO instance with the following settings:

storage:
  mode: "use_existing"
  use_existing:
    type: "minio"
    minio:
      endpoint: "" # must be a URL, HTTP/HTTPS
      user: ""
      password: ""
  buckets:
    symbols: "zymtrace-symdb"

Required fields:

  • endpoint: Complete URL to your MinIO server (e.g., https://minio.example.com or http://192.168.1.100:9000)
  • user: MinIO access key
  • password: MinIO secret key
Bucket Creation​
mc mb minio/zymtrace-symdb
Testing Connectivity​
mc config host add minio http://minio.company.com:9000 ACCESS_KEY SECRET_KEY
mc ls minio/zymtrace-symdb
Configuration Example​
Development Environment​
storage:
  mode: "use_existing"
  use_existing:
    type: "minio"
    minio:
      endpoint: "http://minio.dev.company.com:9000"
      user: "dev-access-key"
      password: "dev-secret-key"
  buckets:
    symbols: "zymtrace-symdb-dev"

Bucket Configuration​

Bucket Names​

zymtrace uses the following bucket for storing symbols:

  • symbols: Stores debugging symbols and related metadata (default: zymtrace-symdb)

Bucket Creation​

Automatic Creation: If you have appropriate permissions, zymtrace can automatically create the required bucket during initialization.

Manual Creation: For production environments, create the bucket manually using your cloud provider's console or CLI tools. See the specific instructions in each storage provider tab above.

Security Considerations​

Access Control​

  • Principle of Least Privilege: Grant only the minimum required permissions
  • Network Security: Use HTTPS/TLS for connections to object storage
  • Access Keys: Use dedicated service accounts with limited scope

Data Encryption​

  • Encryption in Transit: Always use HTTPS for object storage connections
  • Encryption at Rest: Enable server-side encryption on your buckets
  • Key Management: Use managed encryption keys when possible

Troubleshooting​

AWS S3 Authentication Modes​

zymtrace supports two authentication modes for AWS S3:

ModeHow it worksWhen to use
Static KeysLong-lived accessKey / secretKey set in helm valuesSimple setups, non-EKS environments
IAM / IRSAuseIAM: "aws" — credentials from EC2 node role (IMDS) or IRSA tokenEKS deployments — IRSA is recommended

Common Issues​

  1. Connection Timeout: Check network connectivity and firewall rules
  2. Access Denied: Verify credentials and permissions
  3. Bucket Not Found: Ensure the bucket exists and is accessible
  4. SSL/TLS Errors: Verify certificate configuration for HTTPS endpoints

AWS: IMDS Timeout (failed to load region from IMDS)​

Symptom: Logs contain warnings like:

WARN  🚧 [warn]: failed to load region from IMDS | err: failed to load IMDS session token: dispatch failure: timeout: HTTP read timeout

Cause: This occurs when useIAM: "aws" is set and the pod tries to fetch credentials from the EC2 metadata service. EKS nodes default to IMDSv2 with a hop limit of 1, which blocks pods inside containers from reaching IMDS.

Fix A — IRSA (recommended): Use IRSA so credentials are injected directly into the pod via a projected service account token — no IMDS required. See IAM / IRSA setup above.

Fix B — Increase hop limit: Increase the IMDSv2 hop limit to 2 on all nodes so pods can reach IMDS:

for id in $(aws ec2 describe-instances \
  --region <region> \
  --filters "Name=tag:eks:cluster-name,Values=<cluster-name>" \
            "Name=instance-state-name,Values=running" \
  --query "Reservations[*].Instances[*].InstanceId" \
  --output text); do
  aws ec2 modify-instance-metadata-options \
    --instance-id "$id" \
    --http-put-response-hop-limit 2 \
    --http-tokens required \
    --region <region>
done

Also update your node group launch template to persist this setting across node replacements.

AWS: Access Denied with IAM Authentication​

Symptom: S3 operations fail with AccessDenied despite useIAM: "aws" being set.

Checklist:

  • The IAM role has the S3 policy attached (s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket)
  • For IRSA: the role trust policy matches the exact namespace and service account name
  • For IRSA: the service account has the eks.amazonaws.com/role-arn annotation
  • For IRSA: storage.use_existing.s3.serviceAccount is set to the annotated service account name
  • The bucket name in storage.buckets.symbols matches the bucket in the IAM policy resource ARN

Next Steps​

After configuring object storage, proceed to configure the other storage components:

Configure ClickHouse | Configure Postgres