Skip to main content

Production Troubleshooting

This guide provides troubleshooting solutions for common errors and operational challenges.

1. Authentication & Permission Errors​

401 Unauthorized (invalid_credentials)​

  • Symptom: CLI or driver requests fail with BastionAuthError or invalid_credentials.
  • Cause: Incorrect username/password, expired access key, or mistyped secret key.
  • Fix:
    • Verify credentials using liorans3 whoami.
    • If using Access Keys, check if the key was revoked or rotated via liorans3 key list.
    • Reconfigure local profile via liorans3 configure.

403 Forbidden (password_change_required)​

  • Symptom: Mutating operations or bucket creations fail with password_change_required.
  • Cause: Fresh server deployment where the bootstrap admin account has not changed its initial password.
  • Fix: Change the admin password via CLI or SDK: 
    liorans3 user passwd --old admin --new "YourNewPassword123!"

403 Forbidden (forbidden)​

  • Symptom: Attempting to create buckets or manage users fails.
  • Cause: Authenticated account possesses readwrite or readonly role instead of admin.
  • Fix: Re-authenticate using an account with the admin role.

2. Capacity & Storage Errors​

507 Insufficient Storage (quota_exceeded)​

  • Symptom: Single PUT or multipart assembly fails with QuotaExceededError.
  • Cause: The target bucket has exceeded its configured quota_bytes.
  • Fix:
    • Inspect current quota via liorans3 bucket info <bucket>.
    • Increase the quota limit: 
      liorans3 bucket update <bucket> --quota-gb 100
    • Or remove the quota limit entirely: 
      liorans3 bucket update <bucket> --no-quota

507 Insufficient Storage (storage_capacity_exhausted)​

  • Symptom: All mutating writes across all buckets are rejected.
  • Cause: Host storage free space has dropped below BASTION_MIN_FREE_SPACE_BYTES (default: 512 MiB).
  • Fix:
    • Check host disk usage with df -h.
    • Expand host disk volume or purge unnecessary temporary files.
    • Tune BASTION_MIN_FREE_SPACE_BYTES in .env if operating on small disks.

3. Network & Reverse Proxy Issues​

Connection Refused / ConnectionError​

  • Symptom: CLI reports Failed to connect to Bastion server.
  • Cause: bastion-server container is not running, or listening on a different port.
  • Fix:
    • Run docker compose ps to inspect container status.
    • View server logs with docker compose logs bastion.
    • Run liorans3 doctor to diagnose endpoint resolution.

TLS / Certificate Errors​

  • Symptom: Browser or SDK throws certificate verification errors (UNABLE_TO_VERIFY_LEAF_SIGNATURE).
  • Cause: Caddy was unable to obtain an ACME certificate because ports 80/443 were blocked or DNS A/AAAA records do not point to the server.
  • Fix:
    • Inspect Caddy logs: docker compose logs caddy.
    • Verify that inbound TCP traffic on ports 80 and 443 is permitted through your firewall/security group.
    • Verify DNS propagation with dig +short s3.yourdomain.com.