Bootstrap Certificate Management

Overview

This guide covers operational procedures for managing bootstrap certificates used by platform services (cluster-sensor-service and device-interrogation-service) for mTLS authentication.

Certificate Generation

Initial Setup

Generate the platform bootstrap CA (one-time operation):

./scripts/generate-bootstrap-ca.sh

This creates:

  • Platform bootstrap CA certificate and encrypted private key (stored in database)
  • CA certificate available for service configuration

Generate Service Certificates

Generate bootstrap certificates for platform services:

./scripts/generate-bootstrap-certificates.sh

This creates certificates for:

  • cluster-sensor-service
  • device-interrogation-service

Certificates are stored in:

  • Database: platform_bootstrap_certificates table
  • Filesystem: ./bootstrap-certs/ directory

Certificate Files

After generation, the following files are created:

bootstrap-certs/
├── cluster-sensor-service-cert.pem
├── cluster-sensor-service-key.pem
├── device-interrogation-service-cert.pem
├── device-interrogation-service-key.pem
└── bootstrap-ca-cert.pem

Deployment

Development Environment

Certificates are automatically generated by session-init.sh:

  1. Bootstrap CA is created if it doesn't exist
  2. Bootstrap certificates are generated for services
  3. Certificates are mounted into containers via docker-compose.yml

Production Environment

  1. Generate Certificates:

    ./scripts/generate-bootstrap-ca.sh
./scripts/generate-bootstrap-certificates.sh

  2. Store Certificates Securely:

    • Use secrets manager (AWS Secrets Manager, HashiCorp Vault)
    • Store in encrypted storage
    • Never commit to version control

  3. Mount Certificates:

    • Update docker-compose.prod.yml with certificate volume mounts
    • Ensure certificates are available at container startup

  4. Set Permissions:

    chmod 755 bootstrap-certs
chmod 600 bootstrap-certs/*.pem

Kubernetes (Helm chart) Environment

The chart does not yet ship a built-in bootstrapCerts Secret/mount feature. If the bootstrap cert isn't mounted into the cluster-sensor-service pod, the service detects this at startup, logs an informational message, and skips auto-registration silently — it does not crash. Tenants in this state must be registered manually through the Tenant Admin UI's sensor-onboarding flow.

If you want auto-registration on K8s, provide the certs to the pod via a customer-owned overlay:

  1. Create a Kubernetes Secret in the release namespace containing the four PEM files at /app/bootstrap-certs/:
    kubectl -n <release-ns> create secret generic cluster-sensor-bootstrap 
  --from-file=cluster-sensor-service-cert.pem 
  --from-file=cluster-sensor-service-key.pem 
  --from-file=bootstrap-ca-cert.pem
  2. In your values-customer.yaml, attach the Secret to the service via extraVolumes / extraVolumeMounts on backends.cluster-sensor-service (read-only, mode 0600). Use a stricter defaultMode (e.g. 0400) for long-lived environments.
  3. Re-deploy. The startup log should now read 🔄 Registering platform discovery sensor for all tenants... instead of ℹ️ Bootstrap cert not present...; skipping auto-registration.

A native chart-level setting for this is tracked as a follow-up.

Certificate Rotation

When to Rotate

  • Before expiration (bootstrap certs expire in 90 days)
  • After security incident
  • During scheduled maintenance

Rotation Procedure

  1. Generate New Certificates:

    ./scripts/generate-bootstrap-certificates.sh

  2. Update Certificate Files:

    • Replace old certificate files in ./bootstrap-certs/
    • Update secrets manager if used

  3. Restart Services:

    docker compose restart cluster-sensor-service device-interrogation-service

  4. Verify Registration:

    • Check service logs for successful registration
    • Verify services are using new certificates

CA Rotation

If the platform bootstrap CA needs to be rotated:

  1. Generate New CA:

    # This will mark old CA as inactive and create new one
go run scripts/initialize-bootstrap-ca.go 
  -db-url "$DATABASE_URL" 
  -encryption-key "$ENCRYPTION_MASTER_KEY" 
  -action rotate-ca

  2. Regenerate All Certificates:

    ./scripts/generate-bootstrap-certificates.sh

  3. Update Services:

    • Restart all platform services
    • Verify new certificates work

Certificate Revocation

Revoke Compromised Certificate

If a bootstrap certificate is compromised:

# Using Go program
go run scripts/initialize-bootstrap-ca.go 
  -db-url "$DATABASE_URL" 
  -encryption-key "$ENCRYPTION_MASTER_KEY" 
  -action revoke-cert 
  -service-name "cluster-sensor-service" 
  -reason "compromised"

Or directly in database:

UPDATE platform_bootstrap_certificates
SET revoked_at = NOW(),
    revocation_reason = 'compromised'
WHERE service_name = 'cluster-sensor-service';

After Revocation

  1. Generate New Certificate:

    ./scripts/generate-bootstrap-certificates.sh

  2. Update Certificate Files:

    • Replace revoked certificate
    • Restart affected service

  3. Verify:

    • Check service can register with new certificate
    • Verify old certificate is rejected

Monitoring

Certificate Expiration Monitoring

Check certificate expiration dates:

SELECT service_name, expires_at, 
       expires_at - NOW() as days_until_expiration
FROM platform_bootstrap_certificates
WHERE revoked_at IS NULL
ORDER BY expires_at;

Alert on Expiration

Set up alerts for certificates expiring within 30 days:

SELECT service_name, expires_at
FROM platform_bootstrap_certificates
WHERE revoked_at IS NULL
  AND expires_at < NOW() + INTERVAL '30 days';

Troubleshooting

Certificate Not Found

Symptoms: Service fails to start, certificate file not found

Solutions:

  1. Verify certificate files exist in ./bootstrap-certs/
  2. Check volume mount in docker-compose.yml
  3. Regenerate certificates if missing
  4. Verify file permissions

Certificate Validation Fails

Symptoms: Service registration fails with certificate validation error

Solutions:

  1. Verify certificate is signed by platform bootstrap CA
  2. Check certificate expiration date
  3. Verify certificate is not revoked
  4. Check CA certificate matches database

Service Cannot Register

Symptoms: Auto-registration endpoint returns 401 Unauthorized

Solutions:

  1. Verify bootstrap certificate is valid
  2. Check service name in certificate CN
  3. Verify sensor-manager can access bootstrap CA
  4. Check encryption key is correct

Security Best Practices

  1. Certificate Storage:

    • Store certificates securely (secrets manager in production)
    • Use restricted file permissions (600 for keys, 644 for certs)
    • Never commit certificates to version control

  2. Certificate Rotation:

    • Rotate before expiration
    • Rotate after security incidents
    • Document rotation procedures

  3. Access Control:

    • Limit access to certificate generation scripts
    • Restrict database access to bootstrap CA tables
    • Monitor certificate operations

  4. Audit:

    • Log certificate generation events
    • Track certificate usage
    • Monitor for suspicious activity