Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Automating TLS Certificate Rotation
& Expiry Alerts.

At 00:00 UTC, 100% of global web and mobile traffic to api.globalshop.com fails immediately with SSL_ERROR_EXPIRED_CERT_ALERT / ERR_CERT_DATE_INVALID. Yet your internal Prometheus/Kubernetes dashboards are all green—origin ALBs, pods, and health checks return 200 OK because the expired certificate sits on the CDN Edge (and internal monitors bypassed the CDN or used insecureSkipVerify: true). Furthermore, the automated ACME renewal cron failed silently 21 days ago when a DNS TXT record permission was revoked. Design a zero-touch TLS lifecycle and edge-aware observability architecture.
Start the journey5 stages
Automating TLS Certificate Rotation & Expiry Alerts architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

At 00:00 UTC, 100% of global web and mobile traffic to `api.globalshop.com` fails immediately with `SSL_ERROR_EXPIRED_CERT_ALERT` / `ERR_CERT_DATE_INVALID`. Yet your internal Prometheus/Kubernetes dashboards are all green—origin ALBs, pods, and health checks return `200 OK` because the expired certificate sits on the **CDN Edge** (and internal monitors bypassed the CDN or used `insecureSkipVerify: true`). Furthermore, the automated ACME renewal cron failed silently 21 days ago when a DNS TXT record permission was revoked. Design a zero-touch TLS lifecycle and edge-aware observability architecture.

From patterns to decisions.

At 00:00 UTC, 100% of global web and mobile traffic to `api.globalshop.com` fails immediately with `SSL_ERROR_EXPIRED_CERT_ALERT` / `ERR_CERT_DATE_INVALID`. Yet your internal Prometheus/Kubernetes dashboards are all green—origin ALBs, pods, and health checks return `200 OK` because the expired certificate sits on the **CDN Edge** (and internal monitors bypassed the CDN or used `insecureSkipVerify: true`). Furthermore, the automated ACME renewal cron failed silently 21 days ago when a DNS TXT record permission was revoked. Design a zero-touch TLS lifecycle and edge-aware observability architecture.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

TLS termination occurs at the.

TLS termination occurs at the CDN edge before HTTP routing, and internal monitors bypassed the CDN or disabled certificate verification.

ChoiceWhat it buysWhat it costs
Trade-off 1Strict verification catches expired, revoked, and mismatched SAN certs immediately.

Step 1

Inspect CDN edge certificate notAfter via openssl s_client -connect api.globalshop.com:443 -servername api.globalshop.com.

Step 2

Audit synthetic monitors for insecure_skip_verify: true or internal VPC targets.

Alternative 1

External SaaS synthetic monitoring (Datadog/Checkly/Pingdom) from 3+ continents.

Failure Mode 1

Monitoring only the leaf cert and missing an expired intermediate CA cert (probe_ssl_earliest_cert_expiry checks the earliest in the chain).

modules:
  http_2xx_strict_tls:
    prober: http
    http:
      tls_config:
        insecure_skip_verify: false
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Renew at 30 days remaining.

Renew at 30 days remaining, alert on renewal failures immediately, and page at 21d (P2) and 7d (P1) from public edge probes.

ChoiceWhat it buysWhat it costs
Trade-off 1A 30-day renewal buffer gives SREs 9 days to notice a warning before the 21-day alert even fires.

Configure `renewBefore

720h` (30 days) on 90-day certs.

Step 2

Alert on certmanager_certificate_ready_status == 0 so broken DNS-01 IAM roles are caught on day 1.

Step 3

Probe all 45 public domains via blackbox_exporter.

Alternative 1

Using CDN-native managed certificates (e.g., AWS ACM + CloudFront or Cloudflare Universal SSL).

Failure Mode 1

CDN API upload step failing after ACME already issued the cert.

- alert: CertManagerRenewalFailing
  expr: certmanager_certificate_ready_status == 0
  for: 15m
  labels:
    severity: warning
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Prefer CDN-managed certs for single-CDN.

Prefer CDN-managed certs for single-CDN simplicity; use your own ACME + API push when running multi-CDN or strict custom key custody.

ChoiceWhat it buysWhat it costs
Trade-off 1CDN-managed certs remove operational burden but don't synchronize private keys across two different CDN vendors.

Step 1

Check if your architecture uses multi-CDN DNS steering (e.g., Cloudflare + Fastly).

Step 2

Verify whether compliance requires HSM/customer-owned private keys.

Alternative 1

Cloudflare Origin CA for origin-to-edge + Universal SSL for edge.

Failure Mode 1

Assuming CDN-managed certs can't fail—domain validation TXT records can still be accidentally deleted.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: globalshop-edge
spec:
  duration: 2160h # 90d
  renewBefore: 720h # 30d
  issuerRef:
    name: letsencrypt-dns01
    kind: ClusterIssuer
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Use wildcard SAN consolidation to.

Use wildcard SAN consolidation to stay under rate limits and maintain a pre-configured secondary ACME CA (ZeroSSL / Google Trust Services).

ChoiceWhat it buysWhat it costs
Trade-off 1Maintaining External Account Binding (EAB) credentials for a backup CA takes 10 minutes once and eliminates single-CA risk.

Step 1

Consolidate subdomains into wildcard certs (*.globalshop.com) to reduce cert count.

Step 2

Configure a secondary ClusterIssuer with ZeroSSL/GTS EAB credentials ready for 1-line failover.

Alternative 1

Purchasing a 1-year commercial DigiCert backup cert stored in Vault for break-glass use.

Failure Mode 1

Misconfigured CAA DNS records (0 issue "letsencrypt.org") blocking the backup CA from issuing.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: zerossl-fallback
spec:
  acme:
    server: https://acme.zerossl.com/v2/DV90
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Monitor the origin ALB's TLS.

Monitor the origin ALB's TLS endpoint directly from inside the VPC with blackbox_exporter and automate origin cert rotation via AWS ACM or Vault PKI.

ChoiceWhat it buysWhat it costs
Trade-off 1Origin-to-CDN mTLS stops origin-bypass attacks but doubles the number of cert chains you must monitor.

Step 1

Add a dedicated blackbox_exporter target for the origin ALB hostname.

Step 2

Use AWS ACM (auto-renewed) on the origin ALB or automate Vault PKI rotation for mTLS client certs.

Alternative 1

Cloudflare Authenticated Origin Pulls with 10-year origin certs + WAF IP allowlisting.

Failure Mode 1

Rotating the CDN client cert without keeping the old CA in the origin truststore during propagation.

assert check_edge_cert('origin-alb.internal.globalshop.com') > 21, 'Origin cert expiring soon'

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Automate ACME (cert-manager / CDN-managed) renewals at 33% lifetime remaining, probe the public edge (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.
  • 02Trade-off tip: CDN-managed certs remove operational burden but don't synchronize private keys across two different CDN vendors.
  • 03Watch out for: Misconfigured CAA DNS records (0 issue "letsencrypt.org") blocking the backup CA from issuing.
Behavioral Progression · Automated Tls Certificate Rotation

Automated Tls Certificate Rotation — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Automate ACME (cert-manager / CDN-managed) renewals at 33% lifetime remaining, probe the public edge (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.

Synthesis / One connected design

Putting Automating TLS Certificate Rotation & Expiry Alerts Together

Automate ACME (`cert-manager` / CDN-managed) renewals at 33% lifetime remaining, probe the *public edge* (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.

A candidate architecture, not a universal answer

Putting Automating TLS Certificate Rotation & Expiry Alerts Together

Step 1
Step 2
Step 3
  1. Configure renewBefore: 720h (30 days) on 90-day certs.
  2. Alert on certmanager_certificate_ready_status == 0 so broken DNS-01 IAM roles are caught on day 1.
  3. Probe all 45 public domains via blackbox_exporter.
Use wildcard SAN consolidation to stay under rate limits and maintain a pre-configured secondary ACME CA (ZeroSSL / Google Trust Services).
Interview Readiness Check

Use wildcard SAN consolidation to stay under rate limits and maintain a pre-configured secondary ACME CA (ZeroSSL / Google Trust Services).

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is TLS Certificate Lifecycle the right default for a production team?

Key takeaway

Automating TLS Certificate Rotation & Expiry Alerts

Automate ACME (cert-manager / CDN-managed) renewals at 33% lifetime remaining, probe the public edge (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.