At 00:00 UTC, 100% of global web and mobile traffic to `api.globalshop.com` fails immediately with `SSL_ERROR_EXPIRED_CERT_ALERT` / `ERR_CERT_DATE_INVALID`. Yet your internal Prometheus/Kubernetes dashboards are all green—origin ALBs, pods, and health checks return `200 OK` because the expired certificate sits on the **CDN Edge** (and internal monitors bypassed the CDN or used `insecureSkipVerify: true`). Furthermore, the automated ACME renewal cron failed silently 21 days ago when a DNS TXT record permission was revoked. Design a zero-touch TLS lifecycle and edge-aware observability architecture.
From patterns to decisions.
At 00:00 UTC, 100% of global web and mobile traffic to `api.globalshop.com` fails immediately with `SSL_ERROR_EXPIRED_CERT_ALERT` / `ERR_CERT_DATE_INVALID`. Yet your internal Prometheus/Kubernetes dashboards are all green—origin ALBs, pods, and health checks return `200 OK` because the expired certificate sits on the **CDN Edge** (and internal monitors bypassed the CDN or used `insecureSkipVerify: true`). Furthermore, the automated ACME renewal cron failed silently 21 days ago when a DNS TXT record permission was revoked. Design a zero-touch TLS lifecycle and edge-aware observability architecture.
TLS termination occurs at the.
TLS termination occurs at the CDN edge before HTTP routing, and internal monitors bypassed the CDN or disabled certificate verification.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Strict verification catches expired, revoked, and mismatched SAN certs immediately. |
Step 1
Inspect CDN edge certificate notAfter via openssl s_client -connect api.globalshop.com:443 -servername api.globalshop.com.
Step 2
Audit synthetic monitors for insecure_skip_verify: true or internal VPC targets.
Alternative 1
External SaaS synthetic monitoring (Datadog/Checkly/Pingdom) from 3+ continents.
Failure Mode 1
Monitoring only the leaf cert and missing an expired intermediate CA cert (probe_ssl_earliest_cert_expiry checks the earliest in the chain).
modules:
http_2xx_strict_tls:
prober: http
http:
tls_config:
insecure_skip_verify: false
Renew at 30 days remaining.
Renew at 30 days remaining, alert on renewal failures immediately, and page at 21d (P2) and 7d (P1) from public edge probes.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | A 30-day renewal buffer gives SREs 9 days to notice a warning before the 21-day alert even fires. |
Configure `renewBefore
720h` (30 days) on 90-day certs.
Step 2
Alert on certmanager_certificate_ready_status == 0 so broken DNS-01 IAM roles are caught on day 1.
Step 3
Probe all 45 public domains via blackbox_exporter.
Alternative 1
Using CDN-native managed certificates (e.g., AWS ACM + CloudFront or Cloudflare Universal SSL).
Failure Mode 1
CDN API upload step failing after ACME already issued the cert.
- alert: CertManagerRenewalFailing
expr: certmanager_certificate_ready_status == 0
for: 15m
labels:
severity: warning
Prefer CDN-managed certs for single-CDN.
Prefer CDN-managed certs for single-CDN simplicity; use your own ACME + API push when running multi-CDN or strict custom key custody.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | CDN-managed certs remove operational burden but don't synchronize private keys across two different CDN vendors. |
Step 1
Check if your architecture uses multi-CDN DNS steering (e.g., Cloudflare + Fastly).
Step 2
Verify whether compliance requires HSM/customer-owned private keys.
Alternative 1
Cloudflare Origin CA for origin-to-edge + Universal SSL for edge.
Failure Mode 1
Assuming CDN-managed certs can't fail—domain validation TXT records can still be accidentally deleted.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: globalshop-edge
spec:
duration: 2160h # 90d
renewBefore: 720h # 30d
issuerRef:
name: letsencrypt-dns01
kind: ClusterIssuer
Use wildcard SAN consolidation to.
Use wildcard SAN consolidation to stay under rate limits and maintain a pre-configured secondary ACME CA (ZeroSSL / Google Trust Services).
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Maintaining External Account Binding (EAB) credentials for a backup CA takes 10 minutes once and eliminates single-CA risk. |
Step 1
Consolidate subdomains into wildcard certs (*.globalshop.com) to reduce cert count.
Step 2
Configure a secondary ClusterIssuer with ZeroSSL/GTS EAB credentials ready for 1-line failover.
Alternative 1
Purchasing a 1-year commercial DigiCert backup cert stored in Vault for break-glass use.
Failure Mode 1
Misconfigured CAA DNS records (0 issue "letsencrypt.org") blocking the backup CA from issuing.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: zerossl-fallback
spec:
acme:
server: https://acme.zerossl.com/v2/DV90
Monitor the origin ALB's TLS.
Monitor the origin ALB's TLS endpoint directly from inside the VPC with blackbox_exporter and automate origin cert rotation via AWS ACM or Vault PKI.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Origin-to-CDN mTLS stops origin-bypass attacks but doubles the number of cert chains you must monitor. |
Step 1
Add a dedicated blackbox_exporter target for the origin ALB hostname.
Step 2
Use AWS ACM (auto-renewed) on the origin ALB or automate Vault PKI rotation for mTLS client certs.
Alternative 1
Cloudflare Authenticated Origin Pulls with 10-year origin certs + WAF IP allowlisting.
Failure Mode 1
Rotating the CDN client cert without keeping the old CA in the origin truststore during propagation.
assert check_edge_cert('origin-alb.internal.globalshop.com') > 21, 'Origin cert expiring soon'
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Automate ACME (
cert-manager/ CDN-managed) renewals at 33% lifetime remaining, probe the public edge (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA. - 02Trade-off tip: CDN-managed certs remove operational burden but don't synchronize private keys across two different CDN vendors.
- 03Watch out for: Misconfigured CAA DNS records (
0 issue "letsencrypt.org") blocking the backup CA from issuing.
Automated Tls Certificate Rotation — Normal → Load / Fault → Recovery
Automate ACME (cert-manager / CDN-managed) renewals at 33% lifetime remaining, probe the public edge (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.
Putting Automating TLS Certificate Rotation & Expiry Alerts Together
Automate ACME (`cert-manager` / CDN-managed) renewals at 33% lifetime remaining, probe the *public edge* (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.
Putting Automating TLS Certificate Rotation & Expiry Alerts Together
- Configure
renewBefore: 720h(30 days) on 90-day certs. - Alert on
certmanager_certificate_ready_status == 0so broken DNS-01 IAM roles are caught on day 1. - Probe all 45 public domains via
blackbox_exporter.
Interview Readiness Check
Use wildcard SAN consolidation to stay under rate limits and maintain a pre-configured secondary ACME CA (ZeroSSL / Google Trust Services).
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is TLS Certificate Lifecycle the right default for a production team?
Automating TLS Certificate Rotation & Expiry Alerts
Automate ACME (cert-manager / CDN-managed) renewals at 33% lifetime remaining, probe the public edge (not origin) with synthetic TLS expiry checks, and keep a hot-swappable backup CA.
