Infrastructure as Code Pattern: GitOps, Reusable Modules, and Policy Guardrails

Scenario context

Infrastructure as code is useful because it makes infrastructure changes reviewable, repeatable, and testable. It becomes production-grade only when module boundaries, state locking, GitOps flow, and policy checks are treated as operational controls rather than tooling preferences.

TLDR: The SRE question is not “do we use Terraform?” It is “can we explain how infra changes are planned, approved, applied, roll…

This walkthrough focuses on When IaC and GitOps Actually Help.

Before making a design choice, clarify:

• IaC is strongest when infrastructure changes are frequent enough to need guardrails and consistent enough to encode declaratively.
• Use this pattern when: - multiple environments must stay aligned, - teams need audited change history, - infra drift creates repeated surprises, - security and reliability rules should block unsafe changes before apply.

Concepts in play: canary-deployments, git-internals-and-versioning, consistency-replication, system-design-hld-architectures

Stage 1 — How IaC Works in Production

• A practical flow is simple and disciplined: 1.
• Engineers propose infra changes in version control.
• CI validates syntax, module inputs, and generated plan.
• Policy checks block unsafe changes such as public exposure, missing tags, or unrestricted IAM.

Observable outcome: Policy checks block unsafe changes such as public exposure, missing tags, or unrestricted IAM.

Stage 2 — Deep Dive: What Incident Reviews Usually Reveal First

• Field note: one of the highest-signal metrics in mature IaC programs is not “apply succeeded.” It is “how often did someone bypass the path entirely?” Manual change count is often the best predictor of the next surprise.
• Internals: How Shopify, Datadog, and HashiCorp Build IaC Programs Shopify manages Terraform at scale by coupling module ownership to team boundaries.
• Each platform team owns a versioned module set published via a private Terraform registry.
• Any change to a shared "golden module" triggers CI on downstream consumers automatically — not just the module's own tests.

sequenceDiagram
    participant Dev as Developer
    participant Git as Git Repo
    participant ACD as ArgoCD
    participant K8s as K8s Cluster
    Dev->>Git: Push config change
    ACD->>Git: Detect diff (polling)
    ACD->>ACD: Compare desired vs actual
    ACD->>K8s: Apply manifests
    K8s-->>ACD: Sync complete
    ACD-->>Dev: Deployment status OK

Observable outcome: Any change to a shared "golden module" triggers CI on downstream consumers automatically — not just the module's own tests.

Stage 3 — IaC and GitOps Control Flow

• This flowchart maps the full GitOps control flow from a pull request carrying an infrastructure change through syntax validation, plan generation, and policy checks to a human-reviewed apply and ongoing drift monitoring.
• The critical gate is the policy check node: hard-block failures reject the change before it reaches apply, ensuring guardrails run before any blast radius is assessed.
• The takeaway is that drift detection closes the loop — any deviation from declared state feeds back into the same plan-review-apply cycle.

flowchart TD
    A[Pull request with infra change] --> B[Validate syntax and module inputs]
    B --> C[Generate plan or desired-state diff]
    C --> D[Run policy checks]
    D --> E{Checks pass?}
    E -->|No| F[Reject change before apply]
    E -->|Yes| G[Human review of blast radius]
    G --> H[Controlled apply or GitOps merge]
    H --> I[Runtime reconciliation]
    I --> J[Drift detection and alerts]

Observable outcome: The takeaway is that drift detection closes the loop — any deviation from declared state feeds back into the same plan-review-apply cycle.

Stage 4 — Concrete Config Example: CI Gate for Terraform and Policy Checks

• This example demonstrates a GitHub Actions CI pipeline that gates infrastructure changes with Terraform plan generation and automated policy enforcement checks before any human review.
• It was chosen because CI-based IaC validation is the most practical first control to implement — it catches risky changes before apply without requiring a dedicated policy server or manual checklist.
• Read the workflow steps in sequence: checkout, authenticate, generate plan, then enforce policy — each step is a distinct failure boundary with its own blast-radius scope.
• An example OPA (Rego) policy that blocks public S3 exposure and unrestricted security group rules: Conftest evaluates this against the Terraform JSON plan before any human review happens.

name: infra-plan
on:
  pull_request:
    paths:
      - infra/**

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - use
s: actions/checkout@v4
      - ru
n: terraform -chdir=infra/prod init -input=false
      - ru
n: terraform -chdir=infra/prod validate
      - ru
n: terraform -chdir=infra/prod plan -out=tfplan -input=false
      - ru
n: terraform -chdir=infra/prod show -json tfplan > tfplan.json
      - ru
n: conftest test tfplan.json --policy policy/

# policy/deny_public_exposure.rego
package main

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket_acl"
  resource.change.after.acl == "public-read"
  msg := sprintf("BLOCKED: S3 bucket '%v' must not use public-read ACL", [resource.address])
}

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_security_group_rule"
  resource.change.after.cidr_blocks[_] == "0.0.0.0/0"
  resource.change.after.from_port == 0
  msg := sprintf("BLOCKED: '%v' opens all ports to the internet", [resource.address])
}

Observable outcome: An example OPA (Rego) policy that blocks public S3 exposure and unrestricted security group rules: Conftest evaluates this against the Terraform JSON plan before any human review happens.

Stage 5 — Real-World Applications: What to Instrument and What to Audit

Observable outcome: You can explain how “Real-World Applications: What to Instrument and What to Audit” changes the architecture or implementation path.

Stage 6 — Terraform, Pulumi, and AWS CDK: IaC Tooling for Production Infrastructure

• Terraform (HashiCorp) is the most widely adopted declarative IaC tool, using HCL to define infrastructure resources with a plan/apply workflow, remote state, and a vast provider ecosystem.
• Pulumi offers the same capabilities using general-purpose languages (TypeScript, Python, Java, Go) instead of HCL.
• AWS CDK is Amazon's IaC framework using TypeScript or Python that compiles to CloudFormation.
• These tools solve the IaC problem by making infrastructure changes reviewable, diffable, and policy-checkable before any resource is modified.

# terraform/modules/api-service/variables.tf
variable "service_name" {
  type        = string
  description = "Unique name for this service instance — scopes all resources."
}

variable "environment" {
  type        = string
  description = "Deployment environment: dev | staging | prod"
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be one of: dev, staging, prod"
  }
}

variable "min_instances" { type = number; default = 2 }
variable "max_instances" { type = number; default = 10 }

# terraform/modules/api-service/main.tf
resource "aws_autoscaling_group" "api" {
  name             = "${var.service_name}-${var.environment}"
  min_size         = var.min_instances
  max_size         = var.max_instances
  # ...
  tag {
    key                 = "service"
    value               = var.service_name
    propagate_at_launch = true
  }
  tag {
    key                 = "environment"
    value               = var.environment
    propagate_at_launch = true
  }
}

# Callers instantiate the module per environment — one change touches one env
module "payments_api_prod" {
  source        = "./modules/api-service"
  service_name  = "payments-api"
  environment   = "prod"
  min_instances = 4
  max_instances = 20
}

flowchart TD
    A[Write .tf config] --> B[terraform init]
    B --> C[terraform plan]
    C --> D{Review diff}
    D -->|Approved| E[terraform apply]
    D -->|Rejected| A
    E --> F[State file updated]
    F --> G[Infra matches config]
    G --> H[Monitor drift]
    H -->|Drift detected| C

Observable outcome: These tools solve the IaC problem by making infrastructure changes reviewable, diffable, and policy-checkable before any resource is modified.

End-to-end outcome

IaC is strongest when infrastructure changes are frequent enough to need guardrails and consistent enough to encode declaratively.

Alternatives considered

Decision Guide for SRE Teams

If the plan review cannot clearly answer “what might be deleted or replaced?”, the change is not ready.

Failure and degraded modes

When Not to Over-Apply IaC

• Detection: IaC needs judgement: encoding one-off decisions into broad shared modules can make change harder, not safer.
• User or operator impact: IaC needs judgement: encoding one-off decisions into broad shared modules can make change harder, not safer.

Trade-offs & Failure Modes: Pros, Cons, and Alternatives

• Detection: Trade-offs & Failure Modes: Pros, Cons, and Alternatives
• User or operator impact: Trade-offs & Failure Modes: Pros, Cons, and Alternatives