Deploy on Azure with Azure DevOps

The Azure Terraform bundle includes Azure DevOps pipelines in its .azuredevops/ folder. They run the same seven modules as the Terraform guide, from Microsoft-hosted agents, with an approval gate on every write.

Pipeline	Does
pipelines/terraform-azure.yml	Manual. Plans, applies or destroys the selected modules, or sets the Key Vault operator secrets. Every write waits for an approval.
pipelines/environment-up.yml	Scheduled, 09:00 Mon–Fri. Starts the metered resources.
pipelines/environment-down.yml	Scheduled, 18:00 Mon–Fri. Stops them.

Each pipeline step is a script in .azuredevops/bin/. Every script runs by hand under az login with the same variables exported, so you can review and test each one without Azure DevOps.

What a run does

The Operation you pick when queuing decides which stages the run has. The list has no headings, so each operation names its group:

Operation	Stages	Writes
Terraform · Plan only (default)	Plan	nothing
Terraform · Plan & apply	Plan → Apply	the plans (and the migration and repairs, with Run DB migrate)
Terraform · Destroy plan	Plan destroy	nothing
Terraform · Destroy immediately	Plan destroy → Destroy	the destroy plans
Terraform · Show outputs	Show outputs	nothing — prints each selected module’s outputs
Operations · Restart apps	Restart apps	restarts the running apps
Operations · Start apps	Start apps	starts all four apps
Operations · Stop apps	Stop apps	stops all four apps
Operations · Run DB bootstrap	Run DB bootstrap	runs the database bootstrap job (roles, grants, extensions)
Operations · Run DB repair	Run DB repair	runs the database repair job (doctor:repair)
Operations · Set secrets	Set secrets	the operator secrets
Certificates · Import frontend certificate	Import frontend certificate	imports the Application Gateway certificate into Key Vault
Certificates · Create frontend CSR	Create frontend CSR	generates the Application Gateway key pair inside Key Vault and prints its CSR
Certificates · Merge frontend certificate	Merge frontend certificate	completes that key pair with the certificate Cloudflare signed
Debug · Break state lock	Break state lock	releases a stuck Terraform state lock
Debug · Delete ACA environment	Delete ACA environment	deletes the Container Apps environment

Every operation that writes waits for the ADO Environment’s approval. Stage names say what they cover — Plan foundation keyvault, Apply app, migrate and repair DB, Plan destroy edge app.

flowchart LR
    subgraph plan ["Plan, or Plan destroy"]
        P1["Preflight<br/>read-only"] --> P2["Open state account<br/>and Key Vault"] --> P3["init + plan<br/>each module"] --> P4["Restore both<br/>always"]
    end
    subgraph apply ["Apply, or Destroy"]
        A0["Approval"] --> A1["Preflight"] --> A2["Open state account<br/>and Key Vault"] --> A3["apply each<br/>module's plan"] --> A4["Restore both<br/>always"]
    end
    plan --> apply
    subgraph secrets ["Secrets and certificates"]
        S0["Approval"] --> S1["Open Key Vault"] --> S2["Write"] --> S3["Restore<br/>always"]
    end

• Modules 1–7 select the roots a Terraform operation or Break state lock touches. Plans and applies run them in order; destroys run them in reverse, so no module is removed from under one that still depends on it.
• A module with no changes is skipped, and with no changes in any module the Apply or Destroy stage is skipped and asks for no approval (unless Run DB migrate is ticked). The plans are one artifact of the run, tfplans-<attempt>, with a summary.txt of what changed.
• After a failure, use Rerun failed jobs. A failed plan re-runs cleanly. A failed apply re-runs from the module that failed, skipping those already applied — provided the failed module changed nothing before it failed (an expired credential, a closed firewall). If it got part-way, or you changed the code, queue a fresh Plan & apply for that module and the ones below it.
• The operation is fixed when you queue. A plan-only run cannot be approved into an apply later; queue again, and the plans are re-cut against current state.
• Run DB migrate is a checkbox below the modules, used with Terraform · Plan & apply. It stops web and job, applies the app module if it is ticked, runs the migration, then the repairs (bin/rails doctor:repair), and starts web and job again — so no app runs against a schema mid-migration, and no pod of the previous release is left to undo a repair. With app unticked, the run is just stop, migrate, repair, start. If the migration or a repair fails, web and job stay stopped; both are safe to re-run. Run it on every release.
• The destructive operations need a confirmation — Terraform · Destroy immediately, Debug · Break state lock and Debug · Delete ACA environment. Type destroy now in Destroy confirmation (case and spaces ignored), or the run is a single Destructive action refused stage and nothing changes. The approval still applies.
• Start apps and Stop apps start and stop all four apps, the inference adapter first on the way up. An app set to 0 replicas in the environment file stays stopped.
• Delete ACA environment removes the Container Apps environment and every app in it. Use it only for an environment Terraform no longer tracks — one that blocks destroying the foundation module because it still holds its subnet. Otherwise use a destroy.
• Set secrets runs alone, after the keyvault module exists.
• Run DB bootstrap starts the database bootstrap job, which Terraform creates but never starts. It waits for the job to finish, prints its output into the pipeline log, and passes or fails with it. Run it after the database module. Safe to re-run.
• Run DB repair runs the platform’s post-deploy repairs (bin/rails doctor:repair) the same way, without stopping the apps. Run DB migrate already runs them; use this after a release you rolled out without migrating, when its release notes name a repair, and only once each app shows a single active revision. Each repair prints a count, and a second run reports zero.
• Every Apply prints the outputs of each module it applies, and Show outputs prints them any time: the inference adapter certificate’s names, the Application Gateway’s address for your DNS record, the app URLs.
• Import frontend certificate takes the Cloudflare Origin CA certificate and key as a PFX from the variable group (frontendCertPfxB64, base64 on one line, and frontendCertPassword), imports it into Key Vault as frontend-cert, and prints the value for edge.appgw_certificate_key_vault_secret_id. Re-import to rotate it.
• Create frontend CSR → Merge frontend certificate keeps the certificate’s private key inside Key Vault from start to finish. Create generates the key pair in the vault for edge.appgw_frontend_hostname and prints a CSR (also a downloadable artifact of the run). In Cloudflare, SSL/TLS → Origin Server → Create Certificate → Use my private key and CSR, paste it, and keep the certificate PEM it returns. Put it in frontendCertSignedPemB64 and run Merge. This is the preferred route: no private key is ever handled or sent.
• Restart apps makes running apps re-read their Key Vault secrets, which they otherwise read only when they start. Run it after Set secrets or any rotation. Stopped or scaled-to-zero apps are skipped.
• Break state lock releases the lock a cancelled run leaves behind (Error acquiring the state lock), showing who held it. Use it only when no other run is in progress.
• The restore steps run under always(): a failed plan, apply or secret write never leaves a firewall open.

Every module is planned before any is applied, so a downstream plan reflects its upstream modules’ currently applied state. Where an upstream module has real changes, apply it alone first, then run the modules below it.

Preflight

bin/ado-preflight.sh is the first step of Plan and Apply. It is read-only, and it catches the failures that otherwise surface partway through an apply, after resources exist:

Check	Why
Identity	Who the pipeline authenticated as, and that the subscription is the one the variable group names.
Resource providers	All nine are registered. The bundle never registers them itself.
Resource group	Exists, and in which region. If it does not, the subscription-level create right is checked instead.
RBAC on the group	Representative writes, plus roleAssignments/write and roleDefinitions/write, which Contributor does not carry. Prints the grant to ask for. Reports deny assignments, which override every grant.
State account	Visible, its current network posture, and both the control-plane write the firewall step needs and the data-plane write terraform init needs.
Key Vault	vaults/write for the firewall step, and the data-plane secret rights the Secrets stage needs. Owner and Contributor grant no access to secret values on an RBAC vault. Checked from the first run after keyvault exists.
Operator secrets	Which still read REPLACE_ME. A warning, not a failure.
Environment file	environments/<tfWorkspace>.yaml exists.

Run it locally before the first pipeline run:

SUBSCRIPTION_ID=<id> RESOURCE_GROUP=<rg> BACKEND_RG=<rg> \
  BACKEND_STORAGE=<account> TF_WORKSPACE=<env> \
  .azuredevops/bin/ado-preflight.sh

The state account firewall

The Terraform state account rests closed to the internet. Hosted agents reach it over the public endpoint, so each stage:

1. Opens it (bin/ado-open-backend-firewall.sh) as narrowly as the agent allows, records the settings it found, and proves the state container is readable before Terraform starts.
2. Restores it (bin/ado-close-backend-firewall.sh) to exactly the recorded settings — not to an assumed default — so a run never tightens or loosens an account that rests differently.

The Secrets stage does the same for the Key Vault (bin/ado-open-keyvault.sh, bin/ado-close-keyvault.sh).

Same-region agents cannot be allow-listed

A hosted agent in the same region as the storage account cannot be admitted by an IP rule

Azure Storage IP rules do not apply to requests from the account’s own region. That traffic arrives over Azure’s internal network with a private source address, which storage never matches against an IP rule — and the API refuses a private address as a rule value. So no rule admits a same-region agent: not its public IP, which storage never sees, and not its private one, which cannot be added.

Microsoft-hosted agents are assigned a region per job, so one run can get both outcomes — a Plan that works and an Apply that times out, or the reverse.

The open step therefore works out the agent’s region (from the instance metadata service, otherwise by locating its public IP in the AzureCloud.<region> service tags) and picks the narrowest posture that works:

Agent	Posture for the job
Different region, or outside Azure	Public access on, this agent’s IP only, default action Deny.
Same region as the account	A warning, then public access on with default action Allow and no IP rule. Entra ID auth and RBAC are the only controls until the restore step runs.

Set Public network setting to Fail for same-region agents to refuse instead. The job then stops in seconds with the reason. The Key Vault steps apply the same fallback.

Allow-listing Azure service tags does not help. The AzureDevOps tag does not contain hosted agents, the AzureCloud tags exceed the 200-rule limit even per region, and no public-address rule admits a same-region agent anyway.

The lasting fix is a private endpoint on the state account and an agent inside the VNet — self-hosted, or a Managed DevOps Pool pinned to a region and injected into a subnet. The firewall steps then have nothing to do.

The Secrets stage

Run the pipeline with Operation Set secrets to write the operator secrets from the variable group into the vault. This is the pipeline equivalent of setting them by hand after the keyvault module.

Secret	Variable
auth-client-id / auth-client-secret	authClientId / authClientSecret
ghcr-token	ghcrToken
db-adapter-api-key	dbAdapterApiKey (only where a database adapter is configured)
inference-api-key / inference-api-keys	inferenceApiKey / inferenceApiKeys

• Mark every one secret in the variable group. Each also accepts a …B64 variant holding base64, which is how a multi-line value travels intact.
• The inference adapter’s certificate is not set here. Key Vault generates it when the app module is applied. Delete any inferenceTlsCert or inferenceTlsKey (or …B64) variable; the stage warns while one remains.
• The variable group is the source of truth. The stage writes what the group holds, without reading the vault first. Re-writing an identical value is harmless, but a stale value in the group overwrites a secret rotated by hand. If the group is not kept current, don’t use Set secrets.
• Every write is a new Key Vault version. Old versions are kept, so a wrong value is recovered with az keyvault secret list-versions and setting the previous value back. The apps read versionless references.
• It refuses the generated secrets by name — enc-primary-key, enc-index-key, enc-key-derivation-salt and secret-key-base — whatever the group contains. A changed encryption key makes every encrypted column unreadable.
• An undefined variable is a named failure. Azure DevOps passes an undefined variable as its literal macro text ($(authClientSecret)), not as an empty string. The stage checks values by shape, so that text never reaches the vault.

The pipeline’s service principal needs Key Vault data-plane rights: put its service principal object id in keyvault.deployer_object_id.

Scheduled start and stop

environment-up.yml and environment-down.yml turn the metered resources off out of hours. They need no Terraform state, so the firewall steps and the same-region limitation do not apply to them.

Resource	Down	Up
Application Gateway v2	stopped first, so nothing new arrives	started last
Container Apps (web, job, OCR, inference)	stopped	started (inference adapter first), with the replica minimums from environments/<workspace>.yaml; an app at 0 there stays stopped
PostgreSQL Flexible Server	stopped last	started first

NAT gateway, public IPs, Log Analytics, storage and AI Foundry cannot be stopped, or already cost nothing idle.

• The crons are UTC. Azure DevOps YAML schedules have no timezone field. The shipped times are 09:00 / 18:00 at UTC+8 (0 1 * * 1-5 / 0 10 * * 1-5); change both for another timezone, and revisit them twice a year where daylight saving applies.
• always: true is required, and set. Without it a schedule runs only when the branch has changed since the last run.
• Schedules run from main and read the pipeline’s default variable group, since a schedule cannot pass parameters.
• Stopped apps stay stopped through a Terraform apply, until the next up — Terraform does not manage whether an app is running. The database and the gateway are another matter, so still do not deploy inside the off-hours window.
• Every step is idempotent, so a re-run after a partial failure is safe.

Setting up

1. Register the resource providers, once, as someone with subscription rights:

   SUBSCRIPTION=<id> .azuredevops/bin/register-resource-providers.sh

   DRY_RUN=true shows what it would do; WITH_DEFENDER=true adds Microsoft.Security.

2. Create an ARM service connection whose service principal holds Contributor + Role Based Access Control Administrator on the resource group and Storage Blob Data Contributor on the state account.

3. Create the variable group (Pipelines → Library):

   Variable	Example	Holds
   serviceConnection	Enprivacy-Terraform	The service connection’s name
   subscriptionId	00000000-…	Target subscription
   tenantId	00000000-…	The service principal’s tenant
   resourceGroup	rg-enprivacy-prod	The deployment’s resource group — foundation.resource_group_name when adopting
   backendRG / backendStorage / backendContainer	rg-tfstate / contosotfstate / tfstate	The state account
   tfWorkspace	prod	The environment name; needs environments/<name>.yaml
   The Secrets stage variables	—	Only if you use that stage. Mark each secret.

4. Create an ADO Environment (Pipelines → Environments), add an Approvals and checks entry to it, and add its name to the adoEnvironment parameter’s values in the pipeline file.

5. Create the pipelines from the three YAML files. The scheduled pair is optional.

6. Authorise the service connection for each pipeline before the first run. The connection name comes from a variable, so ADO cannot prompt for it at queue time.

7. Run the preflight locally, or queue a plan-only run.

Where each setting lives

	Holds	Set in
Variable group	Pipeline settings: subscription, tenant, service connection, state account, workspace — and the Secrets stage values	ADO Library
environments/<workspace>.yaml	Application settings: public hostname, image digests, replica counts, sizing, feature gates	The repository, applied by Terraform
Key Vault	Secret values only	By hand, or the Secrets stage

The public hostname, for example, is app.public_hostname in the environment yaml, not a pipeline setting.

The run-form choices — Operation, ADO Environment, Public network setting, the Module checkboxes and the destroy confirmation — are pipeline parameters, not variables. To make one a standing default, change its default: in the pipeline file.