Maintenance-Window Upgrades

Most releases upgrade by rolling the new image out and letting the Web service migrate on boot. A few cannot: they rename something the database stores — a task’s class name, a key inside a JSON column, an attachment’s name — and for a few minutes of a rolling deploy the old and new code would read and write the same rows. Neither version survives that. An old Job worker raises on a row the new code renamed, and a new one silently miscounts a row the old code wrote.

Those releases deploy in a maintenance window: nothing runs while the database changes. This page is the procedure, the list of releases that need it, and what to check after each.

Which releases need it. The release notes say so, and the table in Releases that need a window lists them. Every other release upgrades as usual.

What you need

Mission Control access: a user holding admin.jobs with manage. Mission Control is at /mission_control/jobs on your Enprivacy hostname.
A Rails console or runner in a Web or Job container: bin/rails runner from a shell in the container (docker compose exec, kubectl exec, az containerapp exec, or your platform’s equivalent). Steps 2, 3, 6 and 7 use it.
A database backup you can restore, taken in step 4 once nothing is running.
The new image digest, and the old one, for rollback.

Every snippet below runs as bin/rails runner - with the Ruby on standard input. Add EP_SKIP_LIBPOSTAL=true to the environment of any one-off Rails command: the address parser is not needed to manage queues, migrate or repair, and loading it takes about 1.8 GiB.

EP_SKIP_LIBPOSTAL=true bin/rails runner - <<'RUBY'
# snippet
RUBY

How the platform behaves in a window

Read this once; each step relies on it.

Pausing a queue stops claims and nothing else. A paused queue keeps accepting jobs: work enqueued during the window waits as ready rows and runs when the queue resumes. A worker that has already claimed a job finishes it. Pauses are rows in the database, so they survive a restart and an image change. Nothing resumes a queue except you.
The app cannot stop enqueueing work. Every finishing task enqueues its successors, and the recurring scheduler enqueues its sweeps on its own schedule. Pausing every queue is what holds all of it.
The recurring scheduler runs inside the Job service and cannot be stopped on its own while the service runs: the Job supervisor restarts it. With every queue paused, its jobs only queue up. It stops when the Job service stops in step 4. The sweeps that queue up during the drain are idempotent and run after the queues resume.
Nothing is killed because of elapsed time. One long task, such as a large archive being unpacked, can hold the drain open indefinitely. You set the drain’s deadline (step 3).
Work stopped at the deadline is recovered, not lost. A worker stopped with a claimed job fails that job. After the queues resume, the workflow sweep finds the task whose job is gone, records the attempt as failed and retries the task from the start, within its retry budget. From the task graph release on, the sweep runs every minute; before it, every five.
Search runs in Web, not on the queues, and stops when Web stops. Chat turns run on the queues: from the task graph release on, an API (proxy) turn’s redaction runs on the Job service while the request waits for it, so a proxy turn needs both services. Announce the window so nobody is mid-conversation.

The procedure

1. Announce and prepare

Announce the window to users: Enprivacy is unavailable from the start of step 4 until step 7 completes, and ingestion pauses from step 2. Choose a low-load time: the drain is shorter, and the deadline rarely bites.
Disable anything that would deploy for you. An image auto-updater (watchtower and the like), a scheduled start/stop of the services, or a pipeline that rolls new revisions on push would bring code back mid-window. Turn them off until step 7.
Time the migrations if this is the first window on a large install: restore last night’s backup to a scratch database and run bin/rails db:migrate against it with the new image. Some migrations in a window release scale with the size of a table (see the table below). If your migration job has a timeout, it must exceed this.
Record the “before” figures. Run the check snippet and keep its output. Note the Partly read count on one or two sources that hold archives or email.

2. Pause every queue

queues = %w[real_time default batch detection_real_time detection_default detection_batch] |
         SolidQueue::Queue.all.map(&:name)
queues.each { SolidQueue::Queue.find_by_name(it).pause }
puts "paused: #{SolidQueue::Pause.order(:queue_name).pluck(:queue_name).join(', ')}"

The six names are the queues the Job service works (config/queue.yml); the second list adds any other queue that holds jobs. A queue can be paused before it has ever held a job, which is why the names are listed rather than read from the jobs table alone.

Mission Control’s Queues page has a pause button per queue and shows which are paused, but it lists only queues that currently hold jobs. If you pause from there, check the page again just before step 4 and pause any queue that has appeared.

3. Drain to your deadline

Choose a deadline before you start: how long you will wait for in-flight work. Watch it drain:

claimed = SolidQueue::ClaimedExecution.includes(:job, :process).order(:created_at)
puts "claimed: #{claimed.size}"
claimed.each { |c| puts [c.created_at.utc.iso8601, c.job.queue_name, c.job.class_name, c.process&.name].join("  ") }
puts "paused queues: #{SolidQueue::Pause.count}  waiting (ready): #{SolidQueue::ReadyExecution.count}"

Mission Control shows the same under In progress (/mission_control/jobs/in_progress/jobs), with how long each has run.

Drained means claimed: 0. Workflow task runs still reading working in Desk are not in flight if nothing is claimed: a task waiting on its child runs has no job, and a queued task waits as a ready row.
At the deadline, go to step 4 with whatever remains. The jobs still claimed fail when their worker stops, and their tasks are retried from the start after step 7 (see above). Do not retry those failed jobs from Mission Control: the workflow sweep retries the task itself.
The count only falls. No new claims happen on a paused queue. If it rises, a queue is not paused: run step 2 again.

4. Stop Web and Job

Stop every Web and Job process, including any second Job replica and any old revision your platform keeps warm. A leftover Job process is not idle: it polls the database and would run old code against the new schema. If you run jobs inside Puma (SOLID_QUEUE_IN_PUMA), stopping Web stops them.

Give the Job service time to stop gracefully (30 seconds is ample). Confirm nothing is left. A process that stopped gracefully deregisters itself, so this returns no rows; a row left by a killed process has a last_heartbeat_at from before the stop, and the next Job start prunes it.

Back up the database now. Nothing is running, so the backup is consistent and holds everything up to the window, including the paused queues. It is the one rollback that works at any point.

SELECT kind, name, hostname, last_heartbeat_at FROM solid_queue_processes;

5. Deploy the new image and migrate

Point Web and Job at the new image digest, but do not start them. Run the migrations once, as a one-off command from the new image:

EP_SKIP_LIBPOSTAL=true bin/rails db:migrate

(Where a dedicated migration job owns schema changes — SKIP_DB_PREPARE=1 on Web — run that job instead; it runs bin/rails db:prepare, which migrates.)

A migration that cannot be undone checks before it acts. The task graph release copies every run into new tables and then, in the same db:migrate, drops the old ones. The drop first verifies the copy: every run, task and task run it should have carried is in the new tables, and nothing else still points at the old ones. If any check fails, the migration stops with CopyIncomplete and the counts that failed, and nothing is dropped. Leave Web and Job stopped and go to Rollback. The copy is already in, and the old tables are still there.

Each data migration in a window release selects only rows still in the old shape, so a migration interrupted part-way is safe to run again.
An interrupted index build leaves an invalid index, and the re-run fails with “relation already exists”. The check snippet counts invalid indexes; name them with SELECT indexrelid::regclass FROM pg_index WHERE NOT indisvalid;, drop the one the migration was building with DROP INDEX CONCURRENTLY <name>;, and migrate again.
If a migration fails and you cannot fix it forward, go to Rollback. Web and Job stay stopped.

6. Start, repair, verify

Start Web and Job on the new image. The queues are still paused, so the Job service claims nothing yet.
Run the repairs when the release notes name one (the table below does):
Terminal window
```
EP_SKIP_LIBPOSTAL=true bin/rails doctor:repair
```
Each repair is idempotent and prints a count. In a window nothing old wrote after the migrations, so most report zero; a backfill reports what it filled. A second run reports zero.
Verify, before anyone else signs in:
- GET /up answers 200 on Web.
- Mission Control’s Queues page lists every queue as paused, and Workers shows the new Job processes.
- The check snippet shows every “old” count at zero, the matching “new” count equal to the “before” figure, and no invalid indexes or unvalidated keys.
- Desk loads, and the run history of a pre-window ingest opens without an error: that is the renamed rows loading under the new code.
- The Partly read counts match the “before” figures.

Rollback is cheap up to here and expensive after step 7: see Rollback.

7. Resume, then smoke-test

SolidQueue::Pause.pluck(:queue_name).each { SolidQueue::Queue.find_by_name(it).resume }
puts "paused: #{SolidQueue::Pause.count}"

Or Resume on each queue in Mission Control. The work that waited runs, the sweeps queued during the drain run, and within minutes the workflow sweep picks up anything stopped at the deadline.

Then:

Smoke ingest. Upload one small document and one small ZIP to a workspace and watch both runs reach succeeded (or degraded, for a ZIP with a member that cannot be read). Check the release’s own items in the table.
Watch Mission Control’s Failed jobs for the first hour. A failure naming SubclassNotFound means a renamed row was missed: stop and investigate.
Re-enable whatever you disabled in step 1, and end the announcement.

The check snippet

Run it in step 1 and again in step 6. It reads only, and works on any release’s schema: a line whose table that release does not have reads n/a.

q = ->(sql) { ActiveRecord::Base.connection.select_value(sql).to_i rescue "n/a" }
members = "FROM active_storage_attachments a JOIN content_items i ON i.delegate_type = a.record_type " \
          "AND i.delegate_id = a.record_id WHERE i.contained_by_item_id IS NOT NULL"
unread = ->(key) { "SELECT (SELECT count(*) FROM content_documents WHERE metadata ? '#{key}') + " \
                   "(SELECT count(*) FROM content_emails WHERE metadata ? '#{key}')" }
tasks  = ->(type) { "SELECT count(*) FROM tasks WHERE type = '#{type}'" }
{
  "unread_entry_* keys (old)"                       => unread.("unread_entry_count"),
  "unread_member_* keys (new)"                      => unread.("unread_member_count"),
  "Content::UnpackContainerTask (old)"              => tasks.("Content::UnpackContainerTask"),
  "Content::UnpackMembersTask (new)"                => tasks.("Content::UnpackMembersTask"),
  "Content::WriteTransformedDocumentTask (old)"     => tasks.("Content::WriteTransformedDocumentTask"),
  "Content::ExportTreatedItemTask (new)"            => tasks.("Content::ExportTreatedItemTask"),
  "member attachments named file (old)"             => "SELECT count(*) #{members} AND a.name = 'file'",
  "member attachments named cached_file (new)"      => "SELECT count(*) #{members} AND a.name = 'cached_file'",
  "tasks only the new release can load"             => "SELECT count(*) FROM tasks WHERE type IN " \
                                                       "('Content::PrepareItemTask', 'Workflow::AwaitChildRunsTask')",
  "workflow runs (old tables)"                      => "SELECT count(*) FROM workflow_runs",
  "workflow runs (task graph)"                      => "SELECT count(*) FROM task_graph_runs",
  "task graph runs started since the window"        => "SELECT count(*) FROM task_graph_runs WHERE id > " \
                                                       "(SELECT COALESCE(max(id), 0) FROM workflow_runs)",
  "invalid indexes"                                 => "SELECT count(*) FROM pg_index WHERE NOT indisvalid",
  "unvalidated foreign keys on workflow_runs"       => "SELECT count(*) FROM pg_constraint WHERE " \
                                                       "conrelid = 'workflow_runs'::regclass AND contype = 'f' AND NOT convalidated"
}.each { |label, sql| puts format("%-46s %s", label, q.(sql)) }

After a window, each “(old)” line reads 0 and its “(new)” line reads what the “(old)” line read before. “Tasks only the new release can load” reads 0 until new code has built a run; see Rollback.

Releases that need a window

Release	Change	Why it needs the window	Its migrations	Its checks
First release after v3.4.8	Awaiting child runs (#1849)	A new task class, `Workflow::AwaitChildRunsTask`, and a changed workflow driver: an old worker advancing a run that holds one raises	`workflow_runs.parent_task_id` with its index, built concurrently; then the foreign key validated	“Unvalidated foreign keys” reads `0`. After step 7, the smoke ZIP’s run shows an Await child runs task that closes once its members settle
First release after v3.4.8	Coverage keys renamed `unread_entry_` to `unread_member_` (#1902)	Partly read tests for the key’s presence: an item an old worker writes after the migration drops out of the count for good, with no error	One `UPDATE` per key on `content_documents` and `content_emails`	Old keys `0`, new keys equal to the old “before”; Partly read counts unchanged; labels read “members unread”
First release after v3.4.8	Unpack task renamed `Content::UnpackMembersTask` (#1916)	`tasks.type` stores the class name: an old worker raises `SubclassNotFound` on a renamed row, and a new one on an old row	One `UPDATE` over the unpack task rows	Old type `0`; Desk shows “Unpack Members” on pre-window ZIP runs
First release after v3.4.8	Export task renamed `Content::ExportTreatedItemTask` (#1929)	As above. Its reuse signature names the old class name, so existing treated copies still match	One `UPDATE` over the export task rows	Old type `0`; Desk shows “Export Treated Item”; a pre-window treated copy still downloads
First release after v3.4.8	Root items cached; members’ bytes move to `cached_file` (#1941)	An old worker refilling an evicted member would attach it to `file`, which the new eviction rule never removes	`content_texts.source_digest`; one `UPDATE` renaming every archive member’s attachment (no bytes move); a partial index on `workflow_runs (context->>'pinned_blob_id')`, built concurrently	Old attachments `0`, new equal to the old “before”; a new ingest run starts with Prepare Item; a member of a pre-window archive downloads
The task graph release	Workflows move to the task graph engine (#2085)	Every run, task and attempt moves to new tables (`task_graph_*`), and the new code reads only those. An old worker would keep running work the new code cannot see; the old framework’s queued jobs are discarded by the copy, and the new release dispatches what is ready itself	Creates the `task_graph_` tables, then one migration copies* every run, task, task run (as an attempt), edge and log entry, keeping ids. It re-points three foreign keys (`health_checks`, `discoveries`, `content_analyzer_runs`), renames two polymorphic type names, cancels any proxy turn left unfinished by a request that died, and deletes the old framework’s unfinished jobs. Scales with `tasks` and `task_runs`: 1.3 s for 1.3k runs and 13.7k tasks on the Lab. The release’s last migration then drops the old framework’s six tables, irreversibly (#2088). It verifies the copy first and drops nothing if the copy is incomplete (step 5). The whole `db:migrate` took 6.9 s on the Lab’s data	“Workflow runs (task graph)” equals the “before” figure of “workflow runs (old tables)”, and both lines that read the old tables print `n/a`. Desk opens a pre-window run with its tasks and attempts. After step 7 the smoke ZIP’s run lists its member runs, and a proxy (API) chat turn answers with the Job service running

The same release carries changes that are not windows themselves but whose repairs and costs land with it. Run doctor:repair in step 6; it runs every one:

Change	Repair it adds	Expect
Stranded task runs found by job liveness (#1841)	`task_run_job_ids` backfills the job id of runs from before it; any it cannot match are listed for you to judge	The stranded-run sweep, which step 3’s deadline relies on, judges only runs that carry a job id
`impaired` no longer counts as finished (#1846)	`stranded_impaired`, `stranded_analyses`, `health_check_results`	Its migration backfills `health_checks` in batches; that table has no retention window, so it scales with the install’s age
One live extracted text per document (#1898)	`duplicate_texts`	Superseded duplicates are re-assessed by the risk sweep
Declared and resolved content types (#1897)	`declared_content_types`	A batched migration over `content_items`
An item’s place in its source (#1940)	`item_identifiers` fills `content_items.identifier` for every existing item	Fills every existing item on its first run, in batches; nothing reads the column yet

The migrations that scale with your data, and are worth timing in step 1: the two task renames (rows of tasks), the attachment rename (archive members), the index builds on task_runs, workflow_runs and content_items, the health_checks backfill and the content_items type move. The rest add a column or a small table.

After step 7, expect more work than usual for a while. Every item re-extracts once on its next ingest (its text has no source digest yet), each archive member’s cache is read back once, and several risk-score migrations mark scores stale, so the risk sweep recomputes them.

Rollback

Take the decision in step 6, before you resume. Until then no work has run on the new code, and the window is fully reversible.

Before step 7: migrate down, then deploy the old image.

Keep the queues paused. Stop Web and Job.
Check that no run was built by the new code: “tasks only the new release can load” in the check snippet must read 0. If it does not, the old release cannot load that work; restore the backup instead (below). Once the task graph release’s migrations have all run, its old tables are gone: restore the backup. If its drop stopped with CopyIncomplete, the old tables are still there, and you may either restore the backup or migrate down from there. Migrating down empties the task_graph_* tables and points the foreign keys back at the old ones. The old framework’s queued jobs the copy deleted are not restored: the old release’s stranded-run sweep finds those tasks by their missing jobs and retries them.
With the new image (the old one does not have the migrations to undo), migrate down to the last migration of the release you are returning to:
Terminal window
```
EP_SKIP_LIBPOSTAL=true bin/rails db:migrate VERSION=<last migration of the old release>
```
For v3.4.8 that is 20260927064947. For another release, list it from the old image: ls db/migrate | tail -1. Rails reverts each migration in reverse order; every migration between v3.4.8 and the window release is reversible, and the renames put the old names back.
Point Web and Job at the old image, start them, run the check snippet (the old counts are back), then resume the queues.

After step 7, or if the down migration fails: restore the backup from step 4 to the database, deploy the old image, and resume the queues (the backup holds them paused). Work done on the new release is lost; ingest it again. No file bytes move in a window release, so object storage needs no restore.

Prefer rolling forward once work has run on the new release: a fix in a follow-up release keeps that work.