# Upgrading a running deployment > **Guide** (informative) · **For:** operators upgrading a mesh that already exists · **See also:** [Substrate stability](/docs/stability/), [Run a mesh](/docs/run-a-mesh/), [Identity and auth](/docs/identity-and-auth/) [Substrate stability](/docs/stability/) tells you what the version numbers promise. This page is the other half: what to actually do when the deployment already exists, has credentials in it, and cannot simply be recreated. Every release that breaks a running deployment gets a section here, naming what migrates on its own, what does not, and the order to move the pieces in. ## Jcode reports an unreadable Cursor database as `cursor_state_unreadable` A Jcode launch that cannot read Cursor's `User/globalStorage/state.vscdb` used to stop with `private_state`. It now stops with `cursor_state_unreadable` and a reason, after waiting up to 2 seconds for a lock Cursor holds ([Connect Jcode](/docs/connect-jcode/)). One database that used to launch now stops: one where a value under an auth key Jcode reads is not text or is larger than 16 KB. The previous reader copied that value as is. ## Identity-pin cleanup API after 0.131.6 `@cotal-ai/workspace` no longer exports the deprecated `removeIdentityPin`. It removed a pin without locking the record or checking whether another publisher had replaced it. Before upgrading an application that imports this function, replace its cleanup with `removePidPair(pidfilePath, recorded)`. Capture the trimmed pidfile text when you adjudicate the record, and remove the pair only after proving that record's process is gone. Pass that captured text, not a fresh read made at deletion time: if the pidfile has changed, the replacement keeps it and its pin. There is no supported operation for deleting the pin alone. The stock CLI already uses `removePidPair`, so CLI users need no extra migration. This change does not alter the record format, credentials, broker state or manager lifecycle, and requires no state reset or manual pin deletion. ## Bounded multi-space startup Multi-space auth, callout and delivery hosts now admit four lane opens at a time. A ready line may include `queued , ` while the remaining spaces wait for admission. Read per-space health rather than treating the first ready line as proof that the whole manifest is serving. Each batch uses a fresh manifest read; removed queued spaces never start. The 30-second opening deadline starts at admission, and an open still running after that deadline keeps its slot until it settles. No credentials, manifests or timeout settings need to change. ## The pre-1.0 upgrade contract The packages are pre-1.0, so a minor bump may break an API or an on-disk expectation. Five commitments make that survivable for someone with a fleet: - **Pin an exact version.** `0.N.P`, never `^0.N.P`. A range can pull a breaking minor in during an unrelated reinstall. - **Every break that touches a running deployment gets a section on this page**, written in terms of what an operator does, not in terms of which module changed. - **Read the section before you start, not halfway through.** A section names the work up front precisely so the operation does not change shape once it is underway. - **A break that cannot be made automatic says so.** Where credentials or state must be recreated by hand, the section says which ones and when, rather than leaving you to discover it at the moment the first one stops working. - **A change to the shape of a credential, or to who may renew one, is breaking whatever the commit marker says.** This rule is stated because the marker is a judgement made while writing the code and the consequence is felt by someone running it a day later. A fleet that keeps authenticating looks compatible and is not, if nothing in it can renew. Any automated check of this rule would read commit markers, so a break recorded as a feature is the one case it could not see, which is why the rule is written for people first. **The marker held for this release: the 0.49.0 change that caused all of this, `36d177951 feat(core)!`, did carry its `!`.** The rule exists for the next one that does not. What this page does not promise is a rolling upgrade. Nothing in the current line dual-serves two authority versions, so where broker and manager run separately there is a window in which the mesh is down. The sections below give that window's shape so it can be scheduled rather than endured. ## Channel posts keep no age limit after 0.134.0 0.134.0 gave a space a 30-day age limit on channel posts, and the first `cotal up` that started an existing space's broker moved the space to it, which deleted every channel post older than 30 days. This release takes both back. A space keeps a channel post until its sender has posted 1000 newer ones on that channel, with no age limit, as it did up to 0.133.13. `cotal up` changes no space's limit, so an upgrade deletes no post. ### What changes for you - An upgrade from 0.133.13 or earlier changes nothing. - A space that a 0.134.0 `cotal up` already moved keeps its 30-day limit, and the posts that move deleted do not come back. From the space's root on the broker host, `cotal history limit --max-age ` sets a longer age, up to `3650d`. - `cotal history limit` still sets an age, a per-sender count or both. A lower limit deletes the posts it no longer keeps, and in a terminal it asks first. - `cotal up`, `team boot` and `setup --start` no longer report `CHAT_HISTORY_MIGRATED`. - INBOX and DLV, the durable delivery streams, keep their own bound of 1000 entries per subject. ## Multi-space host acknowledgments after 0.116.1 Every multi-space host (`auth-service --spaces`, `auth-service --callout-only --spaces`, `deliver --spaces`) writes `.applied` to say that every space the manifest does not name has no lane in that process. From 0.115.2 a space whose open outlived the pass deadline kept that open running outside the lane map, so a manifest that dropped such a space was acknowledged while its open was still pending. The open then settled, the host took the lane, and the next pass closed it. A writer that treats the acknowledgment as a fence, as Cotal Cloud does before it tears a tenant down, could see a lane come up for the space after it was told there was none. The host now holds the record for such a space until its open has settled and any lane it brought up is closed, the same way it already holds it for a lane whose close did not confirm. It names the space when it starts holding: ``` ✗ delivery host: space(s) "" left the manifest while still opening - acknowledging nothing until each open settles and any lane it opened is closed ``` ### What changes for you - A removal you fence on can now take as long as that space's open does. There is no timeout and no partial answer: the record names the manifest you wrote only once the removal is true. An operator script that waited a fixed few seconds for `.applied` and then proceeded was reading an answer the host could not yet give; give it the time a hung open can take, or stop the host. - The other spaces are unaffected. They keep opening, serving and closing while one space's removal is held, which the host's lane lines report as before. - A space the manifest still names does not hold anything back: a pending open only matters once the manifest has dropped the space. ## Membership snapshot reads after 0.115.4 `CotalEndpoint.readMembership()` now reads the membership feed with batched Direct Gets and binds no consumer, so a dashboard that polls it no longer leaves a consumer per read on the feed's stream or a refused `CONSUMER.DELETE` line in the broker log. The read needs `$JS.API.DIRECT.GET.KV_cotal_membership_`, which the observer and admin profiles now carry. An observer or admin credential minted by an older manager or auth service lacks the row, and its `readMembership()` is refused with the grant it lacks rather than scanned with a consumer. Upgrade the issuer first, then mint the dashboard's and the CLI's observer credentials again; reconnecting with the old credential does not add the grant. This also applies to `down -f` for a ledger written by the retired `spawn -f`. The teardown keeps owned channels when it cannot establish membership, and now prints the read failure and the missing grant. On a user-auth mesh, run it as the signed-in deploying owner with admin scope. Upgrading only the CLI does not add the snapshot grant to a credential issued by an old auth service. Upgrade the issuer before retrying the retained run. On a static-auth mesh, the current CLI mints its short-lived teardown credential with the snapshot read. ## Membership watch entries after 0.115.4 `CotalEndpoint.watchMembership(onChange)` now hands `onChange` each feed entry as a `MembershipChange`: a `member` change with the agent's decoded record (or none, when the key was deleted), a `heartbeat` change with the daemon's `asOf`, and one `replayed` change closing the entries a newly armed or rearmed watch replays (those carry `update: false`). Up to 0.115.4 the callback took no argument, and the documented contract was to re-read `readMembership` on every call; because the delivery daemon rewrites the heartbeat key every poll, an observer following that contract rescanned the whole bucket through a fresh ordered consumer per heartbeat, with nothing changed. A callback written against the old contract still compiles and still works; it is now also called once per replay with the marker. To stop paying the rescan, branch on `change.type`: copy a `heartbeat` into the last snapshot's `asOf`, ignore `replayed`, and read (or apply the record) on a `member` change. The stock dashboard, `cotal view` and `cotal channel members --follow` do this. ## Node 22.10 is the floor after 0.115.4 Every package now declares `"node": ">=22.10"`, where Node 22 was enough up to 0.115.4. The reason is `module.flushCompileCache()`, which arrived in v22.10.0 and which a Claude Code seat's entry calls on every launch. Check the runtime before upgrading a host with `node -v`. On a host whose Node is 22.0 through 22.9, three things change. The `cotal` binary refuses to start, naming the floor, the running version, and the repair for the way it was installed. It checks the version before loading anything else, so an old runtime gets one sentence instead of a parse error from a dependency. A seat launched through `dist/seat.cjs` prints one line naming the floor and exits without serving. `node dist/mcp.cjs` still serves there and pays the bundle's compile on every start, which is the route for a host that cannot move its Node yet. The installer compares the minor version from this release on, so `curl -fsSL https://get.cotal.ai | sh` on a host whose Node is 22.9 downloads and verifies the pinned Node rather than using that one. Up to 0.115.4 it accepted any Node 22. Nothing migrates and no credential or state is recreated. A host already on Node 22.10 or newer has nothing to do. ## The Claude Code seat entry after 0.115.4 The plugin's MCP server is launched through a new file, `dist/seat.cjs`, which enables Node's compile cache before loading the server bundle. Nothing migrates and nothing is recreated by hand. Two things are worth knowing before the first upgraded seat starts. Run `cotal setup` after upgrading, as the normal upgrade step. The installed plugin and the Claude Desktop entry both name the launched file, so a copy written by an older release still points at `dist/mcp.cjs`. That copy keeps working: it serves the same seat and pays the bundle's compile on every start. Until setup runs, `cotal status` reports the Desktop entry as stale. Each seat writes a compile cache under `~/.cotal/compile-cache` (`%LOCALAPPDATA%\Cotal\compile-cache` on Windows), which `COTAL_HOME` does not redirect. It holds under 1 MB per Node version and is safe to delete at any time. A host that gives its seats a read-only home gets one line on stderr per seat and seats that serve without a cache. To turn it off for a host, set `NODE_DISABLE_COMPILE_CACHE=1` in the seat's environment. ## Multi-space host lane opens after 0.115.2 Every multi-space host (`auth-service --spaces`, `auth-service --callout-only --spaces`, `deliver --spaces`) now opens its spaces' lanes together and waits at most 30 seconds for any one of them. Up to 0.115.2 the lanes opened one after another, so one space whose open hung (a broker connect or JetStream request that never answered) held back every later space in the manifest, the ready line, and the acknowledgment a provisioning writer waits on to remove a space. A space whose open outlives the deadline is reported, its open keeps running and is not started again, and its lane is taken when that open settles, or closed if the space left the manifest meanwhile. The ready line names such spaces apart from the ones being retried: ``` ✓ delivery host up (2 space(s): , ; still opening ; retrying ) - one delivery lane per space on ✗ delivery host: space "" did not open within 30s - serving the others without it; its open is still running and is not started again ``` The default startup (`every`) refuses a space that does not open within the deadline as it refuses one that fails to open: `space "" did not open within 30s - no lane was left running`. Nothing has to change before the upgrade; a supervisor that waited more than 30 seconds for a ready line now sees it sooner. A stop (SIGINT, SIGTERM) that lands while a pass is still opening a space now waits for that open, bounded by the same deadline, and closes the lane it produces before the host ends. Up to 0.115.4 the host exited with that lane never closed, so a delivery host stopped in that window left the space's lease row in place until its TTL and a restart was refused with `a live lease already exists`. ## Multi-space callout host startup after 0.115.0 `cotal auth-service --callout-only --spaces ` now starts with the spaces it can serve. It names each space it cannot serve with the reason and retries that space every 5 seconds. Up to 0.114.x the host exited nonzero unless every space in the manifest opened, so one space whose service endpoint admission was open, or whose material was missing, kept the callout of every other space in the manifest unanswered by this host, and a supervisor that restarts it looped. A supervisor, health check or script that read that nonzero exit as "a space could not open" no longer sees it: the host stays up. Read its ready line, which counts the spaces answering and names the spaces it retries, or its per-space lines: ``` ✓ auth callout host up (2 of 3 space(s) answering: , ; retrying ) - answering the callout on ; no HTTP listener, no auth-service.json ✗ auth callout host: space "" could not be served () - retrying in 5s ``` A manifest that does not read or parse whole still refuses startup, and so does a manifest directory the host cannot write. `auth-service --callout-only --space ` still refuses a space it cannot serve and exits nonzero. ### Before the upgrade - Nothing has to change unless something relied on the old exit. - A space whose service endpoint admission is open is now named and retried for the life of the host, one lane attempt every 5 seconds. Keep it off callout host manifests: `admitsServiceEndpoints({ space, server, secrets })` from `@cotal-ai/auth` returns `true` for each such space. ## Loopback host doors share the invalid-capability budget `POST /interactive-lifecycle/retire`, `POST /managed-lifecycle/retire` and `POST /manager-service-authority/verify-enrollment` now count a missing or wrong loopback capability in the same invalid-capability window as `POST /exchange`. The first 30 in a rolling minute answer 401 with the route's own reason. A further one answers 429, `too many invalid-capability attempts - wait a minute and retry`, and each refusal is logged. Before, these three routes answered 401 for every attempt and never filled the window. A correct capability is unchanged, and a running mesh needs nothing. The interactive retirement route still states `outcome: "not-executed"` on that refusal, including the 429, because nothing was retired. The other two routes keep the 401 body they answered before and add no outcome field. ## Service endpoints on a multi-space auth service host after 0.114.0 `cotal auth-service --spaces ` now takes service endpoints per space, from each space's manifest entry, and refuses `--service-endpoints`. This affects a host run on any release from 0.112.0 before 0.114.0 with both `--spaces` and `--service-endpoints`, where the flag turned service endpoints on for every space the manifest named. Started the same way, such a host now stops at startup with nothing opened: ``` auth-service: --spaces takes service endpoints per space - set "serviceEndpoints": true on the manifest entry of each space that hosts them and drop --service-endpoints ``` `auth-service --spaces` without the flag, `auth-service --callout-only --spaces` and `auth-service --space --service-endpoints` are unchanged. ### Before the upgrade - Add `"serviceEndpoints": true` to the entry of each space that keeps service endpoints, and drop `--service-endpoints` from the host's command line. An entry that had a public face keeps it: ```json { "version": 2, "spaces": [ { "space": "", "exchangePublicPort": 7451, "exchangeTrustedProxy": true, "serviceEndpoints": true }, { "space": "" } ] } ``` To keep the old behavior, set it on every entry. - Write the new manifest first, then restart the host on the new version. Until the restart the old host reports the new field as unknown and keeps its lanes as they were. - A space whose entry leaves the field out answers `/service-endpoint-authority` and `/service-endpoint-host` with 404 after the upgrade. Its admission record and registrations stay in its stores, and a callout replica still refuses it. ## Multi-space delivery startup after 0.112.0 `cotal deliver --spaces ` now starts with the spaces it can serve. It names each space it cannot serve with the reason and retries that space every 5 seconds. Up to 0.111.x the host exited nonzero unless every space in the manifest opened, so one space whose lease another daemon still held, or whose credential was missing, kept every other space's delivery down. A supervisor, health check or script that read that nonzero exit as "a space could not open" no longer sees it: the host stays up. Read its ready line, which names the spaces it retries, or its per-space lines: ``` ✓ delivery host up (1 space(s): ; retrying ) - one delivery lane per space on ✗ delivery host: space "" could not be served () - retrying in 5s ``` A manifest that does not read or parse whole still refuses startup. Nothing has to change before the upgrade unless something relied on the old exit. ## Writable manifest directory for multi-space hosts after 0.112.0 Every multi-space host now writes `.applied` beside its manifest before it opens any lane, and refuses to start when it cannot. That covers `auth-service --spaces`, `auth-service --callout-only --spaces` and `deliver --spaces`, the last two of which never wrote there before. A manifest kept in a directory the host's user cannot write, such as a read-only mount or a folder another user owns, now stops the host at startup with nothing opened: ``` ✗ deliver --spaces: cannot write .applied (EACCES: permission denied, ...) - no lane was opened ``` ### Before the upgrade - Give each host's user write access to the directory that holds its manifest, or move the manifest into a directory that user can write and point the unit at the new path. - The file holds `{"version": 1, "sha256": ""}`, the SHA-256 of the manifest the host applied. A tool that stops one space by removing it from the manifest can wait for that hash ([server daemons](/api-reference/cli/#server-daemons)). ## Issued-authority stores after 0.111.0 0.111.0 bounds the two issued-authority stores of every space, `KV_cotal_issued_` and `KV_cotal_accepted_`: each row now ends at its generation's horizon. A store an earlier release created keeps every row forever, so every issuer and resolver of the space refuses it and the space does not start. This cannot be made automatic. Each space's two stores are backed up, deleted and created again once, and every generation issued before that is refused afterwards. 0.111.0 and 0.112.0 name `nats stream backup` and `nats stream rm` for this, and no credential those releases mint can run either command. From 0.113.0 `cotal up --recreate-issued-stores ` does it. Upgrade a mesh that ran 0.110.x or earlier straight to 0.113.0 or later. **A rollback after this step does not restore the old stores with stock tools.** Read [Rolling back](#rolling-back-the-issued-authority-stores) before you start. ### Before the upgrade - Pick a backup directory for each space that does not exist yet, inside one that does. The step creates it. - Copy the connector seed store and the extensions folder, `seed` and `extensions` in the operator config directory (`~/.config/cotal` by default). The first start of the new release migrates the seed store, and an older release refuses to run over the migrated one. - Plan to respawn every static agent a manager runs. Its credential's generation is refused after the step, so its renewals stop. ### The upgrade 1. Stop every issuer and resolver of the space: its stack (`cotal down` from the root), its auth service and every callout replica you started yourself. 2. Install the new release. 3. Start the space once with `--recreate-issued-stores ` added to the `cotal up` that starts it, for example `cotal up --space --recreate-issued-stores ` (with `--user-auth` and the other flags that space starts with). Before any issuer starts, it backs up each store created before 0.111.0, checks each backup against its stream, deletes those stores and creates both again: ``` → backed up KV_cotal_issued_ ( messages) to /KV_cotal_issued_, then deleted it → backed up KV_cotal_accepted_ ( messages) to /KV_cotal_accepted_, then deleted it ``` 4. Remove the flag from any unit or script that starts the space. A later start with it refuses `CONFLICT`, since neither store predates bounded retention any more. 5. Start the callout replicas and other issuers again, and respawn the static agents. A space your own composition provisions, rather than `cotal up`, runs the same step through `removeIssuedStoresBeforeRetention` ([Embedding](/docs/embedding/#issued-authority-stores-from-before-01110)). ### What refuses - Without the flag, `cotal up` refuses `CONFLICT` (exit 2), names the store, and stops the broker it started. 0.111.0 and 0.112.0 report the same refusal as `OPERATION_UNKNOWN` (exit 3); nothing is left running in either case. - The flag refuses `USAGE_INVALID` when the backup directory exists or its parent does not, or with `-f` or `--restore`, and `LIFECYCLE_BLOCKED` while the stack runs or the root has a maintenance attempt to recover. It runs only on a broker that start launches, before any issuer. - A backup that does not match its stream's message count and last sequence, or a store that changes between its backup and the deletes, means an issuer is still writing. Nothing is deleted. Stop it and run the start again with a new backup directory. ### The backup Each store gets `//backup.json` and `stream.tar.s2`, the layout `nats stream backup` writes, so `nats stream restore` reads it. The directories are 0700 and the files 0600. Keep them private: they hold the issued-authority evidence of the space. ### Rolling back the issued-authority stores - 0.110.0 starts on the stores the step created, and its `cotal ps` works there. Restore the seed store and extensions copies first, or it refuses to run. - Rows an earlier release writes while rolled back carry no TTL, so the broker never removes them. A later upgrade does not recreate the stores again, because their shape is already current. - Putting the backed-up stores back needs a credential that holds `STREAM.RESTORE` on them for `nats stream restore`, and no stock command mints one. A rollback to the stores as they were before the upgrade is not supported with stock tools. ## Starting the manager by name after 0.105.0 `cotal manager start` starts the manager for this folder's or the selected space, with the pty runtime and the broker the space is registered with. `manager stop`, `status`, `logs` and `restart` act on it. Nothing has to be added first: the first start saves a service definition named `manager` from the manager preset's defaults. `cotal manager start` after `cotal up` reports the manager `up` started and starts no second one. `cotal service` lists every built-in service, started or not, and `cotal service start ` starts one from its defaults. A preset that needs a file the space does not keep, such as the feedback form's tester keys, refuses and names the flag. `cotal service add --exec ""` adds a program of your own without a definition file. ### What changes for a running deployment `supervise` is deprecated again, see [deprecated daemon entries](#deprecated-daemon-entries). It keeps its flags and behavior and prints one line on stderr. `cotal manager install` is not supported yet and refuses. A unit you write yourself runs `cotal manager start --foreground --space `; a unit that runs `supervise` keeps working. `--execution`, `--spawn`, `--seat-cap` and the resume flags have no `manager start` form yet. A `supervise` run that names one says so instead of naming `manager start`; keep `supervise` for it. ### Before the upgrade - Nothing has to change before the upgrade. A unit, script or container that runs `supervise` keeps working and prints the deprecation line. - A script that read the `presets` array of `cotal service --json` still finds it. The new `services` array lists each saved and built-in service with its state. - A definition a preset-name start saved is an ordinary preset definition, so an earlier release lists, starts and stops it after a downgrade. ## Service definitions record their root after 0.100.2 `cotal service add` now saves each definition with the root of the space it is added in. A definition added in another folder, such as one a forgotten space of the same name left, refuses to start, restart or install there `CONFLICT`, naming that root and `cotal service remove --space --yes`. Definitions saved before the upgrade have no recorded root: they start as before, and `cotal service` lists them as `root not recorded`. An earlier release refuses a definition that records a root `SCHEMA_INVALID`, and with it the commands that read the space's definitions. Before you downgrade, remove each service this release added with `cotal service remove --space --yes`, and add it again after the downgrade. ## Missing $SYS credentials after 0.100.2 `cotal doctor auth` reports a missing `membership-observer` or `connection-evictor` credential as a finding, exit 5, once the root holds the space's trust. Before, it noted that pair as not provisioned and exited 0. A space started before the pair existed, or one whose files were removed, therefore has a finding after the upgrade. Without the pair, the membership feed and live connection eviction do not run, as before the upgrade. A plain `cotal up` does not mint them, because it reuses the existing trust record. Run `cotal down`, then `cotal up --rotate-sys`. The rotation makes earlier full backups unrestorable, so take a new `cotal backup` after it. ## Minted credential lifetime after 0.100.2 `cotal auth mint` gives a fixed-profile credential a 24-hour lifetime when no lifetime flag names one. Before, such a credential had no expiry claim. Credentials minted before the upgrade keep the lifetime they were minted with, and endpoint callers (`auth mint --endpoint`) are unchanged. After the upgrade, a process that reads a newly minted file for longer than a day stops authenticating when the file expires. A dashboard given `--creds` is one. Mint such a file one of two ways: - with `--expires-in ` or `--expires-at `, and mint a replacement before it expires; - with `--no-expiry`, which issues the unbounded credential the old default did and records the request beside the file, so `cotal doctor auth` reports it as a warning rather than a finding. ## Spawn in this terminal after 0.100.0 A bare `cotal spawn []` runs the application in this terminal again, on every kind of mesh, as it did before 0.91.0. No manager is asked, so several live managers no longer stop it. A manager runs a launch only with `--detach` (`-d`), which returns once the agent joins, or with `--on `, which attaches to the agent on that manager. - A script or alias that relied on a bare spawn attaching to a manager's seat adds `--on `, or `-d` and then `cotal attach `. - `--foreground` keeps working and changes nothing. - `--restarts`, `--creds`, `--json` and `--plain` belong to a manager's launch, so without `-d` or `--on` they are refused with a Next that adds `-d`. - On a user-auth mesh that runs elsewhere, a spawn in this terminal provisions its agent through the agent-provisioning endpoint the mesh advertises. A mesh that advertises none refuses it and names `-d` among the remedies. ## Folder spaces capture commands only while running after 0.105.0 A space prepared in a project folder and never started, or one whose stack there is stopped, no longer captures the commands run in that folder. They target the space `cotal use` selected, and the line under the `Mesh:` heading names what the folder holds and the `cotal up` that starts it. A running folder space keeps capturing them, as after 0.100.2. `cotal up`, `cotal down`, `cotal space show` and the folder part of `cotal status` keep their folder scope. Nothing on a running mesh changes. - A script that runs in a folder whose space is prepared or stopped and means that space names it with `--space`, or starts it with `cotal up` first. - `meshes.folder` in `cotal status --json` and `folder` in `cotal space --json` are `null` in such a folder; `target.space` names what the command targeted, and `warnings` carries a `FOLDER_SPACE_NOT_RUNNING` entry naming the folder's space. - A copy of a project, a moved project folder and a folder whose space name a later `up` took over still refuse `TARGET_UNAVAILABLE` and name the folder the stack runs from, as before 0.106.0. - `cotal status` reports what it found at a glance; `cotal status --wide` prints every row in full, and `cotal status --all` dials every space. `--json` is unchanged, with an optional `meshes.catalogs[].account`. ## Space selection inside a project folder after 0.100.2 A command run without `--space` inside a project folder whose root has one registered or prepared space now targets that space, even when another space is selected. The note under its `Mesh:` heading says the selection applies outside the folder. Everywhere else the selection is the target, as before. Nothing on a running mesh changes. - A script that runs in one space's folder and means the selected space names it with `--space`. - `SELECTION_NOT_FOLDER_SPACE` is reported only in a folder whose root has several spaces, none of them the selection. - `cotal up` says when it selects the space it started. ## Bare down beside a running service after 0.100.2 On a legacy root, bare `cotal down` refuses `LIFECYCLE_BLOCKED`, exit 2, while a saved service in the space still runs or cannot be verified, and stops nothing. Before, it stopped those services first through the service controller. A controller stack's `down` already refused this way, and `down --preserve-state` refuses the same way on both. - Stop each service the refusal names with `cotal service stop --instance --space --yes`, then run `cotal down` again. A stopped service does not block. - `cotal down --dry-run` warns with that refusal. Its JSON lists the instances as `serviceChecks`, where `stop-first` marks one that blocks; `serviceStops` is gone. ## Manager instance in the exchange response after 0.100.2 `POST /exchange` no longer answers a `manager-caller` exchange with `managerInstanceId`. Both the human and the agent exchange answer with `token`, `owner` and `exp`. The instance a bearer may address is its signed `act.managerInstanceId` claim, which the callout enforces and the CLI reads. ### What stops working Code of your own that reads `managerInstanceId` from an exchange response reads `undefined`. ### Before the upgrade Read the instance from the bearer's `act.managerInstanceId` claim. ## JSON refusals after 0.100.2 Under `--json`, a command that refuses or fails writes its envelope on stdout and nothing on stderr. Before, it also printed the text report on stderr. The envelope and the exit code are unchanged, and nothing on a running mesh changes. - A script that read the refusal from stderr reads `error.message` and `error.remedy` from the envelope. - The text report still goes to stderr when the envelope cannot be written to stdout. ## Endpoint caller files after 0.99.0 `cotal auth mint --endpoint --out ` from 0.91.0 through 0.99.0 recorded the endpoint's wire token, such as `com_example_reviews`, in `.issuance.json`. `ep describe`, `ep invoke` and `invoke` with `--creds ` then failed with "the endpoint contract read did not complete", so those files could not make a call. A current client refuses that intent with `SCHEMA_INVALID`. Issue a new caller to a new `--out` file; the old output and intent block reuse of their path. ## Personal native connections in 0.92.0 Update the CLI and its Claude plugin together, then reopen the native Code session. Existing static auth connections keep their credentials and gateway tools. Personal signed-in connections are new; the plugin binds a native conversation through its ordinary command hooks before creating one. This works without intercepting the `classic.*` events protected by Claude's built-in guard; keep the guard enabled. Organization policies that disable command hooks or native modules still apply. Remote meshes must advertise conversation identity v1 on the selected registration's HTTPS origin. The existing account signs in through Cotal; do not copy login files or paste a token into Claude. Each signed-in conversation gets its own roleless actor, direct messages and an own-event-only publication grant. The space's required-event policy remains in force. Desktop Chat cannot use this native path because it has no Code transcript binding. Creation persists the request and secret before contacting the authority. Retrying an interrupted creation reuses them. Disconnect can report pending cleanup; retry it for the same connection. Private recovery state remains until the authority confirms that exact lifecycle is retired. Neither an upgrade nor a lost response silently creates replacement authority. ## Cooperative workflow cancellation in 0.92.0 The manager adds the issued, instance-bound `run-cancel` command. Its cluster revision advances to 37, retaining the vitals and keyed-submission commands. Cancellation is available through the generic endpoint client; this change adds no CLI or MCP cancellation shortcut. ### What keeps working Existing run records, admissions, revocation markers and journals keep their formats. Upgrade does not rewrite them, cancel a run or remove a marker. A released run without a marker can still resume under its admission. A revoked run remains revoked. Manager shutdown still requests boundary release; it does not write a cancellation marker or prove an agent process stopped. ### What needs an update - Custom `RunHost` implementations must return a drive with `cancel(reason)`, in addition to `done` and `release(reason)`. It must propagate cooperative root cancellation to in-flight effects. Do not substitute boundary release or process termination. Rebuild and qualify the custom host against the new core contract before enabling it on an upgraded manager. - The built-in runtime and language packages carry the root signal through both engines and the compiled worker. Keep core, auth, manager, runtime and language on the same release; copying one source file or replacing only the manager does not supply the new protocol and worker code. - Existing caller credentials do not gain a new publish permission on installation. Obtain fresh issued credentials through the supported issuer before using `run-cancel`. Static credentials and caller generations must still meet the original lifecycle and grant checks. Never edit a credential to add the permission. - The new command requires a fresh Describe-derived instance and epoch bind. An old issuer lacks its independent request observer and inspect/commit path, so a new signerless manager cannot use that issuer for cancellation. Do not treat missing support as a reason to bypass the issuer. ### Order for a split topology Schedule a maintenance window for the version change; this is not a rolling-compatibility promise. Take the deployment's usual consistent backup of broker state, including run records, admissions and journals, and the issuer's private configuration. Upgrade the issuer, then the manager and its runtime dependencies to the same release, using their normal stop/start procedure. Reinstalling a package does not replace modules already loaded by a running process. Reconnect callers through the upgraded issuer and resolve the manager's current contract before invoking cancellation. There is no broker format migration for this command and no automatic cancellation during upgrade. ### What the result establishes A durable marker denies future admission but is not proof of cessation. Only `stopped` with the captured drive outcome establishes that local drive's result. If the holder recheck fails or the request budget expires, a parked effect can remain running beside a committed marker. Preserve an unknown result; do not blindly resend it or declare cleanup complete. To continue cancellation, a caller can make a fresh explicit request, which observes the marker and current holder before reporting its own result. Pending cancellation intent can also block a local launch after a definite no-dispatch refusal if marker absence was not confirmed within the budget. `run show` now displays revocation independently of recorded state. An unreadable marker yields a partial result and exit 4 in human and JSON output, rather than an apparently complete view. Automation must handle that existing partial-result convention. None of these observations proves model or tool preemption, physical process cleanup or cessation on another host. ## Service authority status selector in 0.92.0 A service endpoint authority status query that reads one operation now names that operation: `target: { kind: "operation", operation, requestId }`, where `operation` is `register`, `activate`, `renew`, `ready` or `revoke`. `@cotal-ai/core` exports the operation names as `ServiceStatusOperation` and the whole selector as `ServiceStatusTarget`. The 0.91.0 selector, `{ kind: "operation", requestId }`, no longer parses. The change is in `parseServiceAuthorityRequest`, `parseServiceAuthorityEnvelope` and the `ServiceAuthorityRequest` type, so it reaches every client and server built on them. A `revoke` operation status parses, but the 0.92.0 auth service does not answer it yet: it is `denied` with reason `unsupported` (HTTP 422), as SPEC states. To follow a revocation, read `target: { kind: "instance" }`, which reports `revoking` with the revoke's operation id. ### What keeps working - Every running mesh. 0.91.0 shipped the request types and the client (the auth provider's `serviceEndpointAuthority` hook and its HTTP client), but no 0.91.0 auth service serves `/service-endpoint-authority`, so no deployment has status traffic on this route to lose. In 0.92.0 only a context opened with `serviceEndpoints` serves the route. That option is off by default and has no CLI flag; without it the route answers 404. - `target: { kind: "instance" }` is unchanged, and so are the register, activate, renew, ready and revoke requests themselves. - No credential, broker or connection is involved. Nothing needs reminting and no broker restarts. ### What does not migrate - **Code that builds an operation status query without `operation`.** The 0.92.0 provider hook and client parse each request before sending it, so the query fails locally with a `bad-request` error, `service-endpoint authority: `, whose detail is `object has missing or unsupported fields`, and nothing is sent. An operation name outside the five gets the detail `unsupported status operation`. - **A 0.91.0 client still sends the old selector.** A 0.92.0 auth service opened with `serviceEndpoints` answers it with HTTP 400, whose body carries the same parser message. The service reads and writes nothing for it and logs nothing. The 0.91.0 provider hook and client never show that message: they cannot parse the 400 body as a protocol result, so they throw a `ServiceAuthorityTransportError` whose message is `service authority returned an unconfirmed response; retain the existing request identity`, with `httpStatus` 400 and outcome `not-executed`. A status query is a read, so nothing ran and nothing needs replaying. An auth service without `serviceEndpoints` answers 404 before it reads the body. - **No legacy operation is inferred.** Neither the client nor the service maps an old selector onto `register` or any other operation. One requestId can key a different request under each operation, so a guessed operation could answer with another request's status. Send the operation the request was made as; the requestId you stored with it stays the same. ### Durable data Nothing stored changes on upgrade, and nothing is rewritten. The selector is a request shape and is never stored. 0.91.0 wrote no service authority request intents or operation records, so there is nothing to convert. A 0.92.0 context opened with `serviceEndpoints` writes its own records from the moment it is enabled: an admission record when it first opens, then request intents, the per-instance intent index, operation, proposal and assignment records, and endpoint gates. Each intent already records its operation. Without `serviceEndpoints` it writes none of them. ### Order for a split topology Upgrade every client that builds status queries together with, or before, the first auth service you open with `serviceEndpoints`. That means code using `ServiceAuthorityRequest` or `parseServiceAuthorityRequest`, and every process that calls the auth provider's `serviceEndpointAuthority`. This change adds no ordering for brokers, managers or connectors; order those as their own sections say. There is no compatibility window to schedule. A 0.92.0 auth service never accepts the old selector, and a 0.91.0 auth service has no handler for the route at all, so a mixed pool is not partly working: every 0.92.0 auth service opened with `serviceEndpoints` refuses an old client's operation status queries with 400, which that client reports as the unconfirmed-response error above, and one without `serviceEndpoints` answers 404. Nothing else is interrupted, so the change has no outage window of its own. ### Before the upgrade - Nothing needs a snapshot for this change, because it writes nothing. If the same upgrade first enables `serviceEndpoints`, take the deployment's usual consistent backup of broker JetStream state and the auth host's private configuration before that step, because the service endpoint host then starts writing to the auth bucket. - Find the old selectors in your code, then name the operation in each: ```sh grep -rn 'kind: "operation"' --include='*.ts' --include='*.js' src/ ``` ```ts // 0.91.0 { ...coordinates, operation: "status", target: { kind: "operation", requestId } } // 0.92.0 { ...coordinates, operation: "status", target: { kind: "operation", operation: "register", requestId } } ``` What this section measured, and what it did not: the local refusal and the accepted and refused selectors are covered by the core parser's own test. The 400, 404 and `unsupported` answers, and the absence of a log line, are read from the auth service's source. The 0.91.0 client's error is read from the 0.91.0 client and core parser source; the client file is unchanged since 0.91.0. No mixed-version deployment was run. ## History reader deletes after 0.86.0 Observer and admin credentials now carry a `CONSUMER.DELETE` row for each of the connection's eight history-reader names, `hist__`, on CHAT (and on DM for admin). `channelHistory`, `multiChannelHistory` and `dmHistory` read under those names and delete each reader when its step ends. Before this, every read left its consumers for five minutes, and a dashboard polling history filled the stream's consumer limit until reads were refused with "maximum consumers limit reached". ### What stops working A static observer or admin credential minted by 0.86.0 or earlier has no reader-delete rows. On it, a current client's history read fails with "it predates the observer and admin reader-delete grant, so remint it", and leaves one reader per stream until the broker's five-minute inactive threshold. A credential the CLI mints at start (`cotal console`, `cotal web` on a static auth mesh) is current after the upgrade. ### Before the upgrade - Remint every observer or admin credential a process reads from a file: `cotal auth mint --profile observer` or `--profile admin`, and any `--creds` file given to a dashboard. - On a user-auth mesh, the auth service mints these views per connection. Upgrade it before, or with, the hosts that read history, so no current client connects through an older auth service. ## Native key issuance recording after 0.85.0 Before upgrading, take a consistent backup of broker JetStream state and the auth host's private configuration and signing material through the deployment's supported backup procedure. The auth bucket must use file storage. Allow space for one issuance record per new callout exchange until its minted expiry plus clock tolerance; static credentials have no such record. Embedders that call `startAuthCallout` must supply the required `recordIssuance` callback. It must finish a durable create-only write for the broker-generated user key and recheck the validated lifecycle and credential authority before returning. A missing callback refuses startup; a failed write or recheck denies the exchange. Do not use the isolated tests' no-op callbacks in a deployment. The stock `startAuthService` authority plane wires the recorder and its pruning automatically. Existing auth records need no manual rewrite, and no tenant or participant grant is added. For split deployments, pause new managed launches and stop every old auth-callout responder before starting the matched new auth service. A mixed responder pool can still issue keys without records. New connections and reconnects cannot authenticate while the callout service is down. Existing broker connections keep their current JWT expiry; this change needs no broker restart. Upgrade the CLI, Manager and connector packages together before admitting new managed launches, using the runtime's supported Manager custody or recovery path. This source change does not promise that running seats survive a Manager restart. The new service prunes expired issuance rows before opening and on a timer. A corrupt row refuses opening or fences the running authority plane; investigate the stored evidence through the normal operator recovery procedure. Do not erase the auth bucket or recreate signing keys to clear it. Rows written before a later authority-check failure can remain until pruning; they do not prove that a client received a JWT or established a connection. Connections admitted before the last old responder stopped, static JWTs and old responders have no issuance record. They are not backfilled or identified from names. Bearers without current lifecycle and credential claims cannot be recorded and are refused; use the supported enrollment or renewal path for the selected release. These records alone do not provide persistent native-key cutoff or complete revocation. Reintroducing an old responder also reopens the recording gap. ## Native Claude MCP discovery after 0.82.2 Before upgrading, snapshot Cotal configuration and the Claude profile on each launch host. Ordinary Claude spawns now load the host's native MCP sources alongside declared Cotal overrides. An empty sharing map no longer suppresses native servers. Audit enabled servers and their credentials before admitting new seats. Use an explicit sharing selection, including `--share-tools none`, to retain isolation. Existing sessions keep their loaded configuration. No broker restart, credential-format migration or cross-host profile copy is required. Upgrade the CLI, core, manager and connector together on each launch host. Pause new managed spawns while adopting the matched packages; use the runtime's supported manager custody or recovery path. Any interruption is that manager transition, not a change to the broker wire contract. Carried private homes stay isolated and do not import the operator's native profile. ## Connector-store protection after 0.82.2 Snapshot the private Cotal state, connector store and mesh configuration before upgrading. Install the new CLI on each CLI and manager host, then verify the selected executable before running setup or store maintenance. The POSIX installer prints its absolute launcher because it cannot change the parent shell's command cache, aliases or functions. An older executable already on disk remains unsafe for a newer store; do not use its force advice to recover. Forward setup and connector seeding migrate built-ins automatically. Same-generation force and reset rebuilds remain supported. An executable older than the store now refuses every backward migration, including force and reset. Select a verified executable at least as new as the store and run normal seeding instead. No credential or mesh-record format needs manual migration. A split deployment needs no broker restart for this change. Keep each host's CLI and connectors on the same release. Pause new managed spawns while adopting it; a running manager keeps its loaded code until restarted. Use the chosen runtime's supported custody or preserve-state path. The interruption covers that manager transition and depends on its runtime, not a broker outage. Do not overwrite a live manager installation or stop an unidentified listener to force adoption. An implicit foreground spawn now chooses a numbered actor when the requested actor is occupied. Existing grants, launch material and sessions stay in place. An explicit name remains pinned and refuses an occupied actor rather than renaming it. Signed enrollments and managed handoffs keep their existing identity semantics. ## Background startup in 0.91.0 `cotal up` now returns after background startup by default. `cotal up --detach` retains that meaning. Use `cotal up --foreground` when terminal lifetime should own the native stack. A systemd unit, launchd KeepAlive job, container entrypoint or script that supervises the `up` process must add `--foreground` before upgrading. [Persisted callers](#persisted-callers-in-0910) gives the literal replacement for each. A project folder an earlier release started keeps its own launcher after the upgrade. Stop it with `cotal down` as usual, upgrade, then run plain `cotal up` in that folder: it starts on the retained launcher with the same identity, store and history. It does not need `space create` or `space add`. Recorded recovery requests retain their original foreground/background intent. They do not reinterpret a prior request using the new default. This change does not migrate a whole stack to owned services. Prepared plain-loopback user-auth roots can use the controller's broker, auth service, delivery and manager members. First enable may return a known partial result after the first three start: the manager needs an existing signed-in, granted CLI login. Sign in and obtain that grant, then run bare `cotal up`; no static caller or new grant is substituted. Auth endpoints come from the creation-pinned child's native provider readiness, not a guessed URL. The controller keeps unknown operations blocked. Existing three-member auth/open journals are read with an unattempted auth row and written with all four roles; old clients do not understand that new record shape. Upgrade the composed CLI together before using it. TLS-required, remote and public-exchange user-auth paths retain their native route. ## Persisted callers in 0.91.0 Some command lines outlive the install that wrote them: OS units, scripts and container commands. After `npm i -g cotal-ai@` they reach the new binary before any `cotal` command has run. This section says what each one does after the upgrade and what to change before it. ### Command mapping | Earlier command | Command in 0.91.0 | |---|---| | `meshes`, `meshes add` | `space`, `space add ` | | `clean store`, `clean all` | `reset store `, `reset space ` | | `mint ` | `auth mint ` | | `mint --signer` | `auth signer --out ` | | `actor list`, `actor grant `, `actor revoke ` | `auth grants`, `auth grant `, `auth revoke ` | | `join [channel]` | `chat [channel]` | | `console` | `watch` | | `channels` | `channel` | | `send dm `, `send msg `, `send ask ` | `send -t `, `send -c `, `send -r ` | | `personas` | `persona` | | `endpoints`, `describe`, `invoke` | `endpoint`, `endpoint describe `, `endpoint invoke ` | | `deregister-instance` | `endpoint deregister ` | | `reconcile-gate`, `seats` | `manager reconcile`, `manager seats` | | `topology view` | `team check --graph` | | `down -f ` | `team remove ` | | `status --components` | `service` | | `run journal`, `run migrate`, `run ps` | `run show`, `run check`, `run` | | `deliver pending ` | `pending ` | | `web` | `service add --preset web`, then `service start ` | | `deliver`, `auth-service`, `feedback-intake`, and `supervise` until 0.101.3 and again after 0.105.0 | Their service presets and `cotal manager start`, under [deprecated daemon entries](#deprecated-daemon-entries) | The sections on this page for each command family give the details of its change. ### What stops working `cotal up` returns after background startup by default. An earlier line such as `cotal up --space --server ` that a supervisor runs as its main process now exits once the stack is started. systemd then treats a `Type=simple` unit as finished and can stop the processes that unit started, and a restart policy starts another attempt. A launchd `KeepAlive` job relaunches it the same way. Add `--foreground` before upgrading. For a systemd unit, replace `ExecStart=/usr/bin/cotal up --space --server ` with: ```ini [Service] Type=simple ExecStart=/usr/bin/cotal up --foreground --space --server ``` For a launchd job, add `--foreground` after `up` in `ProgramArguments`: ```xml ProgramArguments /usr/local/bin/cotal up --foreground --space main --server nats://127.0.0.1:4222 ``` A container command or script makes the same change to its `up` line. When background `up` is the main process of a systemd unit, it prints a warning naming the unit and the `--foreground` remedy after it starts or refreshes the stack. `up -f`, which has no foreground form, and a dry run print nothing. A launchd job gets no warning. `cotal up --detach` keeps its meaning. The warning changes no mode and no unit, and no unit or script is rewritten or restarted. `cotal spawn ` now launches through a manager and attaches. A container or script that ran an agent directly with `spawn ` adds `--foreground`. After 0.100.0 a bare spawn runs in the terminal again; see [Spawn in this terminal after 0.100.0](#spawn-in-this-terminal-after-01000). ### Deprecated daemon entries `deliver`, `auth-service`, `feedback-intake` and `supervise` stay accepted with their existing flags for every caller: units Cotal installed, units and scripts you wrote, and container commands. They are deprecated and are removed in `1.0.0`, one release for all four. Help and completion no longer list them. Each run prints one line on stderr naming its replacement and that release. Stdout, behavior and exit codes are unchanged, so no unit restarts in a loop because of the upgrade. `deliver pending` warns the same way and names `cotal pending `. | Entry | Replacement | |---|---| | `deliver` | `cotal service start delivery`, or `cotal service add --preset delivery` | | `auth-service` | `cotal service start auth-service`, or `cotal service add --preset auth-service` | | `feedback-intake` | `cotal service start feedback-intake`, or `cotal service add --preset feedback-intake` | | `supervise` | `cotal manager start`, or `cotal manager start --foreground` in a unit or container | `cotal service start ` starts the built-in service from its defaults and refuses naming any option it needs that has none; `cotal service add --preset --help` lists every option. The delivery daemon, auth service and manager that `up` starts itself do not warn, so the line in a log always comes from a unit, script or container you can change. `supervise` was deprecated with them in 0.91.0, was not deprecated from 0.101.3, and is deprecated again after 0.105.0, now with `cotal manager start` as its replacement. `manager start` has no `--execution`, `--spawn`, `--seat-cap` or resume flags yet; a `supervise` run that names one prints that instead of the replacement, and those callers keep `supervise` until `manager start` has them. `cotal manager start` uses a registered space, so a container that ran `supervise --server ` runs `cotal space add --server --root ` first. ### Owned units A manager unit that an earlier `cotal service install` wrote keeps calling `supervise`. After the upgrade it starts the new binary and prints the deprecation line. This release does not rewrite that unit, and `cotal manager install` refuses until a Cotal-written unit can run the manager. A unit you write yourself runs `cotal manager start --foreground --space ` with `KillMode=mixed`. A unit that this release's `cotal service install ` writes runs the hidden `__service-unit` entry, which an earlier binary does not have. Before you downgrade, stop it with `cotal service stop ` and remove it with `cotal service uninstall `. After the downgrade, install the unit again with the earlier binary's `service install`. ### Containers The `solo` service in `deploy/docker/compose.yaml` runs `spawn scout --foreground`. The team service runs `supervise` with `--roster`, and the image's default command runs bare `supervise`; both print the deprecation line. An image you already built carries its own CLI and is unaffected until you rebuild it. ### Before the upgrade - Add `--foreground` to every unit, launchd job, container command or script that supervises `cotal up`. - Add `--foreground` to every container command or script that runs `cotal spawn ` directly. - Move units and scripts that call `deliver`, `auth-service` or `feedback-intake` to their service presets, and those that call `supervise` to `cotal manager start`, before `1.0.0`. - Before a downgrade, stop and uninstall every unit this release's `service install` wrote. ## Team commands in 0.91.0 The new grammar names fresh intent `team boot ` and additive intent `team deploy `. `team check --graph` resolves declared access without a broker. `team` and `team show ` read local fresh summaries, additive ledgers and agent/channel returned receipts; `team remove ` is the ID-only additive teardown name. An edited manifest is not authority to remove an old deployment. Boot's `--dry-run` is a declaration view. Deploy's `--dry-run` reads native presence and channel registry for advisory classification against a prior ledger. Remove's preview compares a legacy ledger with native manager PS and display-only membership observations, or a returned channel-only receipt with current KV revisions. No preview authorizes a launch or stop. Bare `team boot ` enters guarded native fresh startup only for a prepared, stopped local root; it refuses a leftover manager record and offers rich, JSON and plain receipts. User-auth boot records an intent before owner/trust preparation and retains uncertainty without retry. Windows boot apply refuses, because protected service-record inventory is not supported there. Bare `team deploy` applies all-new channel-only manifests with explicit live delivery, or one to 100 all-new agents on a static/open mesh, optionally with up to 100 new live channel cards. A mixed deployment records one intent before creating channels and launching agents through one pinned manager/runtime. Each channel revision and original launch request, allocation and matched join is recorded before continuing. A full receipt requires both sequences to match. Unknown effects are retained, not replayed; completed effects are not rolled back. Both routes offer rich, JSON and plain output. Identical settled deployments can return a verified no-op without new effects. Named stale-agent waivers retain the old policy without restarting. User-auth agent apply and existing-card mutation still refuse effects. `team remove ` can retire a returned static agent deployment through its original manager. Review `--dry-run` first; scripts require `--yes`. This needs the native slot query, original UIDs and native retired slot/head plus cleanup evidence, not just stop acknowledgements. A returned channel-only deployment instead uses deletes conditioned on its original card revisions. A static mixed deployment retires agents first and needs a complete retirement receipt before checking and deleting cards, under one confirmation and maintenance lock. Both receipts must exist before its blocker is released. Partial and unknown effects retain their journals without replay. History, grants and readers are unchanged. Open/user-auth agent retirement and legacy removal still refuse. The old `spawn -f` manifest spelling no longer deploys. Native `up -f` still starts a fresh mesh; `down -f --run ` remains the retained route only for an older additive ledger. Move an automated agent deploy or teardown to the new spelling only when its auth mode and manifest are ones these routes accept. ## Credential commands in 0.91.0 Credential minting is `cotal auth mint `. The old `cotal mint` spelling refuses before issuance and names its replacement. A stripped account signer is exported with the explicit `cotal auth signer --out `, not `mint --signer`. The default standing credential label is create-only. Repeated minting does not silently replace it. Choose a new label or a separate private output. An explicit existing output is replaced only when it is an owned private native credential matching the supplied `--identity`. Explicit exports cannot overwrite managed `.cotal` or workstation state. The new receipt reports the actual principal, resulting lifecycle, private destination and native expiry without credentials. Reusing an nkey does not promise reuse of its old lifecycle or bind-only inboxes. An agent-profile re-mint reports its newly allocated UID and warns about that distinction; consuming clients must use the returned UID. Provisioning is explicit and is not evidence that a client joined. Scripts should parse the finite `--json` document rather than scraping the former text lines. Secret and signer files must stay out of support bundles. `auth mint --endpoint --out ` is a separate caller mode. It uses the same native static authority and a live digest-verified untargeted synchronous ephemeral surface. Targeted, journaled or action commands refuse the whole issuance; no unsupported row is dropped. The native agent footprint and exact additional endpoint rows are provisioned with required TLS, without adding spawn/run/admin scope or starting a seat. `--channels` enters both explicit channel ACLs and defaults empty; `--expires-in` remains native seconds. The create-only private `.issuance.json` intent is retained before footprint work and contains public identity/lifecycle/scope metadata only. It is not a native grant or completion receipt. An existing output or intent blocks replay, including when an interrupted operation left no output. Keep that intent with any original file while diagnosing an unknown result. Provisioner close is reported separately. No raw-file caller identity, managed launch, user-auth issuer or environment- export onboarding is added by writing this credential. ## Endpoint deregistration in 0.91.0 Replace `deregister-instance --endpoint --instance ` with `endpoint deregister `. Both positional values are required. The old spelling refuses before effects and has no implicit instance selection. Review the new `--dry-run` output, then use `--yes` in scripts. JSON retains native revision receipts; unknown or superseded deletions are not retried. User-auth and open targets do not receive a static-signing fallback. ## DM pending inspection in 0.91.0 Use `cotal pending ` for a one-document JSON/plain read of native DM durable counts and bounded candidate IDs. `--durable ` targets an older lifecycle directly. The retained `deliver pending` spelling still uses its earlier output shape; scripts must migrate to `result` fields rather than assuming the old per-line format. Candidate IDs are not proof of recipient delivery or complete message history. This route has only the static/open admin read authority; user-auth has no corresponding view yet. ## Live space deletion in 0.91.0 `space forget ` changes only the local registration. It does not delete live broker state. `space delete --dry-run` now reads known first-party native streams, bucket backing streams and presence entries on the registered broker in open or static-auth mode. Static transfer-stream names are read under an additional exact-name grant on a second confirmed connection. This non-atomic read is not an apply plan or evidence that presence records are live. User-auth inspection and deletion remain unavailable. Bare deletion now requires a typed exact-name confirmation on a terminal or the exact positional name with `--yes` in a script. It rereads the native impact and checks the current target before submitting exact stream deletes, stopping on the first refused or unconfirmed reply with acknowledged and remaining names. A native post-read reports any survivor or newly observed transfer target without retrying. No process, registration or local file is removed. ## Stopped reset in 0.91.0 `reset store ` and `reset space ` are the named replacements for `clean store` and `clean all`. `reset store --dry-run` provides a partial local-impact preview of the store path, recorded processes, maintenance journal and broker credless reachability. `reset store` requires confirmation, acquires the maintenance lock, rechecks its exact store identity, legacy processes and all recorded service instances for that space, and calls the existing store-only cleanup. Any recorded supervisor blocks reset. It keeps trust and the registration. TLS-required or TLS-first store targets refuse, because no trusted stopped-broker check covers them. `reset space` now routes a reviewed local identity reset through native clean-all. It requires Linux, an isolated locally started registration and plain-NATS loopback nonresponse. Its preview inventories provider-declared keys and native filesystem targets; apply rechecks the scope under the maintenance lock and constrains each delete to that reviewed inventory. Trust is removed last, followed by the unchanged eligible registration. Unknown effects are not replayed. Service definitions, personas, packages, IdP logins and remote data stay. Unsupported providers, foreign material, TLS-first and non-Linux targets refuse. ## Backup maintenance in 0.91.0 `backup create` supports one-document `--json` and plain reports. The completion receipt follows native attempt cleanup. A published artifact with unconfirmed cleanup is partial; retain its path and original attempt instead of repeating creation. `backup inspect` reads metadata and `backup verify` checks declared file sizes and hashes. Neither proves restoration or authority continuity, starts a broker or writes a staging tree. The canonical stale pre-commit recovery is `backup recover `. Healthy retained old-source cleanup is `backup cleanup-fallback `. Review `--dry-run`, then use `--yes` in scripts. Neither flag bypasses owner, deadline, root, journal revision or store-identity checks. Restore startup stays on its guarded existing route. A controller-owned stack cannot be preserved in 0.91.0. On a root with a stack journal, `down --preserve-state`, `backup create`, `backup restore` and `up --restore` refuse `UNSUPPORTED` before any effect. Stop such a stack with plain `cotal down`; it keeps its store, channels and history for the next `cotal up`. Preserve, backup and restore of a controller-owned stack return in a later release. A root without a stack journal keeps its preserve, backup and restore. ## Feedback input in 0.91.0 Use a protected `feedback --key-file ` rather than placing an intake key on argv. The old `--key` spelling refuses without sending or echoing its value. Public feedback still works with a contact email and requires no environment setup. JSON/plain output is one public receipt, not a copy of the submitted body. Unknown submission outcomes are not retried, and no redirect or alternate credential/intake is substituted. ## Credential diagnosis in 0.91.0 `doctor auth` uses JSON/plain reports and distinguishes local claims from live authorization. A plain diagnostic that encounters material requiring migration refuses without moving it. Explicit `--fix` retains the native static daemon repair, with separate migration/re-sign receipts and no user-auth signer fallback. A local re-sign is not daemon adoption or broker acceptance. Unhealthy, partial and unknown results retain their classified outcomes; scripts must read the new result and error fields instead of scraping the old `auth: healthy` line. Never respawn from a locally ordered older credential file alone. ## Service adapter declarations in 0.91.0 A local-process provider can expose the version-1 workstation service ABI alongside its existing process metadata. It declares a closed option schema, explicit foreground/multi-instance/log capabilities and launch/readiness hooks. Re-add an extension after adding this ABI so its cached service metadata and package fingerprints are recorded. An older descriptor is not imported to guess its schema. Plain process definitions keep their existing meaning. Presets are declarative registry contributions. Their option flags, help, completion and guided questions derive from the adapter schema. Re-add a preset provider to populate its new cache; missing or changed metadata refuses without importing code to guess the definition. The top-level `feedback-intake` launcher is a [deprecated entry](#deprecated-daemon-entries). For a new intake, save a definition with `service add --preset feedback-intake --space --root --keys --creds `, then use `service start ` in that space. An explicit `--store` must be an existing private file. The default is private and instance-keyed. No broker or other service is started, no signer is minted and required TLS is preserved. Native startup receipts are private, request-specific observations rather than native endpoint serving claims. A service instance pins its adapter name, package version and declaration fingerprint. A later package update does not silently adopt a running or retained instance under a different adapter. Stop the exact owned process and explicitly remove its stopped records before choosing changed adapter metadata. Logs remain private local files. Extension removal refuses live, foreign or unresolved service instances, including records outside the current mesh selection. None of these declarations migrate legacy PID records or supply OS-unit ownership. Native manager impact comes from its explicit adapter's attributed current read, not cached metadata. The delivery implementation now registers its service adapter and legacy custody descriptor. Custom composition roots that want those inventory/control rows import `@cotal-ai/delivery`. The full binary already composes it. CLI stop helpers share the workspace descriptor without an implementation-to-implementation import. The `delivery` preset requires the recorded root and existing scoped credential, keeps the native store/account/lease guards and does not adopt an existing legacy daemon. Its hidden child entry is recognized by the same process-role predicate. The old `deliver` machine-facing route remains for the stack callers that still use it. Fresh static-auth and open `team boot` now use controller-owned stack members and accept `--foreground`. Their manager receives a digest-pinned manifest launch spec, with no additive ownership ledger. A recorded stack can resume only after a completed whole-stack stop with its manager never attempted, recorded broker inputs and each member's settled native evidence. Legacy custody, prior team boots, supervisors and unresolved attempts still refuse. User-auth preparation uses ordinary `up --user-auth`, login and an explicit grant, then `down` before `team boot`. No caller or owner grant is substituted. New controller journals record the broker store path, storage cap request and bind host. A later preparation keeps those inputs and refuses differing explicit values. Older journals remain readable. If their broker is stopped, restart requires the original store named with `--store-dir` and the original `--max-file-store` and `--host` values, if set. The store must already be a real directory. Do not omit the cap or host unless the original start set neither. That restart records the inputs; run `down` before the stopped-stack team route. No store path is inferred from an absent field, and bare restart never silently opens a default empty store. The native manager status contract adds optional caller-scoped stop-impact metadata and advances the cluster revision. Upgrade the composed manager and its compatible registration host together; a host that does not accept the current contract must refuse registration. Older managers can omit the observation. Component status then says impact was not reported, rather than inferring that seats survive. An observed policy does not migrate PID records, reserve a stop or confirm cleanup. The manager package adds a separate controller-only `manager-service` adapter and `manager` preset. Existing legacy manager records are not migrated or adopted. Update the composed packages and extension metadata together; an old cached declaration is not rewritten behind a retained instance. Native stops require a fresh full-scope read and unchanged request-bound policy, with native returned evidence plus original process exit. Missing terminal evidence retains its reservation and blocks replacement, without force kill. Cleanup remains not checked. The manager preset accepts `service start --foreground`, as its declared capabilities have since 0.91.0. On an open mesh there is no authenticated caller, so readiness and stop impact come from the pinned child's own request-bound local receipt. Its registration must equal the launch identity bound to that PID. Status shows it as a child-reported local receipt, not an attributed endpoint response. OS installation and non-Linux protected process support are not available. Recorded systemd user units with an original invocation ID use that invocation's native journal for finite/follow logs. Missing invocation evidence refuses instead of reading a stale direct-child file. Native cursors and timestamps are retained; journal retention/access is not proof of complete output or completed service work. No unit install/control support is implied by this read route. Auth providers may implement the optional read-only `preflightServer` hook. It checks public server inputs without creating a directory, pin, key, grant or daemon. The controller's user-auth route requires it before preparation; providers without it refuse rather than skipping validation. `prepareServer` remains authoritative and checks again before provisioning. The auth package also declares an `auth-service` preset over its prepared native key projection. It does not migrate or reclaim existing PID records. Its real child claims custody before opening planes and publishes a private startup phase only after native initialization and signal handlers. This route declares shared foreground control and still refuses remote broker and TLS-required targets. The transport refusal is intentional, because not every native authority connection carries the registered transport requirement. No plaintext substitution or new issuer is used. Manager maintenance now supports finite JSON/plain reports. Gate repair and seat drain default to cancellation in a terminal and require `--yes` in scripts. Use `manager reconcile --dry-run` or `manager seats --drain --dry-run` first. Gate repair requires one registered root/space/broker and carries its TLS requirement through native liveness/eviction. It does not dial a default broker when that target is missing. An interrupted submitted repair or unresolved drain can return exit 3; confirmed native results remain distinct from unconfirmed effects. No process is started. ## Linear reports in 0.91.0 Update the installed Linear extension with the matching CLI/workspace generation. Account add/show, inventory and direct call/resource/prompt JSON now use version-1 envelopes: read the native inventory from `result.inventory` and a native request reply from `result.reply`. Rich/plain reports bound displayed rows and lines without truncating JSON. MCP tool `isError` remains a result with exit 0 and a warning, not a work-success claim. Native not-executed refusals stay distinct from unknown submissions, and neither is replayed automatically. The `linear` service preset runs the same native endpoint implementation from an explicit static-auth root and existing account. It retains native registration, renewal and bounded deregistration; no new generic caller is issued. Its private phase proves handler initialization only, not a current endpoint reply or completed deregistration. Foreground service start uses the shared terminal lifetime and native signal handlers, not the old serve parser. User-auth/open targets and OS-supervised service operation refuse. The older `linear serve` route remains available for installed callers. A bare Linear/account group opens no account. Wrong-mode flags or extra operands refuse before account access. Interactive OAuth remains a human flow without JSON. Foreground serve stays on its native human-output path. Static caller issuance keeps its native authority and now returns shared JSON/plain private-file receipts. Use an existing protected physical parent outside managed state; an existing output refuses before provisioning. Uncertain provisioning/publication is not replayed and no environment- export launch recipe is printed. The supported `auth mint --endpoint` form resolves its scope live; targeted/journaled/action issuance and consuming launch adapters are not supported. These changes do not replace ownership, start a service or print secrets. ## Setup inputs in 0.91.0 `cotal setup` takes four inputs: destination, application, project folder and space. Nothing on a running mesh changes, and there is no outage window. ### What stops working - A `cotal setup --yes` or a setup with no terminal exits 1 with `USAGE_INVALID` unless it names every input as a flag. It no longer sets up every detected connector and seeds the folder it runs in. - Setup sets up one application, the one `--harness` names, and records it as the space's `harness` preference. An application whose executable is not on PATH exits 4 with the rest saved; it used to be skipped with a warning. - The default persona lands in `/.cotal/agents`. Setup no longer looks for a running mesh to seed into. - A repeat `cotal setup` no longer prints a status card. `cotal status` reports the same state. ### Before the upgrade A script or image that ran `cotal setup --yes` names the four inputs: ```bash cotal setup --destination local --harness claude --project --space --no-input ``` For a space that runs elsewhere, register it first with `cotal space add`, or pass the bundle its operator exported with `--connection-file`, and use `--destination self-hosted` with the registration's root as `--project`. Startup remains opt-in. `--start` conflicts with `--configure-only`; a script, JSON or plain invocation must add `--detach`. `--yes` by itself still saves configuration only. Startup failure leaves completed preparation in the result. Do not treat an unknown startup result as permission to submit another launch; inspect the selected space and recorded original request first. ## Reads that install nothing in 0.91.0 A read no longer installs or refreshes the built-in connectors and the web dashboard before it runs. Nothing on a running mesh changes, and there is no outage window. ### What stops working - On a fresh machine, `cotal status`, `cotal ext list`, `cotal ps`, `cotal runtimes`, `cotal docs` and the other reads leave the extension root empty. `cotal setup`, or any command that is not a read, installs the built-ins as before. - After an upgrade, a read does not refresh the built-ins. Where it would load one still at the older version, such as a connector's setup row in `cotal status`, it names the package and its version with `cotal ext seed` instead of loading it. ### Before the upgrade A script that ran a read to install the built-ins, such as `cotal ext list` right after `npm i -g cotal-ai`, runs `cotal ext seed` instead. ## Space selection in 0.91.0 `cotal use` and the target a command picks without `--space` change. Nothing on a running mesh changes, and there is no outage window. Upgrading the `cotal` binary is the whole operation. ### What stops working - `cotal use ` no longer checks the broker. It selects any registered space, including a stopped one, and the command that targets it reports the space not up. A name that is not registered exits 2 with `TARGET_NOT_FOUND` instead of 1. Bare `cotal use` reports the selection instead of exiting 1 with its usage, and `--json` writes one envelope. - A command run without `--space` stays on the selected space when its broker is down and refuses with `TARGET_UNAVAILABLE`, exit 2. It used to fall back to the one other running mesh and print a note that the default mesh was down. A selection that names a space no longer registered now refuses with `TARGET_NOT_FOUND` instead of falling back to the local project or the one running mesh. - `cotal space` lists the registered spaces. `cotal meshes` still lists them, and its `--json` lines keep their shape. ### Before the upgrade A script that relied on the fallback names the space it means with `--space`, or selects it with `cotal use `. A script that read `cotal use` exiting 1 for a stopped space checks the broker with `cotal space show --json` and reads `result.check`. ## Extension listing output in 0.91.0 `cotal ext --json` and `cotal ext list --json` now write one JSON document instead of one JSON line per extension. Nothing on a running mesh changes, and there is no outage window. Upgrading the `cotal` binary is the whole operation. ### What stops working - A script that reads `cotal ext list --json` line by line reads one envelope instead. The rows it read are now `result.items`, each with the same fields, and `result.root` is the install prefix. With `jq`, `cotal ext list --json | jq -c '.result.items[]'` prints the rows as before. An empty prefix writes an envelope with no items instead of nothing. - A usage error from `cotal ext` exits 1 as one `refused` envelope under `--json`, and `--plain` or `--json` on `ext add`, `ext remove`, `ext root` or `ext seed` is refused. `cotal ext root` still prints the path alone. - The human listing is a table now. Human output is not a parsing target; use `--json`. ### Before the upgrade Search scripts for `cotal ext list --json` and `cotal ext --json`, and change line-by-line reads to read `result.items` from the one document. ## History reads in 0.91.0 `cotal history [channel]` reads retained messages. Clearing is now `cotal history clear`, which asks in a terminal and takes `--yes` in a script. This changes the CLI grammar only. Stored messages, credentials, agents and running services do not migrate or restart, and there is no outage window. Upgrade the CLI and its first-party components together through the normal update path; a separate broker or manager can stay running. ### What stops working - Replace a script's `cotal history --force` with `cotal history clear --yes`. `--force` is refused on every `history` form, `history clear` included, rather than clearing anything. Add `--dms` to the explicit clear command only when the purge should include direct messages and the identity holds that authority. - `history clear` accepts no channel argument, read filters or `--follow`. It purges every channel in the selected space; these inputs are refused rather than treated as a selective purge. - Bare `history` reads channels within the caller's read grant. It never clears stored messages. Use `--json` and `result.items` for scripts; the human report is not a parsing target. - A user-auth history read uses the connection's channel grant. DM audit history and stream-sequence cursors are not available through that ordinary human credential. Use a time with `--since`. ### Before the upgrade Find scripts that pass `--force` to `history` and change their explicit purge invocation. No stored state or credential backup is needed for this CLI-only transition. Keep the deployment's usual backup procedure before an intentional history purge; reads do not make a backup or restore data that retention has already removed. ## Status output in 0.91.0 `cotal status` writes one structured document with `--json`, takes `--plain` and `--all`, and exits by the structured outcome table. Nothing on a running mesh changes, and there is no outage window. Upgrading the `cotal` binary is the whole operation. ### What stops working - `cotal status` exits 4 when it could not read part of what it reports, such as a PID record, the folder's trust material, a persona catalog, the mesh registry or one of its records, or a user-auth login. It exited 0. An unreachable broker, a stopped process or an unbound responder still exits 0. - The recorded meshes list checks only the selected mesh and reads `not checked` for the others. Pass `--all` to check every one. - The report no longer lists the roster or prints a `cotal status` title line, and its standalone hints moved to a `Hints` section at the end. Rows are labeled `name: value`. Human output is not a parsing target; use `--json`. - `--components` is refused with `--json`, `--plain` or `--all`. Its exit, `0` to `3` by component health, replaces the report's, so it does not exit 4 for a partial report. ### Before the upgrade Search scripts and monitors for `cotal status`. A check that treats any nonzero exit of bare `status` as an outage should treat 4 as a report with unreadable parts. A script that read the roster from `status` should use `cotal ps` for managed seats. ## Read command output in 0.91.0 `cotal ps --json` now writes one JSON document instead of one JSON line per seat, and `ps`, `models` and `runtimes` exit by the structured outcome table. Nothing on a running mesh changes, and there is no outage window. Upgrading the `cotal` binary is the whole operation. ### What stops working - A script that reads `cotal ps --json` line by line reads one envelope instead. The rows it read are now `result.items`, each still the manager's row unchanged, and `result.managers` names each manager instance and whether it answered. With `jq`, `cotal ps --json | jq -c '.result.items[]'` prints the rows as before. Instance headers no longer go to stderr under `--json`. - An incomplete manager census, where a registered manager did not answer, refused or serves a different contract, exits 4 instead of 1. - A mesh that `ps` or `models` cannot resolve or reach exits 2 instead of 1, and a space catalog refresh that `ps` or `models` needed and could not get exits 2 instead of 1. Under `--json` each is one `refused` envelope whose `error.code` names the reason. - `cotal models` exits 4 when a connector's catalog could not be read, and `cotal runtimes` exits 4 when an installed runtime provider fails to load. Both exited 0. `cotal models --agent ` exits 5 when that one catalog could not be read. OpenCode 2.x, which has no catalog the CLI can list, is reported `supported: false` with the reason in `unavailable` and does not fail the read. - The human reports of `ps`, `models` and `runtimes` are tables now, and `ps --wide` prints one labeled record per seat instead of a facts line under each row. Human output is not a parsing target; use `--json`. ### Before the upgrade Search scripts for `cotal ps --json`, and for checks of the exit status of `ps`, `models` or `runtimes`. Change line-by-line reads of `ps --json` to read `result.items` from the one document. ## Shell completion script in 0.91.0 `cotal completion ` is now `cotal completion script `, and bare `cotal completion` reports which shells have the stub instead of printing usage. Nothing on a running mesh changes, and there is no outage window. Upgrading the `cotal` binary is the whole operation. ### What stops working A startup line written by hand that runs the old form, such as `source <(cotal completion bash)` or the PowerShell profile line `cotal completion powershell | Out-String | Invoke-Expression`, now loads no completion. The command exits 1, prints nothing on stdout, so the shell sources nothing, and names the replacement on stderr. Replace the line with its new form: ```bash source <(cotal completion script bash) # bash source <(cotal completion script zsh) # zsh cotal completion script fish | source # fish cotal completion script powershell | Out-String | Invoke-Expression # PowerShell $PROFILE ``` A cached stub from an earlier release keeps calling the hidden `__complete` entry. The new ownership-aware installer does not adopt it just because its text matches a generated script. Review unrecorded stubs and custom startup wiring manually before replacing them. Nothing automatically rewrites a hand-edited startup file. On Linux and macOS, `completion install powershell` asks the installed shell for its current-user all-hosts profile before adding an owned block; it does not adopt an existing manual line. Windows profile editing still requires a manual review and the printed script. New installs record their stub hash and startup block. Reinstall and update refresh unchanged recorded stubs at the same path. Update adds no startup block and does not restore a stub the user removed. Uninstall removes only unchanged recorded material. Modified or unowned material is preserved with a diagnosis. Reopening the shell is necessary to stop using functions already loaded in its parent process. ## CLI command resolution in 0.91.0 The CLI now resolves and validates a command before it seeds connectors or writes configuration. Nothing on a running mesh changes. Brokers, managers, seats, credentials and mesh records need no migration and no restart, and there is no outage window. Upgrading the `cotal` binary is the whole operation. What changes is what two kinds of command line do. ### What stops working - `cotal -v` no longer prints the version. It exits 1 and names `cotal version` and `cotal -V`. A script that reads the version this way must use `cotal version`, `cotal --version` or `cotal -V`. - The words `sp`, `a`, `s`, `r`, `x` and `h` are now aliases of built-in commands (`spawn`, `attach`, `status`, `run`, `ext` and `history`). An installed extension that provides a command under one of those words still loads, but the built-in wins and every run prints a warning naming the extension. Remove or update that extension to clear it. `cotal ext add` refuses a new extension whose name or alias is already claimed. ### Before the upgrade Search scripts and shell aliases for `cotal -v`, and run `cotal ext list` to see whether an installed extension provides one of the words above. There is nothing to snapshot. The extension manifest an older binary wrote is read as before. ## Messaging commands in 0.91.0 `cotal channels` is now `cotal channel`, and `cotal send` names its destination with a flag instead of a `dm`, `msg` or `ask` word. Nothing on a running mesh changes, and there is no outage window. Upgrading the `cotal` binary is the whole operation. ### What stops working - `cotal channels`, with or without `list`, `set` or `default`, exits 1 as an unknown command. Bare `cotal channel` lists the channels, `cotal channel set ` saves one channel's settings, and `cotal channel defaults` reads or updates the space-wide defaults. - `cotal send dm ""`, `cotal send msg ""` and `cotal send ask ""` exit 1 with `USAGE_INVALID`, because they name no destination flag. Use `cotal send -t ""`, `cotal send -c ""` and `cotal send -r ""`. ### Before the upgrade Search scripts and shell aliases for `cotal channels` and `cotal send`, and change each line to the new form above. ## Managed Claude workspace trust in 0.80.0 Before upgrading, snapshot the manager's private Cotal state and Claude configuration home. Pause new managed spawns while upgrading the connector, core, manager and chosen terminal runtime together. For tmux, cmux, Herdr, Orca and custodial Linux PTY seats, use that runtime's supported manager-reload path and observe its custody/release bounds. Default `auto` selects in-process `pty`, whose seats cannot survive a manager restart. Drain those seats at an approved idle window, or use the documented preserve-state cut and `up` recovery, which resumes them rather than keeping their processes running. This upgrade does not require a broker restart. Managed launches now accept the authorized workspace through Claude's native startup dialog. They no longer require a prior interactive visit to that directory. Claude retains its normal repository and linked-worktree trust semantics, including any trust it persists for later sessions. Foreground launches still prompt. No tool-permission bypass flag is added, but accepting workspace trust enables that checkout's hooks and settings, including its default permission mode unless a launch flag overrides it. Authorize only reviewed workspaces. The launch recipe includes optional startup choices before its final confirmation. A custom runtime must implement `LaunchSpec.confirmBefore`, or refuse a recipe carrying it. Ignoring it leaves an untrusted launch at the native dialog until startup times out. The new launch recipe does not require restarting an existing seat; manager reload continuity remains runtime-dependent as described above. Adopt the new runtime before launching new seats. No credentials, mesh records or operator configuration files need manual migration. ## AG-UI `takeCodePoints` export in 0.79.0 `@cotal-ai/connector-core` no longer exports `takeCodePoints`. The AG-UI preview splitter is its only user, and the package exported it only so the emitter could shorten a `RUN_ERROR` failure detail, a path that never ran and is now removed. Nothing about a running mesh changes, and no shipped connector or host imports it. Only code of your own that imports it from the package is affected. ### What stops working An import of `takeCodePoints` from `@cotal-ai/connector-core` no longer compiles, and an ES module that names it fails to link with `does not provide an export named 'takeCodePoints'`. ### Before the upgrade Take a string's first `n` code points with `Array.from(s).slice(0, n).join("")`, which is what the helper did. ## SecretStore create-only write in 0.78.0 `SecretStore` from `@cotal-ai/core` has a fourth method, `create(key, value)`. It stores the value only while the key is absent and resolves `true`, or resolves `false` and leaves the stored value in place. The auth secret kinds made once per space now write their first value through it: the callout account, the issuer keys, the owner secret and a hosted auth plane's instance identity. Before, two first calls on one space each made a value, the later `put` replaced the earlier one, and the earlier caller kept using a value the store no longer held. The filesystem store implements `create` with an exclusive create, so a CLI root needs nothing, and stored values do not change. ### What stops working A store of your own without `create` no longer compiles as a `SecretStore`. Plain JavaScript gets a `TypeError` at the first call that makes one of those kinds, such as `prepareServer` on a new space or the first `startAuthService` of a hosted context. ### Before the upgrade Add `create` to your store, decided in one atomic step by the backend, such as a conditional create or an insert that fails on an existing key. A read followed by a write brings the race back. ## Manager-service authority refusal budgets in 0.78.0 `POST /manager-service-authority` now counts its refusals against the exchange face's budgets, as `POST /exchange` does. On the loopback face a missing or wrong capability counts in the invalid-capability window, and a refused IdP token or request counts in the refused-exchange window. On the public face a refusal counts in the peer's bucket. A running mesh needs nothing, and a request that succeeds is never throttled. ### What stops working `ManagerServiceAuthorityCtx` from `@cotal-ai/auth` now requires `failures` and `badCaps`, so a host that builds it in an object literal without them fails to compile. The handler now calls the `peerKey`, `throttled` and `recordFailure` of the policy it is given. After 30 refusals in a minute on one face, a further refusal on either route answers 429 without its reason. ### Before the upgrade Pass the `failures` and `badCaps` arrays your `/exchange` route counts in, so both routes share one budget per face. A host that serves only this route can pass new empty arrays. ## Primitive scope kind in 0.78.0 `PrimitiveSpec` from `@cotal-ai/lang` replaces `opensScope: boolean` with `scope: ScopeKind | null`, the scope kind a primitive opens, in the way `kind` names the effect it journals. Both engines now read that kind from the table. Before, they cast the primitive's name to `ScopeKind`, so a scope row missing from the union compiled and reached the journal as a kind its readers did not know. A running mesh and its journals need nothing. ### What stops working Code that reads `opensScope` from `PRIMITIVES` or a `PrimitiveSpec` fails to compile, and plain JavaScript reads `undefined` there, so a check on it never holds. ### Before the upgrade Read `spec.scope !== null` where you read `spec.opensScope`, or `spec.scope` when you need the kind. ## Token verifier clock tolerance in 0.77.0 `validateUserToken`, and the `IdpConfig` that `verifyIdpToken` and `createIdpBridge` take, from `@cotal-ai/auth` no longer accept `clockToleranceSec`. Both verifiers allow a fixed 5 seconds of clock skew on `exp`, `nbf` and `iat`, the default they already applied. Nothing in the repository passed the option, so a running mesh behaves as before. Only code of your own that passes it is affected. ### What stops working A call or `IdpConfig` that passes `clockToleranceSec` in an object literal no longer compiles. Plain JavaScript that keeps it still runs, and the value is ignored. A `NaN` value, such as `Number()` of an unset environment variable, used to turn off the expiry, not-before and issued-at checks in both verifiers, so an expired or post-dated token was accepted. A large value widened those checks by that many seconds. Both kinds of token are now refused. ### Before the upgrade Remove `clockToleranceSec` from each call and each `IdpConfig`. A host whose clock is more than 5 seconds off the issuer's needs its time synchronized. ## Auth context closure in 0.72.0 Existing deployments need no credential migration or restart for these additive APIs. Embedded hosts can now inspect `handle.connections()` and await `handle.closed` after `close()` or `drain()` to prove every owned transport ended, including the callout, replaced readiness readers and short-lived clients. The inventory is a detached snapshot. A transport close failure now rejects with its connection label. The terminal signal stays pending while any connection remains live. Repair the failure and retry `close()` before awaiting `handle.closed`. Closing one hosted context does not close another account's context. Read a space's claim with `readPlaneClaim(kv, space)` on that account's leader-only auth bucket. An unclaimed space returns `undefined`; held and released rows retain their claim identity. Deleted, malformed and foreign-space rows refuse. `PlaneClaimRow` and `PLANE_CLAIM_KEY` are exported. Use `observeAccountLivenessWithCreds({ servers, observerCreds, accountId, options })` with the account-scoped membership-observer credential to list that account's connections. It never widens credentials or evicts connections. Zero rows prove absence only with a complete sweep and the single-server proof. An embedded endpoint's trusted composition can retain transport custody through `EndpointOptions.onConnection`. ## Event plane of an owned spawn in 0.72.1 On a per-user-auth mesh, a spawn-scoped caller can arm the event plane of a child under its own owner without `admin`. This fixes owned spawns in spaces whose registration policy requires the plane. Upgrade the manager to pick up the admission change. No credential or state migration is needed. Cross-owner arming still requires `admin`, and the child's own-channel rule and ledger envelope are unchanged. A silent non-owner caller in a space without the policy still has the plane disarmed, with a notice if provisioning succeeds. ## Hosted auth plane identity in the store in 0.76.0 A hosted context started through `startAuthService` keeps its auth plane instance identity in the injected `SecretStore` under `authInstanceKey(space)`, with the other auth secret kinds. It used to sit under `stateDir` at `.cotal/space./auth-instance.json`, though `stateDir` holds non-secret state and the record holds the plane's private serve seed. The first start of an upgraded context puts that record into the store, removes the file and keeps the instance. A CLI root is unchanged. ### What stops working A start refuses when the store and `stateDir` hold different instance identities, and names both. A store that refuses a put of the new key fails the start. ### Before the upgrade Let the store accept a put of `authInstanceKey(space)`. A copy of `stateDir` taken before the upgrade still holds the serve seed, so delete it or protect it as secret material. ## Injected SecretStore naming in 0.76.0 The manager and the delivery daemon name an injected `SecretStore` through `injectedSecretStoreIdentity` from `@cotal-ai/core`. The rule is unchanged: the store's declared identity, else the coordinate in `COTAL_SECRET_STORE`, else a refusal. A running mesh needs nothing. ### What stops working `reloadStoreIdentityOf` from `@cotal-ai/delivery` takes `{ injected: true, store }` or `{ injected: false, identity }`. A call that passes `injected: true` with no store fails to compile, and plain JavaScript gets a `TypeError`. Both processes refuse an unnamed injected store with one message, which starts `an injected SecretStore must declare its identity`, so a log match on either old message no longer matches. ### Before the upgrade Pass the injected store to `reloadStoreIdentityOf`, or call `injectedSecretStoreIdentity(store)`. ## Manager-service authority policy flag in 0.76.0 `handleManagerServiceAuthority` from `@cotal-ai/auth` no longer reads `allowManagerAuthority` from its policy argument, and the field is removed. Both exchange faces set it to `true`, so it never refused anything. A running mesh needs nothing: the public listener serves `POST /manager-service-authority` as it did before, and the docs now list that route. ### What stops working A host that serves the route itself and passes a policy object literal with `allowManagerAuthority` fails to compile with TS2353. Plain JavaScript that set it to `false` got a 403 from the handler, and its requests now reach the capability and IdP checks. ### Before the upgrade Remove `allowManagerAuthority` from the policy you pass. Where you set it to `false`, leave the route out of that listener's route table instead. ## Hermes model from the environment in 0.68.0 A connector now launches on the model and variant its launcher resolved (the `--model` or `--variant` flag, else the agent file's `model:` or `variant:`) and no longer reads them again from the agent file. The Hermes connector also no longer takes a model from `HERMES_MODEL` in the environment of the process that spawns the seat, including when `spawn.env` lists it. ### What stops working A Hermes spawn whose only model was `HERMES_MODEL` in the spawning environment is refused at launch, and the refusal names both ways to set a model. Spawns that set `--model` or `model:` are unchanged, on every connector. Code that calls a connector's `buildLaunch` directly with only `configPath` now gets no model or variant from that file. Pass them as `model` and `variant`. ### Before the upgrade Move each Hermes seat's model from `HERMES_MODEL` onto its spawn with `--model`, or into its persona's `model:`. ## Run answers on a participant manager in 0.68.0 A participant manager now asks its issuing host for an answering credential by naming the run and step it answers. The host reads the pause's token off that run's journal and no longer accepts a token from the manager. Runs on a mesh with no participant manager are unaffected. ### What stops working While a participant manager and its issuing host run different sides of this release, the host refuses every `cotal run answer` and every amendment that manager serves, because each side refuses the other's request shape. Starting, resuming and reading runs is unchanged. A pause stays waiting through the window, or follows its timeout if it has one. ### Before the upgrade Upgrade the auth service and every participant manager registered with it in the same window, then answer the pauses that waited. ## Headless OpenCode handshake in 0.69.0 With `COTAL_SERVE_HEADLESS=1`, the OpenCode launcher's `[cotal-serve]` line on stdout now carries only `port` and `session`. The server password no longer appears in it, and the 1.x TUI no longer receives the password on its command line. ### What stops working A headless host that read `password` from that line has no password, and the server refuses its requests. Seats with a TUI, and headless seats that no host drives, are unaffected. ### Before the upgrade Have each headless host mint a password and pass it to the launcher as `OPENCODE_SERVER_PASSWORD`, then use it for basic auth as before. Without that variable the launcher mints its own. ## Filesystem store identity in 0.69.0 The delivery daemon's answer to the manager's store check now names a filesystem store by its root and by a random `id` that the store records once in `store.id` inside its own directory: `.cotal/store.id` for a workspace root, or the directory of the file for `cotal deliver --creds `. A manager no longer counts the daemon's store as its own because the two roots have the same path. On a split whose broker host and manager host use one root path, the manager host now stays off the daemon-credential renewal lease, so `cotal doctor auth --fix` on the broker host can renew the daemon credentials. ### What stops working A manager and a delivery daemon on different sides of this release refuse each other's answer to the store check. The manager then remints no daemon credential, and a manager that is booting does not start. This is read from the code and was not measured across two releases. A `cotal deliver --creds ` whose directory is a read-only mount and holds no `store.id` stops at start. So does a `--creds` file that is its directory's `store.id` under any name, and a `store.id` that is a symbolic link or holds anything but a lowercase UUID. ### Before the upgrade Upgrade the broker host and every manager host of a space in the same window. For a `--creds` file on a read-only mount, add a regular `store.id` file beside it that holds a new lowercase UUID and no newline, as `node -e 'process.stdout.write(crypto.randomUUID())' > store.id` writes. Move a `--creds` file named or linked as `store.id` to a file of its own. ## Detached spawns with `--share-tools` in 0.69.0 The manager's `spawn` operation now takes `shareTools` as a list of MCP server names. The CLI parses `--share-tools` into that list before it sends the request, and the manager cluster document moves to revision 22. A cut taken with `cotal down --preserve-state` before the upgrade still resumes: the manager reads its `cotal-manager-resume/v1` inventory and writes new cuts as `cotal-manager-resume/v2`. ### What stops working A CLI and a manager on different sides of this release refuse a detached spawn that passes `--share-tools`, because the CLI checks each request against the contract the manager serves. This is read from the code and was not measured across two releases. A detached spawn without the flag, a foreground spawn and a roster entry are unaffected. A manager older than this release cannot resume a cut that this release took. ### Before the upgrade Upgrade the CLI on every host that runs `cotal spawn --detach` in the same window as the managers it reaches. ## Shared MCP server checks in 0.69.0 The cotal config reader now checks each server under `connectors..mcpServers` when it reads the file, and refuses one that cannot launch as written, naming the file and the field. The rules are in [the config file](/api-reference/config/#the-config-file). ### What stops working A config file that holds such a server refuses every Claude spawn that reads it, including one with `--share-tools none`. Before, a field of the wrong type failed each Claude spawn that shared the server with a `TypeError` that named neither the file nor the server, a spawn that did not share it launched, and a server with no `command` or `url` was passed to `claude`, which never started it. Read from the code and not measured: spawns on other connectors, a manager resume and the step of `cotal setup` that records the shared list read the same files, so each stops at the same refusal. ### Before the upgrade Check `connectors..mcpServers` in the operator-level config file and in each space's `.cotal/config.json`. Give each server a string `command`, or a `type` of `http`, `sse` or `ws` with a string `url`. Write `args` as a list of strings and `env` and `headers` as objects of strings, or remove the server. ## Remote manager family eviction in 0.69.0 A remote manager registered through its host now asks the host to evict up to 256 holders of its credential family in one maintenance request, and the host reads the family once for the whole set. Before, a restart sent one request per holder and the host read the whole family for each one. Meshes with no remote manager are unaffected. ### What stops working While a remote manager and its issuing host run different sides of this release, each side refuses the other's eviction request shape. A restart whose credential family already has holders then fails at its eviction step and leaves the manager's registration gate frozen. A first start, a clean stop and the host's reconciliation of a foreign slot holder are unchanged. ### Before the upgrade Upgrade the auth service and every remote manager registered with it in the same window. A manager that restarted inside the window resumes its frozen registration on its next start once both sides run this release. ## AG-UI emitter holder hooks in 0.69.0 `AguiEmitterHolder` from `@cotal-ai/connector-core` now takes its hooks as one named object after the emitter factory: `new AguiEmitterHolder(startEmitter, { onError, onRunClosed, waitLive, runMeta })`. Only `onError` is required. Nothing about a running mesh changes, and every shipped connector passes its hooks by name. Only a connector of your own that builds a holder is affected. ### What stops working A holder built with positional hooks, such as `new AguiEmitterHolder(start, onError, onRunClosed)`, no longer compiles, because the constructor takes two arguments. Plain JavaScript that keeps the positional form still runs, but the holder calls none of its hooks, so a failure never reaches `onError`. ### Before the upgrade Pass each hook by name, for example `new AguiEmitterHolder(start, { onError, onRunClosed })`, and drop any `undefined` that filled an earlier slot to reach a later hook. ## Worker run failure type in 0.70.0 `WorkerRunFailed`, the failed result of `runInWorker` in `@cotal-ai/lang`, is now a union on `class`: `released`, `held`, `effect`, `too-large`, `rejected` or `error`. A running mesh needs nothing, because the runtime host and the engine thread ship in the same install. A run on the compiled engine whose program throws an object with `code: "L5012"` or `code: "L5025"` used to end released and now ends failed, as it does on the walker. ### What stops working TypeScript code that reads `code`, `reason`, `step`, `pending`, `kind`, `detail` or `tooLarge` on a `WorkerRunFailed` it has not narrowed fails with TS2339. A released, held, too-large or rejected result no longer carries `code`, so JavaScript that branched on `L5012`, `L5025`, `L5006` or `L5010` stops matching with no error. `tooLarge` is gone. ### Before the upgrade Branch on `class` where such code read `code`: `released` for L5012, `held` for L5025, `too-large` for L5006 and `rejected` for L5010. An `effect` or `error` result keeps its `code`. Once narrowed to `too-large`, a result carries the `stepKey`, `bytes` and `bound` that `tooLarge` held. ## Remote manager request builder in 0.70.0 `remoteManagerClient.remoteManagerAuthorityRequest` from `@cotal-ai/manager` now takes an operation's coordinates as one object, and `remoteManagerRegistrationProof` from `@cotal-ai/core` computes the proof from the manager's identity state instead of a request. Nothing about a running mesh changes: the proof digest and the request on the wire are the same, so a manager and a host on different sides of this release still accept each other. Only code that builds remote manager requests itself is affected, in TypeScript and in plain JavaScript. ### What stops working A call that passes the registration proof, contract artifacts, session, retirement or transfer reader as positional arguments after the operation no longer compiles. A call that passes a request to `remoteManagerRegistrationProof` no longer compiles either, because the second argument now names the lifecycle `lifecycleUid`, as the identity state does. Plain JavaScript runs both old calls without an error. The builder drops the positional coordinates, so the host refuses the request with `requires a sha256 registrationProof`. A proof computed from a request leaves out the lifecycle, so the host refuses a request that carries it as a proof mismatch. ### Before the upgrade Name the coordinates, for example `remoteManagerAuthorityRequest(state, "cli", "retire", { registrationProof, retirement })`. Compute the proof as `remoteManagerRegistrationProof(owner, state)`, adding the contract artifacts as a third argument for activation only. A host that recomputes the proof from a received request passes `{ space, instanceId, lifecycleUid: managerLifecycleUid, identities }` from that request. ## Bearer validator lifetime cap in 0.70.0 `validateUserToken` from `@cotal-ai/auth` no longer takes `maxTtlSec`. It caps a bearer's lifetime at the cap of the bearer's view, the same cap the issuer applies when it mints: 900 seconds, or 300 for a `transfer-writer` bearer. The auth callout never passed the option, so a running mesh behaves as before. Only code of your own that calls the validator with `maxTtlSec` is affected. ### What stops working A call that passes `maxTtlSec` in an object literal no longer compiles. Plain JavaScript that keeps it still runs, and the value is ignored. A `NaN` value, such as `Number()` of an unset environment variable, used to turn the lifetime check off and accept a bearer of any lifetime. That bearer is now refused at its view's cap. ### Before the upgrade Remove `maxTtlSec` from each call. A test that needs a bearer to expire sooner mints one with a shorter lifetime. ## Persisted identity records in 0.70.0 The manager instance identity, the manager sibling identities, the auth plane instance identity and a participant manager's remote authority state now share one reader and one first mint in `@cotal-ai/workspace`, exported as `claimIdentityRecord` with the nkey check `identityOf`. Each record is read as a regular file, must hold non-empty nkeys and is created exclusively, so concurrent first starts of a participant manager on one root now settle on one identity where each used to keep its own. `saveManagerInstanceIdentity` and `saveAuthInstanceIdentity` are gone. A running mesh whose records are plain files needs nothing. ### What stops working A manager instance, auth instance or remote authority record that is a symlink, a directory or any other non-regular entry is refused where it used to be followed. The manager, the auth plane and a participant manager fail to start on it, and `cotal reconcile-gate` and `cotal deregister-instance` refuse it. Retirement already refused it. A remote authority record with an empty nkey id or seed is refused too. A first mint that loses its race and cannot read the winner now refuses with `identity-record-create-lost` in place of `manager-instance-identity-create-lost` or `auth-instance-identity-create-lost`. Code that imports either `save` function no longer compiles. ### Before the upgrade Replace a symlinked identity record with a copy of the file it points to. Code that wrote a record with a `save` function plants it with `createManagerInstanceIdentity` or `createAuthInstanceIdentity`, which create the record when it is absent and otherwise return the stored one unchanged. Nothing replaces an overwrite of a stored identity. ## Manager instance in user credentials in 0.70.0 `AuthProvider.userCredentials` from `@cotal-ai/core` no longer returns `managerInstanceId`. A `manager-caller` credential's manager instance is the signed `act.managerInstanceId` claim in its bearer, which the broker verifies and the CLI already used. The reference provider in `@cotal-ai/auth` stops copying the exchange response's field into its result, where nothing compared it with the bearer. The exchange still answers with the field, so a running mesh behaves as before. ### What stops working Code of your own that reads `managerInstanceId` from a `userCredentials` result no longer compiles, and plain JavaScript reads `undefined` there. ### Before the upgrade Read the instance from the bearer's `act.managerInstanceId` claim. ## Auth plane identity location in 0.70.0 The user-auth service keeps its instance identity in the root's `.cotal/space./auth-instance.json`, beside the manager's. It used to sit inside `.cotal/auth`, at `space./.cotal/auth/auth-instance..json`, so a copy of that folder carried it. The first start of an upgraded root moves the record and keeps the instance. A hosted context started through `startAuthService` has its record moved the same way inside its `stateDir`. ### What stops working Code that calls `openAuthAuthorityPlane` without the new `identityRoot` option no longer compiles. A start that finds a record both in `.cotal/space./` and at its older place refuses and names the two files. A start also refuses when the older place of the auth or manager identity holds a symlink, a directory or anything else that is not a regular file. The manager used to skip a dangling symlink there and mint a new identity. ### Before the upgrade Pass `identityRoot` to `openAuthAuthorityPlane`. When `dir` is a workspace root's user-auth state dir, `/.cotal/auth/space.`, pass that root. A plane with no workspace root, as `startAuthService` runs, passes `dir` itself. Either keeps the identity the plane already has: on the first start it moves from `/.cotal/auth/` to `/.cotal/space./`. Never pass a directory inside `.cotal/auth`: the record would land in the folder an operator copies and travel with it again. A copy of `.cotal/auth` taken from a root last run by an older Cotal carries that root's record. Delete `.cotal/auth/space./.cotal/auth/auth-instance..json` from the root you copied it to before the first `cotal up --user-auth` there. ## Per-seat `COTAL_` names in `spawn.env` in 0.71.0 `spawn.env` in the cotal config no longer forwards a `COTAL_` name the launcher sets for each seat, such as `COTAL_ROLE`, `COTAL_MODEL` or `COTAL_SUBSCRIBE`. Before, a seat launched with no value of its own took the spawning process's value and ran under that role, model or read set. The machine-wide knobs a seat already receives, such as `COTAL_HOME`, may still be listed. ### What stops working Every spawn and resume under a config whose `spawn.env` lists such a name is refused before launch, and the refusal names the entry. Code that calls `launchEnv` from `@cotal-ai/connector-core` with such a name in `envAllow` gets the same error. ### Before the upgrade Remove those names from `spawn.env`. Give each seat its role, model and channels with `--role`, `--model` and `--subscribe`, or in its persona's `role:`, `model:` and `subscribe:`. ## Role addresses in 0.71.0 A role must be one `[A-Za-z0-9_-]` token. Before 0.71.0 any other spelling was rewritten into one: ` probe ` reached the `probe` queue and `pro.be` reached `pro_be`, while the message kept the spelling sent. An anycast to `*` was accepted and stored where no holder reads it. ### What stops working An agent whose role is outside the token set no longer starts, however it is launched: `cotal join --role`, `cotal spawn --role`, an agent file's `role:`, `COTAL_ROLE` and an embedded endpoint's `card.role` are all refused before the agent joins. A send to such a role, or to `*`, through `cotal send ask`, `/anycast` or `cotal_anycast` is refused, and nothing is stored. `routeToken` is no longer exported from `@cotal-ai/core`. A role routes as spelled, so code that used it to name a role's queue uses the role itself, and `assertValidRole` checks one. ### What migrates on its own Every task queue. A `svc_` durable was always named from the rewritten token, so its pending requests and its holders carry over. ### Before the upgrade Rename each role outside the token set to the token it already routed to: remove the surrounding spaces and replace every other character outside the set with `_`. Rename it where the holder is launched and in every script or prompt that sends to it. ## IdP URLs on `localhost` in 0.73.0 The IdP URL that `cotal login --idp` and `cotal up --user-auth --idp` take, the JWKS URL the auth service pins from it, the space-catalog link and `cotal sync` now share one rule: `https://`, or `http://` on a loopback IP literal. `localhost` is a name, so it no longer counts: a hosts entry would choose the IdP, and with it the keys the callout trusts. Every loopback literal now passes, so `http://127.0.0.2/api/auth` and `http://[::ffff:127.0.0.1]/api/auth` are accepted where they were refused before. A JWKS URL with any scheme other than `https:` or `http:` is refused. A mesh whose IdP uses HTTPS is unaffected. ### What stops working On a mesh whose IdP is pinned at `http://localhost:/...`, the auth service builds its key resolver from the pinned JWKS URL at start, and that resolver now refuses it with `JWKS origin must be https (or http on a loopback IP literal for dev)`. `cotal login` and `cotal logout` with `--idp http://localhost:/...` are refused with `idp url must be https (or http on a loopback IP literal such as 127.0.0.1 for local dev)`, and so is every command that reads a session cached under that URL. ### Before the upgrade Use `127.0.0.1` (or `::1`) in place of `localhost`. In the space's `idp.json` under the mesh's `.cotal/auth`, change `url` and `jwksUri` to the literal spelling and leave `issuer` and `audience` as they are. Owners derive from the issuer, so existing grants keep matching. Change a manifest's `broker.idp` the same way, because `cotal up` refuses an `--idp` that differs from the pin. Then have each person run `cotal login --idp http://127.0.0.1:/api/auth` again, because sessions are cached under the URL. ## Terminal layout `Pane.confirm` in 0.73.0 `Pane.confirm` is removed from `@cotal-ai/core`, and the tmux and cmux terminal-layout providers no longer press Enter in the panes they open. The flag carried no prompt text, so both providers pressed Enter five times, one second apart: the first dialog a pane showed was answered with its default, and a prompt that never appeared went unreported. Nothing in Cotal sets it, so a running mesh needs nothing. Spawned agents are unaffected, because their runtimes match the startup prompt by its text. ### What stops working A `Pane` object literal that sets `confirm` fails to compile with TS2353. Plain JavaScript that sets it runs without an error, and the pane stays at its prompt. ### Before the upgrade Remove `confirm` from each `Pane`. Start a command that shows a startup prompt with the option that skips it, or answer the prompt in the pane. ## Carrying a resumed Claude session to another host in 0.67.0 `cotal spawn --resume --detach --on ` now carries a Claude session held on the operator's host to the target manager instance. Both sides need this release: an older manager does not serve `transcript-receive`, and the CLI then stops with that manager's refusal instead of launching. The manager cluster document moves to revision 21, and the `ps` row's `resume` object gains `host` and `transferredAt`. A manager host that runs carried seats needs `CLAUDE_CODE_OAUTH_TOKEN`, `ANTHROPIC_AUTH_TOKEN` or a cloud provider selection in its environment, because each carried seat runs in its own Claude home with no stored login. On an authenticated mesh the CLI mints the transfer writer from the space's signing seed, so the carrying host needs that seed, as for any other operator command. On a user-auth mesh it exchanges the operator's login for a `transfer-writer` view instead, so the operator's grant needs scope `admin`, and the auth service must run this release. A remote manager receives a carry once its host serves the manager-service `transferReader` operation. A seat launched without carrying, including any `--resume` whose id this host does not hold, is unchanged. ## Lifecycle head type in 0.67.0 `LifecycleMapping`, the type `parseLifecycleHead` returns, is now a union on `state`. Nothing about a running mesh changes: heads that parsed before parse the same way, and the refusals are unchanged. Only TypeScript code that compiles against `@cotal-ai/core` is affected. ### What stops working An `interface` that extends `LifecycleMapping` fails with TS2312, because an interface cannot extend a union. Code that builds a head in memory no longer compiles when the head is `retiring` without its `op`, or `active` or `retired` with one. The parser already refused those heads. ### Before the upgrade Declare such an interface as an intersection instead, for example `type ActiveMapping = LifecycleMapping & { state: "active" }`. A reader that has checked `state === "retiring"` reads `op` without a guard. ## Issuance gate types in 0.67.0 `EpGateRow` and `EndpointGateRow`, which `parseIssuanceGate` and `parseEndpointGate` return, and `EpGateState`, which an `EpIssuanceGate` or `EpIssuanceBarrier` returns from `observe`, are now unions on `state`. Nothing about a running mesh changes: gates that parsed before parse the same way, and the refusals are unchanged. Only TypeScript code that compiles against `@cotal-ai/core` is affected. ### What stops working An `interface` that extends one of these types fails with TS2312, because an interface cannot extend a union. Code that builds a gate in memory, such as a custom barrier's `observe`, no longer compiles when the gate is `frozen` or `retired` without its `op`, or `open` with one. The gate parsers already refused those rows. ### Before the upgrade Declare such an interface as an intersection instead, for example `type CustomGateRow = EpGateRow & { custom: string }`. ## Lifecycle-blocked refusals in 0.66.0 A refusal that carries `ai.cotal.ep.lifecycle-blocked` now reports only the lifecycle state it read. Nothing about a running mesh changes. A client that branches on the detail must read the new field. ### What stops working A refusal raised at the issuance gate used to carry `headState` without reading the head: `retiring` for a frozen gate and `retired` for a retired one. It now carries `gateState` (`frozen` or `retired`) and no `headState`. A client that treats `headState: "retired"` as a burned uid, or `headState: "retiring"` as a retirement in flight, no longer matches those refusals, and the `[lifecycle ...]` suffix on the error string changes the same way. A custom issuance barrier whose `observe` returns a frozen gate without a valid `op` (a string `opId` and one of the four op kinds) is now refused as `internal` by `registerServiceInstance`. ### Before the upgrade Update such a client to read `gateState` for a gate refusal and `blockedOp` for the operation that holds the gate. `headState` is present only when the refusal read the head, for example an activation refused because the head is still retiring. ## Workflow programs that bind `once` in 0.65.0 `once` is now a scope of the workflow language, so it is a reserved name. A program that declares its own `once` binding (`const once = ...`, a parameter or a function named `once`) is refused at validation with L2002. Nothing else about a running mesh changes. ### What stops working A run whose recorded program binds `once` cannot be resumed after the upgrade, because a resume validates the recorded program again. A new `cotal run start` of such a program is refused before anything is recorded. ### Before the upgrade List the runs with `cotal run ps` and check each program that is still running or held for a binding named `once`. Let those runs finish on the old version before you upgrade the manager, and rename the binding in the program before you start it again. ## From 0.58.0 to 0.59.0 Every connector now publishes a failed run's `RUN_ERROR` on `events..` with the fixed message `run failed` and no `code` or `rawEvent`. The error text and error kind a harness reports can echo a prompt, a peer message or tool output, and that channel has a different read ACL. A reader that showed the message or branched on `code` gets neither after the upgrade. Where a connector reports the error kind as the agent's presence condition, that is unchanged. ### Settle pending event frames before the upgrade Each session's events are frozen in its event write-ahead log before they are published. A session restarted on 0.59.0 whose log still holds an unacknowledged frame with an older `RUN_ERROR` does not republish it: its event emitter halts with `egress-run-error` and publishes nothing further for that session. The broker may or may not already hold that frame, so the halt cannot settle it. 1. Stop the seats cleanly on 0.58.0, with the broker still up. 2. List the logs that still hold a pending frame. The logs live under the events state root (`COTAL_WORKSPACE_ROOT`). Empty output means there is nothing to settle. ```sh find "$COTAL_WORKSPACE_ROOT/.cotal/events" -name wal.json \ -exec jq -r 'select(.pending != null) | input_filename' {} + ``` 3. For each session listed, start it again on 0.58.0 while the broker is reachable, let it recover, stop it, and run step 2 again. Recovery publishes the frame as 0.58.0 would have, error text included, so it only finishes what 0.58.0 had already started. If that start halts with `cas-loss` instead, the agent's subject is no longer at the sequence this log expects, and no restart settles that log, on 0.58.0 or later. A lost acknowledgement is one cause: the broker stored the frame, so it and its error text are already on the channel, and every retry halts the same way because the stream checks the frozen expectation before it deduplicates. The halt message names the other causes, such as a second emitter for the same agent under a different state root, a restored stream or frontier record, or a purged channel. With those the pending frame may never have reached the broker, so a `cas-loss` does not tell you whether it landed. Find and stop any second writer and rule out a restored state first. Clearing the halt then means purging the agent's event channel and removing the agent's directory under the events state root whole (see [Event plane](/docs/connect-claude/#event-plane)). That abandons the pending frame whether or not the broker has it, and the purge also drops the earlier frames of every session of that agent. 4. Upgrade once step 2 prints nothing. If a session halts with `egress-run-error` after the upgrade, go back to step 3 for that session on 0.58.0. Do not edit or delete `wal.json` on its own to get past either halt: clearing the pending frame abandons that epoch, an event the broker never received is lost, and removing part of the directory leaves a state the next start refuses. ## Explicit actor grants in 0.59.0 `cotal actor grant` no longer fills an omitted ACL flag with its wide default. A grant names `--scope`, `--allow-subscribe` and `--allow-publish`, or passes `--full` to give the ones it leaves off their wide defaults (`spawn,role:default`, `>` read, `>` post). Any other grant is refused. The break is in the CLI on the machine that holds the actor ledger, the one that ran `cotal up --user-auth --idp `. No stored row, credential or wire message changes. ### What keeps working Existing actor ledger rows keep the authority they were granted, and their users and agents connect as before. `actor revoke`, `actor list` and a `grant` that names all three ACL flags behave as they did on 0.58.0. Nothing on disk is converted. ### What stops working A grant that leaves off any of the three flags without `--full` exits 1 with `refusing to grant "" with --scope, --allow-subscribe, --allow-publish left off`, naming the flags it is missing, and then prints both accepted forms. It writes no row and does not retire the actor's current lifecycle. An existing row stays as it was, and an actor granted for the first time stays out until the grant is run again. This includes the bare grant printed on 0.58.0 by `cotal login`, `cotal status`, `actor list` and the not-granted refusal. Look for it in provisioning scripts, onboarding runbooks and anything that pastes those hints. ### Upgrade order Change the scripts before the ledger machine is upgraded, and make each grant name all three flags. 0.58.0 and 0.59.0 both accept that form. To keep a wide row, write its defaults out: ```sh cotal actor grant --sub \ --scope spawn,role:default --allow-subscribe '>' --allow-publish '>' ``` Switch to `--full` only once the ledger machine runs 0.59.0. 0.58.0 refuses it with `Unknown option '--full'` before it reads the ledger. Brokers, managers and participant machines need nothing for this break, so their order is the one the section above gives. ### The window This break has no outage. No process restarts for it, and a refused grant changes nothing. The exposure is a grant script that runs against 0.59.0 before it was changed: it fails and grants nothing. ### Snapshot this first Nothing is rewritten, so this break has no state to back up. On the ledger machine, save the output of `cotal actor list` to compare rows after the changed scripts run, and list the scripts that call `cotal actor grant`. ### The upgrade end to end ```sh # on the ledger machine, still on 0.58.0 cotal actor list > actors-before.txt grep -rn 'actor grant' # make every grant name --scope, --allow-subscribe and --allow-publish, run them, then upgrade npm i -g cotal-ai@0.59.0 cotal actor list | diff actors-before.txt - ``` Both refusals quoted here were run on 0.58.0 and on the 0.59.0 code. That brokers, managers and stored rows need nothing is read from the change, which touches only the CLI and its hints, and was not run on a live split deployment. ## Repeated flags refused in 0.59.0 A `cotal` flag given more than once is now a usage error unless the command declares it repeatable. On 0.58.0 the last value won with no message, so `cotal down web --space a --space b` acted on `b` while a wrapper that checked the first `--space` verified `a`. The break is in the command-line parser on the machine that runs the command, including commands added with `cotal ext add`. No stored state, credential or wire message changes. ### What keeps working A command line that gives each flag once parses as it did on 0.58.0, in any order and in the `--flag=value` form. Flags whose help says repeatable, such as `--opt` and `down --session-store`, still collect every value. A flag-shaped word after `--` is still a positional. The daemons, units and agents that `cotal` starts for itself are given each flag once, so a fleet driven only by `cotal` commands typed by hand needs no action. ### What stops working A command line that repeats any other flag exits 1 before the command runs. It prints `Option '--space' cannot be repeated`, or `Option '-f, --file' cannot be repeated` for a flag with a short form, followed by the command's help. `-f` and `--file` count as the same flag. Look for it in scripts, aliases and wrappers that append a flag to override one set earlier, such as a fixed `--space` followed by `"$@"`. ### Upgrade order Change those scripts first so each flag is given once. 0.58.0 and 0.59.0 both accept that form. Brokers, managers and participant machines need nothing for this break, and each machine's CLI applies it when that machine is upgraded, so their order is the one the sections above give. ### The window This break has no outage. No process restarts for it, and a refused command does nothing. The exposure is a script that still repeats a flag when it runs on 0.59.0: it exits 1 instead of acting on the last value. ### Snapshot this first Nothing is rewritten, so this break has no state to back up. List the scripts, aliases and wrappers that call `cotal` so each one can be checked. ### The upgrade end to end ```sh # still on 0.58.0 grep -rn 'cotal ' # give each non-repeatable flag once, then upgrade npm i -g cotal-ai@0.59.0 # run each changed script; a repeat left behind exits 1 with the usage error and does nothing ``` The refusal and its messages were run against the 0.59.0 parser and `cotal topology view`. That the argument lists `cotal` builds for its own processes give each flag once is read from the code, and was not run on a live split deployment. ## Detached spawns from a seat's shell in 0.62.0 On a static or open mesh, `cotal spawn --detach` run inside a managed seat's shell now launches as that seat. On 0.61.0 it minted a one-shot operator instrument, so the manager recorded that instrument as the spawner and the seat's own `cotal_despawn` of the child was refused with `not authorized: was not spawned by (admin tier required)`. The break is in the CLI on the machine where the seats run. No stored state, credential or wire message changes. ### What keeps working `cotal spawn --detach` from an operator terminal or from a script outside any seat launches as before, and so does any call with `--creds`, one aimed at a space other than the seat's own, or a raw open target named with `--server` and an unregistered `--space`. A user-auth mesh is unchanged. A seat with `capabilities: [spawn]` still spawns from its shell, and can now stop that child with `cotal_despawn`. `--on ` from a seat's shell still lands on that manager instance, now as the seat. ### What stops working - On a static mesh, a seat without `capabilities: [spawn]` can no longer spawn from its shell. Its own credential holds no spawn subject, so the broker refuses the request and the command exits 1. - A child launched from a seat's shell is now that seat's child, so the manager stops it when the seat exits, as it does for a `cotal_spawn` child. A child that has to outlive the seat that started it now goes with the seat. - A seat launched without `COTAL_SPACE` is placed by its static credential. Every connector sets that variable, so this only reaches a hand-built launch: from such a seat's shell, a spawn aimed at a static space that holds no credential for the seat is refused instead of running as the operator. ### Upgrade order Only the CLI that seats run from their shell changes, which is the one installed on the host where the seats run. Brokers and managers need nothing for this break, so their order is the one the sections above give. ### The window This break has no outage. No process restarts for it. A child already running when you upgrade keeps the spawner the manager recorded at its launch. ### Snapshot this first Nothing is rewritten, so this break has no state to back up. List the agent files whose seats run `cotal spawn --detach` from their shell, note which of them lack `capabilities: [spawn]`, and note which of their children must outlive the seat. ### The upgrade end to end ```sh # still on 0.61.0: find the seats that spawn from their shell grep -rln 'cotal spawn' .cotal/agents # add `capabilities: [spawn]` to each of those agent files that lacks it, and launch any child # that must outlive its seat from an operator terminal instead npm i -g cotal-ai@0.62.0 ``` The attribution, the despawn, the refusal of a seat without `spawn`, the stop on seat exit and a seat's `--on` spawn were run on a local static mesh, and the attribution and the despawn on a local open mesh. ## From 0.53.0 to 0.54.0 Manager calls now borrow an instance-bound `manager-caller` credential. Followed mutations require `manager.goal-result` on the selected manager, so a compatible issuer, manager and client must be loaded together. An older manager is refused before a followed mutation; upgrading an installed binary alone does not replace code in a running manager, connector or embedded client. ### Preserve state before changing processes Snapshot the broker's durable storage using its supported backup procedure, the host authority and actor ledgers, and each participant's manager identity, runtime custody records, credentials and saved sessions. Include the embedding application's database and configuration under its supported backup procedure. Record the loaded package versions and the CLI path used by bearer helpers. Keep these copies private. Do not change the IdP issuer, regenerate manager identities, rotate agent credentials or recreate tenant storage to make the upgrade pass. No ledger, goal-history or session conversion is required for this change. Existing ordinary messaging credentials retain their normal expiry rules. New manager-caller credentials are obtained on demand from the current grant; old manager-call credentials do not gain the new view automatically. Existing accepted goals remain durable and must not be submitted again merely because observation was interrupted. Fresh remote registration publishes its service status at the current revision and epoch; do not seed that status manually. ### Upgrade the split deployment 1. Stage one pinned 0.54.0 package set for the host and participants, including the embedding SDKs. Pause new manager mutations and let accepted work settle where possible before reloading processes. 2. Upgrade the host issuer and embedding first. Keep the broker, its account identities and durable storage in place. Then load the matching manager release on each participating machine. 3. Preserve active seats through the runtime's supported update path. A Linux custodial runtime may release and re-adopt seats within its 600-second unattended window; verify the actual runtime, custody records and process identities before relying on it. A legacy PTY runtime without release support cannot preserve active seats through a generic manager restart. Drain it at an approved idle window instead of signalling the manager or replacing conversations. 4. Reload the clients and connectors through their session-preserving host controls. Refresh any bearer helper captured from an older immutable CLI path. A transport-only reconnect does not reload JavaScript. Verify authenticated instance selection, a read-only manager command and canonical result recovery before allowing new followed mutations. Treat the interval from issuer reload through compatible manager/client reload as a manager-control outage. Mixed versions can refuse discovery or commands; there is no promised rolling transition. Ordinary agent sessions survive only where their runtime and credentials permit it. If verification fails, keep mutations paused and repair forward from the preserved state rather than resetting it. This release does not add host-backed enrollment or terminal release for stock participant detached agents; see [Remote supervised agents](/docs/run-a-mesh/#remote-supervised-agents). ## From 0.48.2 to 0.49.0 0.49.0 changes how a credential's authority is recorded. A credential is no longer only a signed file: it is an *issuance*, with a generation the issuer chose and durable evidence of the ceiling it was granted under. The important consequence for a running deployment is not at connect time. It is at renewal time. ### What keeps working without any action - **Existing agent credentials keep authenticating.** A credential minted under 0.48.2 is not revoked and is not rejected at connect. Nothing needs to be re-issued to bring the fleet back up after the upgrade. - **The channel registry survives.** Channels, their replay settings, descriptions, and usage text are ordinary durable state and are not rewritten by the upgrade. - **`cotal deliver` is still a standalone command.** Running the delivery daemon as its own process remains supported; it is not restricted to being a child of `cotal up`. - **`cotal join` keeps its flags.** In particular `--lifecycle-uid` is not new in 0.49.0. It has been required alongside `--creds` since well before this release, and the pairing rule did not change here. A scripted external join that worked under 0.48.2 works unchanged. ### What does not migrate **A credential minted before 0.49.0 cannot be renewed.** Managed agent credentials carry a 24-hour lifetime and the manager re-signs one once it passes **75%** of its life, ticking every quarter of the TTL so a tick always lands inside that window. When the manager reaches a credential that carries no issuance, it refuses to renew it and logs the agent by name: ``` ! managed cred renewal : renewManagedStaticCred: carries no issuance; a static credential minted before SPEC 13.15 is not renewed under an unbound generation - respawn the agent - the agent dies loud at this cred's expiry unless it is reminted ``` So the fleet comes up fine, runs normally, and then each agent stops at its own credential's expiry, within roughly a day of the upgrade, one at a time rather than together. The refusal is deliberate: the renewal would otherwise have to invent a generation nobody issued, which is the state the release exists to remove. **Respawn the managed agents as the last step of the upgrade.** For this particular upgrade the respawn is not optional: stopping a 0.48.2 manager ends its agent processes whichever CLI you use, for the reason given under the outage window below. The respawn is how they come back, and it is also what mints each credential as an issuance so it renews from then on. One planned pass over the fleet is the whole job. Skipping it leaves agents stopped and, for any credential that survived into 0.49.0 unminted, brings the renewal cliff above a day later, one agent at a time. ### Credentials you minted yourself **A credential you minted with `cotal mint` is a different case, and it very likely needs nothing.** The distinction that matters is not the word "static", which covers both. It is **what minted the credential and who owns its renewal**. A credential the **manager** minted for an agent it spawned carries a lifetime and is renewed by the manager, so it is the subject of everything above. A credential **you** minted with `cotal mint` and handed to an external peer is issued with **no expiry at all**, and no manager renews it: it is not in the sweep, so there is no renewal to fail. It keeps working after the upgrade, and re-minting it would mean coordinating with a third party for no gain. The manager says which one it is holding. Where a credential has no expiry to reach, the sweep names it and moves on rather than refusing: ``` ! managed cred renewal : credential is unbounded - not renewed (a pre-TTL credential stays as minted until respawn) ``` Re-mint an external peer's credential only if you want it to carry a lifetime, and at a time you choose. ### How to read the boot log A 0.49.0 manager starting over an existing space may print lines like: ``` verified evicted: (3/12) already verified (durable): ✓ boot self-heal: manager/ registration gate reopened at generation ``` These are **not** a credential migration, and reading them as one is the most likely way to conclude the fleet is fine when it is not. They come from the manager repairing **one** endpoint registration gate that a previous restart left frozen, and they enumerate that single gate's credential-family holders as it verifies each one evicted. `already verified (durable)` on a later start is the repair cursor resuming, not a credential that became durable. The repair is real and useful (it is what previously needed `cotal reconcile-gate` by hand), but it says nothing about whether your agent credentials carry issuances. The renewal refusal above is the signal that does. ### Which side to upgrade first in a split topology Move the manager first. The stores 0.49.0 introduces are created by the **manager** at its own boot, not by the broker. They are create-or-verify and idempotent, so a 0.49.0 manager brings the space's authority stores up to the new shape itself, and it does so against whichever broker is answering. Being honest about the evidence behind each direction, because they are not equally established: - **Broker-first was measured on a live 30-agent deployment** (issue #1578). Upgrading the broker first locks the old manager out immediately: `cotal up` re-renders the broker's generated config from the trust record, and after the restart the still-0.48.2 manager is refused on every connection with an `authentication error` naming the Nkey, continuously. That text comes from the broker process, not from a Cotal command, so match on its shape rather than on an exact string. `cotal ps` reports zero agents while the agent processes are still alive, because the manager has lost its view of them, not because they died. Upgrading the manager clears it immediately. - **Manager-first is reasoned from where the new stores are provisioned**, not from a measured fleet upgrade. It is the recommended order because the manager is the component that creates what 0.49.0 adds, but it has not been run end to end on a production split topology at the time of writing. Treat it as the better-supported order rather than a guaranteed one, and keep the rollback below ready either way. Whichever order you pick, **this is not a rolling upgrade**. Between the two steps the mesh is down and the manager cannot see its agents. Go straight through rather than pausing between them, and schedule it as an outage window. ### What the window looks like - **The managed agent processes do not survive step 1, in either order.** This is the one place where the obvious reordering does not rescue you, so it is worth understanding rather than working around. Sparing agents on a bare manager stop is a **handshake**: a 0.49.0 manager publishes a capability file proving it can release its agents, and a 0.49.0 `cotal down` refuses the stop unless it finds one. **A 0.48.2 manager never publishes that file**, because the mechanism ships in the release you are installing. So the old CLI against the old manager sends a plain stop and takes every seat with it, and the new CLI against the old manager either refuses (leaving `--with-agents`, which reaps deliberately) or falls to the legacy path, warns that it cannot verify the manager can spare its agents, and signals it anyway. - **You can confirm which side you are on in one command, without stopping anything.** The flag that marks the newer behaviour is absent from the older CLI, and its summary line makes the difference plain: ``` $ cotal down --help # on 0.48.2 cotal down - stop the whole local stack, or name only the components to stop $ cotal down --help # on 0.49.0 cotal down - stop the whole local stack (managed agents stay running unless --with-agents), ... ``` If your `cotal down --help` does not mention `--with-agents`, stopping the manager stops the agents with it. - **Therefore the respawn in step 5 is mandatory recovery for this upgrade, not an optional pass.** It is also the step that re-mints credentials as issuances, so it is the same action either way. Plan the window to include it rather than treating it as cleanup. - The **manager's view** of them is lost while the two sides disagree, so `cotal ps` reports zero and control commands do not reach seats. - **Messages are not delivered** while the mesh is down. - The window is as long as it takes to restart the second component, plus the manager's own start. It is minutes, not hours, provided you do not stop between the steps. - **Nothing self-heals if you stop halfway.** The refusal is continuous until both sides match. ### Snapshot this before you start Take these while the deployment is still on 0.48.2. The two `cotal` reads are live reads and must happen before anything stops. - **A filesystem or volume snapshot of both containers**, if your platform offers one. This is the only rollback that covers every case, and it is what the reporting deployment used. - **`cotal backup create `**, for the durable space state, **but read the next paragraph before you rely on it**: on a split broker and manager topology it is very likely unavailable to you, and the volume snapshot above is your actual rollback. - **The trust records and credential directory** under `.cotal/auth` on the manager host, including the per-space material directory. These are what a re-mint would otherwise have to replace. - **A copy of the channel registry**, so you can verify it came back rather than assuming it did: `cotal channels list` before and after. - **The output of `cotal ps`**, so you know how many seats you expect to see afterwards and can tell a lost view from a lost agent. #### `cotal backup` on a split topology **`cotal backup create` cannot read a running stack.** It requires a completed cut, and only `cotal down --preserve-state` publishes one: ``` $ cotal backup create ./backup.0482 ✗ backup requires a completed cut; run `cotal down --preserve-state` first ``` **And `cotal down --preserve-state` requires a manager alive on the host you run it from.** It uses that manager to attest that every retained child stopped, and the check is deliberately fail-closed: a manager that is dead or merely uncertain refuses rather than preserving an unproven cut. The check reads a local pidfile, so a **remote** manager does not satisfy it. On a split topology the broker host has no local manager, which means the documented durable-backup path is not available there. **Measured rather than assumed, at 0.48.2**: the backup refusal above is executed output. The preservation requirement is read from `down.ts` at the same tag, where the preserve path asks a manager to prepare an inventory and then requires that manager to be locally alive before it commits. The part not executed end to end is a genuine two-host split, which needs two real hosts. **What to do instead.** Use the filesystem or volume snapshot of both containers. That is the rollback the reporting deployment actually used, it covers the broker's durable state and the manager's credential material together, and it does not depend on either component being able to attest for the other. If you want `cotal backup` as well, take it from a host that does have a live local manager, and understand it is a second copy rather than the primary rollback. **This looks like a product limitation rather than a documentation gap**, and it is written here as one so an operator is not left thinking they mis-typed a command. The upgrade path for the exact topology this page is addressed to cannot use the documented backup command. ### The upgrade end to end ```bash # 0. on 0.48.2, STILL RUNNING: record what you expect to see afterwards. # These two are live reads, so they must happen before anything stops. cotal channels list > channels.before cotal ps > ps.before # 1. manager host. READ THE NOTE BELOW THE BLOCK FIRST: this step ends the # managed agent processes whichever order you choose, and the respawn in # step 5 is how they come back. It is recovery, not tidying. # # STOP THE MANAGER WITH THE 0.48.2 CLI, BEFORE INSTALLING 0.49.0. The # order matters and it is not recoverable once you install: a 0.49.0 # `down manager` REFUSES to stop a 0.48.2 manager whose pid record carries # a start token, which is every manager on a platform that can read one # (Linux can): # refusing bare manager stop: ... does not prove this manager can detach # its agents; use --with-agents or stop the agents explicitly # The refusal names two remedies and NEITHER clears it for this case. The # check reads a capability file that only a 0.49.0 manager writes; it never # counts agents, so stopping them first changes nothing. And `--with-agents` # is whole-stack only, so `down manager --with-agents` is refused by its own # flag rule. See #1592. cotal down manager # the 0.48.2 CLI, still installed. # 0.48.2 has no --with-agents; this # is the whole route. On a host that # runs the whole stack, the 0.49.0 # `cotal down --with-agents` after # installing is the alternative. npm install -g cotal-ai@0.49.0 # ONLY after the stop above # `supervise` RUNS IN THE FOREGROUND and holds the terminal until you stop # it. There is no --detach on this command. Start it under whatever keeps # your manager alive normally (systemd unit, container entrypoint, or a # second terminal), and run the remaining steps from another shell. cotal supervise --space --server nats://:4222 # 2. broker host: stop the stack. # NOT `--preserve-state` on a split topology: it needs a manager alive on # THIS host to attest its children stopped, and yours is on the other one. # Your rollback is the volume snapshot from "Snapshot this before you # start", not `cotal backup`. # See "cotal backup on a split topology" above. cotal down # 3. broker host: install 0.49.0 and start it again npm install -g cotal-ai@0.49.0 # Record the manager log's size BEFORE starting, so step 3a can tell THIS # boot's output from every earlier one. It must be captured here, ahead of # the start: taken afterwards it sits past the new line and the wait hangs. # `` is NOT the space name. It is lowercase hex of the name's # UTF-8 bytes, so space `prod` is `manager.70726f64.log`. Do not guess it: # `cotal up` prints the real path on its launch line. Substituting the # plain name points at a file that does not exist, and the wait below then # burns its full timeout before telling you. LOG=.cotal/manager..log OFF=$( [ -f "$LOG" ] && wc -c < "$LOG" || echo 0 ) cotal up --detach --host 0.0.0.0 --space --no-manager # 3a. SPLIT TOPOLOGY ONLY: `--no-manager` above boots the broker (and the # delivery daemon) with NO local manager on the broker host, so there is # no wait-and-stop step on a current cotal-ai. The rest of this step is # the OLDER-host recipe, kept because the flag is refused there and that # refusal is your signal you are on it: without the flag the `up` also # starts a local manager, and you must wait for the log to show it is up, # then stop it, or you finish the upgrade with two managers and the one # you did not intend is the one nobody is watching. # A bare `grep -q` does NOT wait: it reads once and exits 1 immediately # if the line has not been written yet. Bound the wait instead, so a # manager that never comes up fails loudly rather than reading as ready. # The log is opened APPEND-ONLY, so on any host that has run a manager # before, this file ALREADY carries a `manager up` line from an earlier # boot. Grepping the whole file therefore matches instantly and waits for # nothing. Read only what THIS boot appended, using the $OFF captured in # step 3 above (before the start, which is the only point it is correct): timeout 60 bash -c \ "until tail -c +$((OFF+1)) \"$LOG\" | grep -q '. manager up'; do sleep 1; done" # exit 0 = THIS boot logged it; exit 124 = it never did, so STOP and look. # This manager is 0.49.0 and publishes its own spare-capability file, so # the bare stop below is NOT the refusal case from step 1. cotal down manager # broker + delivery remain # On a current cotal-ai the two commands above are unnecessary (nothing # to wait for, nothing to stop) and `cotal down manager` simply reports # no manager to stop. # 4. verify the mesh is whole again before touching the fleet. # Do NOT compare `cotal ps` against ps.before yet: step 1 ended the agent # processes, so at this point it is EXPECTED to be empty, and an empty # `ps` is also the signature of the broker/manager mismatch described # above. The two are indistinguishable here, so compare what the mesh # itself should have carried across instead: cotal channels list # compare against channels.before: this SHOULD match now cotal ps # expect it to be EMPTY here; ps.before is the target for # step 5, not for this step # 5. the step that is easy to skip: respawn the managed agents so their # credentials are re-minted as issuances and can renew. Persona is a # POSITIONAL argument here, unlike `cotal stop`, which requires --name. # One call per agent: cotal spawn --detach --name --space # then the comparison step 4 could not make: cotal ps # NOW compare against ps.before: seat count should match ``` The mesh is down from step 2 until step 3 finishes. That is the window. On a split topology there is no cut and no backup inside it, so the window is the stop, the install and the restart, nothing more. ## Adding a section for a future release **Every changeset marked breaking adds a section to this page.** A release that changes what an operator must do, in what order, or what stops working, is not finished until the section exists. `scripts/upgrade-section-gate.mjs` grades a commit range for this: run it as `pnpm upgrade-section-gate --base ` and it reds when the range carries a breaking change and adds no new release section. The former Attribution workflow enforced this on each pull request; that workflow and its required-status rule were removed on 22 October 2026. The script remains available for manual checks, and reviewers still check upgrade instructions for breaking changes. Be precise about what the check proves. It proves a section for a release **was written here**. It cannot prove the section is **correct**, or that it describes the break that actually landed, and it cannot see a breaking change that carries no marker at all. Reviewing the words remains a person's job. **Mark the break, or the gate cannot see it.** Any one of these is enough, and they are the only things it reads: - a `!` before the colon in the commit subject, as in `feat(core)!: bind hosted runs to the caller` - a `BREAKING CHANGE:` footer in the commit body - a changeset in `.changeset/` declaring a `major` bump for any package The marker must survive the squash. A `!` that lives only in a commit you squash away is not in the range the gate grades, so put it in the subject that lands on `main`. **The heading is a `##` and names the release**, like `## From 0.48.2 to 0.49.0`. Both matter, and neither is a style preference. Coverage is claimed by a heading, so a heading that names no release claims every release and distinguishes none: `## Notes` with a sentence under it would otherwise satisfy the rule. Naming the release also makes the section the one an operator upgrading that release will search for. Use `###` freely for detail inside a section. Subsections belong to their release rather than counting as separate coverage. Name the release that first carries the change: the next version Changesets publishes, which `pnpm changeset status --verbose` lists. `bin/package.json` on `main` still reads the release already published. If a release is cut while the change is open, the change ships in the release after it, so move the heading before merging. The gate accepts any version in a heading, so before merging a release pull request, check every heading added since the previous tag against the version it publishes. A section is written for the operator, not for the reviewer. It answers, in this order: 1. What keeps working with no action at all. 2. What does **not** migrate, and when that becomes visible. Name the log line if there is one. 3. The order to move components in for a split topology, and why that order. 4. What the outage window looks like, including what survives it. 5. What to snapshot before starting. 6. The commands, end to end. **Where an answer was not measured, say so in the document rather than guessing.** An operator who knows which half of a recommendation is reasoned and which is measured can plan around it; one who finds out afterwards cannot. ## Handle redemption in 0.91.0 `auth redeem ` selects the native auth target, then refuses unsupported conferring-handle redemption without reading the file or issuing credentials. It does not redirect the artifact into agent enrollment or interactive-session redemption. Offline handle inspection still labels signatures, revocation and holder currency not checked. An explicit `--space` requests static-registry signature inspection with a native ephemeral operator reader. It checks signatures, issuer scope, presented links and required revocation records without establishing presenter identity, live epochs or use authority. It confers no holder credential, persists no reader credential, and never turns missing or stale native records into verified offline success. Open/user-auth inspection and conferring redemption remain unsupported. ## Dashboard service transition in 0.91.0 The matching dashboard package now registers a service preset and hidden native child, not a public web command. An older cached command descriptor is hidden and refuses with the matching extension-update remedy before importing provider code or repairing the install. `update` remains explicit; `service start web` never installs a deliberately absent extension. Configure the saved web definition with its existing registered static-auth root, one protected credential file and explicit read-only intent. The service retains required TLS, creates no caller or channel-purger credential, opens no browser and prints no bootstrap secret into logs. The page marks it read-only and hides delete; the server independently rejects that mutation before any alternate connection. The private session file holds the full-launch bootstrap; logs give its reference only. User-auth/open targets and OS installation remain unsupported in this route. Existing legacy process records are still observed, never silently adopted. An already-running legacy dashboard is not restarted or converted by loading metadata. See [dashboard setup](/docs/watch-a-mesh/#web-dashboard). ## Registered lifecycle retirement in 0.91.0 `auth retire [--yes]` has closed UID/space input, but the standalone CLI still lacks a supported native retirement instrument carrying its own current serve identity and gate. It refuses without publishing a request. It does not borrow the manager's identity, treat --yes as a grant, or call the host's interactive/managed loopback doors as a substitute. This remains an explicit authority gate, not a completed retirement operation. ## Local-process `rootedAt` removal in 0.91.0 A local-process extension descriptor no longer has a `rootedAt` field. No stop has read it since the dashboard became a service. Operators have nothing to migrate, snapshot or order, and there is no outage: a manifest that an older install cached with the field still loads, and the next manifest rewrite drops it. An extension author removes the field from each `LocalProcess` declaration, where TypeScript now reports it as an unknown property. ## Earlier section links These links lead to the current guidance for sections renamed in earlier versions. ### Auth context closure in 0.71.0 (unreleased) [#auth-context-closure-in-0710-unreleased] See [Auth context closure in 0.72.0](#auth-context-closure-in-0720). ### Unreleased migrations [#unreleased] See [The pre-1.0 upgrade contract](#the-pre-10-upgrade-contract). ### Hosted auth plane identity in the store (unreleased) [#hosted-auth-plane-identity-in-the-store-unreleased] See [Hosted auth plane identity in 0.76.0](#hosted-auth-plane-identity-in-the-store-in-0760). ### Manager-service authority policy flag in 0.75.0 [#manager-service-authority-policy-flag-in-0750] See [Manager-service authority policy flag in 0.76.0](#manager-service-authority-policy-flag-in-0760).