# Manager maintenance `manager` provides retained-seat inspection and guarded issuance-gate repair. ## `manager reconcile` ```bash cotal manager reconcile [--space ] [--server ] [--endpoint ] [--instance ] [--yes|--dry-run] [--json|--plain] ``` | Flag | Default | Meaning | |---|---|---| | `--space ` | this folder's auth space | Space the frozen gate lives in | | `--server ` | the local mesh | Broker URL | | `--endpoint ` | `manager` | Endpoint whose gate is frozen | | `--instance ` | this folder's persisted manager instance | Instance id | | `--yes` | ask and default to cancellation | Confirm this selected gate repair outside a terminal | | `--dry-run` | off | Resolve the local coordinate without minting or checking native guards | | `--json`, `--plain` | terminal report | Finite structured receipt or plain shared report | ### Frozen gate recovery **When you need this.** A manager restart killed after deregistration begins but before the new incarnation finishes leaves the endpoint's issuance gate *frozen*, held by a process that no longer exists. The freeze is what stops two incarnations serving at once, which is correct. The successor manager now completes that dead registration itself on boot, including on the remote user-auth path. A foreign remote manager blocked by this gate also asks the host to repair it before one registration retry. Both use the same guard this command uses: they act only when the freeze-holder is affirmatively gone under a complete CONNZ sweep (`gone` and `sweepComplete=true`). If that registration's spec write already committed, it finishes the same freeze at the committed registration revision. If the spec did not advance, it abort-reopens the gate at generation+1 with processEpoch unchanged and continues the normal takeover. Live, unknown, unestablishable, and wrong-op-kind still refuse; there is no TTL. Use this command when the automatic path cannot run: the delivery daemon is down, the repair targets a non-manager endpoint, or you want to lift the freeze without starting a manager. It checks that the holder really is gone, then finishes the dead operation the same way as the interrupted restart would have: revoke the old credentials, evict their holders with verification, and reopen the gate. The command revokes the old credentials 16 at a time. It then verifies the holders' eviction in shared sweeps of up to 256 holders on the delivery daemon. Each sweep scans the broker a fixed number of times and kicks live connections 16 at a time, so holders that are already gone add almost nothing and live ones add one broker round trip per 16 connections. The daemon must serve the `evictPrincipals` verb; an older daemon refuses it and the gate stays frozen. Each sweep durably records the holders it verified before the next sweep starts. If a holder is not verified gone, the command leaves the gate frozen with those records kept. An interrupted sweep records nothing, and the sweeps before it stay recorded. A retry still repeats the freeze-holder liveness check, then skips only progress bound to the same registration operation, frozen-gate revision, and holder set. The output reports holders completed before this attempt, completed now, and still remaining. A new freeze or changed holder set starts from zero. Cursor cleanup happens only after reopen; a retained cursor is harmless because its old gate revision cannot authorize a later freeze. The governance slot release after the reopen is cleanup in the same way: a slot it fails to release does not fail the repair, because the reopened gate is already past the slot's generation. **It refuses far more often than it acts, on purpose**, and always says which check stopped it: | Refusal | What it means | What to do | |---|---|---| | `holder-alive` | The freeze-holder still has a live connection: a manager *is* running | Stop that process first. Reconciling would evict a live manager's credentials | | `holder-unknown` | The connection sweep could not prove the holder absent | Not safe to proceed: an unprovable holder is treated as a live one. Re-run once the broker answers completely | | `liveness-unestablishable` | The delivery daemon gave no verdict: it was unreachable, timed out, or refused | Act on the delivery lease line in the refusal (below). Silence is never read as death | | `not-frozen` / `no-gate` | The gate is open, or there is no gate at that coordinate | Nothing to repair: check `--endpoint` / `--instance` | | `wrong-op-kind` | Frozen under a takeover or retirement, not a registration | Out of scope for this command; it will not reinterpret another operation's intent | | `eviction-unverified` | The holder looked gone but eviction could not be verified | The gate is left frozen, unchanged. Investigate the broker before retrying | | `raced` | A newer manager moved the gate mid-repair, and nothing was reopened | Run the command again, which reads the gate afresh | ### Delivery lease observations When the daemon gives no verdict, the refusal also reads the delivery lease (`lease.0`) and names what is blocking the rail: | Lease reading | What to do | |---|---| | absent | No daemon is running. Start it (`cotal up` runs it) and re-run | | unreadable | The daemon cannot be named, so do not assume none is running. Fix the lease read, then re-run | | held, not ready | That holder claimed the shard and has not bound its rails. Wait for it, or stop it so its lease lapses | | held, ready, no answer | The query may have gone to another daemon still subscribed to the rail, such as a stopped one whose lease lapsed. Re-run before stopping anything. If no run gets an answer, stop any other delivery daemon for the space, then stop or restart the holder | | changed hands | The holder took the shard after the query was sent, so it was never asked. Re-run before stopping anything | The command reads the lease before it sends the query and again after the query fails. It names a holder as the blocker only when the same run of the same daemon held the lease both times, and two rows from a daemon too old to record its run never count as the same run. Even then a ready holder may not have been asked: the rail is queue-grouped, so any daemon still subscribed to it can take the query. A row whose times are not valid dates reads as unreadable. A daemon that answered and refused keeps its own reason, followed by the same lease line. The lease line names the holder, whether it is ready, the space account that holds the lease bucket, when that holder acquired the shard, and when the row was last written. A ready holder rewrites the row on every renewal and keeps its acquisition time, which only a successful acquisition sets. A row written by a daemon that predates the acquisition time reports it as unknown. The lease reads never change the outcome: the gate stays frozen and the command exits 2. A manager's boot self-heal uses the same check and reports the same line. `raced` means the repair's own reopen lost its compare-and-set and wrote nothing. A reopen whose outcome is unknown, such as one whose acknowledgement was lost, fails with an `unavailable` error that says it may have committed. Re-run the command: it refuses `not-frozen` if that reopen landed and finishes the repair if it did not. There is no `--force`, and no path that discards gate state: the only way this reopens a gate is by proving the holder is gone and then completing the operation properly. ### Governance slot after repair **What reopening the gate does for the endpoint's governance slot.** A registration takes the endpoint-wide governance slot before it publishes its spec, and holds it until its gate reopens. An instance that died between those two points leaves the slot held with no registration behind it. This command does not write that slot and never has; the registration path is its only writer. What the reopen does is advance the holder's gate past the generation the slot is stamped with, which is what marks the slot abandoned. The next registration for that endpoint then reclaims it as part of its ordinary start. So the repair here is still one command followed by starting the manager, and the slot needs no separate step. ## Manager maintenance `cotal manager`, also `cotal mgr`, shows its two scoped operations. The group and its operations support `--json` and `--plain` through the shared reports. Neither operation installs connectors or starts a manager. `manager seats` lists retained local PTY custody records. `--drain --dry-run` reads the census without submitting a drain. A real drain reviews the local store and defaults to cancellation; scripts require `--drain --yes`. Its scope is all safely childless records in that store when the native pass runs, not a frozen list from the review. Each native reap rechecks process creation identity. Live children stay. An unresolved preflight blocks draining; a partial read returns exit 4. An unconfirmed effect during a submitted drain returns exit 3 and keeps confirmed removals in the result. Raw native diagnostic bodies are omitted. `manager reconcile` uses the existing guarded issuance-gate repair from the owning local mesh root. It resolves the registered broker without an alternate-target fallback and preserves its TLS requirement across repair, liveness and eviction connections. It requires that root's auth material and verified holder liveness. Review defaults to cancellation; scripts require `--yes`. `--dry-run` checks only the local coordinate and issues no credential or native guard query. A pre-effect native guard refusal returns exit 2. Its error names the guard's condition and keeps the guard's own sentence as its native cause, which names the freeze-holder and the next step. A failure after the repair may have submitted work returns exit 3, without replaying mutations or hiding possible revocation/eviction progress. Re-observe that gate and its operation-bound durable cursor before another explicit repair. The final receipt reports the native operation and before/after coordinates, not a running-manager verdict. A reopened gate with retained cursor cleanup returns exit 4. ## Manager stop impact The native manager status contract includes its selected `lifecycleUid` and a caller-scoped `stopImpact` observation. It lists visible managed principals and lifecycle IDs, their runtime modes and the seat actions a default stop or `--with-agents` would request. In-process PTYs without a release hook stop by default; releasable seats lose manager custody. Preservation or pending resume uses the native retained-state policy instead. None of these labels confirms seat exit, authority retirement or cleanup. From manager cluster revision 33, a manager that is not `active` and holds a preservation inventory also reports `preservation`: the attempt and the inventory digest a preserved-exit stop names. A preserved-exit stop request carries that binding with `withAgents: false`; the manager accepts it only when it is fully preserved, whole-manager and quiescent, and its result is `preserved-exit-returned`. The open-space local receipt keeps its revision-32 shape. The manager resolves visibility through the same fresh owner/admin policy as `ps`. An `owner-filtered` observation is not the full manager impact, even if its rows happen to match a count. A refused stop reports the selected manager, visibility, maintenance state and pending-work counts that blocked its preflight, without exposing other owners' agent rows. Owner-filtered visibility requires the stack operator or an authorized admin-scoped caller; the command never substitutes one. On a user-auth space, the refusal's next step is the whole-row `cotal actor grant` that adds `admin` to your current grant, as the auth plane prints it from the ledger row. A stop in the folder that hosts the space reads the manager's own local receipt instead, so its owner stops the whole manager without that scope or a sign-in. No stop policy is accepted while any blocker remains. This is an observation at the reported time, not a reservation, permission to stop or a guarantee that the set cannot change. Older managers can omit it; the native service adapter refuses when the required process/lifecycle correlation is absent. Generic process launch/stop still refuses managers. ### Native manager service definition The `manager` service preset uses the `manager-service` adapter. It requires the selected registered `root` and an explicit installed `runtime`; an optional `server` can repeat the registered broker. The optional `console-port`, `console-host`, `ws-port` and `max-sessions` pass explicit native settings, and an optional `name` names the manager as `supervise --name` does. Omitting the host keeps the native loopback binding; port zero requests an allocated console port. A declared `launch` also requires `launch-sha256`. It names an existing private resolved launch spec, pins its full SHA-256 and must name this space. The child parses those verified bytes before authority preparation and keeps that snapshot through startup. An optional `roster` names an existing [roster file](/docs/define-a-team/#roster-files) by absolute path. Only its owner or root may write it or its directory. The child parses it before startup, and each launch boots the agents it lists the way `supervise --roster` does. A resumed manager takes no roster. Saving these options does not prepare a manifest, provision an agent or prove that requested agents joined. Saving it starts nothing and grants no authority. Its native child starts no broker or delivery daemon, does not adopt an existing legacy manager, and never prints its private console URL. It declares one instance per definition and runs in the background or with `start --foreground`. It is not OS-supervised installation or platform parity. A root that `space add --root` registered has no `.cotal` until a command writes there. A start makes it, as a directory private to you, after its other checks pass and once the root itself is protected against other local writers. It does not replace a `.cotal` that is already there, such as a link, which the child refuses. ### Manager service startup proof The real child exclusively claims its creation-pinned native PID custody before hosted authority preparation or local endpoint initialization. That starting record is not readiness; another existing slot is not adopted or reclaimed. After a crash, the record the killed child left names a dead pid. The manager adapter removes that record before the next launch, so the next `up` relaunches the manager; a live, foreign or unreadable record still refuses. It binds its actual native identity before endpoint construction. A startup phase appears only after native start, settled custody and signal handlers. On authenticated meshes the adapter also requires a current status response pinned to that native instance and epoch, with the same process lifecycle UID. It preserves the invoking managed-seat caller, registered root/broker and required TLS. Missing caller support refuses without an operator substitution. The completed status read's own admission is excluded from stop impact; other admitted work remains counted. An open mesh uses the pinned child's private request-bound local observation, and so does a stop in the folder that hosts a user-auth space. Its service, launch, request and both process creation identities must match, and its registration must match the launch identity. Output labels this a child-local receipt, not a broker-attributed endpoint response. A retained stop handoff blocks readiness adoption on either route instead of treating a live PID as a new serving launch. ### Manager service stop policy A real `service stop` or `restart` reads complete caller-visible impact before confirmation. `--with-agents` requests native agent stops, not generic descendant signalling. An interactive confirmation defaults to cancel and shows every affected identity, lifecycle and requested seat action. Scripts require `--yes`. The offline `--dry-run` still imports no provider and performs no native impact query; it is not stop authorization. The native stop reserves its original PID custody. It reclaims an existing reservation only when the pid it names has exited and no request was published for this launch, and otherwise refuses and names that pid. It rechecks the confirmed scope, binds the full service/launch/stop request and both process creation identities, and uses the existing native shutdown intent for agent reap. The child rechecks the same complete scope before accepting. Changed or incomplete scope refuses without a broader policy. The controller requires immutable accepted/returned evidence and the original process exit before recording a native stopped state. An unconfirmed request retains its reservation and never escalates to force kill, restarts, or treats PID disappearance as completion. A later bare `up` or `down` settles it only from the same evidence, and only then releases the reservation. Reported cleanup remains not checked. Retained native-policy markers prevent a metadata change from selecting generic signals, and an unconfirmed native stop cannot be forgotten just because its process disappeared. Legacy shutdown refuses the native child rather than bypassing that boundary. ## manager ```bash cotal manager start [--runtime ] [--roster ] [--name ] [--foreground|--dry-run] [--space ] cotal manager stop [--with-agents] [--yes|--dry-run] [--space ] cotal manager restart [--with-agents] [--yes|--dry-run] [--space ] cotal manager status [--space ] cotal manager logs [--tail ] [--follow] [--space ] cotal manager install [--dry-run] [--space ] cotal manager uninstall [--yes|--dry-run] [--space ] cotal manager reconcile ... cotal manager seats [--drain] ``` The manager starts and keeps your agents running. `cotal up` starts one with the rest of the stack; `cotal manager start` starts one on its own, for this folder's space or the one `cotal use` selected, with the `pty` runtime and the broker the space is registered at. Nothing has to be added first, and no other service is started for it. `mgr` is an alias. `cotal manager ` is `cotal service manager`: every [`service`](/api-reference/cli-services/#service) rule applies, and its flags and the manager preset's options (`--runtime`, `--roster`, `--name`, `--server`, `--console-port`, `--console-host`, `--ws-port`, `--max-sessions`) are taken on the line. `cotal help manager ` and `cotal manager --help` print the verb's help with those options. The first start saves the space's `manager` service from the preset's defaults and says so on stderr; `cotal service show manager` reads it. A start that names another `--runtime` or other option for a stopped manager saves it again with that option, and says so. A running one refuses `LIFECYCLE_BLOCKED` and names `cotal manager stop` first. On a user-auth space this folder hosts, `start` and `restart` need your signed-in, granted login, as `cotal up` does for its manager. On a user-auth space registered here from another host, they need your sign-in. Before anything is saved, stopped or launched, they confirm the sign-in with its IdP. Without one, or when the IdP rejects it because it expired or was revoked, they refuse `AUTH_REQUIRED` (exit 2), and Next names `cotal login --idp `, then the command again. A refused `restart` leaves a running manager as it was and says the running manager was left running. Any other failure of that check, such as an IdP that does not answer, is left to the start. In a folder that hosts the space, `stop` and `cotal down` need no sign-in: the broker, the records and the process belong to this folder, so the stop reads the manager's own local receipt. The manager these verbs act on is the space's one manager service: `manager` when `cotal manager start` saved it, or `stack-manager` when `cotal up` did. Stderr names the saved service when it is not `manager`. So after `cotal up`, `cotal manager status` and `cotal manager stop` act on the stack's manager, and `cotal manager start` reports it running and starts no second one. Two saved manager services in one space refuse `TARGET_AMBIGUOUS`, naming each with its `cotal service` command. A manager that `cotal up` or `cotal supervise` started without a saved service is reported by `start` and `status` with its PID, and starts nothing beside it. `logs` reads the log `cotal up` opened for it, `/.cotal/manager..log`, and changes nothing. It takes `--tail`, `--follow` and `--json` as a service's `logs` does, and a follow ends when that manager's process is gone. After `cotal down` stops that manager, `logs` still reads the log it left: the text says the manager is stopped, with its PID while its record still names one, and names `cotal up` under Next, and `--json` carries `recordedProcess` with `state` set to `stopped`. With no such log, `logs` refuses `TARGET_NOT_FOUND` and names the file. If the folder's records show a manager ran there (a PID record that names a dead manager, or the instance identity a manager keeps once it starts), the refusal says the log is no longer there and names only `cotal up`. Otherwise it says no manager has been started and names `cotal up` or `cotal manager start`. `cotal service logs stack-manager` reads the same file when no `stack-manager` is saved. A manager that `cotal supervise` runs in a terminal writes to that terminal instead. `stop`, `install` and the other verbs refuse it while it runs, because no saved service owns it; it stops with its stack through `cotal down`, or where `cotal supervise` runs it. `--foreground` keeps the manager in this terminal until Ctrl-C, which stops it through its native stop. A unit or container you write that runs a manager uses `cotal manager start --foreground`. `cotal manager install` refuses `UNSUPPORTED` in this release, because a Cotal-written unit runs only a direct-process service. It names the `ExecStart=` line for a unit of your own instead, whether or not a manager was started, with one exception: while a manager that `cotal up` started with no saved service runs, `install` refuses it as `stop` does and names `cotal down`. A manager that fails while it starts writes one last line that names the cause: `the manager did not start`, then the failure's code and message, the error it wraps and its one next step after `; next:`. `start` fails `OPERATION_FAILED`, exit 5, and quotes that line, whole up to 1000 characters. A longer line is quoted by its start and its `; next:` step, with its length where the middle was cut. `cotal service logs manager --instance ` shows the whole line. When the error ends in a certificate this machine does not trust, such as `UNABLE_TO_VERIFY_LEAF_SIGNATURE`, Node's message stays as the cause, and the next step is to set `NODE_EXTRA_CA_CERTS` to the CA file, or the self-signed certificate, for the command that starts the manager, as the `--tls-cert` row of [up](/api-reference/cli-startup/#up) says. The manager forwards that CA-file path to the seats it launches without a `spawn.env` entry. With `--foreground` the line goes to this terminal on stderr, wrapped at its width as the CLI's own lines are, after the `Mesh:` heading and lifetime notice that `start` prints there first, and the refusal says `its output went to this terminal and was not captured in its log` and names no log. `manager start` and `manager restart` run the first-run connector seed, as `cotal supervise` does, so a fresh machine or container has its connectors before the manager boots; `status`, `logs`, `stop` and the maintenance verbs install nothing. `reconcile` and `seats` are the [manager maintenance](#manager-maintenance) commands. Bare `cotal manager` lists every verb and starts nothing. ## Start a manager for a hosted space A hosted space runs its broker elsewhere, so nothing on your machine starts a manager for it. Start one: ```bash cotal manager start --space --runtime ``` It needs you signed in to the space (`cotal login`) and the space registered on this machine (`cotal meshes add`). A sign-in the IdP rejects because it expired or was revoked counts as none: the start refuses `AUTH_REQUIRED` before it launches anything, as [manager](#manager) says. With the space selected by `cotal use `, a bare `cotal manager start` is enough: it uses the `pty` runtime, which every platform has built in. `orca` and `tmux` need their extension first, such as `cotal ext add @cotal-ai/orca`. The command returns once the manager answers; `--foreground` keeps it in this terminal instead. `cotal ps` then lists it with the agents it runs, and `cotal spawn -d ` launches an agent through it. ### Gate repair refusals See the [current command contract](/api-reference/cli-maintenance/#reconcile-gate) for this behavior.