Manager maintenance

manager provides retained-seat inspection and guarded issuance-gate repair.

manager reconcileCopy Anchor Link

cotal manager reconcile [--space <s>] [--server <url>] [--endpoint <e>] [--instance <id>] [--yes|--dry-run] [--json|--plain]

Frozen gate recoveryCopy Anchor Link

When you need this. A manager restart killed after deregistration begins but before the new incarnation finishes leaves the endpoint's issuance gate frozen, held by a process that no longer exists. The freeze is what stops two incarnations serving at once, which is correct. The successor manager now completes that dead registration itself on boot, including on the remote user-auth path. A foreign remote manager blocked by this gate also asks the host to repair it before one registration retry. Both use the same guard this command uses: they act only when the freeze-holder is affirmatively gone under a complete CONNZ sweep (gone and sweepComplete=true). If that registration's spec write already committed, it finishes the same freeze at the committed registration revision. If the spec did not advance, it abort-reopens the gate at generation+1 with processEpoch unchanged and continues the normal takeover. Live, unknown, unestablishable, and wrong-op-kind still refuse; there is no TTL.

Use this command when the automatic path cannot run: the delivery daemon is down, the repair targets a non-manager endpoint, or you want to lift the freeze without starting a manager. It checks that the holder really is gone, then finishes the dead operation the same way as the interrupted restart would have: revoke the old credentials, evict their holders with verification, and reopen the gate.

The command revokes the old credentials 16 at a time. It then verifies the holders' eviction in shared sweeps of up to 256 holders on the delivery daemon. Each sweep scans the broker a fixed number of times and kicks live connections 16 at a time, so holders that are already gone add almost nothing and live ones add one broker round trip per 16 connections. The daemon must serve the evictPrincipals verb; an older daemon refuses it and the gate stays frozen.

Each sweep durably records the holders it verified before the next sweep starts. If a holder is not verified gone, the command leaves the gate frozen with those records kept. An interrupted sweep records nothing, and the sweeps before it stay recorded. A retry still repeats the freeze-holder liveness check, then skips only progress bound to the same registration operation, frozen-gate revision, and holder set. The output reports holders completed before this attempt, completed now, and still remaining. A new freeze or changed holder set starts from zero. Cursor cleanup happens only after reopen; a retained cursor is harmless because its old gate revision cannot authorize a later freeze. The governance slot release after the reopen is cleanup in the same way: a slot it fails to release does not fail the repair, because the reopened gate is already past the slot's generation.

It refuses far more often than it acts, on purpose, and always says which check stopped it:

Delivery lease observationsCopy Anchor Link

When the daemon gives no verdict, the refusal also reads the delivery lease (lease.0) and names what is blocking the rail:

The command reads the lease before it sends the query and again after the query fails. It names a holder as the blocker only when the same run of the same daemon held the lease both times, and two rows from a daemon too old to record its run never count as the same run. Even then a ready holder may not have been asked: the rail is queue-grouped, so any daemon still subscribed to it can take the query. A row whose times are not valid dates reads as unreadable.

A daemon that answered and refused keeps its own reason, followed by the same lease line. The lease line names the holder, whether it is ready, the space account that holds the lease bucket, when that holder acquired the shard, and when the row was last written. A ready holder rewrites the row on every renewal and keeps its acquisition time, which only a successful acquisition sets. A row written by a daemon that predates the acquisition time reports it as unknown. The lease reads never change the outcome: the gate stays frozen and the command exits 2. A manager's boot self-heal uses the same check and reports the same line.

raced means the repair's own reopen lost its compare-and-set and wrote nothing. A reopen whose outcome is unknown, such as one whose acknowledgement was lost, fails with an unavailable error that says it may have committed. Re-run the command: it refuses not-frozen if that reopen landed and finishes the repair if it did not.

There is no --force, and no path that discards gate state: the only way this reopens a gate is by proving the holder is gone and then completing the operation properly.

Governance slot after repairCopy Anchor Link

What reopening the gate does for the endpoint's governance slot. A registration takes the endpoint-wide governance slot before it publishes its spec, and holds it until its gate reopens. An instance that died between those two points leaves the slot held with no registration behind it. This command does not write that slot and never has; the registration path is its only writer. What the reopen does is advance the holder's gate past the generation the slot is stamped with, which is what marks the slot abandoned. The next registration for that endpoint then reclaims it as part of its ordinary start. So the repair here is still one command followed by starting the manager, and the slot needs no separate step.

Manager maintenanceCopy Anchor Link

cotal manager, also cotal mgr, shows its two scoped operations. The group and its operations support --json and --plain through the shared reports. Neither operation installs connectors or starts a manager.

manager seats lists retained local PTY custody records. --drain --dry-run reads the census without submitting a drain. A real drain reviews the local store and defaults to cancellation; scripts require --drain --yes. Its scope is all safely childless records in that store when the native pass runs, not a frozen list from the review. Each native reap rechecks process creation identity. Live children stay. An unresolved preflight blocks draining; a partial read returns exit 4. An unconfirmed effect during a submitted drain returns exit 3 and keeps confirmed removals in the result. Raw native diagnostic bodies are omitted.

manager reconcile uses the existing guarded issuance-gate repair from the owning local mesh root. It resolves the registered broker without an alternate-target fallback and preserves its TLS requirement across repair, liveness and eviction connections. It requires that root's auth material and verified holder liveness. Review defaults to cancellation; scripts require --yes. --dry-run checks only the local coordinate and issues no credential or native guard query.

A pre-effect native guard refusal returns exit 2. Its error names the guard's condition and keeps the guard's own sentence as its native cause, which names the freeze-holder and the next step. A failure after the repair may have submitted work returns exit 3, without replaying mutations or hiding possible revocation/eviction progress. Re-observe that gate and its operation-bound durable cursor before another explicit repair. The final receipt reports the native operation and before/after coordinates, not a running-manager verdict. A reopened gate with retained cursor cleanup returns exit 4.

Manager stop impactCopy Anchor Link

The native manager status contract includes its selected lifecycleUid and a caller-scoped stopImpact observation. It lists visible managed principals and lifecycle IDs, their runtime modes and the seat actions a default stop or --with-agents would request. In-process PTYs without a release hook stop by default; releasable seats lose manager custody. Preservation or pending resume uses the native retained-state policy instead. None of these labels confirms seat exit, authority retirement or cleanup. From manager cluster revision 33, a manager that is not active and holds a preservation inventory also reports preservation: the attempt and the inventory digest a preserved-exit stop names. A preserved-exit stop request carries that binding with withAgents: false; the manager accepts it only when it is fully preserved, whole-manager and quiescent, and its result is preserved-exit-returned. The open-space local receipt keeps its revision-32 shape.

The manager resolves visibility through the same fresh owner/admin policy as ps. An owner-filtered observation is not the full manager impact, even if its rows happen to match a count. A refused stop reports the selected manager, visibility, maintenance state and pending-work counts that blocked its preflight, without exposing other owners' agent rows. Owner-filtered visibility requires the stack operator or an authorized admin-scoped caller; the command never substitutes one. On a user-auth space, the refusal's next step is the whole-row cotal actor grant that adds admin to your current grant, as the auth plane prints it from the ledger row. A stop in the folder that hosts the space reads the manager's own local receipt instead, so its owner stops the whole manager without that scope or a sign-in. No stop policy is accepted while any blocker remains. This is an observation at the reported time, not a reservation, permission to stop or a guarantee that the set cannot change. Older managers can omit it; the native service adapter refuses when the required process/lifecycle correlation is absent. Generic process launch/stop still refuses managers.

Native manager service definitionCopy Anchor Link

The manager service preset uses the manager-service adapter. It requires the selected registered root and an explicit installed runtime; an optional server can repeat the registered broker. The optional console-port, console-host, ws-port and max-sessions pass explicit native settings, and an optional name names the manager as supervise --name does. Omitting the host keeps the native loopback binding; port zero requests an allocated console port. A declared launch also requires launch-sha256. It names an existing private resolved launch spec, pins its full SHA-256 and must name this space. The child parses those verified bytes before authority preparation and keeps that snapshot through startup. An optional roster names an existing roster file by absolute path. Only its owner or root may write it or its directory. The child parses it before startup, and each launch boots the agents it lists the way supervise --roster does. A resumed manager takes no roster. Saving these options does not prepare a manifest, provision an agent or prove that requested agents joined. Saving it starts nothing and grants no authority. Its native child starts no broker or delivery daemon, does not adopt an existing legacy manager, and never prints its private console URL. It declares one instance per definition and runs in the background or with start --foreground. It is not OS-supervised installation or platform parity. A root that space add --root registered has no .cotal until a command writes there. A start makes it, as a directory private to you, after its other checks pass and once the root itself is protected against other local writers. It does not replace a .cotal that is already there, such as a link, which the child refuses.

Manager service startup proofCopy Anchor Link

The real child exclusively claims its creation-pinned native PID custody before hosted authority preparation or local endpoint initialization. That starting record is not readiness; another existing slot is not adopted or reclaimed. After a crash, the record the killed child left names a dead pid. The manager adapter removes that record before the next launch, so the next up relaunches the manager; a live, foreign or unreadable record still refuses. It binds its actual native identity before endpoint construction. A startup phase appears only after native start, settled custody and signal handlers. On authenticated meshes the adapter also requires a current status response pinned to that native instance and epoch, with the same process lifecycle UID. It preserves the invoking managed-seat caller, registered root/broker and required TLS. Missing caller support refuses without an operator substitution. The completed status read's own admission is excluded from stop impact; other admitted work remains counted.

An open mesh uses the pinned child's private request-bound local observation, and so does a stop in the folder that hosts a user-auth space. Its service, launch, request and both process creation identities must match, and its registration must match the launch identity. Output labels this a child-local receipt, not a broker-attributed endpoint response. A retained stop handoff blocks readiness adoption on either route instead of treating a live PID as a new serving launch.

Manager service stop policyCopy Anchor Link

A real service stop or restart reads complete caller-visible impact before confirmation. --with-agents requests native agent stops, not generic descendant signalling. An interactive confirmation defaults to cancel and shows every affected identity, lifecycle and requested seat action. Scripts require --yes. The offline --dry-run still imports no provider and performs no native impact query; it is not stop authorization.

The native stop reserves its original PID custody. It reclaims an existing reservation only when the pid it names has exited and no request was published for this launch, and otherwise refuses and names that pid. It rechecks the confirmed scope, binds the full service/launch/stop request and both process creation identities, and uses the existing native shutdown intent for agent reap. The child rechecks the same complete scope before accepting. Changed or incomplete scope refuses without a broader policy. The controller requires immutable accepted/returned evidence and the original process exit before recording a native stopped state. An unconfirmed request retains its reservation and never escalates to force kill, restarts, or treats PID disappearance as completion. A later bare up or down settles it only from the same evidence, and only then releases the reservation. Reported cleanup remains not checked. Retained native-policy markers prevent a metadata change from selecting generic signals, and an unconfirmed native stop cannot be forgotten just because its process disappeared. Legacy shutdown refuses the native child rather than bypassing that boundary.

managerCopy Anchor Link

cotal manager start [--runtime <name>] [--roster <file>] [--name <name>] [--foreground|--dry-run] [--space <s>]
cotal manager stop [--with-agents] [--yes|--dry-run] [--space <s>]
cotal manager restart [--with-agents] [--yes|--dry-run] [--space <s>]
cotal manager status [--space <s>]
cotal manager logs [--tail <n>] [--follow] [--space <s>]
cotal manager install [--dry-run] [--space <s>]
cotal manager uninstall [--yes|--dry-run] [--space <s>]
cotal manager reconcile ...
cotal manager seats [--drain]

The manager starts and keeps your agents running. cotal up starts one with the rest of the stack; cotal manager start starts one on its own, for this folder's space or the one cotal use selected, with the pty runtime and the broker the space is registered at. Nothing has to be added first, and no other service is started for it. mgr is an alias.

cotal manager <verb> is cotal service <verb> manager: every service rule applies, and its flags and the manager preset's options (--runtime, --roster, --name, --server, --console-port, --console-host, --ws-port, --max-sessions) are taken on the line. cotal help manager <verb> and cotal manager <verb> --help print the verb's help with those options. The first start saves the space's manager service from the preset's defaults and says so on stderr; cotal service show manager reads it. A start that names another --runtime or other option for a stopped manager saves it again with that option, and says so. A running one refuses LIFECYCLE_BLOCKED and names cotal manager stop first.

On a user-auth space this folder hosts, start and restart need your signed-in, granted login, as cotal up does for its manager. On a user-auth space registered here from another host, they need your sign-in. Before anything is saved, stopped or launched, they confirm the sign-in with its IdP. Without one, or when the IdP rejects it because it expired or was revoked, they refuse AUTH_REQUIRED (exit 2), and Next names cotal login --idp <base>, then the command again. A refused restart leaves a running manager as it was and says the running manager was left running. Any other failure of that check, such as an IdP that does not answer, is left to the start. In a folder that hosts the space, stop and cotal down need no sign-in: the broker, the records and the process belong to this folder, so the stop reads the manager's own local receipt.

The manager these verbs act on is the space's one manager service: manager when cotal manager start saved it, or stack-manager when cotal up did. Stderr names the saved service when it is not manager. So after cotal up, cotal manager status and cotal manager stop act on the stack's manager, and cotal manager start reports it running and starts no second one. Two saved manager services in one space refuse TARGET_AMBIGUOUS, naming each with its cotal service command. A manager that cotal up or cotal supervise started without a saved service is reported by start and status with its PID, and starts nothing beside it. logs reads the log cotal up opened for it, <root>/.cotal/manager.<spaceKey>.log, and changes nothing. It takes --tail, --follow and --json as a service's logs does, and a follow ends when that manager's process is gone. After cotal down stops that manager, logs still reads the log it left: the text says the manager is stopped, with its PID while its record still names one, and names cotal up under Next, and --json carries recordedProcess with state set to stopped. With no such log, logs refuses TARGET_NOT_FOUND and names the file. If the folder's records show a manager ran there (a PID record that names a dead manager, or the instance identity a manager keeps once it starts), the refusal says the log is no longer there and names only cotal up. Otherwise it says no manager has been started and names cotal up or cotal manager start. cotal service logs stack-manager reads the same file when no stack-manager is saved. A manager that cotal supervise runs in a terminal writes to that terminal instead. stop, install and the other verbs refuse it while it runs, because no saved service owns it; it stops with its stack through cotal down, or where cotal supervise runs it.

--foreground keeps the manager in this terminal until Ctrl-C, which stops it through its native stop. A unit or container you write that runs a manager uses cotal manager start --foreground. cotal manager install refuses UNSUPPORTED in this release, because a Cotal-written unit runs only a direct-process service. It names the ExecStart= line for a unit of your own instead, whether or not a manager was started, with one exception: while a manager that cotal up started with no saved service runs, install refuses it as stop does and names cotal down.

A manager that fails while it starts writes one last line that names the cause: the manager did not start, then the failure's code and message, the error it wraps and its one next step after ; next:. start fails OPERATION_FAILED, exit 5, and quotes that line, whole up to 1000 characters. A longer line is quoted by its start and its ; next: step, with its length where the middle was cut. cotal service logs manager --instance <id> shows the whole line. When the error ends in a certificate this machine does not trust, such as UNABLE_TO_VERIFY_LEAF_SIGNATURE, Node's message stays as the cause, and the next step is to set NODE_EXTRA_CA_CERTS to the CA file, or the self-signed certificate, for the command that starts the manager, as the --tls-cert row of up says. The manager forwards that CA-file path to the seats it launches without a spawn.env entry. With --foreground the line goes to this terminal on stderr, wrapped at its width as the CLI's own lines are, after the Mesh: heading and lifetime notice that start prints there first, and the refusal says its output went to this terminal and was not captured in its log and names no log.

manager start and manager restart run the first-run connector seed, as cotal supervise does, so a fresh machine or container has its connectors before the manager boots; status, logs, stop and the maintenance verbs install nothing.

reconcile and seats are the manager maintenance commands. Bare cotal manager lists every verb and starts nothing.

Start a manager for a hosted spaceCopy Anchor Link

A hosted space runs its broker elsewhere, so nothing on your machine starts a manager for it. Start one:

cotal manager start --space <space> --runtime <orca|tmux|pty>

It needs you signed in to the space (cotal login) and the space registered on this machine (cotal meshes add). A sign-in the IdP rejects because it expired or was revoked counts as none: the start refuses AUTH_REQUIRED before it launches anything, as manager says. With the space selected by cotal use <space>, a bare cotal manager start is enough: it uses the pty runtime, which every platform has built in. orca and tmux need their extension first, such as cotal ext add @cotal-ai/orca. The command returns once the manager answers; --foreground keeps it in this terminal instead. cotal ps then lists it with the agents it runs, and cotal spawn -d <persona> launches an agent through it.

Gate repair refusalsCopy Anchor Link

See the current command contract for this behavior.