Endpoint discovery

Request routesCopy Anchor Link

One kind, ep, carries every request under a mode token that says where the request routes, never which verb it is (the verb rides the envelope): one (queue-group anycast, one and only one class member), all (scatter, every instance), and inst (one instance by its stable address). Replies come back on a reply rail keyed to the serving instance and its epoch. Around these sit the sibling planes the composites use: per-goal events, timers, sessions, and the journal that holds durable facts. Every request carries the caller as three forge-locked tokens, owner, actor, and lifecycle uid, plus an unguessable nonce, so the broker polices who is calling in the subject grammar itself. See SPEC §13.2 for the grammar and §13.5 for the verbs (call, cast, watch, claim, scatter).

Lifecycle identityCopy Anchor Link

A principal owner.actor is a reusable routing alias: a despawn frees the actor name and a later spawn may legitimately reuse it, so the alias alone is never authority. Two further coordinates make an identity durable: a lifecycle uid, an unguessable, never-reused id for one managed lifecycle under a principal, and a process epoch, the fenced ownership epoch of the process currently animating it, advanced on every restart or takeover. At most one live epoch owns an identity, and a superseded epoch must stop serving. Durables and credentials key on the lifecycle uid, not the reusable name, which is what lets a supervised restart recover the same lifecycle instead of minting a new one. See SPEC §13.1 and identity & auth.

Service discoveryCopy Anchor Link

No client has compile-time knowledge of any endpoint's commands. cotal describe <endpoint> resolves a registered endpoint's command set off the wire: the reserved describe command answers the registered contract digests, the schemas are fetched from the space's content-addressed contract store, recompiled, and verified against those digests. Each command prints with its capability class and targeting shape. cotal invoke <endpoint> <command> --args '<json>' then calls one command by name, validating the arguments as they will be sent (JSON drops a key whose value is undefined) against the fetched input schema before publish. A refusal at that check means nothing was sent, and it is marked not-executed. An endpoint that refuses a request before running the command states not-executed in the reply's error.outcome too, for example on a wrong envelope version, sender, contract digest or target mode, or a body that does not parse. A refusal the command itself raises carries only the outcome it states. A refusal that does not serialize comes back as internal, and one too large to publish as resource-exhausted, and both keep that outcome. A signed-in user invokes the same surface through their bearer, and the broker enforces each command's existing capability grant. A manager alias supplied through --name resolves through its name-keyed inspect command, so an authorized targeted call does not need the manager-wide ps enumeration grant. Every built-in manager command uses this same trust chain, so there is nothing the built-ins can reach that a described contract cannot. The registered auth endpoint is describable the same way manager is: cotal describe auth lists retire-lifecycle and its exact-mode target shape. See SPEC §13.7 and cli.md.

Listing what a space offersCopy Anchor Link

An agent asks the space's registry what it can call with the cotal_endpoints tool. The registry is a registered endpoint named registry. Its endpoints command lists each registered endpoint where the caller holds at least one command, with the commands and request forms the caller's recorded ceiling covers, and leaves the other endpoints out. It reads the ceiling the issuer recorded for the caller's credential and does not check that the ceiling's sources are still live, so the broker or the endpoint can still refuse a listed command. The answer says this with source: "recorded-ceiling". On an open mesh every registered endpoint is listed and source is no-permission-system. Each endpoint row also lists its registered instances, the state each one recorded, and whether its command declarations verified. A page holds up to 100 endpoints. When the registry does not answer, the tool says so and lists nothing; it does not build a list from describe or presence. A listing drops the session's cached describe of every endpoint it lists. On a user-auth mesh manager commands ride a separate manager-call credential, so cotal_endpoints with view: "manager" opens that credential as a manager call does and lists what it can call. A session whose bearer comes from its host has no manager-call credential, and the tool says so. See SPEC §13.20.

Machine vitalsCopy Anchor Link

The manager's vitals command reports one invocation-time sample of host CPU count and load, available and total memory, and bounded local filesystem capacity. It is untargeted but resolves to one exact Manager instance. The separate manager.vitals capability is issued only when the actor scope contains machine-vitals; spawn, run and admin do not imply it. The response contains mount points and aggregate byte counts, never device paths, remote endpoints, process data, command lines, environment values or persisted history. Unsupported measurement refuses without a partial or zero-valued fallback.

Host directory resolutionCopy Anchor Link

The manager's resolve-cwd command is in the manager.spawn capability class. It accepts an absolute path on that manager's host and returns its canonical directory plus the host name. It refuses a relative, missing or non-directory path with failed-precondition; it creates nothing. spawn applies the same check at admission, before any credentials or durables are minted. It is minted beside spawn in the operator instruments, which are the caller on a static mesh, and in a spawn holder's manager-caller view. The standing agent spawn capability does not carry it. A runtime that delegates seats outside the manager's host cannot use a host cwd. Its manager refuses resolve-cwd with failed-precondition before any filesystem lookup, with the same execution-mode refusal for every path. It returns no host name or canonical path.

Spawn input resolutionCopy Anchor Link

The manager's resolve-agent command is minted beside spawn in the spawn capability and the operator instruments. It accepts the persona fields spawn takes (name, config, agent, defaultAgent) and returns {agent}, the harness spawn on that manager would launch: the flag, then the persona's agent: pin, then the caller's default, then the manager's own default. It loads the persona with the same visibility check as spawn and launches nothing. A detached --resume without --agent asks it before carrying a session, and a manager that does not serve it refuses the call.

On a delegated runtime, both resolve-agent and spawn refuse an explicit persona config before any lookup of that path. Absolute, relative and bare-name config overrides receive the same failed-precondition refusal. Select a named catalog persona through name instead. Locally executing managers retain their config-path behavior. A resolved manifest uses its own validated, host-materialized persona resource and is not a caller-selected config override.

spawn refuses an empty or whitespace-only value in any of its optional string fields, such as role, agent or identity, with bad-request naming the field. To take the persona file's value, omit the field.

Instance routingCopy Anchor Link

A space can run more than one manager. Each manager persists a stable logical instance id across restarts and advances its process epoch when it comes back, so callers address a specific manager without caring which process currently serves it. A start serves only at the epoch its own registration committed, never at the epoch of a later start of the same instance. On a static mesh or from an unbound open caller, an untargeted spawn rides class anycast (any manager may accept, and the acceptance records which one did). An open managed seat's persona definition, list, show and spawn tools default to its launching manager. Its generic manager describe and self-targeted invoke use that launch address too. An explicit existing persona or spawn selector still selects that instance. Static credential grants and user-auth issuer selection are unchanged. For self-stop, persona definition and direct spawn, a static managed seat keeps its class-only credential and accepts only its launching manager's attributed describe reply. These commands retain that bind on the class rail, so a sibling refuses before effects and the existing bounded split repair asks again. An absent launching manager never selects a replacement. The library's describeService accepts an optional instanceId; a pinned describe does not read or populate the endpoint-only class cache. cotal spawn <persona> --detach --on <instance> and cotal_spawn(instance: "<instance>") pin one instance by its exact id, as --on does for cotal ps, cotal models and every hosted cotal run verb, and instance for cotal_run, cotal_persona and cotal_personas. A foreground CLI spawn has no manager to pin and refuses the flag. An MCP pin that does not resolve is refused without falling back to class anycast. There are no ordinal aliases and no short forms: wherever a display names an instance you can address, it prints the whole id, because both surfaces take nothing else.

On a user-auth mesh, manager commands obtain a short-lived manager-caller view from the exchange. It authorizes one concrete manager instance using the caller's current actor grant and the host's registered service records. Discovery and invocation both use that instance's inst route. This view grants no registry scan or class queue. It adds no command capability except the three reads a launch needs: a view whose actor holds spawn may call status, models and resolve-cwd on that instance, subject to the execution-mode limit above. A run-only or no-scope view may not, and ps and slots stay with admin. A user mesh's co-located manager answers every owner's view, so its status counts only the caller's own seats, as ps would list them, unless the caller holds admin. An absent, ambiguous or unauthorized selection refuses before the command is sent. An ambiguous one lists the owner's live manager instance ids and names the caller's way to pick one: --on <instanceId> for the CLI, --manager-instance <instanceId> for cotal agent-bearer --manager-call, and the instance argument for an agent's tools. A pin changes routing, not the caller's grant.

Each status.connectors row includes eventPlane: boolean. The value is true only when the connector declares the event channel it publishes. Installed-extension rows carry the cached fact before the package is imported, and an old inventory without it is refused rather than treated as true. A client that reads a required event policy can use this field to disable an incompatible connector before spawn. The Manager still owns the launch refusal text and applies it at admission.

Managed launches carry COTAL_MANAGER_INSTANCE so their tools address the manager that launched them; a tool's instance argument must name that same manager. The cotal commands run from their shell select it too, unless --on names another instance. Existing unbound sessions can use the exchange's unique authorized selection without replacing their actor or conversation. The connector uses a separate control connection; the standing message connection and its credential source are unchanged. Accepted spawn goals are followed on that renewing connection, using its existing caller-scoped progress grant, so a long readiness budget does not depend on the short-lived control credential. The follower confirms its progress subscription with the broker before submitting on the separate connection. A caller still checks the resolved instance and epoch, and never retries an ambiguous mutation outcome.

The reference cotal_spawn tool retains the published request id when its submission wait ends before acceptance is observed. That id is the manager's goal id if the submission is accepted later. The tool reports an unknown outcome with the id and a same-caller goal-result read, without claiming that acceptance occurred. Do not repeat the spawn. An unanswered describe or a broker-attested no-responder refusal does not create a goal handle. A missing readback is inconclusive while the original submission may still arrive.

Goal result readsCopy Anchor Link

The manager's goal-result command accepts {goalId} and returns {goalId, result?}. It reads only the authenticated caller's owner, actor and lifecycle through the manager's separate trusted goal-writer connection. The caller receives an attributed reply, never a raw JetStream reader grant. Each read is admitted by the connection's broker-enforced command grant. A live user-auth connection remains bounded by its bearer expiry after revocation; a renewed connection is checked against fresh authority. There is no separate per-read ledger check. An absent result means no terminal is recorded; it does not prove the goal is running or permit another submission. The existing trusted goal-writer's leader-served EPF read is space-wide at the broker; the handler confines it to this endpoint and caller triple.

Goal cancellationCopy Anchor Link

The manager's reserved cancel command accepts {goalId, mode?} and returns {goalId, state}. It is served for a turn the manager relays. The goal is the authenticated caller's own, so a caller withdraws only a turn it submitted. The turn ends cancelled, its seat is not shown it again, and a later yield of it is answered with that terminal. A goal that already ended is refused failed-precondition with its cached outcome attached, and a goal this manager does not relay is refused without being changed. So is a second cancel that arrives while a first is still ending the turn; a first that fails leaves the turn pending unless something ended it meanwhile. A workflow run sends it for the turn, ask attempt or escalation of a branch it cancelled.

Request outcomesCopy Anchor Link

A followed mutation requires a manager whose attributed describe includes goal-result. If it is absent, preflight refuses with failed-precondition, an upgrade notice and not-executed before submitting the mutation. Narrowed discovery preserves that refusal. Update the manager, issuer and client together before using that recovery path. Reloading an issuer alone cannot change an already-running participant manager. Recovery re-resolves the accepting instance's epoch, preserves the caller lifecycle and validates the result against the accepted goal and any acceptance fingerprint. Stopping the caller ends its observation, not the already accepted goal.

A followed call resolves the endpoint within its deadline before the submission starts, so a refused or unanswered describe surfaces as its own error. On a user-auth mesh that preparation also obtains the caller's manager-caller view and opens its connection, and a failure there surfaces unchanged. The control connection uses the remaining preparation deadline, including time spent obtaining the bearer, rather than a separate two-second limit. A deadline that passes during those steps reports that the request was not run. The describe then spends what is left of the deadline, so a describe that draws no reply still reports itself as unanswered. A describe or command publish that the broker refuses reports not-executed. Cancellation before submission reports not-executed. Once submission starts, cancellation or a lost reply reports an unknown outcome unless an attributed refusal proves otherwise. A received refusal remains a refusal even when stop races it. Local failures do not invent responder identities. The follower owns its subscription, timers and read cancellation signal. Reconciliation begins before the wait deadline, and late read completions cannot settle an expired observation. Its read callback receives the accepting caller triple, remaining budget and abort signal; borrowed bearer commands and control connections use that signal. An in-flight dial that finishes after cancellation closes without publishing. A local reply-subscription failure prevents publication and is observed by the same request promise, including when the transport is closing or draining. Request cancellation does not revoke or resubmit the accepted operation.

Split topologyCopy Anchor Link

"Only one manager per space" is not the current invariant. A split topology that keeps the broker host manager-free is still a topology choice: cotal up --no-manager on that host starts no manager, and cotal supervise --server runs the manager elsewhere (Run a mesh). A broker host that already runs a manager stops it only after ✓ manager up in .cotal/manager.<spaceKey>.log (detach stdout listing manager is pidfile liveness, not a teardown boundary). A controller stack stops it with cotal service stop stack-manager --space <s> --yes; a legacy stack has no command that stops only its manager. Extra live managers are addressable, not an error.

Discovery deadlinesCopy Anchor Link

The reserved describe bootstrap is the one request the resolver may repeat while waiting: it is read-only, it is re-published under the same request binding, and every attempt stays inside the original deadline. This covers the startup window where Core NATS discards the first request before the manager has subscribed. If the connection closes while the resolver waits, the describe fails cleanly instead of throwing from the retry timer. The resolved command is never repeated by this readiness behavior.

The describe and the contract store reads after it share the resolver's one deadline, so a slow store read fails the resolve with deadline-exceeded when that deadline expires. cotal invoke passes its --timeout as that deadline (default 10000 ms) and gives the reply what the resolve leaves of it. cotal describe has no --timeout and passes 10000 ms. Cancelling a resolve settles a store read in flight instead of waiting for it to return. When one store read fails, the resolve fails with it and fetches no further command contracts; fetches already under way finish on their own. A call for one command fetches only that command's cluster and schema closures. cotal invoke --name also fetches inspect's, because it resolves the target through inspect. describe still fetches the complete visible surface because its result lists every command.

Instance mismatchCopy Anchor Link

The resolve and the invoke are separate trips through the same anycast queue, so in a multi-manager space an unpinned call can land on an instance the caller did not resolve. Every call carries the incarnation it resolved against, and a manager that is not that incarnation refuses before running the command, so the failure an operator sees says the command did not run, and re-issuing it cannot duplicate the effect. That is the difference that matters for a mutation: the older behaviour detected the mismatch on the reply, after the manager had already acted, and could only tell you to go and check. --on still matters for reaching a specific manager (ps, stop, attach, spawn --detach), but it is no longer what stands between a split and a duplicated spawn. Against a manager older than this fence the refusal is still after the fact, and its message says so. The re-issue is automatic only when the refusal states not-executed in its outcome field; a refusal that omits the field, or states unknown, is surfaced to the caller instead of repaired, because neither proves the command did not run. The CLI's manager commands, cotal invoke, the cotal run verbs and the manager row of cotal status re-describe and re-issue an unpinned call after each such refusal, up to 16 times, so a split reaches the operator only when every attempt split. A hosted run's own manager calls use the same bound. A pinned call is never re-issued. An agent's own manager tools, such as cotal_spawn and cotal_despawn, re-describe and re-issue with the same bound, including the goal-result read that follows a spawn to its outcome.

An unpinned targeted call, such as cotal_despawn or a hosted run's turn relay, can also reach a manager that does not host its target, because each manager resolves targets against the agents it runs. That manager refuses with expired and not-executed and says it holds no mapping for the target, and the same re-issue repairs it within the same bound. An agent that no manager hosts still ends in that refusal once the re-issues run out. A pinned call gets the refusal of the instance it named.

Manager availabilityCopy Anchor Link

A verified Manager status reply includes host, the answering process's host name, and version, the running @cotal-ai/manager package version captured when its module loaded. These fields are advisory display metadata. They do not select a Manager, authorize a caller or determine placement. Use the verified responder's instanceId for attribution. Clients must accept status payloads that omit either field and leave the missing value unknown. The optional fields advance the Manager cluster revision to 28.

A manager whose boot inventory marked every declared connector unavailable does not subscribe spawn or launch on the class one rail. Those commands stay on scatter and on this instance's inst rail, so a sibling that can launch them can take an unpinned spawn, and a caller that pins this instance with --on still gets a named harness refusal. describe still lists the commands: the instance rail serves them, and describe itself stays on the class rail (SPEC 13.7). An unpinned spawn can therefore bind-fence: describe may land on the skip member while spawn lands on a sibling, the command was not run, and the caller re-issues or pins --on. status reports classSpawn: false when that skip is in effect. When no manager takes the class queue, the broker answers an unpinned spawn with no responders, and the caller classifies that not-executed (SPEC 13.3). A manager that can launch some connectors keeps the class rail. If the queue hands it a harness its inventory marked unavailable, the refusal names --on because the standing serve credential cannot read sibling inventories. Pin the capable instance (the whole id, as ps prints it).

Scatter livenessCopy Anchor Link

ps and status become a scatter across every registered instance: the caller freezes the expected set from the service registry, invokes each under a shared deadline, and merges the results with per-instance attribution. A non-answering instance is labelled as registered with no answer within the deadline, never silently omitted. See SPEC §13.5 (scatter) and cli.md.

The expected set comes from the registry, which records registration rather than liveness. An instance that crashes never deregisters, so it stays in the set and the gather has nothing left to wait for but an answer that cannot come. It pays the whole deadline, on every scatter, indefinitely. A scatter can therefore be given a per-instance liveness probe: when the broker itself reports that an instance holds no subscription on its own instance rail, the gather stops waiting for it. Only that affirmative report counts. A lapsed presence entry, a probe that timed out, and a probe that failed are all absence of evidence, and treating any of them as death would turn a slow correct answer into a fast wrong one, so they leave the full deadline standing. Nothing about the outcome changes either way: an instance that did not answer is still unreachable, still surfaced, and the scatter is still not complete.

The probe is supplied by the caller, not invented by the scatter. Asking about an instance is a publish on that instance's rail, and a credential that holds no row for it is refused by the broker asynchronously, while the publish itself returns normally. The probe verb watches for that refusal and raises it as permission-denied naming the rail, so it is never mistaken for a quiet instance, and it never burns the probe budget waiting out a refusal. Only the layer that minted the credential knows which ids it may ask about, so that layer asks about those and no others. cotal ps freezes the class on its first connection, re-mints an instrument pinned only to the frozen ids, and scatters on a second; a refusal the broker raises anyway is printed and the instance's row says the probe was refused, which is a fact about the credential, not about the instance.

This does not help against an instance that is connected but not answering. A hung manager holds its subscriptions, so it is indistinguishable from a slow one, and it still costs the full deadline. That is the correct result, not a gap in the probe.

DeregistrationCopy Anchor Link

A probe makes a dead registration cheap to skip; it does not remove it. Removal is the registration's own exit, and there are two explicit routes to it (SPEC §13.5: a deleted svc spec is the deregistration).

A manager that stops cleanly removes its own registration, so an ordinary shutdown leaves no stale row. The delete is pinned to the registration revision that process wrote. When a successor has registered the same instance since then, the stop logs that and leaves the successor's registration alone. It refuses that delete while this instance holds the endpoint governance slot at the live issuance-gate generation (a registration still completing its reopen). A leftover slot whose generation is behind that live generation is not in-flight and does not block the stop. A manager that cannot renew or read its lease keeps serving, stays registered, and retries. If another process holds the same instance key, that process has taken the instance over, so this one logs the conflict and exits without deregistering, leaving the successor's registration alone.

A restart that died mid-registration is a different residue: the issuance gate stays frozen under that op. The successor completes the dead registration on boot when the freeze-holder is affirmatively gone under a complete CONNZ sweep (the same composition as cotal reconcile-gate). A committed spec write is finished under that same freeze; only a definite no-commit abort-reopens and then runs the normal takeover. Once that reopen commits, the registration has completed: a governance slot it then fails to release is left behind the live generation, as on a normal registration, and does not fail the restart. It does not invent a TTL and it does not start a new freeze over a still-held one.

A registration's own reopen fails in one of two ways, and its unavailable error says which. A reopen that lost its compare-and-set wrote nothing: a reconciler or newer barrier moved the gate. A reopen that threw, such as one whose acknowledgement was lost, has an unknown outcome and may have committed. Either way, re-read the gate before retrying.

That residue has a second half, and it is the endpoint governance slot rather than the gate. Every registration takes the endpoint-wide slot before it publishes its spec and holds it until its own gate reopens, which is what serializes registration for the endpoint. An instance that stopped between those two points leaves the slot held with no registration behind it, so the endpoint refuses new registrations while nothing is actually in flight. The slot is stamped with the generation of the gate its holder had frozen when it took it, and a slot is promoted only at that same generation. So once the holder's gate has reopened past the stamp, the slot can never be promoted by anyone, and the next registration for that endpoint replaces it. That reclaim is part of an ordinary start and needs no operator step.

A slot whose holder's gate is still at the stamped generation is a registration that is genuinely in flight, and it keeps refusing. The two states read differently only in the holder's gate coordinate, so reopening that gate is what separates them: the holder's own restart heals it on boot, and cotal reconcile-gate is the operator's route when the boot path cannot run. The registration path is the slot's only writer, and neither repair command writes it. A registration that cannot read the holder's gate at all refuses, because an unreadable gate does not distinguish the two states either. Each of these refusals carries kind = ai.cotal.ep.foreign-slot-held in error.details[] with the holder's instance id and the condition that refused: in-flight for a holder gate still at the stamp, or no-seam, unreadable, garbled or behind when the registration could not read that gate or read it below the stamp. A remote manager asks its host to reconcile the holder only on in-flight, the one condition a gate repair can clear.

For the instance that cannot cooperate, an operator names it: cotal endpoint deregister <endpoint> <full-instance-id> (cli.md). It removes the record only on the same evidence cotal ps acts on: the broker reporting nothing subscribed on that instance's own rail. It refuses if the instance answers a describe, refuses if the probe could not run at all, and refuses if the instance is merely quiet, because a hung process still holds its subscriptions and is therefore not affirmed gone. It also refuses while that instance holds the endpoint governance slot at the live issuance-gate generation (a registration still completing); a leftover slot behind that generation is not in-flight and does not block. Nothing sweeps the registry on an age threshold or on silence. An instance that is deregistered while it is merely wedged re-registers over the tombstone on its next start, which is what makes the operator's decision a recoverable one.