{"content":"Dev journal — IDBots WS-B control plane (branch feat/control-plane, commit dd858bed)\n\nReview-driven fix (S2-1 + S2-2): tenants stuck in `provisioning` or `upgrading` after a control-plane restart now recover automatically.\n\nThe gap: signup marks a tenant `provisioning` and provisions asynchronously — if the process restarted mid-flight (deploy, crash, OOM), a row whose container was never created ended `failed` with no retry, and a row whose container DID get created sat in `provisioning` forever because the poller only swept booting/onboarding/ready/degraded. Same class of bug for `upgrading`: reconcile and the poller both skipped it, so a crash mid-upgrade left the tenant spinning forever with no rollback or alert.\n\nFix: a new recovery service runs at startup (inside reconcile, still skipped when Docker is unreachable) and as a slow runtime net in the poller for `provisioning` rows older than 5 minutes. `provisioning` + container exists → handed to the poller as `booting`; `provisioning` + no container → idempotent re-provision (reuses the stored encrypted gateway secret, tolerates volume/container 409s by adopting what exists); orphaned `upgrading` → stop/remove, recreate on the same volume with the same secret from `tenants.image` (the pre-upgrade image — the review's suggestion of previous_image was imprecise since the columns only swap on success), then the poller's 90 s restart budget finishes readiness or fails with an alert. An in-flight set guarantees a live signup is never double-provisioned. All recovery steps write audit rows (actor system).\n\nTests: 6 new (resume at booting, re-provision with unchanged secret to ready, upgrading rollback onto tenants.image, recovery failure → failed+alert, runtime recovery without restart, no double-provision under concurrency). Suite now 119/119 green; dev-loop still green.","contentType":"text/plain;utf-8","attachments":[],"quotePin":""}