Storage and recovery boundaries

Recorded baseline: 2026-09-24. Check actual mounts, StorageClasses, reclaim policies and backups immediately before storage work. A Ready PVC is not a restore test.

TrueNAS dataset roots recorded by the migration plan are /mnt/homelab/apps/ and /mnt/homelab/media. The exact installed CSI artifact and non-secret settings should be reconciled with platform CSI. The Kubernetes PV/PVC objects live in the cluster; TrueNAS holds the underlying datasets, exports and block devices.

ComfyUI models remain at /mnt/assets/ComfyUI/models on media, exposed read-only through separate dev/prod claims. Writable state is separate under /mnt/assets/comfyui-cluster/dev/state and /mnt/assets/comfyui-cluster/prod/state. The older “786 GB of models” figure was an estimate; owner inventory on 2026-09-16 measured 841 GB used across the whole 4.6 TB assets filesystem, not the model directory alone. Preserve the original models and terminal installation while validating Kubernetes. ComfyUI storage owner.

Actual Budget's production app uses a retained static NFS PV for its existing TrueNAS data. Its successful restart check establishes persistence at that time, not backup recoverability. GitchAlerts uses external PostgreSQL and S3 assets; database, object and app state recovery are separate from deployment rollback.

OpenBao uses a three-replica Raft design with independent local persistence in the platform package, avoiding a bootstrap dependency on the TrueNAS CSI key it eventually serves. Protect unseal/recovery material outside both Git and OpenBao itself. The accepted homelab recovery scope is rebuild-first; verify current backup and restoration records before claiming a tested disaster recovery path. OpenBao/ESO runbook.

External PostgreSQL and S3-compatible object storage are outside Kubernetes and can survive a cluster rebuild, but each needs independent backup, credential, DNS/TLS and restore checks. Keycloak database recovery needs database-native backups. Velero objects are useful only if the external object store and its access can be recovered. The Windows runner and Forgejo remain separate operational boundaries.

Object storage transition — 2026-10-13

Owner-selected direction: use the free SeaweedFS TrueNAS stable app for new object storage, then migrate existing AIStor consumers individually. AIStor Free blocks encryption at rest; configuring another KMS does not remove that license restriction. SeaweedFS supports server-side encryption. The owner selected its native SSE-S3 key mode on 2026-10-14; no separate KMS service is required for that mode. The bounded state-bucket test below verifies initial encrypted readback; application migration and independent recovery remain separate acceptance work. Sources: AIStor licensing and SeaweedFS encryption.

The owner reports SeaweedFS 4.48 / TrueNAS app 1.2.39 installed and running, with persistent datasets owned by UID/GID 3000:3001. The owner confirmed Admin port 30300, S3 port 30304 and no backend S3 certificate. The selected routes are seaweed.app.gitchegumi.com through the existing tenant gateway and seaweed.admin.app.gitchegumi.com through the existing LAN-only admin gateway. The owner chose cert-manager-managed HTTPS termination at Istio, with HTTP from Istio to TrueNAS. No new Gateway or wildcard certificate is intended.

Routing source merged in private Homelab/platform#121, tracked by issue #120; its owning runbook is docs/seaweed-storage.md. On 2026-10-14 the owner clarified that this is configuration of the existing UDS gateways. It needs no standalone SeaweedFS bundle, separate release workflow or new CI deployment identity. Small reviewed routing changes use existing authenticated platform administration; application delivery remains governed by its owning pipelines.

2026-10-14 update: the owner reports routing working. An unauthenticated S3 request returned HTTP 403 after credentials were created; authenticated scoped access was unverified at that checkpoint. The owner reports the provisioning pair saved in OpenBao and WEED_S3_SSE_KEY configured with a private recovery copy. No key value is recorded here. Encrypted object readback was untested at that checkpoint; restart recovery and backup/restore remain untested. Keep an off-system copy of the same encryption key independent of the object store and cluster gateway. Preserve existing AIStor endpoints, consumers and the the-gitchistry bucket until each application's migration is accepted.

State bootstrap verified, 2026-10-15 (private Homelab/terraform-modules run 62, job 1, attempt 1; source bd714dac26e479a5560852a302b1641dfd94fa07). The owning protected workflow successfully authenticated through the scoped OpenBao role, read its bucket-only key and verified homelab-terraform-state: private ACL and policy checks, enabled versioning, default SSE-S3 AES256, encrypted/versioned probe readback and rejection of a conditional overwrite. The bucket is created or reconciled by this operation; the log does not distinguish those two cases. The earlier inspect run 61 supplied the exact public workflow identity, and the owner reported installing its policy/role and saving the scoped key in OpenBao. No key or token value was read or recorded by this documentation work.

Terraform PRs 11 and 12 are merged. The additional OpenBao identity profile is first published in workflows v1.41.0 and retained in promoted v1.42.3, with the RKE2 delivery and state-bootstrap support first published in v1.40.0. Detailed operator steps remain in private Homelab/terraform-modules/docs/state-bootstrap.md and docs/openbao-identities.md.

Central runner setup, 2026-10-15: private Terraform inspect run 64 passed and supplied the selected public claims for its owning provisioning workflow. The owner reports saving both scoped state-key pairs and installing the central plan/apply grants from the privately authenticated pod shell. Private platform inspections 4969/4970/4971 passed at source 566e28aab888d547773b470d67c44c268912b458; their exact observed workflow_call claims merged in Terraform PR #13. The reusable-event fix from workflows #106 is retained in promoted v1.42.3. Concurrent Terraform locking and recovery access independent of the Kubernetes gateway remain unverified.

Plan review history, 2026-10-15: private Terraform run 69 stopped after JSON export because its review guard misread fully known collection markers. The correction from workflows #107 was released/promoted in v1.42.0; run 70 then passed with eight proposed creates. Approved apply run 71 stopped before mutation on a fresh-fingerprint mismatch. Five credential-free mocked Terraform 1.16.5 exports reproduced differing outer orders of identical relevant_attributes UI hints; all other exported fields matched. Neither private live plan was retrieved. The narrow normalization from workflows #108 is merged, released and promoted in v1.42.1. Fresh plan run 72 passed under that contract.

Identity apply verified, 2026-10-15 (private Terraform run 75, job 1, attempt 1): after the owner's specific approval, the owning workflow ran as kiyi on main at source 3f0c7d9d3708bfcd512b89576bfe47c5d760d3f7, using run 72's exact source candidate and plan fingerprint. The fresh plan matched, Terraform applied the eight reviewed creates (three JWT login roles, three policies and two routine Kubernetes issuance roles), and the locked post-apply plan reported no managed-resource changes. This establishes protected apply and convergence for the original declarations. It does not establish concurrent-run locking, independent recovery, routine workload issuance or Kubernetes deployment.

RKE2 bootstrap delivery evidence, 2026-10-15: private platform build run 4972 passed and published signed release rke2-bootstrap-bundle-v0.1.0-r7501 at source 566e28aab888d547773b470d67c44c268912b458. Deployment run 4974 verified its signature, obtained the bootstrap Kubernetes credential and passed all five required permission probes. It stopped before bundle deployment. Its visible HTTP 403 is traced to OpenBao self-revocation: the original RKE2 ACLs omit that permission while their logins omit the default policy. The cleanup error also masked the earlier preflight failure; that original message is unverified.

Self-revocation correction applied, 2026-10-15: Terraform #14 and workflows #109 merged; promotion run 838 moved stable-v1 to v1.42.2 at workflow source 2418f947a9339774a674bf4b7f1a9f577d3882ea. The library preserves an existing operation error when cleanup also fails, while cleanup failure still fails an otherwise successful operation. Owner-authorized private Terraform plan run 80 contained only the three RKE2 policy updates adding update on auth/token/revoke-self. Apply run 81 used that exact reviewed plan at owning source 4e12c39aa2c17f3585ecc2ac5777f09736510ae9, applied those updates and reported no managed-resource changes in its post-apply plan. JWT bindings, credential lifetime and Kubernetes issuance roles are unchanged. Kubernetes bootstrap, routine-role credential validation, temporary bootstrap-login retirement, controller installation, Core/RKE2 upgrades and Harbor provisioning remain pending.

Bootstrap retry verified, 2026-10-15 (private platform run 4975): the owner-authorized retry used v1.42.2 and the unchanged signed bootstrap release, with adoption disabled. Signature/source checks, credential issuance and all five permission probes passed. OpenBao session revocation succeeded. The run stopped before bundle deployment with Preserve installed Zarf init 0.86.0; its log omits the observed package version. The owner's authenticated package-list output on 2026-10-16 confirms init v0.73.0; v0.86.0 is the next Core 1.14.1 target. The owner requires the first RKE2 1.36.5 upgrade before that Core/init transition. Workflows #110 merged and publication run 852 published v1.42.3 from 2f32073eb071e372c93a495d47d90f6468751dd6 on 2026-10-16. It adds the exact Core 1.13.1/init 0.73.0 profile alongside the retained Core 1.14.1/init 0.86.0 profile and reports safe version diagnostics. Promotion 853 attempt 1 stopped because the released commit's library check was still pending; that check later passed. Attempt 2 was rejected because protected workflows require a fresh manual dispatch to verify the initiating actor. Fresh promotion 854 selected v1.42.3 on stable-v1 with passing canary 7545; neither failed attempt changed the selected channel. Platform #130 passed all five head-bound checks (source 4982, worker 4983, security 4984), received PorteGumi's approval, and the owner merged it on 2026-10-16 at ee5874e583e45ef5ef20025ef9aca2c4091dd8ba. The matching platform declaration changes signed inventory, requiring a new bootstrap build/deploy from that merged source. Build 4988 passed and published signed rke2-bootstrap-bundle-v0.1.0-r7557 at that source, using v1.42.3. Owner-authorized deployment 4989 used main, that exact tag, confirm_bootstrap=rke2-bootstrap, and adoption disabled. It verified the signed release/source, credential permissions and installed init: expected=0.73.0; observed=0.73.0. It then stopped in the first, read-only ownership pass with Unowned retained RKE2 resource needs explicit scoped adoption. Session revocation succeeded. No namespace creation, adoption patch or bundle deployment ran. The log does not name the retained object or establish that its fields match the reviewed source. At that checkpoint, retained-resource review was still required and attempts stopped at the owner-authorized failure boundary. The following review and successful deployment supersede that blocker. No additional OpenBao apply is needed. The explicit Core/init upgrade is prepared for the following stage, with exact target version/component postflight checks. These source changes are not live profile acceptance or a cluster upgrade.

Retained-resource review, 2026-10-16: read-only inspection through the owner's authenticated WSL context identified five unowned cluster-scoped leftovers: plans.upgrade.cattle.io (CRD), system-upgrade-controller and system-upgrade-controller-drainer (ClusterRoles), and system-upgrade and system-upgrade-drainer (ClusterRoleBindings). All five match the reviewed source fields, have no Helm owner, and are not being deleted. The other 28 named bootstrap resources, including the namespace, are absent. The local source tree equals release source ee5874e583e45ef5ef20025ef9aca2c4091dd8ba; the checker equals released workflows v1.42.3. Its exact ownership validation passed locally for all 33 objects with adoption enabled and mutation disabled. No Secret was read and no live object was changed by this inspection.

Bootstrap deployed and verified, 2026-10-16: the owner resumed the task through bootstrap completion. Fresh private platform deployment 4992 passed on main using signed rke2-bootstrap-bundle-v0.1.0-r7557, confirm_bootstrap=rke2-bootstrap and adopt_existing=true. The workflow rechecked the named resources, created the reviewed namespace, adopted the five matching retained objects and deployed the exact signed archive. Postflight admission readiness, ownership and target checks passed; init remained 0.73.0 and the OpenBao session was revoked. Sanitized acceptance evidence is retained as rke2-bootstrap-evidence (internal run 7563). No code, permission, signature or field-match guard was relaxed. Work stopped after bootstrap completion. At that checkpoint, routine credential validation and the later delivery stages were still pending; the following evidence supersedes those credential gates.

Routine credential acceptance and retirement, 2026-10-16: private Platform runs 4993 and 4994 passed the controller and Plan credential-validate operations at source ee5874e583e45ef5ef20025ef9aca2c4091dd8ba. Required/forbidden permission probes, target/init checks and OpenBao session revocation passed; no deployment ran. Terraform PR 15 merged rke2-bootstrap.enabled=false at 0a9dd5edca9d3824d23f14abe0d13bf04301d3a2. Plan 86 and apply 87 passed. Apply 87 deleted only the temporary bootstrap JWT login role and policy, then reported no managed-resource changes in its post-apply plan. Routine identities and the existing Core issuance role were not deleted.

Controller image gate, 2026-10-16: private Platform build 4995 (internal run 7574) stopped before signing/publication with 26 blocking image vulnerability occurrences, 29 existing mitigated configuration occurrences and zero scanner errors. Twenty-one blockers are in the kubectl helper and five in the controller. At that checkpoint private base-images PR 50 prepared rebuilt controller/helper inputs with Go 1.27.2 and reviewed dependency locks, without a new mitigation.

Controller image publication and adoption preparation, 2026-10-16: base-images PR 50 merged at source e262e3732e3065296458ee2c76c279e1ac27e7bb. The published release pages and attached receipt fields agree on that source and these pins:

Private Platform #131 and workflows #111 are merged. Their reviewed support preserves the old Rancher receipt shapes and accepts only the two exact Homelab controller/helper repositories. Bootstrap and Plan source inventories are unchanged; this adoption needs no repeat bootstrap or OpenBao provisioning.

Controller delivery and Plan build, verified 2026-10-16: private Platform build 5004 published signed rke2-controller-bundle-v0.1.0-r7636 from merged source 398fa282dbb09456385ae3b7b358d01e6c44bfb6. Deploy 5005 verified the release, rescanned both images and passed permissions/init checks, then stopped at the explicit retained-object adoption gate. Deploy 5007 passed signature/source, fresh scans (zero blocking, zero mitigated, zero scanner errors), all 77 required and 14 forbidden permission probes, init expected=observed=0.73.0, and ownership checks. UDS archive deployment then exited with failure; session revocation passed. The raw CLI error was withheld and not retained. The cause and any partial deployment remain unverified; no controller acceptance is established.

Plan build 5006 failed with 28 blocking findings, 29 mitigated occurrences and zero scanner errors in rancher/rke2-upgrade:v1.36.5-rke2r1, digest sha256:63b54cbceda240d7e280f146eb3f7b4998631b8f2a7b3bcf9f7357ee878fb607. It published no Plan bundle. This image contains its own RKE2 and kubectl binaries; the new controller/helper releases do not update those embedded binaries. There is no verified Plan activation, node upgrade or Core/init transition.

Workflows #114 is merged and released/promoted as v1.44.0; private Platform #132 is merged. The channel adapter binds workflow_release_sha 832a8a460baab40849493e811bc217ca08c65e53 through snapshot 17784193acf44bac66c98ed3d59cca7d3deac610. Fresh controller deploy 5012 at Platform source c3d50cb074a869131d73da19cf66d4baebea609e used automatic signed release selection and passed scans, 77 required/14 forbidden probes, init 0.73.0 and ownership preflight. UDS archive deployment failed with kubernetes_authorization; session revocation passed. That category does not identify the denied verb/resource, and controller acceptance or partial state remain unverified. The owner found Loki, but its search did not surface the denial. Workflows #115 subsequently merged and was promoted as v1.44.1, binding workflow_release_sha=6999b5326fb4c4ac7a06114c592f0d69a0b4873b through snapshot 2716c0bde2095d7a77986fb98b6daffaac38075c.

Verified 2026-10-17: owner-authorized deploy 5015 used the same signed r7636 controller at the same Platform main source. Scans, original 77 required/14 forbidden probes, init and ownership checks passed. UDS then failed with kubernetes_denial(verb=list, resource=exemptions, api_group=uds.dev, namespace=uds-policy-exemptions); session revocation passed. uds.dev names the Kubernetes API group, not a dev environment. Partial state and controller acceptance remain unverified.

The pinned Zarf readiness informer lists and watches chart resources within their namespaces. Workflows #116 and private Platform #133 prepare only exemption list/watch in that namespace for both routine identities, retaining named/admission-constrained writes and Secret restrictions. Generated probes become 79 required/14 forbidden. Local source comparison confirms only two bootstrap Roles change; controller/Plan runtime contracts remain compatible with existing signed releases. Terraform #16 prepares temporary restoration of the retired bootstrap login for this Role update, followed by retirement after verification. No live access change or new provisioning/deployment dispatch is established by these PRs. Review, workflow release/promotion, paired merges, Terraform plan/apply, new signed bootstrap build/deploy and routine validation must precede controller retry; the Plan image gate remains separate.

Harbor dev provisioning is still unfinished. Existing MinIO administrative provisioning modules need a SeaweedFS compatibility review; an S3-compatible data API does not establish compatible identity/policy APIs. Application stacks still select AIStor and are not migrated by state bootstrap.

For every persistent application, record the source of truth, backup destination/schedule, restore test, reclaim behavior, writer coordination, node-loss behavior, and route/data rollback. The platform storage conventions are the implementation guide. No public documentation should contain actual API keys, Secret data, database URLs or recovery shares.