HOMELAB / THE SYSTEMEdit source

Platform components and recorded state

UDS version update, 2026-10-14 (owner confirmation): installed UDS Core is v1.13.1 and the newly installed operator UDS CLI is v0.39.0. Other component records retain their original observation dates. This update is not a fresh cluster-health audit; check the live cluster and the owning repository before maintenance.

ComponentRecorded baselineState / evidence limitPrivate owner reference
RKE2Kubernetes v1.35.7+rke2r1; four nodesEstablished at the UDS readiness gate; current health needs a new checkplatform bootstrap
UDS Corev1.13.1Installed version confirmed by the owner on 2026-10-14; tenant/admin gateways remain established. Capture exact installed flavor and digests before maintenanceOwner confirmation, 2026-10-14; platform Core
Operator UDS CLIv0.39.0Owner reported installation on 2026-10-14; this does not change independently pinned Actions toolingOwner confirmation, 2026-10-14
Zarf initv0.73.0Owner supplied authenticated package-list output on 2026-10-16; v0.86.0 is the next Core 1.14.1 target, not the installed baselineOwner package-list output, 2026-10-16
MetalLB0.16.1, Layer 2Installed; disposable VIP test passedplatform MetalLB
TrueNAS CSIOfficial csi.truenas.ioNFS and iSCSI disposable provisioning tested; exact installed artifact pin needs reconciliationplatform CSI
OpenBao and ESOOpenBao 2.6.2 / ESO 2.10.0 package baselineLater operator records report OpenBao/ESO deployed and managed credential issuance working; independent recovery coverage must be checked against current runbooksplatform secrets
cert-managerv1.21.1 package baselineControllers deployed; staging and production issuance/trust reported for dev and tenant/admin certificates. Future leaf renewal has not been observedcertificate record
Forgejo Linux runnersKEDA 2.20.2, worker bundle 0.1.6Run 4865 deployed signed forgejo-runners-bundle-v0.1.6-r6507 on 2026-10-10 and verified all three in-house image references in both pools. Four ci slots and one deploy slot remain configured; fresh job/artifact, dev deployment and cleanup/scale-to-zero acceptance remain pendingPrivate platform runs 4863/4864/4865; runner operations
Renovaterenovate 0.1.0 standalone UDS bundle; patched Renovate 44.125.1Deployed and schedule enabled 2026-10-04. Run 93 installed signed bundle renovate-bundle-v0.1.0-r6133 with ESO readiness and UDS egress checks. The owner accepted dry runs and onboarding; run 103 resumed the schedule using signed image release renovate-patched-v2026.10.03-r6132. Bot onboarding and a Dependency Dashboard are observed. The bundle contains only Renovate and is independent of Core. Detailed coverage and operating checks continue in renovate issue #15.Private Homelab/renovate runs 93/95/96/98/103, onboarding PR #13 and dashboard #14; platform issue #62 owner comments dated 2026-10-04; owning docs/renovate.md runbook
Media GPURTX 3080 Ti, host driver/toolkitOwner-provided 2026-09-16 inventory had no Kubernetes GPU allocation; later platform #3 records scheduling/CUDA validation. Recheck before app acceptanceGPU runbook

UDS versions

The current owner-confirmed baseline is Core v1.13.1 with operator CLI v0.39.0 as of 2026-10-14. Earlier Core 1.11.1 and CLI 0.35.1/0.38.0 observations remain historical evidence. Platform #128 merged the Core Actions selector for CLI 0.39.0 on 2026-10-15; other pipelines retain their own reviewed caller/tool-image pins. Operator installation and pipeline source changes are separate evidence, and neither proves a new Core deployment.

The owner's authenticated uds zarf package list on 2026-10-16 reports installed init v0.73.0, with zarf-injector and zarf-seed-registry. This corrects the earlier v0.86.0 installed-init statement. The owner clarified that v0.86.0 is the target with Core 1.14.1, and selected preparation of an explicit init upgrade in that stage. The separately observed standalone Zarf CLI 0.83.0 remains historical; CLI and init package versions are independent artifacts.

Priority change, 2026-10-14 (owner direction): update the cluster first, then finish Harbor. Harbor setup is no longer a prerequisite. RKE2 upgrade packaging and Core 1.14.1-registry1 with init v0.86.0 were merged together in platform #128 on 2026-10-15; shared review does not require simultaneous deployment. The Core caller uses CLI 0.39.0. Reusable RKE2 bootstrap/controller/Plan delivery was first published in workflows v1.40.0 and remains available through promoted v1.44.1 as verified on 2026-10-17. Bootstrap deployment passed in run 4992 on 2026-10-16. Controller/Plan delivery and node/Core upgrades remain pending; the bounded evidence is recorded below.

The owner selected reusable Terraform for repeatable OpenBao policy/JWT/issuance-role onboarding on 2026-10-15. Terraform PRs #11/#12/#13 are merged, state bootstrap run 62 passed its bounded checks, and the owner completed the central runner's one-time authorization. Plan 72 and specifically approved apply 75 created/converged the eight original OpenBao resources. Private platform build run 4972 published signed bootstrap release rke2-bootstrap-bundle-v0.1.0-r7501; deployment run 4974 obtained its bootstrap credential and passed five permission probes, then stopped before bundle deployment with a cleanup HTTP 403 that hid the original preflight error. Terraform #14 and workflows #109 merged the minimal self-revocation and error-preservation corrections. Promotion 838 selected v1.42.2; owner-authorized plan 80 and apply 81 updated only the three RKE2 policies and the post-apply plan had no managed-resource changes. See the bounded evidence.

Bootstrap retry 4975 on 2026-10-15 passed signature/source, credential and permission checks, and session revocation succeeded. It stopped before bundle deployment at the old installed-init 0.86.0 guard. The owner's package-list output resolves the mismatch: installed init is 0.73.0. Workflows #110 merged and publication 852 published v1.42.3 with exact Core/init profiles and safe version diagnostics. Promotion 854 selected it on stable-v1, with passing canary 7545. Platform #130 selects Core 1.13.1/init 0.73.0 for the first RKE2 upgrade. The owner merged Platform #130 on 2026-10-16 after PorteGumi approval and all five checks passed (4982/4983/4984). Build 4988 passed and published signed rke2-bootstrap-bundle-v0.1.0-r7557 from merged main under v1.42.3. Owner-authorized deployment 4989 used that exact tag with adoption disabled. It verified init expected=0.73.0; observed=0.73.0, then stopped before adoption or bundle deployment because an unowned retained RKE2 resource needs explicit scoped adoption. Session revocation succeeded. At that checkpoint, retained-resource review was still required and attempts stopped as authorized. The subsequent review and successful run 4992 below resolve that blocker without another Terraform apply. See the bounded evidence.

The exact RKE2 workflow_call claims from private platform inspections 4969/4970/4971 at source 566e28aab888d547773b470d67c44c268912b458 merged in Terraform PR #13. The reusable-event fix from workflows #106 was promoted in v1.41.1 and is retained in v1.42.3. Bootstrap-role login/issuance and revocation passed in runs 4989/4992; Kubernetes bootstrap passed in run 4992. Routine-role issuance passed in runs 4993/4994 on 2026-10-16; Terraform apply 87 retired the temporary bootstrap login/policy and converged. Concurrent Terraform locking, independent recovery and live upgrades remain pending. Detailed steps stay in the private owning repositories; the public reusable contract describes the delivery boundary. Platform owns Kubernetes RBAC, signed delivery and upgrade acceptance.

Bootstrap deployed and verified, 2026-10-16: read-only inspection identified five matching unowned cluster-scoped leftovers and 28 absent bootstrap objects; the released ownership checker passed for all 33 with mutation disabled. After the owner resumed the task, deployment 4992 passed on main with the same signed r7557 release and scoped adoption enabled. It created the namespace, adopted the matching objects, deployed the signed archive and passed admission/ownership/target postflight. Init stayed 0.73.0; session revocation passed. Work stopped after bootstrap completion. Controller/Plan delivery and node/Core upgrades remain pending. The storage evidence lists exact objects and evidence limits.

Controller delivery update, 2026-10-16: the earlier build 4995 image gate is resolved for the controller/helper inputs. Workflows #111 and private Platform #131 are merged. Build 5004 published signed rke2-controller-bundle-v0.1.0-r7636 from Platform main 398fa282dbb09456385ae3b7b358d01e6c44bfb6. Deploy 5005 stopped at the explicit adoption gate; deploy 5007 passed signature/source, fresh image scans (zero blocking findings), permission and ownership checks, then failed during UDS archive deployment. Its raw error was withheld and not retained, so the root cause, partial deployment and controller acceptance remain unverified. Plan build 5006 separately stopped on 28 blocking findings in rancher/rke2-upgrade:v1.36.5-rke2r1 and published no Plan bundle. The new controller/helper images do not replace binaries embedded in that upgrade image. Node/Core/init upgrades remain pending. Workflows #114 is merged and released/promoted in v1.44.0; private Platform #132 is merged. Fresh deploy 5012 used automatic signed release selection and passed scans, all 77 required/14 forbidden permission probes, init and ownership preflight. UDS archive deployment failed with kubernetes_authorization; session revocation passed. The denied operation, partial deployment and controller acceptance remain unverified. Workflows #115 subsequently merged and was promoted in v1.44.1; it grants no permissions. Fresh controller deploy 5015 on 2026-10-17 passed the same checks and identified the denial: list uds.dev/exemptions in uds-policy-exemptions. Session revocation passed. The pinned readiness informer also needs watch. Workflows #116, private Platform #133 and Terraform #16 prepare only the namespaced read correction and temporary bootstrap authorization. A new signed bootstrap deployment, routine credential validation, temporary-login retirement and controller retry remain pending. No node/Core/init upgrade is established. See bounded evidence.

Owner-selected order, 2026-10-16: RKE2 1.36.5+rke2r1; Core 1.14.1 with init 0.86.0; Core 1.15.0; RKE2 1.37.1+rke2r1; RKE2 auto-update setup; Harbor deployment; image migration. Platform #130 merged the first RKE2 profile and Core 1.14.1/init transition source; later versions and automation remain planned. The existing Core job excluded init, so the merged correction includes init,gitchauth-config,core from the exact signed 1.14.1 archive and verifies init afterward. Private platform #123 tracks the upgrade sequence. No upgrade or automation activation occurred in this documentation update.

Runners and credentials

The global disposable Linux runner pools use the standard ci and deploy labels across Sager, OptiPlex and Media. The Windows Hyper-V runner stays separate. Worker bundle 0.1.6 was deployed through UDS on 2026-10-10 with four CI slots and one deployment slot. Postflight verifies both pools' in-house image references; fresh workload and cleanup acceptance remains pending as detailed below. Application onboarding remains subject to its own switch and validation.

Image publication update, 2026-10-07 (owner-supplied release; private Homelab/base-images #2/#28 and Homelab/platform #56). The signed Ubuntu foundation base-ubuntu-v2026.10.06-r6354 is published. The tool definitions were consolidated to tools/main-runner in base-images PR #30 after their requirements proved identical. Source remains in base-images; tool-main-runner-v2026.10.07-r6415 is published in homelab/tool-images/main-runner with passing source/full-image gates and clean fresh ClamAV evidence. Hosted compatibility, reviewed platform delivery and live adoption are pending before both existing ci/deploy labels can use the same published digest. The labels and security pools remain separate. The minimal foundation alone does not contain the tools those jobs require. See the accepted digest and adoption procedure.

Worker adoption source merged, 2026-10-07 (private Homelab/platform PR #99, source b46c3b74c817fc89aa871582032ede51582ad55e). Both Kubernetes pool mappings and the validation override select the published in-house digest in the merged source; worker package and UDS bundle versions advance together to 0.1.5. The PR adds the owning protected manual signed worker delivery caller and a scoped identity proposal. Local repository rendering and 23 evidence/identity regressions passed. The initial signed deployment has no earlier signed worker release. Platform PR #101 (fe2cf185bd7ebae05464c8424ee62b45de5038e4), merged on 2026-10-08 as dc6c562acc6125a946126e205c627cc6dad108f0, makes rollback_release optional and validates any selected baseline; retain the first accepted bundle for future recovery. Its 36 offline regressions passed. Run 4808 failed at job-image signature verification before bundle creation. The correction supplies the operator's existing SIGNING_PUBLIC_KEY for both platform and base-images receipts and reuses CORE_DEPLOY_ACTORS. Run 4821 on 2026-10-08 verified the retained job-image signature and passed the source scan, then stopped before service-image scans because the caller omitted the registry hostname. Platform PR #102, source 60617df303ea16559f29e90d3d9e2018126326d4, prepares the existing REGISTRY_HOST mapping to the shared scanner; all 37 offline regressions passed, including the actual pinned scanner's credential contract with mocked commands. Hosted image scans and worker bundle publication remain pending after merge. Its k9s OpenBao setup prepares public role/policy files and uses the existing verified issuer; it does not install Kubernetes RBAC. Hosted validation, privileged identity bootstrap, initial registry pull and live adoption remain pending. No cluster change or new live worker version is established.

Worker service-image gate, 2026-10-08 (private platform run 4826; base-images #32). The registry-host correction is merged. The run verified main-runner's signed evidence, then rejected the separately packaged upstream services: 28 runner vulnerability occurrences and 157 Docker-in-Docker vulnerability occurrences plus one configuration finding. No worker bundle was published or deployed by that run. The owner selected two additional in-house Ubuntu-based tool images, forgejo-runner and docker-engine, under base-images tools/ and the existing homelab/tool-images/ registry namespace. Their definitions and updated source/dependency locks are prepared for review. Full image scans, signed publication, exact-digest platform adoption and both-pool runtime acceptance remain pending. These services have different roles from the published main-runner job environment.

Service images published, 2026-10-08 (owner-supplied registry links and base-images release attachments). Base-images PR #33 is merged at 4ccd604d4071467a2d3060aabadde5c54eb0c01b. Both distinct in-house services now have published releases:

RolePublished signed evidenceExact digest
One-job runner servicetool-forgejo-runner-v2026.10.09-r6495sha256:a74f6f8a465a14f83c93ca54156523f1d6dc2d43c94579d842c0b2f9140321b7
Private Docker engine/runtimetool-docker-engine-v2026.10.09-r6496sha256:3820b599e9574671a610aaa818f81b2b7257345e6874acc5936f647b9ed3c14f

The release reports record passing complete-image gates and clean ClamAV results. Docker engine retains two owner-approved GO-2026-5932 occurrences with their original November 6 expiry; other lower-severity findings remain visible. Platform PR #103, merged at 7547d4a2f9fe299a96ae27d98258ac96f923c0bb, adds both service pins and their signed release verification in worker package/bundle 0.1.6, retaining the published main-runner job digest. Its consuming pipeline verifies signatures with the existing trusted public key and re-evaluates current mitigation approvals. Platform bundle publication, scoped identity/bootstrap verification and fresh both-pool runtime acceptance remain pending in private platform #56 and base-images #32. The release tags' October 9 date does not establish a different local publication date or any live deployment.

Runner identity blocker confirmed, 2026-10-08 (private platform run 4835 and the operator's in-pod OpenBao diagnostic; platform #104). The workflow passed OpenBao JWT login, then failed to issue its Kubernetes credential. Kubernetes denied the existing openbao/openbao service account permission to create serviceaccounts/token for platform-provisioning/forgejo-runner-deployer. The OpenBao engine role exists; its authorization does not install the Kubernetes permission binding.

Platform PR #105, source b09aff22c7674665c092c701c5faf31bd1ab3531, prepares an image-free runner identity UDS bundle and two operations in the existing Core delivery workflow. It reuses the established Core deployment authority to install and verify only the runner service account and scoped RBAC. Twenty bootstrap, 44 worker delivery and 74 Core delivery offline regressions passed. Review/merge, signed identity-bundle publication, operator dispatch and fresh worker credential validation remain pending. This is prepared source, not evidence that the missing permission or new worker images are deployed. The sequence is recorded in delivery and operations.

Identity deployed and verified, 2026-10-09 (private platform run 4855, source a8fb5f89789ee3668cdebca5228db75214908eaf; deployment.json). The operator deployed signed release forgejo-runner-identity-bundle-v0.1.0-r6544 through the existing Core UDS caller. Signature, archive, current-policy and target checks passed; postflight verified all 13 service-account/RBAC objects, including the OpenBao TokenRequest issuer binding. The temporary Core OpenBao session was revoked. Fresh worker credential issuance and its required/forbidden permission probes remain pending in platform #104. Worker release forgejo-runners-bundle-v0.1.6-r6507 is already published from reviewed source 7547d4a2f9fe299a96ae27d98258ac96f923c0bb; its retained receipt and inventory select all three in-house images. The later identity bootstrap does not change that worker bundle. Reuse the exact release after credential validation, subject to deployment's signature, hash and current-policy checks. Image rollout and both-pool runtime acceptance remain pending in platform #56/base-images #32.

Credential validation passed; deployment blocked, 2026-10-09 (private platform runs 4856/4857, source a8fb5f89789ee3668cdebca5228db75214908eaf). Fresh runner issuance and the original permission contract passed. Deployment verified worker release forgejo-runners-bundle-v0.1.6-r6507 and all three image receipts, pushed its two service images, then failed in Helm post-rendering: Zarf could not patch the named private-registry Secret in uds-policy-exemptions. Worker rollout and postflight remain unverified. Platform prepares identity bundle 0.1.1 with a separate Role/RoleBinding for only that named-secret patch, preserving the 13 installed identity objects and adding required/forbidden permission probes. After review, deploy a new signed identity release and validate the revised contract, then reuse worker release forgejo-runners-bundle-v0.1.6-r6507. The worker bundle needs no rebuild for this permission correction; no additional OpenBao setup is required. Platform #104 remains open for the correction and revised credential acceptance.

In-house worker bundle deployed, 2026-10-10 (private platform runs 4863/4864/4865; retained identity and worker deployment.json). Run 4863 deployed signed identity release forgejo-runner-identity-bundle-v0.1.1-r6562 from reviewed source 615a105e05b66b58953fcf248fbd8d69f7a80184 and verified all 15 account/RBAC objects. Run 4864 passed fresh worker credential validation with all 188 required and 22 forbidden permission checks, including the named registry-secret patch and its scoped denials. Platform #104 is closed and its Roadmap card is Done.

Run 4865 verified and deployed the existing signed worker release forgejo-runners-bundle-v0.1.6-r6507, built from 7547d4a2f9fe299a96ae27d98258ac96f923c0bb. Postflight at 00:57 CDT confirms both CI/deploy pools select the published main-runner digest recorded in delivery and operations and the two service digests above. This supersedes the earlier identity and worker deployment blockers. The existing worker bundle required no rebuild.

The original worker evidence left three runtime checks open: a fresh CI job with artifact retrieval, a fresh dev deployment, and worker cleanup and scale-to-zero. On 2026-10-10, the operator completed the checklist and closed base-images #32; its Roadmap card is Done. This records operator acceptance, without claiming additional runtime metrics from the earlier deployment record. ComfyUI integration/timing follow-up stays with its owning issue, Comfy-UI #1; the initial base-image meta milestone is closed/Done at the owner's direction. Retain the signed worker release as the accepted deployment baseline.

Application base publication accepted, 2026-10-10 (owner confirmation, published registry packages and receipt-bound source/full-image decisions; private Homelab/base-images PR #35, issues #3/#4, merged source 57b92a1c30c0c9541b76ff3fe241250b43701764). With the worker bundle deployed, Python 3.15.0 and Node 26.11.1 base definitions are merged on the accepted Ubuntu foundation. Both live under images/ and publish separately from the runner tools. The owner requires the newest stable upstream runtimes, including Node's stable Current series; that selection rule is recorded in base-images AGENTS.md. Forty-seven local regressions and source checks passed. Accepted releases are base-python-v2026.10.10-r6609 and base-node-v2026.10.10-r6608, including Node's libatomic1 correction. Their registry digests agree with the release receipts, source/full-image decisions pass, and fresh complete-image ClamAV scans are clean with no mitigations. Signed receipts and SBOMs are retained. Base-images #3/#4 are closed and their Roadmap cards are Done; Ubuntu #2 remains closed/Done. These publications supersede the earlier candidate and startup blockers. The owner accepted closure of platform #56 and moved the meta card to Done for the completed initial image-family milestone. Worker acceptance issue base-images #32 was subsequently closed by the operator; ComfyUI follow-up stays with Comfy-UI #1; this does not claim new workload or timing evidence. See the exact release/digest evidence in delivery and operations.

Runner startup recovered on rerun, 2026-10-10 (private base-images run 142, attempt 2; platform #55). Setup completed in 47s using the same main-runner digest; the pull log still has an empty username and the masked git.***.com hostname. The container started and the definition checks passed. The unmerged diagnostic proposal in platform PR #110 was not needed for this recovery. The original pull failure remains unexplained; this successful job does not establish image build/scan, artifact retrieval, dev deployment or cleanup acceptance. Platform #55 returns to Backlog for the intermittent-pull history; worker acceptance issue base-images #32 was subsequently closed by the operator. Python/Node publication was subsequently accepted as recorded above.

OpenBao/ESO supplies pool credentials; global queue polling uses a separate read-only instance-admin permission. Forgejo OIDC to OpenBao and short-lived Kubernetes credentials were tested in Actual runs 95/96, including permission probes and session revocation. This is not a standing cluster-admin kubeconfig for every app.

Certificates and secret delivery

The selected chain is Cloudflare DNS-01 → cert-manager Certificate → Istio gateway Secret. Preserve each gateway's existing UDS ownership when renewing or adopting TLS; do not create a competing writer for gateway-tls. The dev certificate and tenant/admin issuance have recorded trusted-chain checks, but first issuance is not a future renewal test. The admin gateway and OpenBao UI stay on the internal VIP.

Runtime values belong in OpenBao, with namespace-scoped ESO consumption. CI signing, registry and limited release-read credentials belong in Forgejo Actions secrets. The platform secrets runbook holds the exact non-secret contracts and adoption sequence. Never commit a TrueNAS CSI bootstrap Secret file, exported UDS TLS key, kubeconfig, OpenBao unseal share, or live Secret data.

Installation and ownership

UDS packages/bundles are the intended deployment units. The platform repository owns shared resources and their detailed runbooks; application repositories own app resources. Existing raw manifests are transitional comparison or migration inputs. Adopt a live resource into Helm/Zarf only after comparing current values and field ownership; do not delete serving shared gateways to force a release.