Verify and Handoff

What You Will Learn

• How to commit, push, and trigger Flux reconciliation
• How to verify every resource is running correctly
• How to troubleshoot common deployment issues
• What to include in a client handoff

---

Step 1: Register the Tenant in the Root Kustomization

Before committing, you need to tell Flux about the new tenant directory. Open the root kustomization file at clients/kustomization.yaml and add the new tenant:

# clients/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - base/hosting
  - biz.bizdom
  - com.junovy
  - com.junovy.omni
  - com.junovy.open-chat
  - com.junovy.open-cloud
  - com.junovy.open-talk
  - com.junovy.world-famous
  - com.tomofamsterdam
  - com.junovy.radical-faeries        # ← Add this line
  - eu.faerues.riverland
  - studio.drakedesign

Without this line, Flux will not know the new directory exists.

---

Step 2: Commit and Push

Stage all the new files and commit. Remember: all commits must be GPG-signed.

# Navigate to the repo root
cd ~/workspace/junovy/dds-k8s-cluster

# Check what you are about to commit
git status

# Stage the new tenant directory and the updated root kustomization
git add clients/com.junovy.radical-faeries/
git add clients/kustomization.yaml

# Commit with a descriptive message (GPG signing happens automatically)
git commit -m "feat(clients): onboard radical-faeries tenant with BookStack and Rallly"

# Push to the remote
git push origin main

Once you push, Flux will detect the changes on its next reconciliation cycle (typically within a few minutes).

---

Step 3: Trigger Flux Reconciliation

You can wait for Flux to pick up the changes automatically, or force an immediate reconciliation:

# Force Flux to reconcile the junovy-hosting kustomization immediately
flux reconcile kustomization junovy-hosting --with-source

# Watch the reconciliation status
flux get kustomizations --watch

The --with-source flag tells Flux to pull the latest Git changes first, then reconcile. Without it, Flux uses whatever it last pulled.

---

Step 4: Verification Checklist

Work through this checklist from top to bottom. Each check confirms one layer of the deployment. If a check fails, fix it before moving on.

Namespace

# Verify the namespace exists and has the correct labels
kubectl get namespace hst-radical-faeries --show-labels

Expected: namespace exists with labels tier=hosting, customer=radical-faeries.

Secrets

# Verify ExternalSecrets synced successfully
kubectl get externalsecrets -n hst-radical-faeries

Column	Expected Value
NAME	bookstack-secrets, rallly-secrets
STORE	vault
REFRESH INTERVAL	1h
STATUS	SecretSynced

If STATUS shows SecretSyncedError, the Vault path may be wrong or Vault access may be denied.

# Verify the K8s Secrets were created from the ExternalSecrets
kubectl get secrets -n hst-radical-faeries

You should see bookstack-secrets and rallly-secrets in the list.

HelmReleases

# Check HelmRelease status
kubectl get helmreleases -n hst-radical-faeries

Column	Expected Value
NAME	bookstack, rallly
READY	True
STATUS	Release reconciliation succeeded

If READY shows False, check the HelmRelease events for details:

kubectl describe helmrelease bookstack -n hst-radical-faeries

Pods

# Verify all pods are running
kubectl get pods -n hst-radical-faeries

Pod	Expected Status	Notes
bookstack-*	Running	Should show 2/2 containers (app + MySQL sidecar)
rallly-*	Running	Should show 1/1 container

If a pod shows CrashLoopBackOff or Error, check its logs:

# Check BookStack logs
kubectl logs -n hst-radical-faeries deployment/bookstack --all-containers

# Check Rallly logs
kubectl logs -n hst-radical-faeries deployment/rallly

Ingress

# Verify ingress resources exist
kubectl get ingress -n hst-radical-faeries

Column	Expected Value
NAME	bookstack-ingress, rallly-ingress
HOSTS	bookstack.radical-faeries.junovy.com, rallly.radical-faeries.junovy.com

DNS and TLS

# Check if DNS records were created (may take a few minutes)
dig bookstack.radical-faeries.junovy.com +short
dig rallly.radical-faeries.junovy.com +short

# Both should resolve to the cluster's external IP

TLS certificates are provisioned automatically by Traefik's ACME resolver. This can take a few minutes on first deployment.

Browser Test

Open both URLs in your browser:

https://bookstack.radical-faeries.junovy.com
https://rallly.radical-faeries.junovy.com

Check	What to Look For
BookStack loads	You see the BookStack login page
Rallly loads	You see the Rallly home page
HTTPS padlock	The browser shows a secure connection (no certificate warnings)
Default admin login works	BookStack: admin@admin.com / password (change immediately!)

---

Step 5: Troubleshooting

If something is not working, use this table to find the most common issues:

Symptom	Likely Cause	How to Fix
Namespace does not exist	Missing from root kustomization.yaml	Add com.junovy.radical-faeries to clients/kustomization.yaml
ExternalSecret shows SecretSyncedError	Vault path is wrong or Vault token expired	Run vault kv get secret/radical-faeries/bookstack to verify the path; check vault token lookup
K8s Secret is empty	ExternalSecret secretKey does not match Vault property	Compare the ExternalSecret YAML with vault kv get output
HelmRelease shows False	Chart version not found or values syntax error	Run kubectl describe helmrelease <name> -n hst-radical-faeries and read the error message
Pod in CrashLoopBackOff	App cannot connect to database or missing env var	Check logs: kubectl logs -n hst-radical-faeries deployment/<name>
Pod in ImagePullBackOff	ECR credentials expired or image tag wrong	Check ecr-dockerconfig secret exists; verify image tag in ECR
Ingress exists but site unreachable	DNS not propagated yet	Wait 5 minutes; run dig <domain> +short to check
TLS certificate warning	ACME challenge not completed	Wait a few minutes; check Traefik logs in the traefik namespace
BookStack shows 500 error	APP_KEY is missing or malformed	Verify APP_KEY starts with base64: in Vault

---

Step 6: Client Handoff

Once everything is verified, prepare the handoff email or message for the Radical Faeries. Include:

Item	Details
BookStack URL	https://bookstack.radical-faeries.junovy.com
Rallly URL	https://rallly.radical-faeries.junovy.com
BookStack default admin	Username: admin@admin.com, Password: password
First action required	Change the default admin password immediately
Create user accounts	BookStack Settings > Users > Add New User
Support contact	Your team's support email or channel
SLA / uptime expectations	Per the client's plan (basic tier)

Remind the client to change the default BookStack admin password on first login. This is a security requirement, not optional.

---

You Did It

Take a moment to appreciate what you just built. You:

1. Planned a multi-app tenant with two databases and 14 secrets
2. Created a namespace with proper labels and conventions
3. Stored secrets securely in HashiCorp Vault
4. Wrote ExternalSecrets that bridge Vault and Kubernetes
5. Deployed two applications using HelmReleases with Flux CD
6. Set up Ingress resources with automatic TLS and DNS
7. Verified everything end-to-end
8. Prepared a professional client handoff

This is the real workflow. This is what onboarding a Junovy client looks like. Every future client follows the same pattern -- new directory, namespace, secrets, HelmReleases, Ingress, verify, handoff.

---

Key Takeaways

• Always register the new tenant in clients/kustomization.yaml before pushing
• All commits must be GPG-signed -- no exceptions
• Use flux reconcile kustomization junovy-hosting --with-source to trigger immediate deployment
• Verify layer by layer: namespace, secrets, HelmReleases, pods, ingress, DNS, TLS, browser
• The troubleshooting table covers the most common issues you will encounter
• Always change default admin credentials before handing off to the client

What Is Next

Next up: Chapter 11 (Day-to-Day Workflows) where you will learn the ongoing tasks of managing tenants -- scaling, updating, debugging, and responding to alerts.