VENI-AIDOCS
03 / Develop · Deployment

Deployment ​

How the platform ships: Kubernetes with GitOps via ArgoCD, manifests that live with each app, and a branch-and-tag promotion flow from Dev to Prod. There is no kubectl apply in the pipeline — Argo syncs from git.

GitOps with ArgoCD ​

A single ApplicationSet in VENIZIA-AI/infra enumerates every (app, environment) pair and generates one Application per pair, syncing deploy/k8s/overlays/<env> from that app's own repo into namespace veni-<env> with auto-sync, prune and self-heal.

  • Manifests live with the app — deploy/k8s/base + overlays/{dev,uat,prod}, assembled with Kustomize (not Helm). Bumping a release is a one-line images[].newTag change. Images are published to registry.venizia.ai; UIs inject env at runtime rather than baking it in.
  • Secrets are never in git — Vault + the External Secrets Operator materialize them into Kubernetes Secrets the app consumes via envFrom.
  • Migrations run as an initContainer (small apps) or a one-shot Job (large apps, to avoid OOM). All migration SQL is idempotent.

Environments ​

EnvironmentNamespaceHost pattern
Devveni-dev<app>.dev.venizia.ai
UATveni-uat<app>.uat.venizia.ai
Prodveni-prod<app>.veni.ai

Promotion ​

Promotion flows through branches and tags:

feature/* → develop deploys to dev; an rc tag promotes to UAT and stamps the passing commit; tagging that same commit for prod runs release-to-prod.yml, which builds prod images and opens a PR in VENIZIA-AI/infra — merging it lets Argo sync. Prod is gated on the matching UAT-passed commit.

Deploying by hand when CI can't run

The pipeline builds the image, pushes it, and bumps the overlay tag for you. When CI is unavailable — a blocked GitHub Actions account, a runner outage — you do those steps yourself. The trap is that every stage fails silently: skip one and Argo still reports Synced/Healthy while serving the old build.

Do all four, in order, for a dev deploy of <app>:

bash
# 1 · Build from the branch you're deploying — NOT the release branch,
#     which doesn't have your commit yet. Tag = dev-<short-sha of that commit>.
git switch develop && SHA=$(git rev-parse --short=7 HEAD)
docker build --build-arg APP_VERSION=dev-$SHA -f api/Dockerfile -t registry.venizia.ai/<app>-api:dev-$SHA .

# 2 · Verify the image carries your change before pushing — grep the built
#     artifact, don't trust "build succeeded" (a wrong-branch build also succeeds).
docker run --rm --entrypoint sh registry.venizia.ai/<app>-api:dev-$SHA -c 'grep -c <marker> <path-in-image>'

# 3 · Push it — the tag must exist in the registry, or the bump ImagePullBackOffs.
docker push registry.venizia.ai/<app>-api:dev-$SHA

# 4 · Bump the overlay tag on release/dev — a NEW tag, never overwrite a live one
#     (same tag = no manifest change = Argo does nothing).
git switch release/dev && git merge --ff-only develop   # or open the promotion PR
sed -i "s|newTag: dev-.*|newTag: dev-$SHA|" deploy/k8s/overlays/dev/kustomization.yaml
git commit -am "chore(k8s/dev): <app> → dev-$SHA [skip ci]" && git push

Argo can lag a few minutes; force it with kubectl -n argocd annotate application <app>-dev argocd.argoproj.io/refresh=hard --overwrite. Then confirm the live tag and a content marker on the running pod — the deploy isn't done until the pod actually serves the new bytes.

Common snags ​

SymptomFix
A new image didn't roll outNothing auto-detects a new tag — bump images[].newTag in that env's overlay and commit; Argo syncs the one-line change.
Argo says Synced/Healthy but the app serves old contentThe tag in the overlay points at an image whose bytes are stale — commonly a tag that was overwritten, or a manual build off the wrong branch. Argo verifies the tag matches, not that its content is fresh. Use a new tag per build and verify the image before pushing.
Prod won't take a commitProd is gated on the matching UAT-passed commit — promote through UAT first (an rc tag), then tag that commit for prod.
A secret is missing at runtimeIt must exist in Vault and be wired through the External Secrets Operator — envFrom only reads the materialized Secret, it can't invent one.
A large migration OOMsRun it as a one-shot Job, not an initContainer, so it isn't bounded by the pod's memory.

See also ​