Skip to content

Troubleshooting

A field guide to what can go wrong, what it looks like, and how to fix it. The failure-mode summary is from SPECIFICATION §14.

Failure modes at a glance

Scenario Behavior you'll see Where to look
No compatible JDK on any ready node Pod stays Pending; event NoCompatibleJDK; scheduler skips nodes → JDK issues
Requested launcher not installed Pod stays Pending; event NoCompatibleLauncher; scheduler skips nodes → launcher issues
OCI artifact missing/unauthorized ImagePull-style failure on the pod → artifact pull
JavaApplication replacement rollout is incomplete Ready=False/Progressing even while old replicas remain ready → rollout status
JVM OOM ExitOnOutOfMemoryError → exit → kubelet restart → OOM
Node provisioning fails Node not labeled ready; condition/event ProvisionFailed → provisioning
No provisioner pod on a node at all Node absent from status.assignedNodes; no DaemonSet pod scheduled → placement
NodeProfile is invalid Ready=False/InvalidProfile; deletion with possible host state remains CleanupBlocked → source policy
NodeProfile deletion does not finish Profile stays with Ready=False/CleanupPending, CleanupTeardown, or CleanupBlocked → deletion
Reinstall after Helm uninstall is refused Brewlet CRDs already exist; install is fresh-install-only → retained CRD review and recovery
Shim crash containerd reports task failure; pod restarts → shim
cgroup v1-only node Provisioner refuses; node not marked ready → provisioning
containerd 1.x node Provisioner refuses; node not marked ready → provisioning

JavaApplication is not Ready during a rollout

Healthy old replicas can keep serving while a replacement image fails to start. The JavaApplication remains Ready=False/Progressing until the current Deployment generation reaches its desired updated, ready, and available counts and extra active replicas are gone.

kubectl get javaapplication <app> -n <namespace> -o yaml
kubectl get deployment <app> -n <namespace> -o yaml
kubectl rollout status deployment/<app> -n <namespace> --timeout=120s
kubectl get pods -n <namespace> -l app=<app>

Compare Deployment metadata.generation with status.observedGeneration, then inspect updatedReplicas, readyReplicas, availableReplicas, and the Progressing condition. An Available=True Deployment or a high JavaApplication readyReplicas count alone does not prove that the replacement rollout succeeded. Check failing new Pods for image-pull, scheduling, probe, or startup errors. Availability may also wait for minReadySeconds.

When HPA is enabled, compare with the Deployment's live spec.replicas. For automation, also require the JavaApplication Ready condition's observedGeneration to match its current generation. See rollout readiness.

Node never becomes ready

Symptom: kubectl get nodes -L brewlet.sh/runtime shows no ready, or brewlet.sh/provision-state=Failed.

# Is the node opted in?
kubectl get node <n> -o jsonpath='{.metadata.labels.brewlet\.sh/provision}{"\n"}'
# Provisioner pod on that node:
kubectl get pods -n brewlet -o wide | grep <n>
kubectl logs -n brewlet <provisioner-pod>
kubectl get events --field-selector reason=ProvisionFailed
# The stable reason code, then the human detail:
kubectl get node <n> -o jsonpath='{.metadata.annotations.brewlet\.sh/provision-error}{"\n"}'
kubectl get node <n> -o jsonpath='{.metadata.annotations.brewlet\.sh/provision-error-message}{"\n"}'

brewlet.sh/provision-error is a stable code you can alert on; the full enumeration is in SPECIFICATION §14.2. brewlet.sh/provision-error-message is free-form operator text and is not a contract — do not parse it.

Common causes & fixes:

  • cgroup v1-only node — the provisioner refuses it (cgroup v2 is required). Move the node to a cgroup v2 kernel/config, or exclude it.
  • containerd 1.x node — the provisioner reports unsupported-containerd-version; upgrade the server to containerd 2.0 or newer. Config file version = 2 remains supported on containerd 2 and is not the containerd server version.
  • Runtime source preflight failed — a required indexed entry is missing, an image is not a canonical SHA-256 digest reference, a path/name is malformed, an entry is duplicated, or a mirror is malformed/unapproved. The provisioner exits before pulling or mounting source content. See NodeProfile source-policy failures.
  • JDK copy-from-image failed — the node can't reach the verified digest-pinned JDK image. Verify the containerd socket mount and image pull access, or mirror the exact digest through an approved destination. See JDK management.
  • can't reach the registry / socket — verify the containerd socket mount and that the node can pull the JDK image.
  • JDK or launcher validation failed — brewlet.sh/provision-error identifies the failed component. Confirm each configured JDK has an executable <javaHome>/bin/java that passes java -version; confirm each launcher source is a regular non-symlink file and the staged copy is executable. Arbitrary launchers are not executed as readiness probes. A failure removes stale runtime and capability labels.
  • containerd config validation failed — in validated mode, Brewlet runs containerd config dump and requires the exact brewlet handler before activation. Inspect the provisioner log and the effective host configuration; Brewlet restores/removes the rejected render automatically.
  • containerd restart or health check failed — validated mode restores the prior primary config or drop-in, restarts containerd, and verifies recovery. Reasons such as restart-failed, containerd-health-check-failed, or runtime-handler-health-check-failed indicate the failed stage; rollback-failed means automated recovery also failed and the node needs immediate operator attention.
  • RBAC — the provisioner needs get/patch on nodes to label them; confirm the ServiceAccount/ClusterRole from the manifest/chart are present.

No provisioner pod is scheduled

Symptom: the profile's DaemonSet exists but has no pods on the node you expected, and kubectl get nodeprofile <p> -o jsonpath='{.status.assignedNodes}' does not count it. This is placement, not provisioning: the provisioner never started, so there is no brewlet.sh/provision-error to read.

# Which nodes may this profile touch?
kubectl get ds brewlet-node-provisioner-<profile> -n brewlet \
  -o jsonpath='{.spec.template.spec.affinity.nodeAffinity}{"\n"}'
kubectl get ds brewlet-node-provisioner-<profile> -n brewlet \
  -o jsonpath='{.spec.template.spec.tolerations}{"\n"}'
# Why was this node not eligible?
kubectl get node <n> --show-labels
kubectl get node <n> -o jsonpath='{.spec.taints}{"\n"}'

Common causes & fixes:

  • It is a control-plane node. Every profile requires node-role.kubernetes.io/control-plane and node-role.kubernetes.io/master to be absent, because the provisioner is privileged. This bites on kind and Docker Desktop, whose single node carries the label without the taint — so nothing at all is provisioned. Set spec.nodePool.includeControlPlane: true (Helm: provisioner.includeControlPlane=true) if you really intend it.
  • The node is tainted and the profile says nothing about it. Nothing is tolerated implicitly. Add the matching entry to spec.tolerations (Helm: provisioner.tolerations); every entry must name a key.
  • The node is not in the profile's pool. Compare the node's label for the resolved pool key (status.resolvedPoolKey) against spec.nodePool.names. A profile whose pool matches nothing reports Ready=False with reason EmptyPool.

NodeProfile source-policy failures

The admission webhook validates sources and mirrors for immediate feedback, but the reconciler repeats the same checks. A stored invalid profile therefore fails closed even if admission is disabled or bypassed: its provisioner DaemonSet is withheld/deleted and only that profile's owned node advertisements are removed.

Repair an invalid profile before requesting deletion. If it is already deleting, the operator stops its provisioner and cleanup workers, waits for them to terminate, and withdraws its node advertisements. It does not run host cleanup under an invalid policy. If recorded or claimed host state may remain, both the cleanup finalizer and node ownership claims are retained, with Ready=False/CleanupBlocked, even after all workers are gone.

Repair the profile's spec, source/mirror policy, or pool conflict to resume cleanup of its recorded targets. Normal finalization requires successful host cleanup and worker teardown. Missing ownership or cleanup-policy records must be restored; their absence is not proof that a node was cleaned.

Only an invalid profile proven to be unprovisioned may finalize without host cleanup, after fresh node and worker checks. Do not remove finalizers, ownership labels, or status records to force deletion.

kubectl get nodeprofile <name> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" message="}{.message}{"\n"}{end}'
kubectl describe nodeprofile <name>
kubectl logs -n brewlet deploy/brewlet-operator

Correct the condition message:

  • replace every tagged spec.jdks[].source.image with a fully qualified, tagless repository@sha256:<64 lowercase hex> reference;
  • supply source.image and source.javaHome for every JDK;
  • supply name, source.image, and source.path for every optional launcher;
  • add the exact mirror destination host, including any explicit port, to security.allowedSourceMirrorHosts, or remove the mapping; and
  • remove duplicate/self mirror mappings and conflicting named-pool ownership.

To inspect a provisioner source declaration directly:

/opt/brewlet-dist/brewlet-source-policy validate-ref \
  --image docker.io/library/azul-zulu@sha256:2e230d906cffcc7bb7360ce82836f2ff0e0be74a1d5ebaf929e4e6ac99d61bf2
/opt/brewlet-dist/brewlet-source-policy validate-path \
  --path /usr/lib/jvm/zulu21

An air-gapped mirror must preserve the referenced OCI manifest/index bytes and digest.

NodeProfile deletion does not finish

A deleted NodeProfile stays until the operator has stopped its provisioner, resolved every recorded target by host cleanup or accepted external-retirement evidence, and torn down the cleanup workers. Follow progress per node:

brewlet k8s profile delete <name> --wait --wait-timeout 15m
  • CleanupPending with nodes stuck in pending or cleaning. Check the cleanup worker Pods (kubectl get pods -n brewlet -l app=brewlet-cleanup,brewlet.sh/nodeprofile=<name> -o wide) for image-pull or scheduling problems, and make sure every target node is Ready and the operator is running.
  • CleanupTeardown. Targets were resolved; the operator is waiting for the cleanup workers to terminate.
  • CleanupBlocked. The profile's spec, source/mirror policy, or pool ownership is invalid, a target node is unavailable or was replaced, or a competing or unfenced worker was found. Read the condition message, then repair the cause as described in source-policy failures or blocked cleanup recovery; repairing a deleting profile resumes cleanup.

For an AKS or other managed-node replacement, Kubernetes readiness of the new Nodes and a successful pool operation do not prove retirement of the original hosts. Preserve the original UID ledger, verify permanent destruction of each original instance, and follow external-retirement recovery. The same-name replacement is not a cleanup target for the old UID. An unreachable original Node still requires connectivity repair, not an attestation bypass. For a non-deleting profile, RetirementPending=True and status.detachedRetirements retain missing-host cleanup obligations without blocking distinct replacement provisioning. A healthy Ready condition does not mean this history is resolved. Deletion and Helm uninstall still wait for every obligation; inspect the detached records and retained evidence rather than only the current Node list. Known providerID/systemUUID conflicts exclude returning hosts from fresh claims even under another name, and the operator withdraws any Brewlet runtime/capability labels from such Nodes regardless of their ownership metadata. brewlet k8s profile list shows the unresolved count as PENDING RETIREMENTS; the brewlet_nodeprofile_detached_retirements metric reports the same value for alerting.

Never remove the node.brewlet.sh/cleanup finalizer, ownership labels, or status to force deletion: that leaves runtimes and containerd changes on nodes with no record of them.

Pod is Pending with NoCompatibleJDK

Symptom: the pod won't schedule; a NoCompatibleJDK event is recorded.

The pod requested a JDK (spec.jvm.version via JavaApplication, or raw pod annotation brewlet.sh/jdk) that no ready node advertises.

# What does the pod ask for?
kubectl get pod <pod> -o jsonpath='{.metadata.annotations.brewlet\.sh/jdk}{"\n"}'
# What do nodes offer?
kubectl get nodes -o custom-columns=NODE:.metadata.name,JDKS:.metadata.annotations.brewlet\\.sh/jdks

Fix: either request a JDK the fleet has, or add the JDK to the inventory:

provisioner:
  jdks:
    - distribution: temurin
      feature: 21
      source:
        image: docker.io/library/eclipse-temurin@sha256:<reviewed-digest>
        javaHome: /opt/java/openjdk

Apply the values with helm upgrade ... -f values.yaml, wait for brewlet.sh/jdks to update, and re-deploy.

Pod is Pending with NoCompatibleLauncher

Same as above, for launchers. The pod requested spec.jvm.launcher or brewlet.sh/launcher: <name> that no ready node has.

kubectl get nodes -o custom-columns=NODE:.metadata.name,LAUNCHERS:.metadata.annotations.brewlet\\.sh/launchers

Fix: add a structured launcher entry with name, digest-pinned source.image, and absolute source.path, then re-provision; or drop the request to use the vanilla java launcher (omit the annotation). See Launchers.

Pod is Pending after a "no ready node currently provides" warning

Symptom: creating the pod printed Warning: brewlet: no ready node currently provides ..., and the pod stays Pending with a FailedScheduling event.

Admission accepted the request because a valid NodeProfile declares it (or, for brewlet.sh/arch, because architecture never denies), but no ready node offers it yet. The pod waits for capacity:

  • Autoscaled pool at zero: check that the autoscaler is enabled for the pool and its node-template labels match the profile (see Capability labels and autoscaling). An AKS-managed Cluster Autoscaler pool cannot scale from zero for Brewlet requests (NotTriggerScaleUp ... didn't match Pod's node affinity/selector); keep --min-count 1 or use Node Auto Provisioning with the Brewlet startup taint.
  • Node stuck with the startup taint: the profile must tolerate startup-taint.cluster-autoscaler.kubernetes.io/brewlet, and the provisioner removes it only after publishing readiness; check the provisioner logs.
  • Fixed-size pool: add a node to a pool the named profile targets, or check the profile's Ready condition and provisioner logs.
  • Architecture: no pool provides that kubernetes.io/arch; add one or drop spec.arch if the JAR is architecture-neutral.

ImagePull-style failure

Symptom: the pod can't fetch the OCI image.

  • Wrong ref / not pushed — verify the image exists: oras manifest fetch <ref> (or brewlet inspect <ref> for the local layout).
  • Unauthorized — add/verify imagePullSecrets (or artifact.pullSecrets in the JavaApplication).
  • Not a Brewlet artifact — confirm it was pushed with the brewlet media types (Reference), not as a plain image.

Pod restarts / OOMKilled

Symptom: the pod restarts under load or is OOMKilled.

  • The heap likely has no non-heap headroom. Set -XX:MaxRAMPercentage=75.0 (not 100%) so Metaspace/threads/code-cache/direct buffers fit under limits.memory. See Resource requests, limits & JVM tuning.
  • Add -XX:+ExitOnOutOfMemoryError so an OOM is a clean restart, not a hang.
  • Or switch to jaz, which sizes the heap from the cgroup automatically.
  • Raise limits.memory if the workload genuinely needs more.

Task / shim failures

Symptom: containerd reports a task failure; the pod restarts or won't start.

kubectl describe pod <pod>                 # events from kubelet/containerd
# On the node:
journalctl -u containerd | grep -i brewlet
ls -l /opt/brewlet/bin/containerd-shim-brewlet-v2
grep -A2 runtimes.brewlet /etc/containerd/config.toml
test ! -f /etc/containerd/config.toml.d/99-brewlet.toml ||
  cat /etc/containerd/config.toml.d/99-brewlet.toml
containerd --config /etc/containerd/config.toml config dump | grep -A4 runtimes.brewlet
  • Shim not on containerd's PATH — confirm it's installed to /usr/local/bin and the effective config dump contains the runtimes.brewlet block. In validated mode the provisioner restarts and probes the handler itself; use its logs and brewlet.sh/provision-error rather than manually replacing config.
  • JDK root missing — confirm /opt/brewlet/jdks/<dist>-<feature>/bin/java exists on the node.
  • Arch mismatch — the shim is arch-specific; ensure the provisioner image matches the node arch (the build compiles it per-arch).

Webhook / admission problems

Symptom: compatibility hints aren't stamped, or scheduling isn't steered.

  • With admission.failurePolicy: Ignore (default), a webhook outage silently lets pods through without stamping/steering — check the webhook is healthy. If the shim still cannot resolve the workload image from containerd metadata, it fails closed rather than guessing: 
    kubectl get pods -n brewlet -l app=brewlet-admission
kubectl logs -n brewlet -l app=brewlet-admission
  • caBundle/cert mismatch in the default mode should self-heal after helm upgrade, which rotates the generated cert and caBundle together. With cert-manager enabled, inspect Certificate, CertificateRequest, and issuer readiness plus the cert-manager.io/inject-ca-from annotations.
  • Webhook timeouts after enabling NetworkPolicies usually mean networkPolicy.admission.apiServerCIDRs and networkPolicy.admission.ingressFrom do not match the sources used by the control plane. Temporarily disable the policies, confirm those sources with your Kubernetes provider/CNI documentation, and re-enable with corrected peers. On AKS, an error such as proxy error from localhost:9443 while dialing <pod-ip>:9443, code 504 means the konnectivity-agent pods in kube-system cannot reach the webhook. CNIs such as Azure CNI powered by Cilium never match those pods with CIDRs, even 0.0.0.0/0, so add them as a namespace+pod selector in networkPolicy.admission.ingressFrom.
  • helm upgrade on AKS fails with conflict with "admissionsenforcer" on .webhooks[name="pods.brewlet.sh"].namespaceSelector: AKS's Admissions Enforcer co-owns webhook selectors, and Helm 4 upgrades use server-side apply. Current charts render the enforcer's expressions through admission.platformNamespaceExclusions, so upgrading to them resolves it. If AKS injects different expressions, copy them into that value; --force-conflicts is a one-off fallback. See the chart README's "AKS Admissions Enforcer" section.
  • Control-plane pods fail readiness after enabling NetworkPolicies when networkPolicy.healthProbes.ingressFrom does not match the source addresses used by kubelets. Configure the cluster's node CIDRs or equivalent trusted peers so port 8081 remains reachable.
  • Metrics scrape timeouts with NetworkPolicies enabled mean the Prometheus namespace/pod or source CIDR does not match networkPolicy.metrics.ingressFrom, or the cluster CNI does not enforce NetworkPolicy as expected.
  • Non-brewlet pods are intentionally passed through untouched.

Local development issues

  • The harness cannot find a component directory — run it from a complete microsoft/brewlet monorepo checkout. For external component checkouts, set BREWLET_CORE_DIR and BREWLET_KUBERNETES_DIR.
  • Tier 2 cannot build a fixture — check JAVA_HOME points at a full JDK 21+.
  • Core or Kubernetes build fails — Go 1.26+ is required.
  • Tier 3 skips or fails before runc — Docker must be installed and reachable; the tier runs a privileged Linux container.
  • Non-Linux dev host — only the portable bundle-assembly core builds locally; use integration-test tier 3 to exercise the Linux/runc path. See Getting started.

Still stuck?