Gorse Worker Health Probe Fix
Problem
Gorse worker pods remain in not-ready state (0/1 Ready) even though the worker process
is running correctly and processing jobs. This breaks HPA scaling and service health.
Context / Trigger Conditions
- gorse-worker pods show
0/1 Ready but Running status
- Worker logs show normal operation:
"msg":"complete ranking recommendation"
- Pod never transitions to Ready state
- Deployment uses exec-based readiness probe with
pgrep -f gorse-worker
- Using the official
zhenghaoz/gorse-worker Docker image
Root Cause
The zhenghaoz/gorse-worker image is a minimal/distroless image that doesn't include
common utilities like pgrep, ps, or procps. The exec probe silently fails because
the command doesn't exist.
# This probe FAILS silently - pgrep doesn't exist in the container
readinessProbe:
exec:
command: ["pgrep", "-f", "gorse-worker"]
Solution
Replace the pgrep-based probe with a process check using shell built-ins that exist
in the minimal image:
# Use kill -0 to check if PID 1 (main process) is alive
livenessProbe:
exec:
command: ["sh", "-c", "kill -0 1"]
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
exec:
command: ["sh", "-c", "kill -0 1"]
initialDelaySeconds: 10
periodSeconds: 5
The kill -0 <pid> command checks if a process exists without sending any signal.
PID 1 is the main container process (gorse-worker).
Verification
After applying the fix:
kubectl get pods -n gorse | grep worker
# Should show 1/1 Ready
kubectl describe pod <worker-pod> -n gorse | grep -A5 "Readiness:"
# Should show passing readiness checks
Example
Kustomize patch to fix worker probes:
patches:
- target:
kind: Deployment
name: gorse-worker
patch: |-
- op: replace
path: /spec/template/spec/containers/0/livenessProbe
value:
exec:
command: ["sh", "-c", "kill -0 1"]
initialDelaySeconds: 30
periodSeconds: 10
- op: replace
path: /spec/template/spec/containers/0/readinessProbe
value:
exec:
command: ["sh", "-c", "kill -0 1"]
initialDelaySeconds: 10
periodSeconds: 5
Notes
- This issue affects only gorse-worker; gorse-master and gorse-server have HTTP endpoints for probes
- The worker doesn't expose any HTTP endpoints, so exec probes are required
- Alternative: use TCP probe on the gRPC port if the worker exposes one
- This pattern applies to any minimal/distroless container without procps utilities