---

Your app is fine, but your database hiccuped for exactly three seconds. In response, Kubernetes panicked and killed every single pod in your cluster, turning a minor blip into a massive cascading failure. If this sounds familiar, you’ve just discovered why improperly configured health checks are a developer’s worst nightmare.

Welcome back to Part 4 of our Docker and Kubernetes series! In Part 3, we covered how to expose your application to the world using Services. However, simply running a pod doesn’t mean it’s ready to handle traffic—or that it’s healthy enough to stay alive. To maintain true high availability, Kubernetes needs to understand the actual state of your application, not just whether the container process is running.

What You’ll Learn

• The critical differences between readiness, liveness, and startup probes.
• How the Kubernetes control loop uses these probes to manage your application’s lifecycle.
• A deep dive into production-ready health endpoints using Node.js and Express.
• Why separating your health checks is essential for cluster stability.
• Common pitfalls that lead to disastrous cascading failures and how to avoid them.

Conceptual Overview: The Kitchen Analogy

Before we look at code, let’s understand the theory. Think of your Kubernetes cluster as a busy, high-end restaurant. The Kubernetes Service acts as the hostess managing the flow of customers, and your Pods are the chefs in the kitchen preparing the meals.

To keep the restaurant running smoothly, the hostess needs to know the status of every chef. This is where probes come in. Probes are simply checks performed periodically by the kubelet—the Kubernetes agent running on every node. The kubelet constantly monitors these probes in a continuous control loop, taking immediate action based on the results.

Readiness Probes: “Is this chef ready for a new ticket?”

A Readiness probe checks if a container is fully prepared to accept incoming network traffic. Imagine a chef who has just arrived and is still putting on their apron, or perhaps they are temporarily overwhelmed by a complex order. They are alive and working, but they shouldn’t receive any new tickets right now.

In Kubernetes, if a readiness probe fails, the kubelet doesn’t kill the container. Instead, it temporarily removes the pod’s IP address from the Service endpoints. The pod stops receiving new requests, giving it time to catch up, initialize, or recover in peace. Once the probe succeeds again, the pod is added back to the rotation.

Liveness Probes: “Is this chef still breathing?”

A Liveness probe is much more drastic. It checks if the container is fundamentally healthy or if it has entered an unrecoverable state, like a deadlock. Going back to our restaurant, if a chef passes out on the floor, you can’t just stop giving them tickets—you need to carry them out and hire a replacement.

When a liveness probe fails, the kubelet takes lethal action: it forcefully terminates the container and restarts it. It’s a blunt instrument designed for absolute worst-case scenarios when the application cannot recover on its own.

Startup Probes: “Has this chef finished training?”

Finally, we have the Startup probe. Some legacy applications, heavy Java/Spring Boot services, or massive data-processing apps can take 30 seconds or more just to boot up. During this time, they will naturally fail both readiness and liveness checks.

A startup probe temporarily suspends all other probes. It asks, “Has the application finished its initial boot sequence?” Once the startup probe succeeds for the first time, it steps aside permanently, handing control over to the liveness and readiness probes. This prevents the kubelet from prematurely murdering an application that is just taking its time to start.

Implementation Deep Dive

Now that we understand the theory, let’s look at how this is implemented in our actual codebase.

The Application Endpoints

In our Node.js application, we need to expose HTTP endpoints for the kubelet to check. Here’s what our index.js looks like:

app.get('/ready', (req, res) => res.status(200).send('ready'));
app.get('/health', (req, res) => res.status(200).send('healthy'));

These might look incredibly simple, but their simplicity is entirely intentional. Let’s break down exactly why we architected them this way.

Why are /ready and /health SEPARATE endpoints? Mixing readiness and liveness logic is a recipe for disaster because they serve fundamentally different purposes. Readiness controls traffic routing, while liveness controls the container lifecycle. If you use a single endpoint for both, you conflate “I am too busy for traffic right now” with “My process is dead.” If a pod is temporarily overwhelmed, a failed readiness check gracefully removes it from the load balancer. If that same check is tied to a liveness probe, Kubernetes will brutally murder the struggling pod, throwing away in-flight requests and forcing a cold restart.

Why does the health endpoint intentionally NOT check the database? You’ll notice our /health endpoint doesn’t ping a database, a Redis cache, or any external API. It simply returns a 200 OK if the Express server can process the route.

This is crucial. If your database experiences a brief network partition, you do not want your application pods to report themselves as “dead.” If they do, the liveness probe fails, and Kubernetes will restart every single pod simultaneously. Restarting your pods does absolutely nothing to fix the database outage. Worse, when the database comes back online, it will immediately be hammered by dozens of pods executing heavy boot-up sequences at the exact same time, creating a secondary outage. By keeping external dependencies out of the liveness probe, we prevent transient glitches from escalating into catastrophic cluster-wide failures.

The Kubernetes Configuration

Next, let’s look at how we tell Kubernetes to use these endpoints. Here is the exact probe configuration from our k8s/deployment.yaml (lines 36-47):

          readinessProbe:
            httpGet:
              path: /ready
              port: 6789
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: 6789
            initialDelaySeconds: 10
            periodSeconds: 20

Let’s dissect this YAML structure and the precise timing configurations. Notice the indentation: the probes are defined under the specific container spec, meaning these checks apply individually to each container inside the pod.

Why use httpGet instead of exec or tcpSocket? Kubernetes allows you to check health by running a command inside the container (exec) or by checking if a port is open (tcpSocket). However, httpGet is almost always the superior choice for web applications. A simple TCP check only verifies that the operating system is listening on the port. The actual application process might be completely frozen. An HTTP GET request ensures that the Node.js event loop is actively processing requests and the Express routing layer is functional.

Why is initialDelaySeconds: 5 for readiness but 10 for liveness? The initialDelaySeconds dictates how long the kubelet waits after the container starts before performing the first check. We set the readiness delay lower (5 seconds) because there is no penalty for failing it. If the app isn’t ready at 5 seconds, it simply doesn’t get traffic yet. However, the liveness delay is set higher (10 seconds) because the penalty for failure is severe. If we set the liveness delay too low, the kubelet might check while the container is still executing its startup scripts, decide it’s dead, and kill it—throwing it into an endless crash loop before it ever finishes booting.

Why is periodSeconds: 10 for readiness but 20 for liveness? The periodSeconds defines how frequently the checks run. Readiness checks need faster feedback loops (every 10 seconds). If a pod becomes overwhelmed or loses its connection to a critical internal component, we want it pulled out of the Service rotation as quickly as possible to prevent user requests from failing. Liveness checks run less frequently (every 20 seconds). Restarting a pod is a highly disruptive operation that consumes significant CPU and memory. We want to be absolutely sure the pod is dead before pulling the trigger. Running liveness checks too frequently increases the risk of false positives from momentary network latency.

Step-by-Step: When Things Go Wrong

What exactly happens in the Kubernetes control plane when these probes fail?

1. Readiness Failure: The application’s connection pool to the database fills up. The /ready endpoint logic (if we implemented deeper checks there) begins returning a 503 status.
2. The kubelet, checking every 10 seconds, sees the failure.
3. After the configured failure threshold (defaults to 3 consecutive failures), the kubelet notifies the Endpoints controller.
4. The pod’s IP is removed from the Service. New requests are routed only to healthy pods.
5. The overwhelmed pod clears its backlog, and the /ready endpoint returns 200 OK.
6. The pod is instantly re-added to the Service and resumes taking traffic.

Contrast that with a liveness failure:

1. Liveness Failure: A bug causes a permanent infinite loop in a background worker, locking up the Node.js main thread.
2. The /health endpoint times out entirely.
3. The kubelet sees the failure on its 20-second interval check.
4. After 3 consecutive failures, the kubelet issues a SIGTERM to the container.
5. If the container doesn’t exit gracefully within the termination grace period (default 30s), a SIGKILL is issued.
6. The kubelet starts a brand new container instance from scratch.

Gotchas and Common Mistakes

[!WARNING] Checking external services in Liveness Probes: As discussed, this is the most common cause of self-inflicted outages. Never check your DB or cache in a liveness probe. Let the application code handle connection retries natively.

[!CAUTION] Setting initialDelaySeconds too low without a Startup Probe: If your application occasionally takes 45 seconds to boot, but your liveness probe checks after 30 seconds, Kubernetes will kill it before it can start. Always use a startupProbe for heavy applications rather than blindly increasing the initialDelaySeconds of your liveness probe, which would delay failure detection during normal runtime.

[!NOTE] Relying solely on TCP checks: Just because the port is accepting connections doesn’t mean your app is healthy. A deadlocked process might still hold the TCP socket open. Always use HTTP probes for web services to verify the application layer is responsive.

Conclusion and Key Takeaways

Properly configured health checks are the foundation of a self-healing Kubernetes cluster. Get them right, and you’ll sleep soundly through minor blips. Get them wrong, and you’ll be debugging cascading failures at 3 AM.

• Separate your logic: Readiness is for traffic routing; liveness is for container restarts. Do not use the same logic for both.
• Keep Liveness dumb: A liveness probe should only check if the core process is running. Never check external dependencies.
• Protect slow starters: Use startup probes for heavy applications to avoid premature termination during boot.
• Tune your timings: Give liveness probes a longer initial delay and a slower polling period to prevent unnecessary restarts.

What’s Next?

Now that your pods are stable, routing traffic perfectly, and recovering from failures intelligently, how do you prevent them from hoarding all your cluster’s memory and CPU? If one memory-leaking pod takes down an entire node, all your health checks won’t save you.

In our final post, Part 5: Right-Sizing Your Pods with Resource Requests and Limits, we’ll dive into the secret to maximizing your cluster’s efficiency without causing resource starvation. See you there!