Kubernetes Debugging Basics: Debugging Clusters, Pods & Services

What Is Kubernetes Debugging? 

Kubernetes debugging refers to the process of diagnosing and fixing issues that arise when deploying, running, and managing applications in a Kubernetes cluster. Kubernetes is a complex system that can have many components, and debugging can be challenging when something goes wrong.

Kubernetes debugging typically involves collecting information about the state of the cluster, identifying the source of the issue, and then using that information to resolve the problem.

This is part of a series of articles about Kubernetes monitoring

Challenges of Debugging Kubernetes Applications 

Debugging Kubernetes-based applications can be a challenging task due to the complexity and distributed nature of the Kubernetes platform. Some of the challenges include:

• Local debugging: The debugging environment should mirror the production environment to ensure that the issues are reproducible and fixable. However, this can be difficult in Kubernetes since the infrastructure and security conditions in Kubernetes are not always reproducible in a local Docker Compose or other local debugging solutions. This can make it challenging for developers to reproduce and fix issues.
• Familiarity with Kubernetes commands: Developers need to be familiar with the Kubernetes command-line interface (CLI) and other tools like kubectl, which can be daunting for developers who are new to the Kubernetes ecosystem.
• Ephemeral nature of pods: Kubernetes is designed to be highly scalable and dynamic, with pods being created and destroyed as needed. This can make debugging challenging since the debugging session may need to be started and stopped several times during the lifetime of the pod, which can make it difficult to identify the root cause of the problem.

Debugging Your Kubernetes Applications 

Debugging can be done at the pod or cluster level.

Cluster-Level Debugging

Here is an overview of what to do to troubleshoot and debug a Kubernetes cluster.

Step 1: Obtaining cluster information

The first step when debugging a Kubernetes cluster is to obtain cluster information. You can do this by running the following command:

kubectl cluster-info

Step 2: Obtaining cluster components’ health status

The next step is to check the health status of the Kubernetes cluster components, such as the control plane and worker nodes. You can use this command to view the health status of the control plane components:

kubectl get componentstatus

This command will display the health status of the control plane components, including the etcd, kube-apiserver, kube-controller-manager, and kube-scheduler (depending on the components installed for your cluster).

To check the health status of the worker nodes, you can use the following command: 

kubectl get nodes

Step 3: Viewing the events in the cluster

The final step is to view the events in the cluster. Events can be used to track changes and actions that occur in the cluster, including the creation, modification, and deletion of resources. To view the events in the cluster, you can use the following command:

kubectl get events

This command will display a list of events that have occurred in the cluster, including their type, reason, message, and timestamp. You can use this information to troubleshoot issues in the cluster, such as resource allocation problems or unexpected resource deletions.

You can also filter events based on specific criteria by using this command:

kubectl get events --field-selector involvedObject.kind=Pod

This command will display events that are related to pods. You can replace Pod with other resource types, such as Service or Deployment, to filter events based on specific resource types.

Debugging Pods

Here is an overview of how to debug a pod using kubectl commands.

Step 1: Check the pod status

The first step when debugging a pod is to check the status of the pod. The following command allows you to view the status of all pods in the cluster:

kubectl get pods

This command will display a list of all the pods in the cluster, along with their status. If the pod is in a pending or waiting state, it may be experiencing issues that are preventing it from running correctly.

Step 2: Check pod events

The next step is to check the events associated with the pod. You can use the following command to view the events associated with the pod:

kubectl describe pod <name>

This will display details about the pod, including any events associated with the pod. You can use this information to identify any issues that may be causing the pod to be in a pending or waiting state.

Step 3: Check the pod logs

The next step is to check the logs associated with the pod. You can use the following command to view the logs associated with the pod:

kubectl logs <pod-name>

This command will display the logs associated with the pod. You can use this information to identify any issues that may be causing the pod to be in a pending or waiting state.

Step 4: Check pod resource requests and limits

If the pod is in a pending state, it may be experiencing issues related to resource allocation. You can use the following command to view the resource requests and limits associated with the pod:

kubectl describe pod <pod-name>

This command will display detailed information about the pod, including the resource requests and limits associated with the pod. You can use this information to identify any issues related to resource allocation.

Step 5: Check the pod configurations

If the pod is in a waiting state, it may be experiencing issues related to its configurations. You can use the following command to view the configurations associated with the pod:

kubectl describe pod <pod-name>

This command will display detailed information about the pod, including its configurations. You can use this information to identify any issues related to configurations.

Debugging Services

Services enable load balancing between pods. Debugging a Kubernetes service involves verifying the service’s configurations, checking the service endpoints, examining the logs and events, and testing the connectivity. 

You can use the commands kubectl describe service <service-name>, kubectl get endpoints <service-name>,and kubectl logs <pod-name> to verify the service configurations, check the service endpoints, and view the logs of the pods associated with the service.

Additionally, you can use tools such as curl or telnet to test the connectivity to the service. Debugging a Kubernetes service requires an understanding of its dependencies, including the pods and nodes that the service is connected to.

Debugging StatefulSets

Debugging a StatefulSet involves checking the status of the StatefulSet and its associated pods, examining the logs and events, and verifying the StatefulSet’s configurations. You can use the commands kubectl describe statefulset <statefulset-name>, kubectl get pods, and kubectl logs <pod-name> to check the StatefulSet status, examine the logs of its pods, and view the events.

Additionally, you can use kubectl edit statefulset <statefulset-name> to verify the StatefulSet’s configurations. Debugging a StatefulSet involves a similar process to debugging a pod, but with additional considerations due to the StatefulSet’s unique characteristics, such as stable network identities and persistent storage.

Kubernetes Debugging Best Practices 

Kubernetes debugging can be a complex and time-consuming process, but there are several best practices that can help make it more efficient and effective:

• Keep logs and events: Logs and events are valuable sources of information when debugging Kubernetes clusters. It’s important to keep logs for a sufficient period of time and to store them in a central location for easy access.
• Use descriptive and meaningful labels: Labels can be used to categorize and identify resources in a Kubernetes cluster. Using descriptive and meaningful labels can make it easier to find the resources you are looking for when debugging.
• Monitor resource usage: Monitoring resource usage can help identify performance bottlenecks and resource constraints in the cluster. This information can be used to diagnose issues and to make adjustments to the cluster configuration.
• Isolate resources: When debugging, it’s important to isolate resources to minimize the impact on the rest of the cluster. For example, you may want to stop or delete a pod or service to determine if it is the source of the issue.
• Use automation: Automated tools can help streamline the debugging process and reduce the time and effort required to diagnose and resolve issues. For example, you can use tools like kubectl and kubectl plugins to automate common debugging tasks.
• Document your process: Keeping a detailed record of the steps you take when debugging can help you reproduce the process in the future and make it easier for others to help you if necessary.
• Collaborate with others: Collaborating with others, such as Kubernetes administrators and developers, can help you identify and resolve issues more quickly. Sharing information and working together can also help you learn from each other and improve your overall debugging process.

Related content: Read our guide to kubernetes monitoring tools.

Kubernetes Debugging with Lumigo

Lumigo is a troubleshooting platform, purpose-built for microservice-based applications. Developers using Kubernetes to orchestrate their containerized applications can use Lumigo to monitor, trace and troubleshoot issues fast. Deployed with zero-code changes and automated in one-click, Lumigo stitches together every interaction between micro and managed service into end-to-end stack traces. These traces, served alongside request payload data, give developers complete visibility into their container environments. Using Lumigo, developers get:

• End-to-end virtual stack traces across every micro and managed service that makes up a serverless application, in context
• API visibility that makes all the data passed between services available and accessible, making it possible to perform root cause analysis without digging through logs
• Distributed tracing that is deployed with no code and automated in one click
• Unified platform to explore and query across microservices, see a real-time view of applications, and optimize performance

To try Lumigo for Kubernetes, check out our Kubernetes operator on GitHub.