Skip to content

020 · CRI Debugging (crictl)

Lab 020 - CRI Debugging with crictl

Module 020

  • Goal: Master crictl - the standard CLI for Kubernetes Container Runtime Interface (CRI).
  • Objective: Learn to inspect, debug, and troubleshoot pods, containers, and images from the CRI perspective - the exact interface kubelet uses to manage your workloads.

What you’ll learn

  • What CRI is and how crictl differs from ctr and docker
  • How to inspect pods and containers as kubelet sees them
  • How to view container logs, exec into containers, and check stats
  • How to troubleshoot image pull issues
  • How to debug pod sandbox and container lifecycle problems
  • How to configure crictl for different runtimes

Background

Where crictl fits
graph TD
    A["kubectl / k8s API<br/><i>You interact here</i>"] --> B
    B["kubelet<br/><i>Node agent</i>"] -->|"CRI gRPC"| Crictl["crictl<br/><i>crictl connects HERE</i>"]
    B --> C["containerd<br/><i>CRI plugin</i>"]
    C --> Ctr["ctr<br/><i>ctr connects HERE (Lab 019)</i>"]
    C --> D["runc<br/><i>OCI runtime</i>"]
crictl vs ctr vs docker
Feature crictl ctr docker
Level CRI (kubelet interface) containerd API Docker Engine API
Sees K8s pods? Yes - first class No - only containers No - only containers
Pod sandbox concept Yes No No
Image build No No Yes
Designed for K8s node debugging containerd debugging Developer workflow
Config file /etc/crictl.yaml N/A ~/.docker/config

Rule of Thumb

  • Debugging Kubernetes pod issues on a node → use crictl
  • Debugging containerd internals → use ctr (Lab 019)
  • Building and testing images locally → use docker

01. Setup and Configuration

Web Terminal

This lab includes an interactive xterm.js web terminal that connects directly to your Kind cluster nodes via docker exec. You get tabbed terminals for each node (control-plane, worker, worker2) with the lab documentation side-by-side.

graph TD
    subgraph Browser["Browser UI"]
        Docs["Lab 020 Docs<br/>(MkDocs iframe)"]
        subgraph Tabs["xterm.js Tabbed Terminals"]
            T1["CP"]
            T2["worker"]
            T3["worker2"]
        end
    end
    Tabs -->|WebSocket| Server["server.js<br/>node-pty"]
    Server -->|"docker exec -it"| CP["control-plane"]
    Server -->|"docker exec -it"| W1["worker"]
    Server -->|"docker exec -it"| W2["worker2"]
cd Labs/020-Crictl
npm install
npm start

Open http://localhost:3015 in your browser. Each tab gives you a shell inside a Kind node where you can run crictl commands directly.

Step 1.1 - Enter a kind node

Using the web terminal tabs above, or manually:

# Enter the kind worker node
docker exec -it workshop-worker bash
Step 1.2 - Check crictl is installed
crictl --version

Expected:

crictl version: v1.29.x

Step 1.3 - Check runtime endpoint configuration
cat /etc/crictl.yaml

Expected:

runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

Step 1.4 - Configure crictl (if needed)
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
Step 1.5 - Verify connectivity
crictl info

This returns a JSON blob with containerd runtime status, including: - Runtime version - Storage driver - CNI plugin status - Streaming server config

# Pretty print
crictl info | python3 -m json.tool | head -30

02. Pod Management

In CRI, a pod sandbox is the isolation boundary (network namespace, IPC, etc.) that holds one or more containers. This maps 1:1 to a Kubernetes Pod.

Step 2.1 - List all pods
crictl pods

Output columns:

Column Meaning
POD ID CRI sandbox ID (not the Kubernetes UID)
CREATED When the sandbox was created
STATE Ready or NotReady
NAME Pod name (from Kubernetes)
NAMESPACE Kubernetes namespace
ATTEMPT Number of sandbox creation attempts
RUNTIME Runtime handler used
Step 2.2 - List pods with filters
# Filter by namespace
crictl pods --namespace workshop

# Filter by pod name (substring match)
crictl pods --name apphub

# Filter by state
crictl pods --state ready

# Filter by label
crictl pods --label app=apphub

# Combine filters
crictl pods --namespace workshop --state ready
Step 2.3 - List pods (quiet mode - IDs only)
crictl pods -q
crictl pods -q --namespace workshop
Step 2.4 - Inspect a pod sandbox
# Get a pod ID
POD_ID=$(crictl pods --name apphub -q | head -1)

# Inspect it (full JSON output)
crictl inspectp "${POD_ID}"

# Pretty print key fields
crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
p = json.load(sys.stdin)
status = p.get('status', {})
print('Pod ID:     ', status.get('id', 'N/A')[:12])
print('State:      ', status.get('state', 'N/A'))
print('Created:    ', status.get('createdAt', 'N/A'))
print('Network:')
net = status.get('network', {})
print('  Namespace:', net.get('namespace', 'N/A'))
for ip_info in net.get('additionalIps', []) + [net]:
    print('  IP:       ', ip_info.get('ip', 'N/A'))
print('Labels:')
for k, v in status.get('labels', {}).items():
    print(f'  {k}: {v}')
"
Step 2.5 - Get pod sandbox networking
crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
info = json.load(sys.stdin)
ns_opts = info.get('info', {}).get('runtimeSpec', {}).get('linux', {}).get('namespaces', [])
for ns in ns_opts:
    print(f\"{ns['type']:12s} → {ns.get('path', '(host)')}\")
"

03. Container Management

Step 3.1 - List all containers
crictl ps

This shows running containers. Key columns:

Column Meaning
CONTAINER Container ID
IMAGE Image reference
CREATED When the container started
STATE Running, Exited, or Created
NAME Container name (from pod spec)
ATTEMPT Restart count
POD ID Parent pod sandbox ID
Step 3.2 - Show all containers (including stopped)
crictl ps -a
Step 3.3 - Filter containers
# By name
crictl ps --name apphub

# By pod ID
crictl ps --pod "${POD_ID}"

# By state
crictl ps --state running
crictl ps -a --state exited

# By image
crictl ps --image apphub

# By label
crictl ps --label io.kubernetes.container.name=apphub

# Quiet mode (IDs only)
crictl ps -q
Step 3.4 - Inspect a container
CONTAINER_ID=$(crictl ps --name apphub -q | head -1)

# Full inspection
crictl inspect "${CONTAINER_ID}"

# Extract key info
crictl inspect "${CONTAINER_ID}" | python3 -c "
import json, sys
c = json.load(sys.stdin)
status = c.get('status', {})
info = c.get('info', {})
print('Container ID:', status.get('id', 'N/A')[:12])
print('State:       ', status.get('state', 'N/A'))
print('Image:       ', status.get('image', {}).get('image', 'N/A'))
print('Started At:  ', status.get('startedAt', 'N/A'))
print('Exit Code:   ', status.get('exitCode', 'N/A'))
print('Mounts:')
for m in status.get('mounts', []):
    print(f\"  {m.get('hostPath', '?')} → {m.get('containerPath', '?')}  (ro={m.get('readonly', False)})\")
print('Env:')
env_vars = info.get('runtimeSpec', {}).get('process', {}).get('env', [])
for e in env_vars[:10]:
    print(f'  {e}')
if len(env_vars) > 10:
    print(f'  ... and {len(env_vars)-10} more')
"
Step 3.5 - View container logs
# Last 20 lines
crictl logs --tail 20 "${CONTAINER_ID}"

# Follow logs (like tail -f)
crictl logs -f "${CONTAINER_ID}"
# Press Ctrl+C to stop

# Logs since a timestamp
crictl logs --since "2024-01-01T00:00:00Z" "${CONTAINER_ID}"

# Logs with timestamps shown
crictl logs --timestamps "${CONTAINER_ID}" | head -10
Step 3.6 - Exec into a container
# Interactive shell
crictl exec -it "${CONTAINER_ID}" sh

# Inside the container:
env | grep -i app
ls /app
cat /proc/1/cmdline | tr '\0' ' ' ; echo
exit
Step 3.7 - Run a one-off command
# Non-interactive exec
crictl exec "${CONTAINER_ID}" cat /etc/hostname
crictl exec "${CONTAINER_ID}" ls /app
crictl exec "${CONTAINER_ID}" env | head -10

04. Container Stats and Resource Usage

Step 4.1 - Live container stats
crictl stats

Output columns:

Column Meaning
CONTAINER Container ID
CPU % CPU usage percentage
MEM Memory usage (bytes)
DISK Writable layer disk usage
INODES Inode usage
Step 4.2 - Stats for a specific container
crictl stats --id "${CONTAINER_ID}"
Step 4.3 - Pod-level stats
crictl statsp

# Filter by namespace
crictl statsp --namespace workshop
Step 4.4 - Stats in JSON (for scripting)
crictl stats -o json | python3 -m json.tool | head -40

05. Image Management

Step 5.1 - List images
crictl images
Column Meaning
IMAGE Image reference
TAG Image tag
IMAGE ID Content digest
SIZE Image size on disk
Step 5.2 - List images with filters
# Filter by reference
crictl images | grep alpine

# Quiet mode (IDs only)
crictl images -q

# With digests
crictl images --digests
Step 5.3 - Inspect an image
IMAGE_ID=$(crictl images -q | head -1)
crictl inspecti "${IMAGE_ID}"

# Extract useful info
crictl inspecti "${IMAGE_ID}" | python3 -c "
import json, sys
img = json.load(sys.stdin)
status = img.get('status', {})
print('ID:    ', status.get('id', {}).get('image', 'N/A')[:12])
print('Size:  ', status.get('size', 0) // 1024 // 1024, 'MB')
for ref in status.get('repoTags', []):
    print('Tag:   ', ref)
for ref in status.get('repoDigests', []):
    print('Digest:', ref[:80])
"
Step 5.4 - Pull an image
crictl pull docker.io/library/busybox:latest
Step 5.5 - Remove an image
crictl rmi docker.io/library/busybox:latest
Step 5.6 - Image filesystem info
crictl imagefsinfo

This shows the filesystem used by the image store - useful for debugging disk pressure.


06. Troubleshooting Common Issues

Scenario 1 - Pod stuck in ContainerCreating
# 1. Find the pod
crictl pods --state notready --namespace workshop

# 2. Get the sandbox ID
POD_ID=$(crictl pods --state notready --namespace workshop -q | head -1)

# 3. Check sandbox details
crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
p = json.load(sys.stdin)
print('State:', p['status']['state'])
print('Network:', json.dumps(p['status'].get('network', {}), indent=2))
"

# 4. Check for containers in 'created' state (not running)
crictl ps -a --pod "${POD_ID}"
Scenario 2 - Container CrashLoopBackOff
# 1. Find exited containers
crictl ps -a --state exited --namespace workshop

# 2. Check the exit code
EXITED_ID=$(crictl ps -a --state exited -q | head -1)
crictl inspect "${EXITED_ID}" | python3 -c "
import json, sys
c = json.load(sys.stdin)
print('Exit Code:', c['status'].get('exitCode', 'N/A'))
print('Reason:   ', c['status'].get('reason', 'N/A'))
print('Message:  ', c['status'].get('message', 'N/A'))
print('Finished: ', c['status'].get('finishedAt', 'N/A'))
"

# 3. Check the logs of the crashed container
crictl logs "${EXITED_ID}"
Scenario 3 - Image Pull Failures
# 1. Check which images are present
crictl images | grep -i <expected-image>

# 2. Try pulling manually
crictl pull <image>:<tag>

# 3. If it fails, check containerd logs
journalctl -u containerd --since "5 minutes ago" | tail -30
Scenario 4 - High Resource Usage
# 1. Get resource stats for all containers
crictl stats

# 2. Find the top CPU consumer
crictl stats -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
stats = data.get('stats', [])
stats.sort(key=lambda s: s.get('cpu', {}).get('usageCoreNanoSeconds', {}).get('value', 0), reverse=True)
for s in stats[:5]:
    cid = s['attributes']['id'][:12]
    cpu = s.get('cpu', {}).get('usageCoreNanoSeconds', {}).get('value', 0)
    mem = s.get('memory', {}).get('workingSetBytes', {}).get('value', 0)
    print(f'{cid}  CPU={cpu:>15}ns  MEM={mem // 1024 // 1024:>6}MB')
"
Scenario 5 - Checking pod networking
POD_ID=$(crictl pods --name apphub -q | head -1)

# Get the network namespace path
NET_NS=$(crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
info = json.load(sys.stdin)
for ns in info.get('info',{}).get('runtimeSpec',{}).get('linux',{}).get('namespaces',[]):
    if ns['type'] == 'network':
        print(ns.get('path',''))
        break
")

# Inspect the network namespace
nsenter --net="${NET_NS}" ip addr
nsenter --net="${NET_NS}" ss -tlnp
nsenter --net="${NET_NS}" ip route

07. Runtime Debugging

Step 7.1 - Runtime info
crictl info | python3 -c "
import json, sys
info = json.load(sys.stdin)
status = info.get('status', {})
print('Runtime:', info.get('config', {}).get('containerd', {}).get('defaultRuntimeName', 'N/A'))
conditions = status.get('conditions', [])
for c in conditions:
    status_str = '✓' if c.get('status') else '✗'
    print(f\"  {status_str} {c.get('type', '?')}: {c.get('message', '')}\")
"
Step 7.2 - Check runtime readiness
# Quick readiness check
crictl info | python3 -c "
import json, sys
info = json.load(sys.stdin)
for c in info.get('status',{}).get('conditions',[]):
    if c['type'] == 'RuntimeReady':
        print('Runtime Ready:', c['status'])
    if c['type'] == 'NetworkReady':
        print('Network Ready:', c['status'])
"
Step 7.3 - Enable debug mode temporarily
# Edit crictl config
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
EOF

# Now all commands will show gRPC debug output
crictl pods 2>&1 | head -20

# Disable debug when done
sed -i 's/debug: true/debug: false/' /etc/crictl.yaml

08. Practical Exercises

Exercise 1 - Pod and Container Mapping

Map every Kubernetes pod to its CRI containers:

echo "=== Pod → Container Mapping ==="
for pod_id in $(crictl pods --namespace workshop -q); do
  pod_name=$(crictl inspectp "${pod_id}" | python3 -c "import json,sys; print(json.load(sys.stdin)['status']['metadata']['name'])")
  echo ""
  echo "Pod: ${pod_name} (${pod_id:0:12})"
  crictl ps --pod "${pod_id}" -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
for c in data.get('containers', []):
    name = c.get('metadata', {}).get('name', 'N/A')
    state = c.get('state', 'N/A')
    image = c.get('imageRef', 'N/A')[:60]
    print(f'  Container: {name}  State: {state}  Image: {image}')
"
done
Exercise 2 - Log Collection

Collect last 50 log lines from all workshop containers:

echo "=== Workshop Container Logs ==="
for cid in $(crictl ps --namespace workshop -q); do
  name=$(crictl inspect "${cid}" | python3 -c "import json,sys; print(json.load(sys.stdin)['status']['metadata']['name'])" 2>/dev/null || echo "unknown")
  echo ""
  echo "──── ${name} (${cid:0:12}) ────"
  crictl logs --tail 5 "${cid}" 2>&1 || echo "(no logs)"
done
Exercise 3 - Resource Report

Generate a resource usage report:

echo "=== Workshop Resource Report ==="
printf "%-14s %-8s %-10s %-10s\n" "CONTAINER" "CPU%" "MEM(MB)" "DISK(MB)"
echo "─────────────────────────────────────────────"

crictl stats -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
for s in data.get('stats', []):
    cid = s['attributes']['id'][:12]
    cpu_ns = s.get('cpu', {}).get('usageCoreNanoSeconds', {}).get('value', 0)
    mem = s.get('memory', {}).get('workingSetBytes', {}).get('value', 0) // 1024 // 1024
    disk = s.get('writableLayer', {}).get('usedBytes', {}).get('value', 0) // 1024 // 1024
    print(f'{cid:<14} {\"N/A\":>8} {mem:>10} {disk:>10}')
"
Exercise 4 - Health Check Script

Write and run a node health check using crictl:

#!/bin/bash
echo "=== Node Health Check ==="

# Runtime status
echo ""
echo "1. Runtime Status:"
crictl info | python3 -c "
import json, sys
for c in json.load(sys.stdin).get('status',{}).get('conditions',[]):
    sym = '✓' if c['status'] else '✗'
    print(f\"   {sym} {c['type']}\")
"

# Pod summary
echo ""
echo "2. Pod Summary:"
total=$(crictl pods -q | wc -l)
ready=$(crictl pods --state ready -q | wc -l)
echo "   Total: ${total}  Ready: ${ready}  NotReady: $((total - ready))"

# Container summary
echo ""
echo "3. Container Summary:"
running=$(crictl ps -q | wc -l)
exited=$(crictl ps -a --state exited -q | wc -l)
echo "   Running: ${running}  Exited: ${exited}"

# Image store
echo ""
echo "4. Image Store:"
images=$(crictl images -q | wc -l)
echo "   Images: ${images}"
crictl imagefsinfo | python3 -c "
import json, sys
info = json.load(sys.stdin)
for fs in info.get('status', []):
    ts = fs.get('timestamp', 0)
    used = fs.get('usedBytes', {}).get('value', 0)
    inodes = fs.get('inodesUsed', {}).get('value', 0)
    print(f'   Used: {used // 1024 // 1024} MB  Inodes: {inodes}')
" 2>/dev/null || echo "   (imagefsinfo unavailable)"

echo ""
echo "=== Done ==="

Quick Reference

crictl Cheat Sheet
# ── Configuration ──
crictl info                              # runtime info & status
crictl version                           # crictl + runtime versions

# ── Pods ──
crictl pods                              # list pods
crictl pods --namespace <ns>             # filter by namespace
crictl pods --name <name>                # filter by name
crictl pods --state ready|notready       # filter by state
crictl pods --label <key>=<value>        # filter by label
crictl pods -q                           # IDs only
crictl inspectp <pod-id>                 # inspect pod sandbox

# ── Containers ──
crictl ps                                # list running containers
crictl ps -a                             # include stopped containers
crictl ps --pod <pod-id>                 # containers in a pod
crictl ps --name <name>                  # filter by name
crictl ps --state running|exited|created # filter by state
crictl ps -q                             # IDs only
crictl inspect <container-id>            # inspect container

# ── Logs ──
crictl logs <container-id>               # view logs
crictl logs --tail 50 <container-id>     # last 50 lines
crictl logs -f <container-id>            # follow logs
crictl logs --timestamps <container-id>  # with timestamps
crictl logs --since <timestamp> <cid>    # since time

# ── Exec ──
crictl exec -it <container-id> sh        # interactive shell
crictl exec <container-id> <cmd>         # run command

# ── Stats ──
crictl stats                             # container resource usage
crictl stats --id <container-id>         # single container
crictl statsp                            # pod-level stats
crictl stats -o json                     # JSON output

# ── Images ──
crictl images                            # list images
crictl images --digests                  # with digests
crictl images -q                         # IDs only
crictl inspecti <image-id>               # inspect image
crictl pull <image>                      # pull image
crictl rmi <image>                       # remove image
crictl imagefsinfo                       # image filesystem info

# ── Troubleshooting ──
crictl info                              # overall status
crictl pods --state notready             # broken pods
crictl ps -a --state exited              # crashed containers

Summary

Concept What you learned
CRI Standard interface between kubelet and container runtimes
crictl The CLI tool that speaks CRI - sees pods as kubelet does
Pod sandboxes crictl pods / inspectp - inspect the pod isolation layer
Containers crictl ps / inspect - container lifecycle and metadata
Logs crictl logs - direct access to container stdout/stderr
Exec crictl exec - run commands inside containers
Stats crictl stats / statsp - CPU, memory, disk per container
Images crictl images / pull / rmi - CRI image management
Troubleshooting Exit codes, sandbox state, image pull errors, resource usage

Labs Complete

You can now debug containers at every level of the stack:

  • kubectl - Kubernetes API level
  • crictl - CRI level (kubelet’s view)
  • ctr - containerd level (runtime internals)