020 · CRI Debugging (crictl)
Lab 020 - CRI Debugging with crictl
Module 020
- Goal: Master
crictl- the standard CLI for Kubernetes Container Runtime Interface (CRI). - Objective: Learn to inspect, debug, and troubleshoot pods, containers, and images from the CRI perspective - the exact interface kubelet uses to manage your workloads.
What you’ll learn
- What CRI is and how
crictldiffers fromctranddocker - How to inspect pods and containers as kubelet sees them
- How to view container logs, exec into containers, and check stats
- How to troubleshoot image pull issues
- How to debug pod sandbox and container lifecycle problems
- How to configure
crictlfor different runtimes
Background
Where crictl fits
graph TD
A["kubectl / k8s API<br/><i>You interact here</i>"] --> B
B["kubelet<br/><i>Node agent</i>"] -->|"CRI gRPC"| Crictl["crictl<br/><i>crictl connects HERE</i>"]
B --> C["containerd<br/><i>CRI plugin</i>"]
C --> Ctr["ctr<br/><i>ctr connects HERE (Lab 019)</i>"]
C --> D["runc<br/><i>OCI runtime</i>"] crictl vs ctr vs docker
| Feature | crictl | ctr | docker |
|---|---|---|---|
| Level | CRI (kubelet interface) | containerd API | Docker Engine API |
| Sees K8s pods? | Yes - first class | No - only containers | No - only containers |
| Pod sandbox concept | Yes | No | No |
| Image build | No | No | Yes |
| Designed for | K8s node debugging | containerd debugging | Developer workflow |
| Config file | /etc/crictl.yaml | N/A | ~/.docker/config |
Rule of Thumb
- Debugging Kubernetes pod issues on a node → use
crictl - Debugging containerd internals → use
ctr(Lab 019) - Building and testing images locally → use
docker
01. Setup and Configuration
Web Terminal
This lab includes an interactive xterm.js web terminal that connects directly to your Kind cluster nodes via docker exec. You get tabbed terminals for each node (control-plane, worker, worker2) with the lab documentation side-by-side.
graph TD
subgraph Browser["Browser UI"]
Docs["Lab 020 Docs<br/>(MkDocs iframe)"]
subgraph Tabs["xterm.js Tabbed Terminals"]
T1["CP"]
T2["worker"]
T3["worker2"]
end
end
Tabs -->|WebSocket| Server["server.js<br/>node-pty"]
Server -->|"docker exec -it"| CP["control-plane"]
Server -->|"docker exec -it"| W1["worker"]
Server -->|"docker exec -it"| W2["worker2"] Open http://localhost:3015 in your browser. Each tab gives you a shell inside a Kind node where you can run crictl commands directly.
Step 1.1 - Enter a kind node
Using the web terminal tabs above, or manually:
Step 1.2 - Check crictl is installed
Expected:
Step 1.3 - Check runtime endpoint configuration
Expected:
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
Step 1.4 - Configure crictl (if needed)
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
Step 1.5 - Verify connectivity
This returns a JSON blob with containerd runtime status, including: - Runtime version - Storage driver - CNI plugin status - Streaming server config
02. Pod Management
In CRI, a pod sandbox is the isolation boundary (network namespace, IPC, etc.) that holds one or more containers. This maps 1:1 to a Kubernetes Pod.
Step 2.1 - List all pods
Output columns:
| Column | Meaning |
|---|---|
| POD ID | CRI sandbox ID (not the Kubernetes UID) |
| CREATED | When the sandbox was created |
| STATE | Ready or NotReady |
| NAME | Pod name (from Kubernetes) |
| NAMESPACE | Kubernetes namespace |
| ATTEMPT | Number of sandbox creation attempts |
| RUNTIME | Runtime handler used |
Step 2.2 - List pods with filters
# Filter by namespace
crictl pods --namespace workshop
# Filter by pod name (substring match)
crictl pods --name apphub
# Filter by state
crictl pods --state ready
# Filter by label
crictl pods --label app=apphub
# Combine filters
crictl pods --namespace workshop --state ready
Step 2.3 - List pods (quiet mode - IDs only)
Step 2.4 - Inspect a pod sandbox
# Get a pod ID
POD_ID=$(crictl pods --name apphub -q | head -1)
# Inspect it (full JSON output)
crictl inspectp "${POD_ID}"
# Pretty print key fields
crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
p = json.load(sys.stdin)
status = p.get('status', {})
print('Pod ID: ', status.get('id', 'N/A')[:12])
print('State: ', status.get('state', 'N/A'))
print('Created: ', status.get('createdAt', 'N/A'))
print('Network:')
net = status.get('network', {})
print(' Namespace:', net.get('namespace', 'N/A'))
for ip_info in net.get('additionalIps', []) + [net]:
print(' IP: ', ip_info.get('ip', 'N/A'))
print('Labels:')
for k, v in status.get('labels', {}).items():
print(f' {k}: {v}')
"
Step 2.5 - Get pod sandbox networking
crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
info = json.load(sys.stdin)
ns_opts = info.get('info', {}).get('runtimeSpec', {}).get('linux', {}).get('namespaces', [])
for ns in ns_opts:
print(f\"{ns['type']:12s} → {ns.get('path', '(host)')}\")
"
03. Container Management
Step 3.1 - List all containers
This shows running containers. Key columns:
| Column | Meaning |
|---|---|
| CONTAINER | Container ID |
| IMAGE | Image reference |
| CREATED | When the container started |
| STATE | Running, Exited, or Created |
| NAME | Container name (from pod spec) |
| ATTEMPT | Restart count |
| POD ID | Parent pod sandbox ID |
Step 3.2 - Show all containers (including stopped)
Step 3.3 - Filter containers
# By name
crictl ps --name apphub
# By pod ID
crictl ps --pod "${POD_ID}"
# By state
crictl ps --state running
crictl ps -a --state exited
# By image
crictl ps --image apphub
# By label
crictl ps --label io.kubernetes.container.name=apphub
# Quiet mode (IDs only)
crictl ps -q
Step 3.4 - Inspect a container
CONTAINER_ID=$(crictl ps --name apphub -q | head -1)
# Full inspection
crictl inspect "${CONTAINER_ID}"
# Extract key info
crictl inspect "${CONTAINER_ID}" | python3 -c "
import json, sys
c = json.load(sys.stdin)
status = c.get('status', {})
info = c.get('info', {})
print('Container ID:', status.get('id', 'N/A')[:12])
print('State: ', status.get('state', 'N/A'))
print('Image: ', status.get('image', {}).get('image', 'N/A'))
print('Started At: ', status.get('startedAt', 'N/A'))
print('Exit Code: ', status.get('exitCode', 'N/A'))
print('Mounts:')
for m in status.get('mounts', []):
print(f\" {m.get('hostPath', '?')} → {m.get('containerPath', '?')} (ro={m.get('readonly', False)})\")
print('Env:')
env_vars = info.get('runtimeSpec', {}).get('process', {}).get('env', [])
for e in env_vars[:10]:
print(f' {e}')
if len(env_vars) > 10:
print(f' ... and {len(env_vars)-10} more')
"
Step 3.5 - View container logs
# Last 20 lines
crictl logs --tail 20 "${CONTAINER_ID}"
# Follow logs (like tail -f)
crictl logs -f "${CONTAINER_ID}"
# Press Ctrl+C to stop
# Logs since a timestamp
crictl logs --since "2024-01-01T00:00:00Z" "${CONTAINER_ID}"
# Logs with timestamps shown
crictl logs --timestamps "${CONTAINER_ID}" | head -10
Step 3.6 - Exec into a container
# Interactive shell
crictl exec -it "${CONTAINER_ID}" sh
# Inside the container:
env | grep -i app
ls /app
cat /proc/1/cmdline | tr '\0' ' ' ; echo
exit
Step 3.7 - Run a one-off command
# Non-interactive exec
crictl exec "${CONTAINER_ID}" cat /etc/hostname
crictl exec "${CONTAINER_ID}" ls /app
crictl exec "${CONTAINER_ID}" env | head -10
04. Container Stats and Resource Usage
Step 4.1 - Live container stats
Output columns:
| Column | Meaning |
|---|---|
| CONTAINER | Container ID |
| CPU % | CPU usage percentage |
| MEM | Memory usage (bytes) |
| DISK | Writable layer disk usage |
| INODES | Inode usage |
Step 4.2 - Stats for a specific container
Step 4.3 - Pod-level stats
Step 4.4 - Stats in JSON (for scripting)
05. Image Management
Step 5.1 - List images
| Column | Meaning |
|---|---|
| IMAGE | Image reference |
| TAG | Image tag |
| IMAGE ID | Content digest |
| SIZE | Image size on disk |
Step 5.2 - List images with filters
# Filter by reference
crictl images | grep alpine
# Quiet mode (IDs only)
crictl images -q
# With digests
crictl images --digests
Step 5.3 - Inspect an image
IMAGE_ID=$(crictl images -q | head -1)
crictl inspecti "${IMAGE_ID}"
# Extract useful info
crictl inspecti "${IMAGE_ID}" | python3 -c "
import json, sys
img = json.load(sys.stdin)
status = img.get('status', {})
print('ID: ', status.get('id', {}).get('image', 'N/A')[:12])
print('Size: ', status.get('size', 0) // 1024 // 1024, 'MB')
for ref in status.get('repoTags', []):
print('Tag: ', ref)
for ref in status.get('repoDigests', []):
print('Digest:', ref[:80])
"
Step 5.4 - Pull an image
Step 5.5 - Remove an image
Step 5.6 - Image filesystem info
This shows the filesystem used by the image store - useful for debugging disk pressure.
06. Troubleshooting Common Issues
Scenario 1 - Pod stuck in ContainerCreating
# 1. Find the pod
crictl pods --state notready --namespace workshop
# 2. Get the sandbox ID
POD_ID=$(crictl pods --state notready --namespace workshop -q | head -1)
# 3. Check sandbox details
crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
p = json.load(sys.stdin)
print('State:', p['status']['state'])
print('Network:', json.dumps(p['status'].get('network', {}), indent=2))
"
# 4. Check for containers in 'created' state (not running)
crictl ps -a --pod "${POD_ID}"
Scenario 2 - Container CrashLoopBackOff
# 1. Find exited containers
crictl ps -a --state exited --namespace workshop
# 2. Check the exit code
EXITED_ID=$(crictl ps -a --state exited -q | head -1)
crictl inspect "${EXITED_ID}" | python3 -c "
import json, sys
c = json.load(sys.stdin)
print('Exit Code:', c['status'].get('exitCode', 'N/A'))
print('Reason: ', c['status'].get('reason', 'N/A'))
print('Message: ', c['status'].get('message', 'N/A'))
print('Finished: ', c['status'].get('finishedAt', 'N/A'))
"
# 3. Check the logs of the crashed container
crictl logs "${EXITED_ID}"
Scenario 3 - Image Pull Failures
# 1. Check which images are present
crictl images | grep -i <expected-image>
# 2. Try pulling manually
crictl pull <image>:<tag>
# 3. If it fails, check containerd logs
journalctl -u containerd --since "5 minutes ago" | tail -30
Scenario 4 - High Resource Usage
# 1. Get resource stats for all containers
crictl stats
# 2. Find the top CPU consumer
crictl stats -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
stats = data.get('stats', [])
stats.sort(key=lambda s: s.get('cpu', {}).get('usageCoreNanoSeconds', {}).get('value', 0), reverse=True)
for s in stats[:5]:
cid = s['attributes']['id'][:12]
cpu = s.get('cpu', {}).get('usageCoreNanoSeconds', {}).get('value', 0)
mem = s.get('memory', {}).get('workingSetBytes', {}).get('value', 0)
print(f'{cid} CPU={cpu:>15}ns MEM={mem // 1024 // 1024:>6}MB')
"
Scenario 5 - Checking pod networking
POD_ID=$(crictl pods --name apphub -q | head -1)
# Get the network namespace path
NET_NS=$(crictl inspectp "${POD_ID}" | python3 -c "
import json, sys
info = json.load(sys.stdin)
for ns in info.get('info',{}).get('runtimeSpec',{}).get('linux',{}).get('namespaces',[]):
if ns['type'] == 'network':
print(ns.get('path',''))
break
")
# Inspect the network namespace
nsenter --net="${NET_NS}" ip addr
nsenter --net="${NET_NS}" ss -tlnp
nsenter --net="${NET_NS}" ip route
07. Runtime Debugging
Step 7.1 - Runtime info
crictl info | python3 -c "
import json, sys
info = json.load(sys.stdin)
status = info.get('status', {})
print('Runtime:', info.get('config', {}).get('containerd', {}).get('defaultRuntimeName', 'N/A'))
conditions = status.get('conditions', [])
for c in conditions:
status_str = '✓' if c.get('status') else '✗'
print(f\" {status_str} {c.get('type', '?')}: {c.get('message', '')}\")
"
Step 7.2 - Check runtime readiness
# Quick readiness check
crictl info | python3 -c "
import json, sys
info = json.load(sys.stdin)
for c in info.get('status',{}).get('conditions',[]):
if c['type'] == 'RuntimeReady':
print('Runtime Ready:', c['status'])
if c['type'] == 'NetworkReady':
print('Network Ready:', c['status'])
"
Step 7.3 - Enable debug mode temporarily
# Edit crictl config
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
EOF
# Now all commands will show gRPC debug output
crictl pods 2>&1 | head -20
# Disable debug when done
sed -i 's/debug: true/debug: false/' /etc/crictl.yaml
08. Practical Exercises
Exercise 1 - Pod and Container Mapping
Map every Kubernetes pod to its CRI containers:
echo "=== Pod → Container Mapping ==="
for pod_id in $(crictl pods --namespace workshop -q); do
pod_name=$(crictl inspectp "${pod_id}" | python3 -c "import json,sys; print(json.load(sys.stdin)['status']['metadata']['name'])")
echo ""
echo "Pod: ${pod_name} (${pod_id:0:12})"
crictl ps --pod "${pod_id}" -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
for c in data.get('containers', []):
name = c.get('metadata', {}).get('name', 'N/A')
state = c.get('state', 'N/A')
image = c.get('imageRef', 'N/A')[:60]
print(f' Container: {name} State: {state} Image: {image}')
"
done
Exercise 2 - Log Collection
Collect last 50 log lines from all workshop containers:
echo "=== Workshop Container Logs ==="
for cid in $(crictl ps --namespace workshop -q); do
name=$(crictl inspect "${cid}" | python3 -c "import json,sys; print(json.load(sys.stdin)['status']['metadata']['name'])" 2>/dev/null || echo "unknown")
echo ""
echo "──── ${name} (${cid:0:12}) ────"
crictl logs --tail 5 "${cid}" 2>&1 || echo "(no logs)"
done
Exercise 3 - Resource Report
Generate a resource usage report:
echo "=== Workshop Resource Report ==="
printf "%-14s %-8s %-10s %-10s\n" "CONTAINER" "CPU%" "MEM(MB)" "DISK(MB)"
echo "─────────────────────────────────────────────"
crictl stats -o json | python3 -c "
import json, sys
data = json.load(sys.stdin)
for s in data.get('stats', []):
cid = s['attributes']['id'][:12]
cpu_ns = s.get('cpu', {}).get('usageCoreNanoSeconds', {}).get('value', 0)
mem = s.get('memory', {}).get('workingSetBytes', {}).get('value', 0) // 1024 // 1024
disk = s.get('writableLayer', {}).get('usedBytes', {}).get('value', 0) // 1024 // 1024
print(f'{cid:<14} {\"N/A\":>8} {mem:>10} {disk:>10}')
"
Exercise 4 - Health Check Script
Write and run a node health check using crictl:
#!/bin/bash
echo "=== Node Health Check ==="
# Runtime status
echo ""
echo "1. Runtime Status:"
crictl info | python3 -c "
import json, sys
for c in json.load(sys.stdin).get('status',{}).get('conditions',[]):
sym = '✓' if c['status'] else '✗'
print(f\" {sym} {c['type']}\")
"
# Pod summary
echo ""
echo "2. Pod Summary:"
total=$(crictl pods -q | wc -l)
ready=$(crictl pods --state ready -q | wc -l)
echo " Total: ${total} Ready: ${ready} NotReady: $((total - ready))"
# Container summary
echo ""
echo "3. Container Summary:"
running=$(crictl ps -q | wc -l)
exited=$(crictl ps -a --state exited -q | wc -l)
echo " Running: ${running} Exited: ${exited}"
# Image store
echo ""
echo "4. Image Store:"
images=$(crictl images -q | wc -l)
echo " Images: ${images}"
crictl imagefsinfo | python3 -c "
import json, sys
info = json.load(sys.stdin)
for fs in info.get('status', []):
ts = fs.get('timestamp', 0)
used = fs.get('usedBytes', {}).get('value', 0)
inodes = fs.get('inodesUsed', {}).get('value', 0)
print(f' Used: {used // 1024 // 1024} MB Inodes: {inodes}')
" 2>/dev/null || echo " (imagefsinfo unavailable)"
echo ""
echo "=== Done ==="
Quick Reference
crictl Cheat Sheet
# ── Configuration ──
crictl info # runtime info & status
crictl version # crictl + runtime versions
# ── Pods ──
crictl pods # list pods
crictl pods --namespace <ns> # filter by namespace
crictl pods --name <name> # filter by name
crictl pods --state ready|notready # filter by state
crictl pods --label <key>=<value> # filter by label
crictl pods -q # IDs only
crictl inspectp <pod-id> # inspect pod sandbox
# ── Containers ──
crictl ps # list running containers
crictl ps -a # include stopped containers
crictl ps --pod <pod-id> # containers in a pod
crictl ps --name <name> # filter by name
crictl ps --state running|exited|created # filter by state
crictl ps -q # IDs only
crictl inspect <container-id> # inspect container
# ── Logs ──
crictl logs <container-id> # view logs
crictl logs --tail 50 <container-id> # last 50 lines
crictl logs -f <container-id> # follow logs
crictl logs --timestamps <container-id> # with timestamps
crictl logs --since <timestamp> <cid> # since time
# ── Exec ──
crictl exec -it <container-id> sh # interactive shell
crictl exec <container-id> <cmd> # run command
# ── Stats ──
crictl stats # container resource usage
crictl stats --id <container-id> # single container
crictl statsp # pod-level stats
crictl stats -o json # JSON output
# ── Images ──
crictl images # list images
crictl images --digests # with digests
crictl images -q # IDs only
crictl inspecti <image-id> # inspect image
crictl pull <image> # pull image
crictl rmi <image> # remove image
crictl imagefsinfo # image filesystem info
# ── Troubleshooting ──
crictl info # overall status
crictl pods --state notready # broken pods
crictl ps -a --state exited # crashed containers
Summary
| Concept | What you learned |
|---|---|
| CRI | Standard interface between kubelet and container runtimes |
crictl | The CLI tool that speaks CRI - sees pods as kubelet does |
| Pod sandboxes | crictl pods / inspectp - inspect the pod isolation layer |
| Containers | crictl ps / inspect - container lifecycle and metadata |
| Logs | crictl logs - direct access to container stdout/stderr |
| Exec | crictl exec - run commands inside containers |
| Stats | crictl stats / statsp - CPU, memory, disk per container |
| Images | crictl images / pull / rmi - CRI image management |
| Troubleshooting | Exit codes, sandbox state, image pull errors, resource usage |
Labs Complete
You can now debug containers at every level of the stack:
- kubectl - Kubernetes API level
- crictl - CRI level (kubelet’s view)
- ctr - containerd level (runtime internals)