019 · Containerd CLI (ctr)
Lab 019 - Containerd CLI (ctr)
Module 019
- Goal: Master the
ctrcommand-line tool - the native client for containerd. - Objective: Learn to manage images, containers, namespaces, content, and snapshots directly through containerd, bypassing higher-level runtimes.
What you’ll learn
- What containerd is and where
ctrfits in the container stack - How to list, pull, push, tag, and remove images
- How to run, list, inspect, and delete containers and tasks
- How to work with containerd namespaces
- How to inspect content and snapshots at a low level
- How to export and import images as OCI tarballs
Background
The Container Stack
graph LR
A["kubectl / k8s API<br/><i>orchestrator</i>"] --> B
B["kubelet<br/><i>node agent</i>"] --> C
C["CRI (containerd shim)<br/><i>container runtime interface</i>"] --> D
D["containerd<br/><i>container runtime</i>"] --> E
E["runc<br/><i>OCI runtime - creates Linux cgroups/namespaces</i>"] - containerd is the core container runtime used by Docker and Kubernetes.
ctris containerd’s built-in CLI - it talks directly to the containerd API.ctris not meant for production container management - it’s a debugging and low-level inspection tool.- In Kubernetes nodes,
ctrlets you see what’s happening beneath the kubelet/CRI layer.
When to use ctr vs other tools
| Tool | Use case |
|---|---|
docker | Developer workflow, build images |
crictl | Kubernetes CRI-level debugging (Lab 020) |
ctr | Low-level containerd debugging/inspection |
nerdctl | Docker-compatible CLI for containerd |
Web Terminal
This lab includes an interactive xterm.js web terminal that connects directly to your Kind cluster nodes via docker exec. You get tabbed terminals for each node (control-plane, worker, worker2) with the lab documentation side-by-side.
Architecture
graph TD
subgraph Tabs["Kind Cluster"]
T1["CP"]
T2["worker"]
T3["worker2"]
end
Tabs -->|WebSocket| Server["server.js<br/>node-pty"]
Server -->|"docker exec -it"| CP["control-plane"]
Server -->|"docker exec -it"| W1["worker"]
Server -->|"docker exec -it"| W2["worker2"] Start the Web Terminal
Open http://localhost:3014 in your browser. Each tab gives you a shell inside a Kind node where you can run ctr commands directly.
01. Getting Started with ctr
Step 1.1 - Check ctr availability
On a Kubernetes node (or inside a kind node), use the web terminal tabs above or manually:
# Enter the kind worker node
docker exec -it workshop-worker bash
# Verify ctr is installed
ctr --version
Expected output:
Step 1.2 - ctr help and subcommands
Key subcommands:
| Subcommand | Purpose |
|---|---|
images, i | Manage images |
containers, c | Manage containers |
tasks, t | Manage tasks (running processes) |
namespaces, ns | Manage containerd namespaces |
content | Manage content store |
snapshots, sn | Manage snapshots |
run | Run a container |
plugins | List containerd plugins |
version | Print version |
02. Containerd Namespaces
What is a ctr namespace?
A containerd namespace is a logical isolation boundary inside the containerd daemon itself. It keeps images, containers, snapshots, and content completely separate between different consumers of containerd - so they can share the same daemon process without stepping on each other.
Not Kubernetes namespaces
Containerd namespaces are not Kubernetes namespaces. A Kubernetes namespace (kubectl get ns) isolates API objects in the K8s control plane. A containerd namespace isolates resources inside a single containerd process on a single node. They operate at completely different layers of the stack.
Why does containerd need namespaces?
On a typical Kubernetes node, multiple clients talk to the same containerd socket (/run/containerd/containerd.sock):
graph TD
kubelet["kubelet<br/>(CRI shim)"] --> containerd
buildkit["BuildKit<br/>(builds)"] --> containerd
ctr["ctr / you<br/>(debug)"] --> containerd
subgraph containerd["containerd daemon"]
ns1["k8s.io<br/>namespace"]
ns2["buildkit<br/>namespace"]
ns3["default<br/>namespace"]
end Each client uses its own namespace:
| Namespace | Owner / Client | Contains |
|---|---|---|
k8s.io | kubelet (via CRI) | All pod containers, pause containers, and images pulled by K8s |
buildkit | BuildKit (image builds) | Build cache, intermediate layers |
default | Anything that doesn’t specify a namespace | Whatever you manually pull/run without -n |
| (custom) | You | Anything you create for testing/debugging |
What does a namespace isolate?
Everything inside containerd is namespace-scoped:
- Images - An image pulled into
k8s.iois invisible indefaultand vice versa. - Containers - A container created in one namespace cannot be listed or modified from another.
- Tasks - Running processes are tied to their container’s namespace.
- Snapshots - Filesystem layers and writable overlay mounts belong to a namespace.
- Content - Blob data (manifests, configs, layers) is scoped to the namespace.
- Leases - Garbage-collection leases that protect content from being cleaned up.
This means if you run ctr images list without -n k8s.io, you will see nothing - because Kubernetes images live in the k8s.io namespace, not default.
Practical implications
# ❌ Wrong - queries the "default" namespace (empty on most K8s nodes)
ctr images list
# ✅ Correct - queries the Kubernetes namespace
ctr -n k8s.io images list
You can also set an environment variable to avoid typing -n every time:
Quick rule of thumb
If you are debugging Kubernetes containers, always use -n k8s.io. If you are doing standalone containerd work (e.g., testing image pulls), use -n default or a custom namespace.
Step 2.1 - List namespaces
Expected output:
Step 2.2 - Always specify the namespace
Almost every ctr command requires -n <namespace>:
# List images in the k8s.io namespace (Kubernetes images)
ctr -n k8s.io images list
# List images in the default namespace
ctr -n default images list
Common Mistake
Forgetting -n k8s.io is the #1 reason people think “there are no images” on a Kubernetes node.
Step 2.3 - Create a custom namespace
Step 2.4 - Remove a namespace
03. Image Management
Step 3.1 - List all images
The output shows image references, types, sizes, and platforms.
Step 3.2 - List images with filters (quiet mode)
# Show only image references
ctr -n k8s.io images list -q
# Count total images
ctr -n k8s.io images list -q | wc -l
Step 3.3 - Pull an image
Step 3.4 - Inspect image details
# Show image metadata
ctr -n k8s.io images check
# Get detailed image info
ctr -n k8s.io images list | grep alpine
Step 3.5 - Tag an image
Step 3.6 - Remove an image
Step 3.7 - Export an image to a tarball
Step 3.8 - Import an image from a tarball
04. Running Containers
Step 4.1 - Run a container (foreground)
Breakdown: - run - create + start a container - --rm - remove container after exit - docker.io/library/alpine:3.19 - image reference - test-container - container ID (must be unique) - echo "Hello from ctr" - command to execute
Step 4.2 - Run a container (interactive shell)
Inside the container:
Step 4.3 - Run a container (detached / background)
Step 4.4 - List running containers
Step 4.5 - Inspect a container
This outputs the full container spec as JSON - image, mounts, environment, process, etc.
Step 4.6 - Get container info (pretty print)
05. Task Management
A task in containerd represents the running process of a container. Containers and tasks are separate concepts:
- Container = configuration + filesystem
- Task = running process within a container
Step 5.1 - List running tasks
Step 5.2 - View task processes
Step 5.3 - Exec into a running task
Inside:
Step 5.4 - Pause and resume a task
# Pause
ctr -n k8s.io tasks pause bg-container
ctr -n k8s.io tasks list # Status: PAUSED
# Resume
ctr -n k8s.io tasks resume bg-container
ctr -n k8s.io tasks list # Status: RUNNING
Step 5.5 - Kill a task
Step 5.6 - Delete the stopped task and container
06. Inspecting Kubernetes Containers
This is where ctr becomes a powerful debugging tool - inspecting what Kubernetes is actually running.
Step 6.1 - List all Kubernetes containers
Step 6.2 - Find a specific pod’s containers
Step 6.3 - Inspect a Kubernetes container’s spec
# Get the container ID from the list above
CONTAINER_ID=$(ctr -n k8s.io c ls | grep apphub | head -1 | awk '{print $1}')
# Inspect it
ctr -n k8s.io containers info "${CONTAINER_ID}" | python3 -m json.tool | head -50
Step 6.4 - Check the image used by a container
ctr -n k8s.io containers info "${CONTAINER_ID}" | python3 -c "
import json, sys
info = json.load(sys.stdin)
print('Image:', info.get('Image', 'N/A'))
print('Snapshotter:', info.get('Snapshotter', 'N/A'))
print('SnapshotKey:', info.get('SnapshotKey', 'N/A'))
"
07. Content Store
The content store is containerd’s content-addressable storage for image layers, configs, and manifests.
Step 7.1 - List content
Step 7.2 - Get content by digest
# Get a digest from the list above
DIGEST=$(ctr -n k8s.io content list | head -2 | tail -1 | awk '{print $1}')
# Read the content (for manifests/configs - will be JSON)
ctr -n k8s.io content get "${DIGEST}" | python3 -m json.tool 2>/dev/null || \
echo "(binary content - likely a layer)"
Step 7.3 - Check content store usage
08. Snapshots
Snapshots are the filesystem layers that make up a container’s rootfs.
Step 8.1 - List snapshots
Step 8.2 - View snapshot info
# Get a snapshot key
SNAP=$(ctr -n k8s.io sn ls | head -2 | tail -1 | awk '{print $1}')
ctr -n k8s.io snapshots info "${SNAP}"
Step 8.3 - View snapshot tree
This shows the parent-child relationship of layers - useful for understanding image layer sharing.
09. Plugins
Step 9.1 - List all containerd plugins
Step 9.2 - Filter by type
# Show only snapshot plugins
ctr plugins list | grep -i snapshot
# Show only content plugins
ctr plugins list | grep -i content
10. Practical Exercises
Exercise 1 - Image Investigation
On the kind worker node:
docker run -it felipecruz/debug-ctr-extension:latest
# 1. How many images are loaded in the k8s.io namespace?
ctr -n k8s.io images list -q | wc -l
# 2. Find all workshop app images
ctr -n k8s.io images list -q | grep -E "app[1-4]|apphub"
# 3. Export a workshop image as a tarball
ctr -n k8s.io images export /tmp/apphub.tar $(ctr -n k8s.io images list -q | grep apphub | head -1)
# 4. Check the tarball size
ls -lh /tmp/apphub.tar
Exercise 2 - Container Deep Dive
# 1. List all running tasks
ctr -n k8s.io tasks list
# 2. Find the apphub container
ctr -n k8s.io c ls | grep apphub
# 3. Get its PID on the host
ctr -n k8s.io tasks list | grep apphub
# 4. Inspect the process from the host
APPHUB_PID=$(ctr -n k8s.io t ls | grep apphub | awk '{print $2}')
ls -la /proc/${APPHUB_PID}/root/ 2>/dev/null || echo "Check PID manually"
Exercise 3 - Run a Debug Container
# 1. Pull a network debugging image
ctr -n k8s.io images pull docker.io/nicolaka/netshoot:latest
# 2. Run it in the k8s.io namespace
ctr -n k8s.io run --rm -t docker.io/nicolaka/netshoot:latest debug-net bash
# Inside the container:
ip addr
nslookup kubernetes.default.svc.cluster.local
curl -v http://apphub.workshop.svc.cluster.local:5000/health 2>&1 || echo "Expected: won't resolve without K8s networking"
exit
Exercise 4 - Snapshot Forensics
# 1. List all snapshots
ctr -n k8s.io sn ls | wc -l
# 2. Show the snapshot tree to understand layer sharing
ctr -n k8s.io sn tree | head -40
# 3. Find how many snapshots are "active" (writable) vs "committed" (read-only)
ctr -n k8s.io sn ls | awk '{print $3}' | sort | uniq -c
Quick Reference
ctr Cheat Sheet
# ── Namespaces ──
ctr ns list # list namespaces
ctr ns create <name> # create namespace
ctr ns remove <name> # remove namespace
# ── Images ──
ctr -n k8s.io i list # list images
ctr -n k8s.io i list -q # list image refs only
ctr -n k8s.io i pull <ref> # pull image
ctr -n k8s.io i tag <src> <dst> # tag image
ctr -n k8s.io i remove <ref> # remove image
ctr -n k8s.io i export out.tar <ref> # export to tar
ctr -n k8s.io i import in.tar # import from tar
ctr -n k8s.io i check # check image status
# ── Containers ──
ctr -n k8s.io c list # list containers
ctr -n k8s.io c info <id> # inspect container
ctr -n k8s.io c delete <id> # delete container
# ── Tasks ──
ctr -n k8s.io t list # list tasks
ctr -n k8s.io t ps <id> # show processes
ctr -n k8s.io t exec --exec-id s1 -t <id> sh # exec into task
ctr -n k8s.io t pause <id> # pause task
ctr -n k8s.io t resume <id> # resume task
ctr -n k8s.io t kill <id> # kill task
ctr -n k8s.io t delete <id> # delete stopped task
# ── Run ──
ctr -n k8s.io run --rm <image> <id> <cmd> # run (foreground)
ctr -n k8s.io run --rm -t <image> <id> sh # run (interactive)
ctr -n k8s.io run -d <image> <id> <cmd> # run (detached)
# ── Content & Snapshots ──
ctr -n k8s.io content list # list content objects
ctr -n k8s.io content get <digest> # read content by digest
ctr -n k8s.io sn list # list snapshots
ctr -n k8s.io sn info <key> # snapshot info
ctr -n k8s.io sn tree # snapshot tree
# ── Misc ──
ctr plugins list # list containerd plugins
ctr version # containerd version
Summary
| Concept | What you learned |
|---|---|
ctr | Native containerd CLI for low-level debugging |
| Namespaces | k8s.io isolates Kubernetes-managed resources |
| Images | Pull, tag, export, import - all at the containerd level |
| Containers | Inspect container specs, find K8s pods at runtime |
| Tasks | Running processes - pause, resume, exec, kill |
| Content Store | Content-addressable storage for layers and manifests |
| Snapshots | Filesystem layers - tree view shows layer sharing |
Next Step
Continue to Lab 020 · crictl to learn the CRI-level debugging tool that sits between kubelet and containerd.