Skip to content

019 · Containerd CLI (ctr)

Lab 019 - Containerd CLI (ctr)

Module 019

  • Goal: Master the ctr command-line tool - the native client for containerd.
  • Objective: Learn to manage images, containers, namespaces, content, and snapshots directly through containerd, bypassing higher-level runtimes.

What you’ll learn

  • What containerd is and where ctr fits in the container stack
  • How to list, pull, push, tag, and remove images
  • How to run, list, inspect, and delete containers and tasks
  • How to work with containerd namespaces
  • How to inspect content and snapshots at a low level
  • How to export and import images as OCI tarballs

Background

The Container Stack
graph LR
    A["kubectl / k8s API<br/><i>orchestrator</i>"] --> B
    B["kubelet<br/><i>node agent</i>"] --> C
    C["CRI (containerd shim)<br/><i>container runtime interface</i>"] --> D
    D["containerd<br/><i>container runtime</i>"] --> E
    E["runc<br/><i>OCI runtime - creates Linux cgroups/namespaces</i>"]
  • containerd is the core container runtime used by Docker and Kubernetes.
  • ctr is containerd’s built-in CLI - it talks directly to the containerd API.
  • ctr is not meant for production container management - it’s a debugging and low-level inspection tool.
  • In Kubernetes nodes, ctr lets you see what’s happening beneath the kubelet/CRI layer.
When to use ctr vs other tools
Tool Use case
docker Developer workflow, build images
crictl Kubernetes CRI-level debugging (Lab 020)
ctr Low-level containerd debugging/inspection
nerdctl Docker-compatible CLI for containerd

Web Terminal

This lab includes an interactive xterm.js web terminal that connects directly to your Kind cluster nodes via docker exec. You get tabbed terminals for each node (control-plane, worker, worker2) with the lab documentation side-by-side.

Architecture
graph TD
    subgraph Tabs["Kind Cluster"]
        T1["CP"]
        T2["worker"]
        T3["worker2"]
    end
    Tabs -->|WebSocket| Server["server.js<br/>node-pty"]
    Server -->|"docker exec -it"| CP["control-plane"]
    Server -->|"docker exec -it"| W1["worker"]
    Server -->|"docker exec -it"| W2["worker2"]
Start the Web Terminal
cd Labs/019-Ctr
npm install
npm start

Open http://localhost:3014 in your browser. Each tab gives you a shell inside a Kind node where you can run ctr commands directly.


01. Getting Started with ctr

Step 1.1 - Check ctr availability

On a Kubernetes node (or inside a kind node), use the web terminal tabs above or manually:

# Enter the kind worker node
docker exec -it workshop-worker bash

# Verify ctr is installed
ctr --version

Expected output:

ctr containerd.io 1.7.x

Step 1.2 - ctr help and subcommands
ctr --help

Key subcommands:

Subcommand Purpose
images, i Manage images
containers, c Manage containers
tasks, t Manage tasks (running processes)
namespaces, ns Manage containerd namespaces
content Manage content store
snapshots, sn Manage snapshots
run Run a container
plugins List containerd plugins
version Print version

02. Containerd Namespaces

What is a ctr namespace?

A containerd namespace is a logical isolation boundary inside the containerd daemon itself. It keeps images, containers, snapshots, and content completely separate between different consumers of containerd - so they can share the same daemon process without stepping on each other.

Not Kubernetes namespaces

Containerd namespaces are not Kubernetes namespaces. A Kubernetes namespace (kubectl get ns) isolates API objects in the K8s control plane. A containerd namespace isolates resources inside a single containerd process on a single node. They operate at completely different layers of the stack.

Why does containerd need namespaces?

On a typical Kubernetes node, multiple clients talk to the same containerd socket (/run/containerd/containerd.sock):

graph TD
    kubelet["kubelet<br/>(CRI shim)"] --> containerd
    buildkit["BuildKit<br/>(builds)"] --> containerd
    ctr["ctr / you<br/>(debug)"] --> containerd

    subgraph containerd["containerd daemon"]
        ns1["k8s.io<br/>namespace"]
        ns2["buildkit<br/>namespace"]
        ns3["default<br/>namespace"]
    end

Each client uses its own namespace:

Namespace Owner / Client Contains
k8s.io kubelet (via CRI) All pod containers, pause containers, and images pulled by K8s
buildkit BuildKit (image builds) Build cache, intermediate layers
default Anything that doesn’t specify a namespace Whatever you manually pull/run without -n
(custom) You Anything you create for testing/debugging
What does a namespace isolate?

Everything inside containerd is namespace-scoped:

  • Images - An image pulled into k8s.io is invisible in default and vice versa.
  • Containers - A container created in one namespace cannot be listed or modified from another.
  • Tasks - Running processes are tied to their container’s namespace.
  • Snapshots - Filesystem layers and writable overlay mounts belong to a namespace.
  • Content - Blob data (manifests, configs, layers) is scoped to the namespace.
  • Leases - Garbage-collection leases that protect content from being cleaned up.

This means if you run ctr images list without -n k8s.io, you will see nothing - because Kubernetes images live in the k8s.io namespace, not default.

Practical implications
# ❌ Wrong - queries the "default" namespace (empty on most K8s nodes)
ctr images list

# ✅ Correct - queries the Kubernetes namespace
ctr -n k8s.io images list

You can also set an environment variable to avoid typing -n every time:

export CONTAINERD_NAMESPACE=k8s.io
ctr images list   # now queries k8s.io

Quick rule of thumb

If you are debugging Kubernetes containers, always use -n k8s.io. If you are doing standalone containerd work (e.g., testing image pulls), use -n default or a custom namespace.

Step 2.1 - List namespaces
ctr namespaces list

Expected output:

NAME    LABELS
k8s.io

Step 2.2 - Always specify the namespace

Almost every ctr command requires -n <namespace>:

# List images in the k8s.io namespace (Kubernetes images)
ctr -n k8s.io images list

# List images in the default namespace
ctr -n default images list

Common Mistake

Forgetting -n k8s.io is the #1 reason people think “there are no images” on a Kubernetes node.

Step 2.3 - Create a custom namespace
ctr namespaces create workshop-debug
ctr namespaces list
Step 2.4 - Remove a namespace
ctr namespaces remove workshop-debug

03. Image Management

Step 3.1 - List all images
ctr -n k8s.io images list

The output shows image references, types, sizes, and platforms.

Step 3.2 - List images with filters (quiet mode)
# Show only image references
ctr -n k8s.io images list -q

# Count total images
ctr -n k8s.io images list -q | wc -l
Step 3.3 - Pull an image
ctr -n k8s.io images pull docker.io/library/alpine:3.19
Step 3.4 - Inspect image details
# Show image metadata
ctr -n k8s.io images check

# Get detailed image info
ctr -n k8s.io images list | grep alpine
Step 3.5 - Tag an image
ctr -n k8s.io images tag docker.io/library/alpine:3.19 my-alpine:latest
Step 3.6 - Remove an image
ctr -n k8s.io images remove my-alpine:latest
Step 3.7 - Export an image to a tarball
ctr -n k8s.io images export alpine.tar docker.io/library/alpine:3.19
ls -lh alpine.tar
Step 3.8 - Import an image from a tarball
ctr -n k8s.io images import alpine.tar

04. Running Containers

Step 4.1 - Run a container (foreground)
ctr -n k8s.io run --rm docker.io/library/alpine:3.19 test-container echo "Hello from ctr"

Breakdown: - run - create + start a container - --rm - remove container after exit - docker.io/library/alpine:3.19 - image reference - test-container - container ID (must be unique) - echo "Hello from ctr" - command to execute

Step 4.2 - Run a container (interactive shell)
ctr -n k8s.io run --rm -t docker.io/library/alpine:3.19 shell-test sh

Inside the container:

hostname
cat /etc/os-release
ls /
exit

Step 4.3 - Run a container (detached / background)
ctr -n k8s.io run -d docker.io/library/alpine:3.19 bg-container sleep 3600
Step 4.4 - List running containers
ctr -n k8s.io containers list
# OR short form
ctr -n k8s.io c ls
Step 4.5 - Inspect a container
ctr -n k8s.io containers info bg-container

This outputs the full container spec as JSON - image, mounts, environment, process, etc.

Step 4.6 - Get container info (pretty print)
ctr -n k8s.io containers info bg-container | python3 -m json.tool

05. Task Management

A task in containerd represents the running process of a container. Containers and tasks are separate concepts:

  • Container = configuration + filesystem
  • Task = running process within a container
Step 5.1 - List running tasks
ctr -n k8s.io tasks list
# OR short form
ctr -n k8s.io t ls
Step 5.2 - View task processes
ctr -n k8s.io tasks ps bg-container
Step 5.3 - Exec into a running task
ctr -n k8s.io tasks exec --exec-id shell-1 -t bg-container sh

Inside:

ps aux
cat /proc/1/cmdline
exit

Step 5.4 - Pause and resume a task
# Pause
ctr -n k8s.io tasks pause bg-container
ctr -n k8s.io tasks list   # Status: PAUSED

# Resume
ctr -n k8s.io tasks resume bg-container
ctr -n k8s.io tasks list   # Status: RUNNING
Step 5.5 - Kill a task
ctr -n k8s.io tasks kill bg-container
ctr -n k8s.io tasks list   # Status: STOPPED
Step 5.6 - Delete the stopped task and container
ctr -n k8s.io tasks delete bg-container
ctr -n k8s.io containers delete bg-container

06. Inspecting Kubernetes Containers

This is where ctr becomes a powerful debugging tool - inspecting what Kubernetes is actually running.

Step 6.1 - List all Kubernetes containers
ctr -n k8s.io containers list
Step 6.2 - Find a specific pod’s containers
# Find containers related to apphub
ctr -n k8s.io containers list | grep -i apphub
Step 6.3 - Inspect a Kubernetes container’s spec
# Get the container ID from the list above
CONTAINER_ID=$(ctr -n k8s.io c ls | grep apphub | head -1 | awk '{print $1}')

# Inspect it
ctr -n k8s.io containers info "${CONTAINER_ID}" | python3 -m json.tool | head -50
Step 6.4 - Check the image used by a container
ctr -n k8s.io containers info "${CONTAINER_ID}" | python3 -c "
import json, sys
info = json.load(sys.stdin)
print('Image:', info.get('Image', 'N/A'))
print('Snapshotter:', info.get('Snapshotter', 'N/A'))
print('SnapshotKey:', info.get('SnapshotKey', 'N/A'))
"

07. Content Store

The content store is containerd’s content-addressable storage for image layers, configs, and manifests.

Step 7.1 - List content
ctr -n k8s.io content list | head -20
Step 7.2 - Get content by digest
# Get a digest from the list above
DIGEST=$(ctr -n k8s.io content list | head -2 | tail -1 | awk '{print $1}')

# Read the content (for manifests/configs - will be JSON)
ctr -n k8s.io content get "${DIGEST}" | python3 -m json.tool 2>/dev/null || \
  echo "(binary content - likely a layer)"
Step 7.3 - Check content store usage
ctr -n k8s.io content list | wc -l
echo "Total content objects in store"

08. Snapshots

Snapshots are the filesystem layers that make up a container’s rootfs.

Step 8.1 - List snapshots
ctr -n k8s.io snapshots list
# OR with short alias
ctr -n k8s.io sn ls
Step 8.2 - View snapshot info
# Get a snapshot key
SNAP=$(ctr -n k8s.io sn ls | head -2 | tail -1 | awk '{print $1}')

ctr -n k8s.io snapshots info "${SNAP}"
Step 8.3 - View snapshot tree
ctr -n k8s.io snapshots tree

This shows the parent-child relationship of layers - useful for understanding image layer sharing.


09. Plugins

Step 9.1 - List all containerd plugins
ctr plugins list
Step 9.2 - Filter by type
# Show only snapshot plugins
ctr plugins list | grep -i snapshot

# Show only content plugins
ctr plugins list | grep -i content

10. Practical Exercises

Exercise 1 - Image Investigation

On the kind worker node:

docker run -it felipecruz/debug-ctr-extension:latest

# 1. How many images are loaded in the k8s.io namespace?
ctr -n k8s.io images list -q | wc -l

# 2. Find all workshop app images
ctr -n k8s.io images list -q | grep -E "app[1-4]|apphub"

# 3. Export a workshop image as a tarball
ctr -n k8s.io images export /tmp/apphub.tar $(ctr -n k8s.io images list -q | grep apphub | head -1)

# 4. Check the tarball size
ls -lh /tmp/apphub.tar
Exercise 2 - Container Deep Dive
# 1. List all running tasks
ctr -n k8s.io tasks list

# 2. Find the apphub container
ctr -n k8s.io c ls | grep apphub

# 3. Get its PID on the host
ctr -n k8s.io tasks list | grep apphub

# 4. Inspect the process from the host
APPHUB_PID=$(ctr -n k8s.io t ls | grep apphub | awk '{print $2}')
ls -la /proc/${APPHUB_PID}/root/ 2>/dev/null || echo "Check PID manually"
Exercise 3 - Run a Debug Container
# 1. Pull a network debugging image
ctr -n k8s.io images pull docker.io/nicolaka/netshoot:latest

# 2. Run it in the k8s.io namespace
ctr -n k8s.io run --rm -t docker.io/nicolaka/netshoot:latest debug-net bash

# Inside the container:
ip addr
nslookup kubernetes.default.svc.cluster.local
curl -v http://apphub.workshop.svc.cluster.local:5000/health 2>&1 || echo "Expected: won't resolve without K8s networking"
exit
Exercise 4 - Snapshot Forensics
# 1. List all snapshots
ctr -n k8s.io sn ls | wc -l

# 2. Show the snapshot tree to understand layer sharing
ctr -n k8s.io sn tree | head -40

# 3. Find how many snapshots are "active" (writable) vs "committed" (read-only)
ctr -n k8s.io sn ls | awk '{print $3}' | sort | uniq -c

Quick Reference

ctr Cheat Sheet
# ── Namespaces ──
ctr ns list                              # list namespaces
ctr ns create <name>                     # create namespace
ctr ns remove <name>                     # remove namespace

# ── Images ──
ctr -n k8s.io i list                     # list images
ctr -n k8s.io i list -q                  # list image refs only
ctr -n k8s.io i pull <ref>               # pull image
ctr -n k8s.io i tag <src> <dst>          # tag image
ctr -n k8s.io i remove <ref>             # remove image
ctr -n k8s.io i export out.tar <ref>     # export to tar
ctr -n k8s.io i import in.tar            # import from tar
ctr -n k8s.io i check                    # check image status

# ── Containers ──
ctr -n k8s.io c list                     # list containers
ctr -n k8s.io c info <id>                # inspect container
ctr -n k8s.io c delete <id>              # delete container

# ── Tasks ──
ctr -n k8s.io t list                     # list tasks
ctr -n k8s.io t ps <id>                  # show processes
ctr -n k8s.io t exec --exec-id s1 -t <id> sh  # exec into task
ctr -n k8s.io t pause <id>              # pause task
ctr -n k8s.io t resume <id>             # resume task
ctr -n k8s.io t kill <id>               # kill task
ctr -n k8s.io t delete <id>             # delete stopped task

# ── Run ──
ctr -n k8s.io run --rm <image> <id> <cmd>      # run (foreground)
ctr -n k8s.io run --rm -t <image> <id> sh       # run (interactive)
ctr -n k8s.io run -d <image> <id> <cmd>          # run (detached)

# ── Content & Snapshots ──
ctr -n k8s.io content list               # list content objects
ctr -n k8s.io content get <digest>       # read content by digest
ctr -n k8s.io sn list                    # list snapshots
ctr -n k8s.io sn info <key>              # snapshot info
ctr -n k8s.io sn tree                    # snapshot tree

# ── Misc ──
ctr plugins list                         # list containerd plugins
ctr version                              # containerd version

Summary

Concept What you learned
ctr Native containerd CLI for low-level debugging
Namespaces k8s.io isolates Kubernetes-managed resources
Images Pull, tag, export, import - all at the containerd level
Containers Inspect container specs, find K8s pods at runtime
Tasks Running processes - pause, resume, exec, kill
Content Store Content-addressable storage for layers and manifests
Snapshots Filesystem layers - tree view shows layer sharing

Next Step

Continue to Lab 020 · crictl to learn the CRI-level debugging tool that sits between kubelet and containerd.