Debugging even simple containerized applications is challenging. Debugging applications that run in distroless containers is hard. And debugging distroless containers running in a Kubernetes cluster is close to impossible. Unless you know a trick a two. This series will teach you how to use different container debugging tools and techniques to troubleshoot your containerized workloads.

Using a special sidecar to debug a potentially distroless container.

Docker: How To Debug Distroless And Slim Containers

Slim containers are faster (less stuff to move around) and more secure (fewer places for vulnerabilities to sneak in). However, these benefits of slim containers come at a price - such containers lack (the much-needed at times) exploration and debugging tools. It might be quite challenging to tap into a container that was built from a distroless or slim base image or was minified using DockerSlim or alike. Over the years, I've learned a few tricks how to troubleshoot slim containers, and it's time for me to share.

Read more

Kubernetes Ephemeral Containers and kubectl debug Command

Last week at KubeCon, there was a talk about Kubernetes ephemeral containers. The room was super full - some people were even standing by the doors trying to sneak in. "This must be something really great!" - thought I and decided to finally give Kubernetes ephemeral containers a try.

So, below are my findings - traditionally sprinkled with a bit of containerization theory and practice 🤓

TL;DR: Ephemeral containers are indeed great and much needed. The fastest way to get started is the kubectl debug command. However, this command might be tricky to use if you're not container-savvy.

Read more

Containers 101: attach vs. exec - what's the difference?

The difference between docker (or podman, or containerd) attach and exec commands is a common source of confusion. And it's understandable - these two commands have similar arguments and, at first sight, similar behavior. However, attach and exec aren't interchangeable. They aim to cover different use cases, and the implementation of the commands also differs. But still, it might be hard to remember when to use which command.

Since I'm no fan of brute memorization, here is my recipe of how I managed to internalize the difference. Long story short, connecting the dots between the knowledge of what containers really are under the hood and these two commands helped to grasp the difference almost instantly. And like any true understanding, it freed me from relying only on my memory and gave me a chance to extrapolate the knowledge on a similar tech such as Kubernetes 😉

Read more

Why and How to Use containerd From Command Line

containerd is a high-level container runtime, aka container manager. To put it simply, it's a daemon that manages the complete container lifecycle on a single host: creates, starts, stops containers, pulls and stores images, configures mounts, networking, etc.

containerd is designed to be easily embeddable into larger systems. Docker uses containerd under the hood to run containers. Kubernetes can use containerd via CRI to manage containers on a single node. But smaller projects also can benefit from the ease of integrating with containerd - for instance, faasd uses containerd (we need more d's!) to spin up a full-fledged Function-as-a-Service solution on a standalone server.

Docker and Kubernetes use containerd

However, using containerd programmatically is not the only option. It also can be used from the command line via one of the available clients. The resulting container UX may not be as comprehensive and user-friendly as the one provided by the docker client, but it still can be useful, for instance, for debugging or learning purposes.

containerd command-line clients (ctr, nerdctl, crictl)

Read more

Docker: How To Extract Image Filesystem Without Running Any Containers

A container image is a combination of layers where every layer represents some intermediary state of the final filesystem. Such a layered composition makes the building, storage, and distribution of images more efficient. But from a mere developer's standpoint, images are just root filesystems of our future containers. And we often want to explore their content accordingly - with familiar tools like cat, ls, or file. Let's try to see if we can achieve this goal using nothing but the means provided by Docker itself.

Container image to filesystem.

Read more

KiND - How I Wasted a Day Loading Local Docker Images

From time to time I use kind as a local Kubernetes playground. It's super-handy, real quick, and 100% disposable.

It just so happened that virtually all the scenarios I've been testing so far were based on publicly available images. But recently I found myself in a situation when I needed to run a pod with a container image that I've just built on my laptop.

One way of doing it would be pushing the image to a local or remote registry accessible from inside the kind Kubernetes cluster. However, kind still doesn't spin up a local registry out of the box (you can vote for the GitHub issue here) and I'm not a fan of sending stuff over the Internet without very good reasons.

Read more