Debugging even simple containerized applications is challenging. Debugging applications that run in distroless containers is hard. And debugging distroless containers running in a Kubernetes cluster is close to impossible. Unless you know a trick a two. This series will teach you how to use different container debugging tools and techniques to troubleshoot your containerized workloads.
Slim containers are faster (less stuff to move around) and more secure (fewer places for vulnerabilities to sneak in). However, these benefits of slim containers come at a price - such containers lack (the much-needed at times) exploration and debugging tools. It might be quite challenging to tap into a container that was built from a distroless or slim base image or was minified using DockerSlim or alike. Over the years, I've learned a few tricks how to troubleshoot slim containers, and it's time for me to share.
Last week at KubeCon, there was a talk about Kubernetes ephemeral containers. The room was super full - some people were even standing by the doors trying to sneak in. "This must be something really great!" - thought I and decided to finally give Kubernetes ephemeral containers a try.
So, below are my findings - traditionally sprinkled with a bit of containerization theory and practice 🤓
TL;DR: Ephemeral containers are indeed great and much needed. The fastest way to get started is the kubectl debug command. However, this command might be tricky to use if you're not container-savvy.
The difference between docker (or podman, or containerd) attach and exec commands is a common source of confusion. And it's understandable - these two commands have similar arguments and, at first sight, similar behavior. However, attach and exec aren't interchangeable. They aim to cover different use cases, and the implementation of the commands also differs. But still, it might be hard to remember when to use which command.
Since I'm no fan of brute memorization, here is my recipe of how I managed to internalize the difference. Long story short, connecting the dots between the knowledge of what containers really are under the hood and these two commands helped to grasp the difference almost instantly. And like any true understanding, it freed me from relying only on my memory and gave me a chance to extrapolate the knowledge on a similar tech such as Kubernetes 😉
containerd is a high-level container runtime, aka container manager. To put it simply, it's a daemon that manages the complete container lifecycle on a single host: creates, starts, stops containers, pulls and stores images, configures mounts, networking, etc.
containerd is designed to be easily embeddable into larger systems. Docker uses containerd under the hood to run containers. Kubernetes can use containerd via CRI to manage containers on a single node. But smaller projects also can benefit from the ease of integrating with containerd - for instance, faasd uses containerd (we need more d's!) to spin up a full-fledged Function-as-a-Service solution on a standalone server.
However, using containerd programmatically is not the only option. It also can be used from the command line via one of the available clients. The resulting container UX may not be as comprehensive and user-friendly as the one provided by the docker client, but it still can be useful, for instance, for debugging or learning purposes.
A container image is a combination of layers where every layer represents some intermediary state of the final filesystem. Such a layered composition makes the building, storage, and distribution of images more efficient. But from a mere developer's standpoint, images are just root filesystems of our future containers. And we often want to explore their content accordingly - with familiar tools like cat, ls, or file. Let's try to see if we can achieve this goal using nothing but the means provided by Docker itself.
From time to time I use kind as a local Kubernetes playground. It's super-handy, real quick, and 100% disposable.
Up until recently, all the scenarios I've tested with kind were using public container images. However, a few days ago, I found myself in a situation where I needed to run a pod using an image that I had just built on my laptop.
One way of doing it would be pushing the image to a local or remote registry accessible from inside the kind Kubernetes cluster. However, kind still doesn't spin up a local registry out of the box (you can vote for the GitHub issue here) and I'm not a fan of sending stuff over the Internet without very good reasons.
Don't miss new posts in the series! Subscribe to the blog updates and get deep technical write-ups on Cloud Native topics direct into your inbox.
