The difference between docker (or podman, or containerd)
exec commands is a common source of confusion. And it's understandable - these two commands have similar arguments and, at first sight, similar behavior. However,
exec aren't interchangeable. They aim to cover different use cases, and the implementation of the commands also differs. But still, it might be hard to remember when to use which command.
Since I'm no fan of brute memorization, here is my recipe of how I managed to internalize the difference. Long story short, connecting the dots between the knowledge of what containers really are under the hood and these two commands helped to grasp the difference almost instantly. And like any true understanding, it freed me from relying only on my memory and gave me a chance to extrapolate the knowledge on a similar tech such as Kubernetes 😉
Slack is full of Prometheus alerts? Try Robusta - an open-source platform that makes Kubernetes troubleshooting faster. Robusta identifies alerting patterns, enriches alerts with contextual data, and helps you get rid of false positives. Go check it out!
Container management - look inside
First, a quick recap of how docker architecture looks like:
Three key takeaways:
- Container management architecture is layered
- A container is
a regular processan isolated and restricted environment + a process
- There is a shim component in between the container manager and the container.
attach command do
Inside a container, there is a regular Linux process. As every normal process, it has stdio streams - stdin, stdout, and stderr. But what happens to these streams when a container is started in the detached (i.e., daemon-like) mode:
$ docker run -d nginx
Back in the day, when you started a process as a daemon (i.e., detaching it from the starter process), it would be reparented to PID 1, and its stdio streams would be simply closed. However, we all know how handy it is to use stdout and stderr streams for logging nowadays, thanks to containers!
Docker came up with a clever idea of putting an extra process between the container and the rest of the system, called a container runtime shim. In the example above, the container manager actually starts a shim process, that in turn, uses an OCI-compatible runtime (e.g., runc) to start the actual container.
This is the shim process that becomes a daemon - it's reparented to PID 1, and its stdio streams are closed:
ps axfo pid,ppid,command
However, the shim takes control over the container's stdio streams!
The daemonized shim process reads from the container's stdout and stderr and dumps the read bytes to the log driver. By default, the shim closes the container's stdin stream, but it can keep it open if
-i was passed to the corresponding
docker run command.
Container runtime shim actually acts as a server! It provides RPC means (for instance, a UNIX socket) to connect to it. And when you do so, it starts streaming the container's stdout and stderr back to your end of the socket. It also can read from this socket and forward data to the container's stdin. Hence, you kind of attach to the container's stdio streams!
So, finally, when you run
docker attach <container>, you basically create a relay:
terminal <-> docker <-> dockerd <-> shim <-> container's stdio streams
On the diagram above,
docker attach streams the container's logs back to the terminal. However, the
docker logs command does a similar thing. So, what's the difference?
logs command provides various options to filter the logs while
attach in that regard acts as a simple
tail. But what's even more important is that the stream established by the
logs command is always unidirectional and connected to the container's logs, not the container's stdio streams directly.
logs command simply streams the content of the container's logs back to your terminal, and that's it. So, regardless of how you created your container (interactive or non-interactive, controlled by a pseudo-terminal or not), you cannot accidentally impact the container while using the
attach is used:
- If a container was created in the interactive mode (
-i), everything you type in the terminal after
attach-ing to the container will be sent to its stdin.
- You can (intentionally or accidentally) send a signal to the container - for instance, hitting
ctrl+con your end while attached sends
SIGINTto the container.
Read more about it → Linux PTY - How docker attach and docker exec Commands Work Inside.
exec command do
I hope the
attach command has been sorted out by now. So, it's time to tackle the
exec command is actually a totally different story. In the case of
attach, we were connecting our terminal to an existing container (read, process). However, the
exec command starts a totally new container! In other words,
exec is a form of the
run command (which itself is just a shortcut for
💡 Fun fact - The OCI Runtime Spec doesn't define
exec commands. Check out the issues #345 and #388 for an interesting discussion of how the
exec functionality is actually redundant and can be reproduced in runtimes implementing just
But why then
docker exec --help says:
Usage: docker exec [OPTIONS] CONTAINER COMMAND [ARG...] Run a command in a running container
The trick here is that the auxiliary container created by the
exec command shares all the isolation boundaries of the target container! I.e., the same net, pid, mount, etc. namespaces, same cgroups hierarchy, etc. So, from the outside, it feels like running a command inside of an existing container.
The confusion of
exec commands arises because the
exec-uted command is also a process with its own stdio streams. So, you can choose whether to
exec in detached mode, whether to keep the stdin open, whether to allocate a pseudo-TTY, etc. Also, when
exec-ing, the relay looks quite similarly:
terminal <-> docker-cli <-> dockerd <-> shim <-> command's stdio streams
The above diagrams and examples were mostly about docker, but other container managers, such as containerd or cri-o behave similarly when it comes to
Podman is probably the most prominent example of a daemon-less container manager. However, even podman employs container runtime shims. There is just one less hop in the relay when you
attach to a podman's container.
The interesting specimen is Kubernetes, though. Kubernetes doesn't manage containers directly. Instead, every cluster node has a local agent, called kubelet, that in turn expects a compatible container runtime to be present on the node. However, on the lowest level, there are still the same shims and processes:
Much like docker, Kubernetes' command-line client (
kubectl) also provides
logs commands with a similar UX. The difference is that Kubernetes operates in terms of Pods and not containers. Luckily, pods are just groups of semi-fused containers, so all the stuff we learned so far is still applicable.
exec-ing works on the container level, every
kubectl logs, and
kubectl exec needs to specify the target container (
-c <name>) in addition to the pod name. Unless the pod was annotated with
kubectl.kubernetes.io/default-container, of course.
So, to summarize:
- Containers are isolated and restricted execution environments.
- Conventionally, there is one main process per container.
- Containers are typically started in the detached mode (i.e., like daemons).
- Container runtime shim wraps the container process and streams its stdout and stderr to logs.
- Runtime shim allows
attach-ing to it to wire the terminal with the container's stdio stream.
- It's possible to start a container reusing the isolation primitives of an already running container.
execcommand is similar to the
runcommand with all the namespaces and cgroups reused from another container.
exec-ing by default happens in the attached mode, it might look similar to the
attachcommand, but its purpose and implementation are quite different.
Join more than a thousand happy subscribers receiving my Cloud Native round-up and get deep technical write-ups from this blog direct into your inbox.