Containers 101: attach vs. exec - what's the difference?

The difference between docker (or podman, or containerd) attach and exec commands is a common source of confusion. And it's understandable - these two commands have similar arguments and, at first sight, similar behavior. However, attach and exec aren't interchangeable. They aim to cover different use cases, and the implementation of the commands also differs. But still, it might be hard to remember when to use which command.

Since I'm no fan of brute memorization, here is my recipe of how I managed to internalize the difference. Long story short, connecting the dots between the knowledge of what containers really are under the hood and these two commands helped to grasp the difference almost instantly. And like any true understanding, it freed me from relying only on my memory and gave me a chance to extrapolate the knowledge on a similar tech such as Kubernetes 😉

Container management - look inside

First, a quick recap of how docker architecture looks like:

Three key takeaways:

• Container management architecture is layered
• A container is a regular process an isolated and restricted environment + a process
• There is a shim component in between the container manager and the container.

What does attach command do

Inside a container, there is a regular Linux process. As every normal process, it has stdio streams - stdin, stdout, and stderr. But what happens to these streams when a container is started in the detached (i.e., daemon-like) mode:

$ docker run -d nginx

Back in the day, when you started a process as a daemon (i.e., detaching it from the starter process), it would be reparented to PID 1, and its stdio streams would be simply closed. However, we all know how handy it is to use stdout and stderr streams for logging nowadays, thanks to containers!

Docker came up with a clever idea of putting an extra process between the container and the rest of the system, called a container runtime shim. In the example above, the container manager actually starts a shim process, that in turn, uses an OCI-compatible runtime (e.g., runc) to start the actual container.

This is the shim process that becomes a daemon - it's reparented to PID 1, and its stdio streams are closed:

However, the shim takes control over the container's stdio streams!

The daemonized shim process reads from the container's stdout and stderr and dumps the read bytes to the log driver. By default, the shim closes the container's stdin stream, but it can keep it open if -i was passed to the corresponding docker run command.

Container runtime shim actually acts as a server! It provides RPC means (for instance, a UNIX socket) to connect to it. And when you do so, it starts streaming the container's stdout and stderr back to your end of the socket. It also can read from this socket and forward data to the container's stdin. Hence, you kind of attach to the container's stdio streams!

So, finally, when you run docker attach <container>, you basically create a relay:

terminal <-> docker <-> dockerd <-> shim <-> container's stdio streams

Difference between attach and logs

On the diagram above, docker attach streams the container's logs back to the terminal. However, the docker logs command does a similar thing. So, what's the difference?

The logs command provides various options to filter the logs while attach in that regard acts as a simple tail. But what's even more important is that the stream established by the logs command is always unidirectional and connected to the container's logs, not the container's stdio streams directly.

The logs command simply streams the content of the container's logs back to your terminal, and that's it. So, regardless of how you created your container (interactive or non-interactive, controlled by a pseudo-terminal or not), you cannot accidentally impact the container while using the logs command.

However, when attach is used:

• If a container was created in the interactive mode (-i), everything you type in the terminal after attach-ing to the container will be sent to its stdin.
• You can (intentionally or accidentally) send a signal to the container - for instance, hitting ctrl+c on your end while attached sends SIGINT to the container.

Read more about it → Linux PTY - How docker attach and docker exec Commands Work Inside.

What does exec command do

I hope the attach command has been sorted out by now. So, it's time to tackle the exec counterpart!

The exec command is actually a totally different story. In the case of attach, we were connecting our terminal to an existing container (read, process). However, the exec command starts a totally new container! In other words, exec is a form of the run command (which itself is just a shortcut for create + start).

But why then docker exec --help says:

Usage:  docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

Run a command in a running container

The trick here is that the auxiliary container created by the exec command shares all the isolation boundaries of the target container! I.e., the same net, pid, mount, etc. namespaces, same cgroups hierarchy, etc. So, from the outside, it feels like running a command inside of an existing container.

The confusion of attach and exec commands arises because the exec-uted command is also a process with its own stdio streams. So, you can choose whether to exec in detached mode, whether to keep the stdin open, whether to allocate a pseudo-TTY, etc. Also, when exec-ing, the relay looks quite similarly:

terminal <-> docker-cli <-> dockerd <-> shim <-> command's stdio streams

Other implementations

The above diagrams and examples were mostly about docker, but other container managers, such as containerd or cri-o behave similarly when it comes to run, exec, attach, or logs commands.

Podman is probably the most prominent example of a daemon-less container manager. However, even podman employs container runtime shims. There is just one less hop in the relay when you attach to a podman's container.

The interesting specimen is Kubernetes, though. Kubernetes doesn't manage containers directly. Instead, every cluster node has a local agent, called kubelet, that in turn expects a compatible container runtime to be present on the node. However, on the lowest level, there are still the same shims and processes:

Much like docker, Kubernetes' command-line client (kubectl) also provides exec, attach, and logs commands with a similar UX. The difference is that Kubernetes operates in terms of Pods and not containers. Luckily, pods are just groups of semi-fused containers, so all the stuff we learned so far is still applicable.

Since attach-ing, logs-ing, and exec-ing works on the container level, every kubectl attach, kubectl logs, and kubectl exec needs to specify the target container (-c <name>) in addition to the pod name. Unless the pod was annotated with kubectl.kubernetes.io/default-container, of course.

Conclusion

So, to summarize:

• Containers are isolated and restricted execution environments.
• Conventionally, there is one main process per container.
• Containers are typically started in the detached mode (i.e., like daemons).
• Container runtime shim wraps the container process and streams its stdout and stderr to logs.
• Runtime shim allows attach-ing to it to wire the terminal with the container's stdio stream.
• It's possible to start a container reusing the isolation primitives of an already running container.
• The exec command is similar to the run command with all the namespaces and cgroups reused from another container.
• Since exec-ing by default happens in the attached mode, it might look similar to the attach command, but its purpose and implementation are quite different.

Further reading

• Linux PTY - How docker attach and docker exec Commands Work Inside
• Debunking Container Myths (series)
• Implementing Container Manager (series)
• Learning Containers From The Bottom Up (learning path)