Introduction

Podman is a container engine sponsored by RedHat.

They qualify it as "daemonless", which is accurate in the sense that there is no central daemon (as with the docker daemon) that handles everything when it comes to container lifecycles and configuration.

More importantly, that central daemon that Docker uses has to run as root — a podman container could be started with no process running as root being involved at all except for systemd which is already required and running on most modern Linux systems anyway.

This brings us to one of the main features of podman: it allows us to run containers as a specific user, without involving any root privileges or a special group (the "docker" group is often used to allow access to the Docker daemon).

As we'll verify later on, running containers as a non-root user will save everything inside the home directory of said user and won't involve any global path whatsoever, completely isolating the different users on the system.

Of course, running containers as root is still allowed and behaves pretty much like Docker would.

Another podman feature is in the name and is the ability to run pods, which are groups of containers that share a "localhost" network, among other things. It's pretty much copied from the notion of pod in Kubernetes.

We'll have a separate article later on talking about pods within podman.

The main disadvantage of using podman is that it's still a young project at this time, lacking documentation and options for many advanced things or just relying on systemd for these.

It also won't run on Windows at all and I'd probably advise against using it on Mac, even though there seems to be a Mac version. In short, it's really meant for Linux only — Which is fine for our server uses.

Installing Podman

Installing podman is very easy and doesn't involve adding any extra repository when using one of the popular Linux distributions.

Its only requirements are a modern kernel and systemd.

For anything Debian-based, this seems to install everything needed:

apt install podman

On some distributions you might need to manually install the userspace networking package which is usually called slirp4netns.

This is another sleeper advantage I find for Podman over Docker, as most Docker setups will use firewall rules (iptables - as root) for networking, port bindings etc. And that can interfere with the server firewall in some cases. The way Podman does it is more flexible.

Adding registries

The Debian installation of podman appears to get installed with no container registries set up out of the box, which means you can't search for images or download them from a registry like the Docker hub.

I assume this is done to promote neutrality as most popular registries are owned by some big company or another.

To add registries, edit the file /etc/containers/registries.conf (as root) and add or modify the line:

unqualified-search-registries = ['docker.io', 'quay.io']

Here we're adding two of them for the example, but that means you'll either have to qualify where you want to pull images from every time, or the podman command line utility will ask you to pick a registry when the image is found on multiple ones.

For basic use and being as close as possible to what you'd do with Docker, the following suffices:

unqualified-search-registries = ['docker.io']

To check what registries are currently configured for lookup, you can use:

podman info

Rootless containers

It's important to note that, when running a container as a non-root user using podman:

• There will be a UID 0 (root) available inside the container, making it think it's root but in reality the container is running as your current user, and the other container users are using subuids of your current user;
• The container images (downloaded or running), states, logs, etc. are all stored in your user's home directory — More precisely inside of ~/.local/share/containers/storage;
• Logging your user out will kill the container, unless a specific option is set as root for your user (explained later) or the container is ran through a global systemd service. This isn't really an issue when using containers on a workstation for development or lab work.

Let me demonstrate how the UIDs look by running a simple Nginx container:

podman run -d --rm -p 8080:80 nginx

And now investigating the users involved in the container processes:

sudo ps -ef | grep nginx
william     1986  0.0  0.1  10664  6020 ?        Ss   11:47   0:00 nginx: master process nginx -g daemon off;
100100      2012  0.0  0.0  11068  2684 ?        S    11:47   0:00 nginx: worker process
100100      2013  0.0  0.0  11068  2684 ?        S    11:47   0:00 nginx: worker process

We see the parent Nginx process (which runs as root inside the container) actually runs as my user "william", and the two worker processes (which usually run as womething like www-data) are using UID 100100 which is a subuid of my user.

These subuids are listed in /etc/subuid, one user per line. For instance, this is my user's:

william:100000:65536

Which means subuids for that user start at 100000 and a total of 65536 possible ones are allowed (which means we can go up to subuid 165535).

Containers ran as root won't be killed on log out and will store their info globally in /var/lib/containers/storage.

It's possible to take this even further and make the user inside the container something that isn't root, because the intra-container-root-user, even though it's got limited privileges, still has more privileges than a limited user inside the container.

The issue is that some images, like the Nginx one, won't start unless the container user is root. Some others will run fine though, here's an example with a Redis server running rootless and with the user inside of the container being non-root as well:

podman run -u 1000 -d --rm redis

Where 1000 is my user ID.

If you have a look at the process from the host to see what user ID is actually running the Redis process, you'll see it's one of the subuids (100999) for the specified user (in the case of Nginx presented before, the Nginx master process was running as UID 1000 as in "my user").

ps -ef | grep redis
100999      1859    1856  0 14:36 ?        00:00:00 redis-server *:6379

Running containers this way adds another extra layer of security. Just keep in mind that some images won't work in their original state when the container user isn't root.

Running container at system start

As mentioned previously, rootless containers won't survive the user session closing — Unless you use forking but I consider that confusing and bad practice since we'll want an easy way to stop and start that service and keep it alive anyway, which we can do with systemd (as intended, might I add).

There is a way to run containers using podman with the root account being completely and entirely unnecessary for the whole process. However, it doesn't really fit server use (although there's a command that can fix everything but it requires root access) so we'll present our ways of running containers as a user, as a service, in a server context and then present the recommended userspace way.

In a server context

Let's say we want to run a service as a specific user and have it start with the computer.

First, create that user, simplest way to do so being:

adduser <USERNAME>

Next you'll want to enable a permanent systemd session for that user (requires root privileges):

sudo loginctl enable-linger <USERNAME_OR_UID>

Now log in as the other user (or use su) and prepare the container, using podman run or podman create and give it a name. For example:

podman run -d --name nginx-server -p 8080:80 nginx

Where we create an empty Nginx container named nginx-server meant to run in the background and binding public port 8080 to container port 80.

Note: As a non-root user you can't bind privileged ports (ports below 1000 or 2000 in certain cases). There are multiple solutions to this, the easiest is to add relevant firewall rules (requires root privileges) or rely on an external load balancing and/or reverse proxy service.

Now that we got the user container created, let's create a systemd service in /etc/systemd/system, call it <CONTAINER_NAME>.service and add the following content:

[Unit]
Description=<YOUR_DESCRIPTION>
Wants=syslog.service
[Service]
User=<YOUR_USER>
Group=<YOUR_GROUP>
Restart=always
ExecStart=/usr/bin/podman start -a <CONTAINER_NAME>
ExecStop=/usr/bin/podman stop -t 10 <CONTAINER_NAME>
[Install]
WantedBy=default.target

Where we added the -a option to the start command so that container output goes to journalctl — which means container output logging will effectilvely appear in two places, we'll have a small section about logging later on but having some of it available through systemd is always nice for a server service.

You can now enable it to run at start:

systemctl enable <SERVICE_NAME>

Of course you can start, stop, restart, check the status of your service:

systemctl status <SERVICE_NAME>
systemctl start <SERVICE_NAME>
systemctl stop <SERVICE_NAME>
# Tail the logs using journalctl:
journalctl -u <SERVICE_NAME>.service -f

In a workstation, fully userspace context

What we explain here still requires an active systemd session for the user that will start the containers.

For a workstation on which you log in with you user in a graphical session anyway, you'll get that systemd session anyway and your userspace containers will start at that time automatically.

In a more "background" server context, it's still possible to use the same technique but systemd has to be told to keep a session for the user, as we did previously, and that requries root privileges:

sudo loginctl enable-linger <USERNAME_OR_UID>

Repeating it just in case: you don't need to do this on a workstation-type system, especially when your own user is the one running the containers.

Let's use the echo-server for demonstration purposes, and this time use podman create (which infers the container is meant to be ran in the background) and use the systemd service generation tool included in podman (which works best for these kind of user services):

podman create --name echo-server -p 3000:80 ealen/echo-server
podman generate systemd echo-server --restart-policy=always -t 5 -n > echo-server.service

Where:

• The file is printed out in the standard output, hence the redirect into a file we called echo-server.service
• -t is the stop timeout
• –restart-policy mimicks the same docker option — Default is “on-failure” but we usually want “always” on hosted services
• -n Use the name of the container to name the systemd service

Then:

mkdir -p ~/.config/systemd/user
mv ./echo-server.service ~/.config/systemd/user/

And finally enabling the service to run at session start (make sure you're not using sudo):

systemctl enable --user echo-server.service

Note: You may get an error message at this point due to some environment variables being absent, in which case this should work in fixing it (retry the sytsemctl) command afterwards:

export XDG_RUNTIME_DIR=/run/user/$UID

Review and configure logging

At the moment it isn't easy to find information about podman and logging. Let's quickly review the basics.

I'm pretty sure the command line logging options to the podman command (which are copied from the docker one) do not do anything at this time (version 3).

Container output will be savec by default in one of these locations depending on the container being respectively ran by a regular user or root:

~/.local/share/containers/storage/overlay-containers/<CONTAINER_ID>/userdata/ctr.log
/var/lib/containers/storage/overlay-containers/<CONTAINER_ID>/userdata/ctr.log

By default these grow indefinitely regarless of what extra logging you may be doing (we've been using journalctl in a section before) and there is no rotation.

We recommend limiting their growth by the means of a global configuration file.

The easiest way is to edit or create the file at /etc/containers/containers.conf.

Many Linux distribution provide a commented example file here: usr/share/containers/containers.conf.

There is an option called log_max_size which takes a value in bytes. For instance, to limit container logging to 10MB:

log_size_max = 10485760

The easiest way to get more permanence with your logging and possibly rotation would be to use systemd services logging options, or just use journalctl as is with systemd services.

Conclusion

There are a lot of topics we didn't touch on: pods, as mentioned in the intro, but also networking and what happens with docker-compose, ...

We tried to demontrate how you can transparently swap in podman to fill any basic docker use you have.

As they say themselves, you could safely alias podman as docker in CLI by adding the following line to your ~/.bashrc (or equivalent for your shell):

alias docker=podman

And then source your shell config file to apply the new alias.

We'll be switching to Podman ourselves for most of our use cases. For instance, we'll probably post an update of our procedure to setup an isolated SFTP server.