The other day I decided to update my server-side rendering example, and I ran into a strange problem. While my compose.yaml hasn’t changed, my containers could not talk to each other. This was most apparent with the nginx container, which immediately crashed out with [emerg] 1#1: host not found in upstream. What gives?

After a quick debugging including on a different computer, it became clear that my compose.yaml should be fine: it worked without issues on the computer that used Docker, but didn’t work with Podman. The issue with Podman was soon clear too: looking at podman network inspect the "dns_enabled": false was an obvious culprit. That’s where things got interesting.

Because it was extremely not obvious how to “switch on” DNS on the virtual networks. Adding depends_on to the nginx container didn’t work (in retrospect naturally, since that doesn’t have anything to do with the network’s DNS settings), and for a similar reason network aliases were a dead end too.

Looking at podman help network create it had an option for disabling DNS (surprisingly --disable-dns), but none to enable it. I had to assume that it’s supposed to be enabled by default. Except quite apparently it was not.

On many forums where I found questions like this, the common suggestion was to install the podman-plugins package. Then in podman info you’d see something like this:

  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1

My network backend was CNI too, but I couldn’t for my life find this “podman-plugins” package. I don’t know if it’s a RHEL-only thing or all those forum posts are just too old and this package shouldn’t even be necessary anymore.

While at the point of writing Podman 5.3 is the latest, my Ubuntu 24.04 install has 4.9. Before running off the apt-get rails and installing it manually, I tried something else. Some people out there mentioned using netavark for backend, so I gave that a try. Since I already had both netavark and aardvark-dns installed (not sure why), I dove into containers.conf (in my case it was at /usr/share/containers/containers.conf and not at /etc/containers/containers.conf) and set the network_backend to netavark.

After service podman restart I confirmed with podman info that my settings are updated:

  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64

I also had to recreate the compose network (by running podman compose down once), but after that things finally worked just fine. I still don’t know how to achieve the same with the default CNI backend or if it’s possible at all (feels like it definitely should be though).

Another thing to watch out for, that unlike Docker, you must have a networks section in your compose.yaml for containers to be able to talk to each other. Which is a pretty nasty hidden “feature” since even if it’s not there, at startup it still says something like “Creating network “hoge_default” with the default driver”. Except of course containers can’t resolve each other without explicitly adding a default network like this:

networks:
  default:

This was an extremely frustrating journey, a large part of which I just kept staring at that "dns_enabled": false and ripping my hair out trying to find some, any documentation on how to make that true. In the end I couldn’t find any, but switching to netavark instead of cni luckily solved the issue.