Percona Monitoring and Management Container, Are You Ready Yet?

My boss asked me if there is a way to wait for the Percona Monitoring and Management (PMM) Server container to be fully ready so his automation script can start adding monitoring instances. I wanted to give him a general solution, not a PMM-specific one. Turned out, it is not so simple.

The root of the problem is that some big containers, including PMM Server, take some time to boot up. We can’t assume that it is ready once docker run -d exited, or once the network port is ready to accept connections. For example, PMM Server contains nginx, and it is ready to accept connections almost immediately, while other components are still starting. We want to know when it is “fully” ready.

Let’s start with the PMM-specific solution. In both PMM 1.x and 2.x we expose a separate endpoint for that purpose – /ping. You can use it with curl and a simple shell loop:

$ until curl -f http://127.0.0.1/ping; do sleep 1; done
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused
curl: (52) Empty reply from server
curl: (22) The requested URL returned error: 503 Service Unavailable
{}

-f (--fail) flag causes curl to exit with code 22 if the HTTP response code is 400 or above. In the example above, you can see three failed attempts with different exit codes, and then the successful response (empty JSON object).

Is there a more generic way for other Docker containers? It turned out, there is: Docker 1.12, released more than 4 years ago, introduced HEALTHCHECK Dockerfile instruction for that purpose. It can be used like this:

HEALTHCHECK --interval=3s --timeout=2s --start-period=10s --retries=3 CMD curl -f http://127.0.0.1/ping || exit 1

Unlike the previous example, that command runs inside Docker container, not outside. The meaning of those parameters is documented in Docker’s reference. We add || exit 1 because CMD should return either 0 or 1, nothing else.

Surprisingly, there is no built-in docker subcommand to wait for a container to become healthy, so we have to use a shell loop again:

until [ "`docker inspect -f {{.State.Health.Status}} pmm-server`" = "healthy" ]; do sleep 1; done

Unfortunately, not many images contain the HEALTHCHECK instruction. In fact, while writing that blog post, I noticed that PMM Server does not have it! So I added it, and it will be released as part of PMM 2.4. But we definitely don’t want to change all images we want to use that don’t contain this instruction. Luckily, HEALTHCHECK functionality is also available via docker run flags:

docker run --health-cmd='curl -f http://127.0.0.1/ping || exit 1' --health-interval=3s --health-interval=2s --health-start-period=10s --health-retries=3 …

Then we can use the same “until docker inspect” loop to wait for a container to become healthy.

Hopefully, that information will be useful to you for running third-party containers without health checks. But please, add the HEALTHCHECK instruction to your own Dockerfiles, like I did.