Understanding Multi-Arch Containers, Benefits, and CI/CD Integration

We'll cover multi-architecture container images, how to create and promote them and share an example code for building one in a CI/CD pipeline.

Have you ever seen the “exec /docker-entrypoint.sh: exec format error” error message on your server while running any docker image or Kubernetes pods? This is most probably because you are running some other CPU architecture container image on your server, or did you ever use --platform linux/x86_64 option on your Apple silicon M1, M2 MacBook? If yes, then you are not getting the native performance of Apple silicon, and it may be draining your MacBook battery. To avoid this kind of error and performance issue, we need to run the correct multi-arch container image, or we may need to build our own image because all container public image does not have the multi-arch image available.

In this blog post, we will learn what are multi-arch container images. How does it work? How to build and promote them? and we will write a sample code for building a multi-arch image in the CI/CD pipeline.

What Is a Multi-Arch Container Image?

A multi-arch Docker image is a list of images that has references to binaries and libraries compiled for multiple CPU architectures. This type of image is useful when we need to run the same application on different CPU architectures (ARM, x86, RISC-V, etc) without creating separate images for each architecture.

Multi-Arch Container Use Cases

• Performance and Cost Optimization: Container multi-arch is used to optimize performance on different CPU architectures. By building and deploying images that are optimized for a specific architecture, we can achieve better performance and reduce resource usage. Using Karpenter, we can easily deploy our workload to arm64 and get the benefit of AWS Graviton’s performance and cost savings.
• Cross-Platform Development: If you are developing an application that needs to run on multiple platforms, such as ARM and x86, you can use buildx to build multi-arch Docker images and test the application on different architectures.
• IoT Devices: Many IoT/Edge devices use ARM processors, which require different binaries and libraries than x86 processors. With multi-arch images, you can create an image that can run on ARM, x86, and RISCV devices, making it easier to deploy your application to a wide range of IoT devices.

Benefits of Using Multi-Arch Container Image

Several advantages of using multi-arch container images are:

• Ability to run Docker images on multiple CPU architectures.
• Enables us to choose eco-friendly CPU architecture.
• Seamless migration from one architecture to another.
• Better performance and cost saving using arm64.
• Ability to support more cores per CPU using arm64.

How To Build Multi-Arch Container Image?

There are multiple ways to build a multi-arch container, but we will be focusing on widely used and easy methods.

• Traditional Docker build command.
• Using Docker buildx

Using Traditional Docker Build Command

In this tutorial, we will manually build both images on different CPU architecture machines and push them to the container registry (e.g., Dockerhub) and then create the manifest file, which has both image references. A manifest file is a simple JSON file containing the index of container images and their metadata like the size of the image, sha256 digest, OS, etc. We will see more about the manifest file later in this blog.

For example, this is our basic Dockerfile.

HTML

FROM nginx

RUN echo “Hello multiarch” > /usr/share/nginx/html/index.html

HTML

########## on amd64 machine ##########

docker build -t username/custom-nginx:v1-amd64 .

docker push username/custom-nginx:v1-amd64

########## on arm64 machine ##########

docker build -t username/custom-nginx:v1-arm64 .

docker push username/custom-nginx:v1-arm64

########## Create a manifest index file ##########

docker manifest create \

    username/custom-nginx:v1 \

    username/custom-nginx:v1-amd64 \

    username/custom-nginx:v1-arm64

########## Push manifest index file ##########

docker manifest push username/custom-nginx:v1

Using Docker Buildx

With buildx, we just need to run one single command with parameterized architecture.

HTML

docker buildx build \

--push \

--platform linux/arm64,linux/amd64 \ 

-t username/custom-nginx:v1 .

In the background, the Docker buildx command uses buildkit, so when we run the above command, it creates one container with moby/buildkitd image, which has QEMU binary for multiple CPU architectures, which are responsible for the emulating CPU instruction sets. We can view these QEMU binaries by running ls /usr/bin/buildkit-qemu-* inside the running buildkit container.

In the above command, we passed --platform linux/arm64,linux/amd64 so it uses the /usr/bin/buildkit-qemu-aarch64 QEMU binary for building linux/arm64 image and linux/amd64 are natively built on the host machine. Once both images are built, then it uses the --push option to create the manifest file and pushes both images to the registry server with the manifest file.

By inspecting the manifest file, we can see “Ref” contains the actual image link, which will be fetched when platform[0].architecture matches the host system architecture.

HTML

$ docker manifest inspect -v nginx

[

        {

                "Ref": "docker.io/library/nginx:latest@sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",

                "Descriptor": {

                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",

                        "digest": "sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",

                        "size": 1570,

                        "platform": {

                                "architecture": "amd64",

                                "os": "linux"

                        }

                },

                "SchemaV2Manifest": {

                        "schemaVersion": 2,

                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",

                        "config": {

                                "mediaType": "application/vnd.docker.container.image.v1+json",

                                "size": 7916,

                                "digest": "sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b"

                        },

….

        {

                "Ref": "docker.io/library/nginx:latest@sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",

                "Descriptor": {

                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",

                        "digest": "sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",

                        "size": 1570,

                        "platform": {

                                "architecture": "arm64",

                                "os": "linux",

                                "variant": "v8"

                        }

                },

                "SchemaV2Manifest": {

                        "schemaVersion": 2,

                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",

                        "config": {

                                "mediaType": "application/vnd.docker.container.image.v1+json",

                                "size": 7932,

                                "digest": "sha256:f71a4866129b6332cfd0dddb38f2fec26a5a125ebb0adde99fbaa4cb87149ead"

                        }

We can also use buildx imagetools command to view the same output in a more human-readable format.

HTML

$ docker buildx imagetools inspect sonarqube:10.0.0-community

Name:      docker.io/library/sonarqube:10.0.0-community

MediaType: application/vnd.docker.distribution.manifest.list.v2+json

Digest:    sha256:51588fac6153b949af07660decfe20b5754da9fd12c82db5d95a0900b6024196

Manifests:

  Name:      docker.io/library/sonarqube:10.0.0-community@sha256:8b536568cd64faf15e1e5be916cf21506df70e2177061edfedfd22f255a7b1a0

  MediaType: application/vnd.docker.distribution.manifest.v2+json

  Platform:  linux/amd64

  Name:      docker.io/library/sonarqube:10.0.0-community@sha256:2163e9563bbba2eba30abef8c25e68da4eb20e6e0bb3e6ecc902a150321fae6b

  MediaType: application/vnd.docker.distribution.manifest.v2+json

  Platform:  linux/arm64/v8

If you are having any issues building multi-arch images, you can run the following command to reset the /proc/sys/fs/binfmt_misc entries.

HTML

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

We can also build multi-arch container images using Buildah as well.

How Do Multi-Arch Container Images Work?

pipeline

{

    agent {

        label 'worker1'

    }

    options{

        timestamps()

        timeout(time: 30, unit: 'MINUTES')

        buildDiscarder(logRotator(numToKeepStr: '10'))

    }

    environment {

        DOCKER_REGISTRY_PATH = "https://registry.example.com"

        DOCKER_TAG = "v1"

    }

    stages

    {

        stage('build-and-push')

        {

        steps{

          script{

            docker.withRegistry(DOCKER_REGISTRY_PATH, ecrcred_dev){

            sh '''

              ####### check multiarch env ###########

              export DOCKER_BUILDKIT=1

              if [[ $(docker buildx inspect --bootstrap | head -n 2 | grep Name | awk -F" " '{print $NF}') != "multiarch" ]]

              then

                docker buildx rm multiarch | exit 0

                docker buildx create --name multiarch --use

                docker buildx inspect --bootstrap

              fi

              ####### Push multiarch ###########

              docker buildx build --push --platform linux/arm64,linux/amd64 -t "$DOCKER_REGISTRY_PATH"/username/custom-nginx:"$DOCKER_TAG" .

            '''

            }

          }

        }

      }

    }

}

name: docker-multi-arch-push

on:

  push:

    branches:

      - 'main'

jobs:

  docker-build-push:

    runs-on: ubuntu-20.04

    steps:

      - name: Checkout Code

        uses: actions/checkout@v3

      - name: Set up QEMU

        uses: docker/setup-qemu-action@v2 

      - name: Set up Docker Buildx

        uses: docker/setup-buildx-action@v2 

      - name: Login to docker.io container registry

        uses: docker/login-action@v2

        with:

          username: $

          password: $

      - name: Build and push

        id: docker_build

        uses: docker/build-push-action@v3

        with:

          context: .

          file: ./Dockerfile

          platforms: linux/amd64,linux/arm64

          push: true

          tags: username/custom-nginx:latest

$ skopeo login --username $USER docker.io 

$ skopeo copy -a docker://dev-account/custom-nginx:v1 docker://prod-account/custom-nginx:v1

####### Pull DEV images ###########

docker pull --platform=amd64 "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG"

docker pull --platform=arm64 "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG"

####### Tag DEV image with STAGE ###########

docker tag "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG" "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64

docker tag "$DOCKER_IMAGE_NAME_DEV":"$DOCKER_TAG" "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64

####### Push amd64 and arm64 image to STAGE ###########

docker push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64

docker push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64

####### Create mainfest and push to STAGE ###########

docker manifest create \    "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG" \    --amend "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-amd64 \    --amend "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"-arm64

docker manifest push "$DOCKER_IMAGE_NAME_STAGE":"$DOCKER_TAG"

####### Pull amd64 image and scan ###########

docker pull --platform=amd64 nginx:latest trivy image nginx:latest 

####### Pull arm64 image and scan ###########

docker pull --platform=arm64 nginx:latest trivy image nginx:latest