Accelerating Docker Deployments in CI/CD Pipelines with Caching Strategies
In the rapidly evolving realm of software development, optimizing efficiency and velocity is paramount. A pivotal component for automating the software delivery process is the implementation of Continuous Integration and Continuous Deployment (CI/CD) pipelines. These pipelines empower teams to seamlessly integrate code changes with increased frequency and reliability. Yet, extended deployment durations can pose a substantial bottleneck, particularly in the context of containerized applications. This article delves into our approach to addressing the imperative of diminishing deployment times through the strategic utilization of Docker caching within our CI/CD pipeline.
The Challenge: Lengthy Deployment Times
Our CI/CD pipeline, constructed using cdk pipelines, required approximately 25 minutes for execution, which was suboptimal for rapid iteration. Upon closer inspection, we identified that a significant portion, up to 40%, of this duration was attributed to the Docker image build process (deployment package for our AWS Lambda function).
An analysis revealed that most of the time, only the final layer of the Docker image, containing the source code, underwent changes. This observation led us to recognize the potential time savings by implementing caching for the other layers, mirroring the functionality on our local machines.
However, the on-demand nature of our deployment environment resulted in a loss of context with each run. Consequently, we were compelled to construct the Docker image from scratch for every deployment, hindering our efficiency.
Example DockerFile:
FROM public.ecr.aws/lambda/python:3.10-x86_64
COPY requirements.txt .
RUN pip3 install –no-deps -r requirements.txt –target “${LAMBDA_TASK_ROOT}”
ADD ./src ${LAMBDA_TASK_ROOT}
CMD [ “app.app” ]
The Solution: Docker Caching
After researching the possible solutions, we narrowed down to two approaches, both of which involved utilising the –cache-from option during the docker build process. Both approaches would allow us to reuse the unchanged layers, thereby reducing build times.
We considered two potential solutions:
Creating a separate AWS CodeBuild step in the pipeline dedicated to building the Docker image and pushing it to the correct ECR repository.
Using DockerImageAsset with the cache_from argument in AWS CDK.
The later solution was suboptimal because AWS CDK doesn’t support pushing images to specific repositories. Images can be pushed only to the one created during the CDK bootstrap process. This limitation (among others) was unacceptable for our needs, and using the cdk-ecr-deployment to copy images from the general ECR repository to specific ones introduced additional overhead. Therefore, we decided to go with the first solution.
Implementing Caching in CDK Pipelines
We established an AWS CodeBuild phase dedicated to constructing the DockerImage, incorporating caching whenever feasible. It was imperative to introduce this step post the initial deployment, ensuring the ECR repository for our Docker images was already established.
The CodeBuild step necessitates comprehensive permissions for both pulling from and pushing to the ECR image repository. The provided buildspec snippet encompasses all essential steps and commands for Docker image construction with layered caching. Notably, in the final step (post_build), the image undergoes two distinct tag-based pushes. This dual-tagging strategy facilitates the creation of uniquely identified images, simplifying the identification of the latest one. This latest image serves as a cache in subsequent deployments.
phases:
pre_build:
commands:
– export DOCKER_BUILDKIT=1
– docker pull $IMAGE_REPO_NAME:latest || true
build:
commands:
– docker build –build-arg BUILDKIT_INLINE_CACHE=1 -t $IMAGE_REPO_NAME:latest -t $IMAGE_REPO_NAME:$COMMIT_ID –cache-from $IMAGE_REPO_NAME:latest .
post_build:
commands:
– docker push $IMAGE_REPO_NAME:latest
– docker push $IMAGE_REPO_NAME:$COMMIT_ID
version: ‘0.2’
We added the CodeBuild step as a pre action in the deployment wave of the pipeline:
wave = codepipeline.add_wave(
“DeployWave”,
pre=[lambda_docker_build],
)
wave.add_stage(…)
Important remarks:
Buildkit must be used → DOCKER_BUILDKIT=1
According to the docker documentation
To use an image as a cache source, cache metadata needs to be written into the image on creation. This can be done by setting –build-arg BUILDKIT_INLINE_CACHE=1 when building the image. After that, the built image can be used as a cache source for subsequent builds.
Results and Lessons Learned
The outcomes are remarkable: builds now consistently achieve a 40% acceleration. The sole exception arises when rebuilding the Docker image due to alterations in layers, excluding the one copying the source code.
We learned some valuable lessons along the way:
– With Docker versions greater than 23.0 every second deploy is not cached, due to a known caching issue (refer to GitHub issue).
– Solution 1, which we did not use in the end, only works with latest AWS CodeBuild environments (Docker versions ≥ v23.0):
– ✅ aws/codebuild/standard:7.0 (Docker v23.0)
– ✅ aws/codebuild/amazonlinux2-x86_64-standard:5.0 (Docker v23.0)
– ❌ aws/codebuild/standard:6.0 (Docker v20.0)
We have also experimented with GitHub Actions, but found that pulling the image was too slow to effectively utilize the cache-from option, taking about 7 minutes just for the pull. Additionally, the built-in cache did not work with the minimal effort we were willing to invest.
Conclusion
Enhancing CI/CD pipelines to expedite deployments can notably enhance the development workflow. The incorporation of Docker caching strategies enabled us to markedly reduce build times, facilitating quicker iterations and optimizing resource utilization. Despite potential challenges and learning curves, the advantages of an optimized pipeline justify the investment of effort. As we refine our methodologies, we aspire to assist others on their journey toward more efficient CI/CD practices by sharing our experiences.
