What You Should Know Before Setting Up Your First CI/CD Pipeline
You want to set up your first automatic deployment pipeline that
builds, tests, and deploys code changes to your target cloud environment.
You’ve spent days reading docs and blogs to figure out what your automatic
deployment pipeline should comprise. But it all seems really overwhelming. They
mention various tools like AWS, Azure, GitHub Actions, Ansible, Jenkins, CircleCI, Terraform, and Kubernetes — the
list is endless. And you’re not sure which one is necessary for your initial
automatic deployment pipeline.
What does your first continuous delivery pipeline need to do
and not do?
What are the necessary components of such a minimum viable
Which tools are the right choice for your initial pipeline?
This article focuses on laying down the thought process behind
how you should go about answering the above questions so that you can quickly
start building your first delivery pipeline.
The post recommends the thought process for building your first
CI/CD pipeline assuming the following:
want to host your application on the cloud.
startup like yours wants to optimize for speed while retaining the
flexibility to customize your pipeline as you scale.
Continuous integration is the process of
merging changes to the main branch as often as possible. These changes are then
validated by creating a build and running automated tests against the build.
Continuous delivery is an extension of continuous
integration since it automatically deploys all code changes to testing and/or
production environments after the build stage. This means that while your build
and testing are automated, the deployment trigger (say a button-click) is
manual. But once the deployment is started, there is no need for manual
Continuous deployment is like continuous
delivery, except that the trigger to deployment is also automated.
delivery is part of continuous deployment.
Continuous integration is part of both continuous
delivery and continuous deployment.
The rest of the post uses CD to refer to both continuous
delivery and continuous deployment. Your thought process and the choice of
tools to set them up will be the same.
The guiding principle while deciding what your initial pipeline
(also called a minimum viable pipeline) should be is to solve current problems
and leave the theoretical problems for the future. Take small steps, and don’t
try to build a fully mature pipeline at the start.
While what your small steps should be is a function of your
application use-cases, listed below are the most necessary steps — divided into
The prerequisite for building any CI/CD pipeline is that
developers regularly commit their code to a central repository. For feature
branches that remain unmerged to the main branch for a long time, developers
should keep it up to date by merging upstream as often as possible.
1. Pull the
latest code of the branch from your VCS.
2. Run unit
tests on the branch code to check that the application is not broken because of
new commits pushed into the branch.
3. Trigger a
build of the branch code whenever an event, that you configure, occurs.
4. Use a build
tool to run a build of your code on a server.
5. Create one
or more build artifacts of the code as part of the build process.
6. Store the
build artifact(s) in a secure and accessible cloud location
triggering the deployment of the build artifact(s)/application to the target
cloud environment (testing/staging/production and so on).
2. On manually
triggering the deployment, it automatically deploys the application to the
target environment without any downtime.
3. Provide an
easy way to determine if the deployment succeeded or failed, and give into the
Note: Your initial pipeline need not implement continuous
Since implementing every step of the pipeline costs time, it is
worthwhile to do a cost-benefit
analysis of every step that you want to automate beyond the ones mentioned in the
previous section. In our experience, in the beginning, most applications don’t
need a fully automated CD workflow for the following:
and managing resources using infrastructure as code
or multi-cloud deployment
to dynamically add or remove instances
many testing phases like performance testing, UI testing, and so on
Note: While determining the responsibilities of your minimum
viable CD pipeline, recognize that you may need to add one or more steps
depending on your application’s use case. For example, a payment processing
application might be more sensitive to a bad deployment than an employee-leaves
management software. In such a case, the minimum viable CD pipeline for the
former should have a step to quickly roll back a bad deployment, while the
latter can skip it at first.
Given what an initial pipeline needs to do, let’s take a look at
the necessary components to build such a pipeline.
3. A CI tool:
To build and run tests on the application code
4. A CD tool:
To deploy the application code to a target environment
For every component of your initial CD pipeline, you have a
plethora of tools to choose from. We recommend that you choose tools that have
the following characteristics:
A fully managed service manages the resources required to do its
job, along with any infrastructure, so you don’t have
to invest the time and people to do it. Only if your application requires
strict compliance that demands tight control and regulated access to your
code or data, should you consider self-hosting while building your initial
be easily modified and extended by using code or integrating with
plugin ecosystem: An extensive pool of plugins provides support
to accelerate the automation of your pipeline.
for one or more stages of your pipeline: The more steps that
a tool solves, the fewer tools you need to integrate with to build your
pipeline. For example, GitHub provides both a version control system and
direct integration with its own CI/CD.
Cost: The pricing
structure of the tools should be such that it fits in your budget even
after you scale.
While building your first delivery pipeline, you have two
categories to choose from.
Pipeline as code means that you configure the steps in your
deployment pipeline — build, test, and deploy, with code that you store in a
repository, such as Git. It enables you to track and manage changes to these
configurations in the same way as you manage your application code — using
version control and pull requests.
We recommend that you choose one of the three options specified
enable you to build, test, and deploy directly from their platform without
the need for integrating with any other tool.
all offer both:
services: they provision, manage, and scale your build servers to scale
continuously and process multiple builds concurrently so that your builds
are not left waiting in a queue.
runners: you can point your builds to run on a machine that you specify.
This can be a server that you host yourself behind the firewall or on a
private cloud that you manage.
templates for common tasks.
- Fully-managed services: they provision, manage, and scale your build servers to scale continuously and process multiple builds concurrently so that your builds are not left waiting in a queue.
- Self-hosted runners: you can point your builds to run on a machine that you specify. This can be a server that you host yourself behind the firewall or on a private cloud that you manage.
a steep learning curve involved when writing a deployment pipeline. It
takes time to learn how to properly write one with the correct syntax and
pipelines can get to more than 1000 lines of code, and updating and
maintaining them could become a headache.
BitBucket only if your team also uses other Atlassian tools like Jira and
Confluence, as it integrates smoothly with them.
Note: Solutions like AWS CodePipeline and Azure Pipeline are also examples of pipelines
as code, and are easy to set up. However, they aren’t customizable because
integration with non-AWS or non-Azure tools/libraries is difficult. They also
make you completely dependent on the specific cloud provider. When you are
setting up your first pipeline it might be wiser to keep your options open
about how you’d like to evolve your pipeline and then decide whether you’d like
to be completely dependent on a specific cloud provider or not.
Release automation platforms remove the need to write code for
creating pipelines. They provide a layer of abstraction over pipeline as code.
This abstraction further simplifies the creation and management of your
Argonaut is one such tool. You can get your first
CI/CD pipeline running in less than five minutes in your own cloud.
Once you’ve built an initial pipeline, keep track of what you
still do manually, and how often. You can evolve your initial pipeline by
automating the following:
coverage analysis: Add a code coverage tool in your build step to
determine the percentage of code being covered by the tests, and fail your
build if it’s below a certain threshold. Assuming that the developers have
added useful tests, this ensures code quality.
environments: It’s hard to test how your app interacts with other
services, queues, and databases, in your local development environment.
Use staging environments to test this before deploying to production.
integrations: Use tools like Snyk to monitor your
application dependencies for vulnerabilities.
deployment rollback: Use a deployment strategy that ensures that
if your application is unusable right after the deploy, you can simply
choose the last version you want to restore it to, and immediately roll
back to that version. Since that version was already previously tested, it
shouldn’t have to go through the pipeline stages once again. You can
choose from various deployment methods like canary
and so on.
hotfix release: Some production bugs may require you to commit and push
the fix by bypassing your pipeline stages like testing. While risky, you
need to configure your pipeline such that it supports the deployment of a
hotfix in a few seconds.
As your software scales, you should adopt GitOpspractices — Git to
version control and manage your infrastructure and application
At the early stages of product development, when you’re racing
against time to release product updates, you might be tempted to postpone
automating your CI/CD pipeline. While shipping new features to your customer is
the topmost priority, taking small steps toward building a continuous CI/CD
pipeline will help you release features much faster and more reliably.
one step at a time. Don’t try to build a mature pipeline on day 1.
the most repetitive tasks before others, like the build step before
tools that solve your current need, are quick to get started with, and
don’t lock you in so that you can easily modify your pipeline as you
We Provide consulting,
implementation, and management services on DevOps, DevSecOps, Cloud, Automated
Ops, Microservices, Infrastructure, and Security
Services offered by us: https://www.zippyops.com/services
Our Products: https://www.zippyops.com/products
Our Solutions: https://www.zippyops.com/solutions
For Demo, videos check out YouTube Playlist: https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro
If this seems interesting, please email us at [email protected] for a call.
Leave a Comment
We will be happy to hear what you think about this post