5 Tips For Building A CI/CD Pipeline

Continuous integration continuous delivery ammeon

Continuous Integration, Continuous Delivery and Continuous Deployment (CI/CD), is a common practice in modern software development. CI/CD is the automation of anything between a developer pushing code and the customer receiving the latest version of a software product. This can include packaging the product into a deliverable form; running the product through a suite of tests to ensure robustness, to deploying the product to a live environment. Having a robust CI/CD pipeline (the common term for this multistage process) comes with immense benefits most importantly:

  • having a higher quality product,
  • reducing developer time spent on tedious, repetitive tasks, 
  • provides a quick feedback loop to the developer. This lowers costs by reducing workload and catching bugs early.

This CI/CD leads to maximising the relevance of the product by quick delivery of value to the customer, which also enables a fast feedback loop on the product. The following tips should help build a robust pipeline, regardless of the Continuous Integration tools chosen.

Continuous Integration, Continuous Delivery and Continuous Deployment

Firstly, it's important to make sure to have a good understanding of the meaning of Continuous Integration, Continuous Delivery and Continuous Deployment. Without understanding all these concepts, you may fall into the trap of having an under-developed pipeline. 

Continuous Integration, or CI, typically involves steps including packaging and testing the software. These are the main components that people consider when creating a CI/CD pipeline (and is where some stop).

The next step beyond CI is Continuous Delivery. A fully tested, working version of a product should be produced at the end of the CI portion of the pipeline. The Continuous Delivery portion of the pipeline is responsible for automatically delivering this working version of the product to customers. This could be as simple as, for example, making the product available on a download page and sending out a notification of its release. This saves time delivering the product produced by the pipeline to the customer manually.

Finally, Continuous Deployment covers the automation of deploying the product onto a live environment. A good example of a use case of Continuous Deployment would be automating the publishing of an update to a website. Here we are trying to automate any steps between the software being delivered and the user having the latest version in a usable form. Even if there is a reason preventing full Continuous Deployment to a live site, it can still be useful to have your pipeline automatically deploy to a staging environment.

Having a full end-to-end pipeline that concludes with Continuous Deployment may not always be applicable to your needs. However, maximising the number of steps automated by your pipeline will reduce time spent performing repetitive tasks, reduce the time taken to deliver value to customers and establish the fastest possible feedback loop. Keep each section of a full CI/CD pipeline in mind and automate wherever possible.


Read more about the benefits of a full CI/CD pipeline, instead of just a CI pipeline.


Pipeline as code

As the CI/CD space matures, so has the concept of pipeline as code. This is the idea that we store our pipeline configuration alongside the code that we are building with it. The most notable advantage when it comes to storing pipeline as code is having access to Version Control System (VCS) tools such as Git. In the past, some Continuous Integration tools were limited to managing pipeline configurations via their GUI. This leads to problems tracking changes made to the pipeline over time and being able to reproduce the state of the pipeline for a version of the software to be built.

By storing our pipeline configuration as code, the exact configuration of the pipeline used to build the software at the time will be stored with the code in VCS. Beyond this, it can be far more intuitive for those with a developer background to automate changes via updating a file in VCS. Having pipeline configurations stored as code will likely save a lot of time and effort in the long run. If it is required to build an old version of the software, there should be little to no time needed to accommodate this in the pipeline. This also comes with the inherent benefits of VCS, by providing us with a stored history of our pipeline configuration.

Keeping things modular

Some programming best practices, such as Don’t Repeat Yourself (DRY), translate very well over to CI/CD pipelines. When automating a task, try to avoid adding too many steps to any phase. Depending on your chosen CI tools the phasing may vary (e.g. in Jenkins there are projects), but for simplicity, the word "phase" is used here to describe a configuration that will tell the tool to do something. For example, one "phase" could be to run a code analyser tool like SonarQube on your repository. The idea here is very similar to functions/methods in programming. Each phase should do one thing and should be made with reusability in mind. 

One may think it is a good idea to create a "code quality check" phase, which may run a linter, a security scanner and some other checks on the code. However, there are some issues with this. Typically, each check in the phase is performed sequentially. This can hinder the concept of "failing fast", which is discussed in more detail below, which ultimately slows down the feedback loop. In addition, there may be instances where we only want to run one of these checks instead of all three. Finally, and possibly most importantly, coupling all the checks together reduces the reusability of the phase. By splitting the checks up into their own phases, we have a higher chance of being able to reuse these same phases elsewhere in the pipeline.

Keeping everything as modular as possible in a CI pipeline may take more upfront effort and planning but is likely to save time spent on maintenance in the long run. 

Maintaining the pipeline

To quote an episode of Futurama "when you do things right, people won't be sure you've done anything at all". This has always summed up my philosophy regarding CI/CD, as in an ideal world, everything should just work. At one end, a developer pushes (good) code, some time passes, and then on the other end, a new, tested version of the software is produced. We live in a world where requirements are ever-changing, bugs are ever-present and sometimes things just break. It is the responsibility of those maintaining the pipeline to make up for its shortcomings.

When taking on the view that a CI/CD pipeline should be like a well-oiled machine that follows certain steps to achieve a goal, it can be easy to forget that we should value "individuals and interactions over processes and tools". Literal failures are not the only shortcomings of any pipeline. Notifications could be improved, perhaps phases could be reordered or maybe the feedback loop is too slow. It's important to remember that a CI/CD pipeline exists to improve the quality of life for developers and customers. As such, try to seek out and engage with feedback from the users of the CI/CD pipeline. Any woes they may have are potential improvements that can be made. Continuous Integration in agile is important, but we should try to continue to be agile in our Continuous Integration as well. Constantly improving your CI/CD pipeline will ensure a robust system that will speed up production and improve the quality of your product.

Fail Fast

A major advantage of having a CI pipeline is having a constant feedback loop. To take full advantage of this, having failures happen as early as possible is ideal. Imagine a simple pipeline where we prepare an environment, deploy our code onto the environment and then run some tests. If there is an issue with the code, there won't be any sign of an issue until we are deploying our code. If the environment preparation is significant, we're potentially losing all this time in the event our code has a simple issue such as a syntax error. In this instance, it would be prudent to use a static code scanner or implement some other method of ensuring code quality prior to the building of the environment. 

There are countless examples where issues in a pipeline could potentially be caught much earlier. By failing fast, issues can be addressed as quickly as possible, which can drastically improve time taken to deliver features to the product, which in turn reduces the cost of the value provided.

Conclusion

There are many ways to create a CI/CD pipeline. How it looks will vary wildly depending on the Continuous Integration tools used, whether it is only a Continuous Integration pipeline, a Continuous Delivery pipeline or has full Continuous Deployment, as well as the specifics of the product itself. Regardless, these tips should apply to most situations and assist in building a robust pipeline that will save a lot of pain, time and money.


Author
Alan Conroy
DevOps Engineer - Ammeon


Chaos Engineering : Explained

Ammeon | October 21, 2020

In order to prevent the chaotic nature of production environments that lead to outages, chaos engineering is the practice of facilitating controlled experiments to uncover weaknesses and help instil confidence in the system’s resiliency. Read more to know everything about Chaos Engineering

READ MORE

Containerization : A Brief Introduction

Ammeon | October 15, 2020

Containers are revolutionizing the way we develop and deliver applications. Here is everything you need to know about containerization, virtualization and development life cycle with containers.

READ MORE

Interview Mistakes that can cost you Job Opportunity

Ammeon | October 13, 2020

A lot has happened around the world this year that has impacted jobs of many. Here are some tips from our technical & operations recruiter that can help you avoid some common interview mistakes that can cost you job opportunity

READ MORE