How to design a modern CI/CD Pipeline 

I can't understand... why you have to build the image before testing? is it not supposed to work opposite: testing your code first, building the image after testing? that's because building the image spends time which, if the code under test fails, becomes a waste, so you end with a tagged image which is unusable. Let's distinguish: testing the code isn't equals to test the image: testing the code is simply the fact of proving the code works the way we expect it works, and, for this, the unit test must be implemented in such way that has no dependencies over external resources (i.e. databases, queues, etc) but is only a logical testing which supplies mock entities for isolating the code; testing the image is testing the application within the environment it is expected to work into, so is different to unit test the code, than testing the container from an integration point of view, because, when you test the container, you have to supply environment variables which fill the required definitions for making it work into a desired environment, but it has nothing to do with the code, because, at that point in time, the code must be proven to work. That's my point of view so I would love any feedback on this subject.

Since they do not require compiling, you would copy those files into the docker image you are building in the build stage

1. You should set the build to fail. You should also build in some notifications to notify your developers. Slack/Email are most commonly used. 2. Release stage is making your container image/software available on a registry. Deployment stages are after the release stage, and actually get the software to run on your servers 3. Some people don't like it, but it's made to save your Devs time. It follows the "fail fast" moto of pipelines. If there are linting/formatting errors, we want that to fail as soon as possible, not during a PR check or during a pipeline run (which comes later). 4. Depends on the company, it doesn't really matter. It's usually something you build and always improve on. 5. Some do, but many don't. I usually just run the pipeline after any changes and make sure it still passes. I would probably do something more if people keep breaking the pipeline with changes they make to it.

@Aleks-fp1kq I would 100% include precommit. It is the first gate for the code to go through. It will eliminate countless PR request updates that say 'fixed linting / formatting'. Also, if you include a precommit hook that checks for secrets like gitleaks or yelp, you are preventing any sensitive information ever being committed in the first place.

Pre commit is really useful to prevent the red mark of shame. I try to put them in every project I dev on so that you fail as early as possible.

yes, this is the place to which you push or from which you pull container images

Linting is more for quality of life during code reviews. Sometimes there are unnecessary formatting stuff that pollutes the PR. Sometimes (and you’d be surprised) devs will push without actually building their code locally. It’s annoying.

I guess it depend how you run unit test in container or not but if we run in container that may take more time then just running unit-test directly when you build and also you dont need to download container image if unit test fails

@@florimmaxhuni4718 Depends on in which format you ship the application and how you test. The norm would currently be to ship release images which include the runtime and the compiled/transpiled binary for execution and not much else. For isolation/parallelization purposes on the build server it also makes sense to run every step of CI pipeline in predefined containers. The 'CI pipeline' image definition would be part of the project repository and could include everything you need for testing, debugging, development (can be used locally by devs). The 'release' image which you want to push into the registry and which is then used as the startable application in production could be way leaner, because no testing framework is needed.

We still ship release images that include compiled/transpiled binary but when build is succussed and all the test pass otherwise if test fails thats not a releasable I assume for dev or other environments thats why I think it may take a lot of time specially when creating PR-s. (I guess you have methods that help to use cache but still). By the way Im talking about some languages that I work and github actions has great support for them (one command to build or run test/integration--test)@@T.P.87

It's a preference. You can UT before or after building the image. My preference is to make use of multistage dockerfiles and run my CI tests within the image. When dockerfiles are built correctly it doesn't take long to build an image, and it's something you can bring to any CI environment. Here's a good post about it www.reddit.com/r/docker/s/2o8TBzsBtK

exactly, and container registry is where we have the container images with tags, right?

sometimes you must first compile the UTs (like in C++), however, I don't see why we should keep compiled UTs in the image

I have the same question UTs and coverage after the build . I don’t understand

It's a preference. You can UT before or after building the image. My preference is to make use of multistage dockerfiles and run my CI tests within the image. When dockerfiles are built correctly it doesn't take long to build an image, and it's something you can bring to any CI environment. Here's a good post about it www.reddit.com/r/docker/s/2o8TBzsBtK

@@DevOpsJourney, instead of multistage dockerfile can’t we just run unit test first after building the code. Copy the artifacts into Dockerfile and just build docker image?

eraser.io. I got a link to the diagram in the description

This is because the test you have seen\used are written badly.

​@@Aleks-fp1kq If you don't think too many unit tests is a net negative you've either not refactored often enough, or not worked in an environment where sloc and code coverage were used as performance metrics. Prioritizing unit tests where they're unnecessary complicates development and adds pointless busy work.

@@ZTGallagher why would anyone write "too many unit test"? you mean multiple tests that test the same thing? How coverage be a performance metric?

@@Aleks-fp1kq 100% code coverage is not just impossible, it's a hindrance. So many functions simply do not need unit tests. They're too small, straightforward, or simple. They don't need it. You can cover more of the system with good integration tests that get you the same reliability. But people will write unit tests arbitrarily, and often bad hacky ones, because it's less important to have good code or develop efficiently, and more important to say "yeah we have high code coverage". Prioritizing code coverage means prioritizing volume, not quality. A system is better with just well-written integration tests. Unit tests aren't bad. They're simply extremely over-valued.

@@ZTGallagher I agree, and as I said in my first reply, you have used/encountered bad tests/practices. there is no such things as prioritizing unit tests. you need unit tests if you are doing TDD (bottom-up\top-down). you write the unit tests to test your SUT, you don't write more than that. While doing that we follow patterns such as equivalence partitions, triangulation and principles ....

😂