Software Testing Quality Assurance using xUnit, TestCafe along with Docker and Jenkins: an automated approach 3/3

For those who are following me, welcome back and thank you for your support! For those who are visiting for the first time, welcome and enjoy the ride!

Now we are going to close this series of three posts with a golden key. Catching up from our last post, when we saw how important CI/CD is to Software Development focused on Quality Assurance, we met Docker Compose as a useful alternative to a pure docker run command. We also discussed Jenkins and its Pipeline concept; we also saw how to extend Docker images. We also gave some best practices, hints and VSCode outstanding extensions to increase our productivity and avoid bugs. All those matters supported by a Software Testing Quality Assurance, Test-Driven Development overview and a little bit of talking about some important tools such as TestCafe we covered in the first post.

Jenkins Docker TestCafe

Provided you are more acquainted with Docker, Jenkins, TestCafe, we can push them a little bit further.  To automate the stages described in the Jenkins Workflow graph, we extended the Jenkins official Docker image by installing .NET Core SDK 3.1 in it. It is a good approach, but based on the Design Pattern Separation of Concerns, there is one more interesting than that. Why not let Jenkins image do what it does best without having any other necessary software component installed on it? What I am trying to say is, we don’t really need .NET Core SDK 3.1 installed in Jenkins image to restore, clean, build and unit test the eShopOnWeb solution. I will show you why and how right after this important discussion about Design Pattern.

Design Pattern – Separation of concerns

Separation of concerns is a pattern/principle of software architecture design for splitting an application into different parts, so each section addresses a separate concern. The ultimate purpose of the separation of concerns is to construct a well-organized structure. Each component fulfils a significant and intuitive function while optimizing its capacity to adapt to change.

The creation of boundaries achieves the separation of concerns in software architecture. Any logical or physical restriction that determines a given set of duties is a boundary. The use of methods, artifacts, modules, and services to describe core actions within an application would provide some examples of boundaries, tasks, solutions, folder hierarchies for source organization, application layers and organization processing levels.

Separation of concerns – advantages

  1. The lack of repetition and the uniqueness of the individual components’ function makes it easier to manage the overall structure.
  2. As a byproduct of improved maintainability, the system becomes more robust.
  3. The strategies needed to ensure that each element only concerns itself with a single set of coherent obligations often lead to natural points of extensibility.
  4. The decoupling that results from requiring components to concentrate on a single function leads to more easily reused components within the same system in other systems or different contexts.

Remember that, if not all, most of the well-known Design Patterns that we use in Software Engineering apply not only to coding as we know it but also to define the infrastructure and architecture to test and assure the quality of our application.

Going back to the point that I said you would not need .NET Core SDK 3.1 installed in the Jenkins image. The answer is simple: there is a way to run the restore, clean, build and functional, integration and unit tests of eShopOnWeb in a separate container, which can be called by Jenkins container. Furthermore, this container will exist only during the processing time of the task.

Enough theory, let’s make it happen!

We will use the Jenkins project we created in the previous post. Open it in VSCode; let’s create a directory called image2 in the Jenkins root directory and copy Dockerfile, build.sh, docker-compose.yaml, start.ps1 and stop.ps1 to this directory. We will make a few changes to them.

In build.sh we will change the name of the image to sitk/jenkins2 to preserve the original image as shown below.

sitk-jenkins2 image on build script

In docker-compose.yaml file, make the changes highlighted in the image below.

changes to docker-compose

The change in the image and container_name is expected since we want to preserve the previous ones.

The change in the mapped port in the host side to 8081 is not to conflict with the one we set up in the previous post.

We added user: root and /var/run/docker.sock:/var/run/docker.sock to allow Jenkins container to access a remote Docker Server which resides in the Host machine to build images, start, stop, create and remove siblings containers. This involves mounting the host machine’s docker socket to the Jenkins container to start new sibling containers ( note, we are using the word siblings here instead of child containers because the newly created container will run alongside the Jenkins container rather than running inside the Jenkins container).

The reason to change the volume to jenkins-home-volume2 is because we don’t want to interfere in the existing one that is persisted in the host from the previous post.

In Dockerfile we removed .NET Core SDK 3.1 installation and included Docker Community Edition CLI installation since we will start siblings containers from within Jenkins container as stated above. To do this, copy the code below to Dockerfile which is inside image2 and save it.

FROM jenkins/jenkins:lts
 # Switching to root user to install .NET Core SDK
USER root

# Show the distro information during compilation!
RUN uname -a && cat /etc/*release

# Install docker
RUN apt-get update -qq \
    && apt-get install -qqy apt-transport-https ca-certificates curl gnupg2 software-properties-common 
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/debian \
   $(lsb_release -cs) \
   stable"
RUN apt-get update  -qq \
    && apt-get install docker-ce docker-ce-cli containerd.io -y
RUN usermod -aG docker jenkins    

# Switching back to the jenkins user.
USER jenkins

The start.ps1 and stop.ps1 remain the same.

Now let’s build the new image, go to Windows Terminal, open a bash terminal and under the /mnt/c/sitk/jenkins/image2 run ./build.sh. If everything went well, you should see something like this.

compiling sitk-jenkins2 image

Now we start a Jenkins container based on the newly created image by running start.ps1 in Windows Terminal under c:\sitk\jenkins\image2.

start new Jenkins container

Let’s access Jenkins at http://localhost:8081/ and set it up as we did in the second post. Notice that at this time, we will not need to install the .NET SDK Support plugin for Jenkins. Instead, we will need the Docker Pipeline plugin for Jenkins to dynamically build images, create, start, stop and remove containers in the Jenkins pipeline.

As soon as Jenkins is ready, go to Manage Jenkins -> Manage plugins, click on the Available tab, and in the search field, type “docker” and select Docker Pipeline and click on Install the Without Restart button.

Docker Pipeline plugin

Jenkins will also install the Docker Commons plugin since there is a dependency.

Making the most of the Jenkins plugins ecosystem, I want to present you Allure Test Report.

Allure Test Report

Allure Test Report is a flexible, lightweight multi-language test reporting tool. The test report tool demonstrates a succinct representation of what has been tested in a tidy web report form and helps those involved in the development process to gain maximum valuable knowledge from the regular execution of tests.

How it works

Allure is based on standard xUnit results output but adds some supplementary data. Any report is generated in two steps. In the first step during test execution, a small library called adapter attached to the testing framework saves information about executed tests to XML files. They already provide adapters for popular Java, PHP, Ruby, Python, Scala and C# test frameworks.

To install the Allure Report plugin, go to Manage Jenkins -> Manage plugins, click on the Available tab, and in the search field, type “allure” and select Allure and click on the Install Without Restart button.

Next, go to Manage Jenkins -> Global Tool configuration, scroll until you find the Allure Commandline section. In the Name field inside the Allure Commandline panel, type “Allure 2.13.7”. The version should match with the latest version you select in From Maven Central field as shown below.

setup allure in global tool configuration

We need to install the so-called adapter to the xUnit, the C# unit testing framework we are using on eShopOnWeb, as we mentioned in the first post.  For doing so, install XunitXml.TestLogger package to generate the XML files Allure will need to generate the reports.

Go to the Windows Terminal and under c:\sitk\eShopOnWeb run the following commands.

dotnet add ./tests/FunctionalTests/FunctionalTests.csproj package XunitXml.TestLogger --version 2.0.0
dotnet add ./tests/IntegrationTests/IntegrationTests.csproj package XunitXml.TestLogger --version 2.0.0
dotnet add ./tests/UnitTests/UnitTests.csproj package XunitXml.TestLogger --version 2.0.0

adding XunitXml-TestLogger

To run the Jenkins pipeline using the Docker and Allure plugins, we need a new Jenkinsfile. Therefore, create a new directory called Jenkins under the eShopOnWeb root directory and place a Jenkinsfile with the content below.


pipeline {
    agent none
    stages {
        stage('Checkout') {
            agent { 
                docker { image 'mcr.microsoft.com/dotnet/core/sdk:3.1-bionic'} 
            }
            steps {
                checkout([$class: 'GitSCM', branches: [
                    [name: '*/master']
                ],
                userRemoteConfigs: [
                    [url: 'https://github.com/sitknewnormal/eShopOnWeb.git']
                ]
                ])
            }
        }
        stage('Restore') {
            agent { 
                docker { image 'mcr.microsoft.com/dotnet/core/sdk:3.1-bionic'} 
            }
            steps {
                sh "dotnet restore --packages ./.nuget/packages eShopOnWeb.sln"
            }
        }
        stage('Clean') {
            agent { 
                docker { image 'mcr.microsoft.com/dotnet/core/sdk:3.1-bionic'} 
            }
            steps {
                sh "dotnet clean eShopOnWeb.sln"
            }
        }
        stage('Build') {
            agent { 
                docker { image 'mcr.microsoft.com/dotnet/core/sdk:3.1-bionic'} 
            }
            steps {
                sh "dotnet build eShopOnWeb.sln --no-restore --configuration Release"
            }
        }
        stage('Functional, integration and unit tests (xUnit)') {
            agent { 
                docker { image 'mcr.microsoft.com/dotnet/core/sdk:3.1-bionic'} 
            }
            steps {
                sh "dotnet test ./tests/FunctionalTests/FunctionalTests.csproj --configuration Release --logger xunit --no-build --no-restore --results-directory ./allure-results/FunctionalTests"
                sh "dotnet test ./tests/IntegrationTests/IntegrationTests.csproj --configuration Release --logger xunit --no-build --no-restore --results-directory ./allure-results/IntegrationTests"
                sh "dotnet test ./tests/UnitTests/UnitTests.csproj --configuration Release --logger xunit --no-build --no-restore --results-directory ./allure-results/UnitTests"
            }
        }
        stage('End 2 end eShopOnWeb tests with TestCafe') {
            agent { 
                docker { 
                    image 'testcafe/testcafe'
                    args '--entrypoint=\'\''
                } 
            }
            steps {
                sh "testcafe chromium:headless tests/e2eTests/*_test.js -r spec,xunit:allure-results/e2eTests/TestResults.xml" 
            }
        }
        stage('Publish Reports') {
            agent { 
                docker { image 'openjdk'} 
            }
            steps{
                script {
                    allure ([
                        includeProperties: false, 
                        jdk: '', 
                        results: [
                            [path: 'allure-results/FunctionalTests'],
                            [path: 'allure-results/IntegrationTests'],
                            [path: 'allure-results/UnitTests'],
                            [path: 'allure-results/e2eTests']
                        ]
                    ])
                }            
            }
        }
    }
}

It should look like this.

Jenkinsfile - pipeline with using docker and Allure

Don’t forget to rename the eShopOnWeb git URL if you forked or git cloned the project. You most probably have to change in this URL “https://github.com/sitknewnormal/eShopOnWeb.git” only the sitknewnornal highlighted in red your GitHub username. And most importantly, don’t forget to commit and push it to your repository because Jenkins will poll it from there when the pipeline runs.

There are some important changes I want to point out if we compare the previous Jenkinsfile with the new one.

comparing Jenkinsfiles

In the first Jenkinsfile on the left-hand side, we have all the pipeline stages running on any agent based on the previous post. It means that they will run on Jenkins master or one of the slaves’ containers if there were any. Since we know that there were no additional slaves in the Jenkins we set up previously, it will run in the Jenkins master container.

On the other hand, on the right-hand side, we defined one agent that will run using the official mcr.microsoft.com/dotnet/core/sdk:3.1-bionic for each stage in the pipeline. It means that Jenkins will pull the image from Docker Hub in the first run, create a sibling container that will run along with Jenkins, perform the step and then remove it. It will repeat this process for each stage.

If we focus on the Functional, integration and unit tests (xUnit) stage, we will notice that it will run in a container using the same image used in the previous stages. The logger highlighted in the image below will log the test results in the xunit format in the respective project directories under  ./allure-results directory so that the Jenkins Allure plugin can access and generate the report.

unit tests stage in jenkins pipeline depicted

For the End 2 end eShopOnWeb tests with TestCafe stage, we will run all 5 end-to-end tests in a container using testcafe/testcafe the official image. Notice that to use the testcafe chromium tests/e2eTests -r spec,xunit:allure-results/e2eTests/TestResults.xml command in the step, we needed to override the image entry point using the following argument --entrypoint=\'\' since, as we’ve seen in our previous post, the syntax is a little bit different. The parameter -r spec,xunit:allure-results/e2eTests/TestResults.xmltells TestCafe to report the test result to stdout using the spec argument and xunit the format file for the Allure use and place it in allure-results/e2eTests directory.

And finally, the last stage will publish the reports from all tests using the Jenkins Allure plugin. For doing so, it will run a container based on the OpenJDK official image. Notice that the mentioned plugin will use the results saved in the allure-results paths to generate the reports.

Now that we understood what the Jenkins Pipeline would do, let’s run it.

On Jenkins’s home page, select New Item.

Jenkins New Item

Enter eShopOnWeb as the item name, select Pipeline and click OK.

enter intem name select pipeline and click ok

We suggest you give it a brief description and click on the Pipeline tab.

basic cicd pipeline for eShopOnWeb solution

Select the Pipeline script from SCM option from the Definition field.

select pipeline script from SCM

Select the Git option from the SCM field.

select git from SCM field

Copy eShopOnWeb git URL https://github.com/sitknewnormal/eShopOnWeb.git to the Repository URL field, and make sure the Script Path field is “Jenkins/Jenkinsfile,” and click on the Save button.

Jenkinsfile script path

Before we run this Jenkins Pipeline, we need to run the eShopOnWeb application since there will be end-to-end tests with TestCafe.

For doing so,  make sure you have the MSSQL Server 2019 running. We explained how to do it here.

First, let’s run eShopOnWeb without debugging by selecting the menu Run and then Run Without Debugging as shown below and wait until it automatically opens the browser with the URL https://localhost:5001.
run eShopOnWeb whitout debugging

As soon as the application starts, go back to Jenkins and click on the Build Now button. It will take around 5min to finish the whole Pipeline, which includes the Allure Report publishing.

In the end, you will see something like this.

Jenkins pipeline eShopOnWeb result

If everything went well up to the Publish Report stage, click on the Allure Report icon, a colourful one that appears on the right side of the last execution on the Build History section, which In my case, is #3.

You will see the Allure Report showing that 79 out of 79 test cases have passed. It is not usual to have 100% of the test cases passed, but you will have a number that will help you to work on for sure.

Allure overview

There is much useful information in the Allure Report that you can get most of. It is not the purpose of this post to dig deep into Allure so, take time to explore Allure Report, and be positively surprised by how it can help you work on Software Quality Assurance.

Congratulations, we have made it! We went through some of the most important steps in the Software Development lifecycle using some powerful, cutting-edge tools and concepts to help us develop software better!! The Jenkins project and eShopOnWeb source code are on our GitHub account.

If you need any help or something is going wrong, let me know. It will be my pleasure to help you. If you want to extend some point, we have discussed, tell me so we can cover in the next posts.

We also have pro bono projects just in case you are interested in learning more about them.

Be aware of why we started this project by clicking here.

Learn more about other posts here.

Contact us for any suggestions. And follow us on FacebookInstagram and Twitter.

If you are a good reader like myself, I recommend the following readings:

  1. Docker Quick Start Guide: Learn Docker like a boss, and finally own your applicationsDocker Quick Start Guide: Learn Docker like a boss, and finally own your applications
  2. Docker for Developers: Develop and run your application with Docker containers using DevOps tools for continuous deliveryDocker for Developers: Develop and run your application with Docker containers using DevOps tools for continuous delivery
  3. C# and .NET Core Test-Driven Development: Dive into TDD to create flexible, maintainable, and production-ready .NET Core applications, Kindle Edition
    C# and .NET Core Test-Driven Development: Dive into TDD to create flexible, maintainable, and production-ready .NET Core applications, Kindle Edition
  4. Modern Web Testing with TestCafe: Get to grips with end-to-end web testing with TestCafe and JavaScript 1st Edition, Kindle Edition
    Modern Web Testing with TestCafe: Get to grips with end-to-end web testing with TestCafe and JavaScript 1st Edition, Kindle Edition
  5. Continuous Delivery with Docker and Jenkins: Create secure applications by building complete CI/CD pipelines, 2nd Edition, Kindle Edition

See you in the next post!