Software Testing Quality Assurance using xUnit, TestCafe along with Docker and Jenkins: an automated approach 1/3
One of the most important steps in the Sofware Development life cycle is Software Testing Quality Assurance. There are many ways to deliver software with quality in all phases of its development process. It’s undeniable that the earlier you start introducing QA methods and techniques to your development process, the most cost-effective it will be. Furthermore, you will most likely finish it on time.
What is Software Testing Quality Assurance?
Software Testing Quality Assurance is a process to ensure the quality of software products or services offered by an organization to its customers. Quality assurance focuses on improving software production and making it effective and efficient in compliance with software products’ quality standards. Quality Assurance is commonly referred to as QA Testing.
It is usually based on Plan-Do-Check-Act steps, which are evaluated and improved periodically by the organization, most likely supported by tools.
To ensure that the product is designed and delivered with proper protocols, a company must use Quality Assurance. This allows, in the final product, to reduce problems and errors.
Quality Assurance best practises:
- Create a robust testing environment
- Select the release criteria carefully.
- To save money, apply automated testing to high-risk areas. This helps to fasten the whole process.
- Allocate appropriate time for each phase
- Prioritizing bug fixes based on the use of apps is important.
- Form a dedicated team for security and quality testing
- Simulate tests in a production environment
There is much more to talk about Software Testing Quality Assurance, but it is not our focus on this post.
Software Testing – theory into practice
Lets’s use eShopOnWeb, a sample ASP.NET Core 3.1 reference application running on a docker container and persisting data in a SQL Server database for our Software Testing experiment. You can find more detailed info about preparing this environment here. It is strongly recommended since there are some pre-requisites steps we are not going to cover here.
Hit granularity test with xUnit
xUnit is the collective name for several unit testing frameworks that derive their structure and functionality from Smalltalk’s SUnit. SUnit, designed by Kent Beck in 1998, was written in a highly structured object-oriented style, which lent easily to contemporary languages such as Java and C#.
Some of the must-have best practices while developing software are building functional, integration and unit tests. In the image below, you will find out that eShopOnWeb BasketAddItem Unit Test uses xUnit, a free, open-source, community-focused unit testing tool for .NET Framework.
To run all the eShopOnWeb solution tests, go to VSCode Terminal, which will lead to a bash terminal since the solution is running in a Linux container. Make sure you are in the solution root directory, which is /workspaces/eShopOnWeb$ and type
dotnet test eShopOnWeb.sln and hit Enter.
Developing into a container will give lots of advantages and huge flexibility if you happen to use different languages and or dependencies in your projects, not to mention that your workstation or laptop will not look like it is cluttered with many stuff installed. To learn more about this, visit our first post of a series of four.
Using TDD (Test Driven Development) approach in the development process is one of the most effective ways to develop functional, integration and unit tests. This is only a useful recommendation!
What is Test-Driven Development?
In software development, it’s a design process. It is based on a concise development cycle being replicated, and the criteria are converted into particular test cases.
These are TDD steps:
- Write a unit test that fails.
- Write enough code to make the test pass — at this step, we don’t care about good code.
- Refactor your code from the previous step.
We will not cover this technique here either, since the focus is to give an in-depth approach to Software Testing using xUnit, TestCafe along with Docker and Jenkins.
TestCafe: testing software from a user perspective
Even though building functional integration and unit tests are important for software testing for back-end and business tier, it is not enough when we are talking about web applications software quality assurance. We will most likely need to test the application in an end-to-end approach. Hence, we cover the frontend and its behaviour in different browsers, devices and screen resolutions simulating a test from a user perspective.
To run tests from a user perspective, there are many options, but let’s choose TestCafe, an open-source project which is a pure NodeJS end-to-end solution for web app testing. It takes care of all phases: starting browsers, running tests, collecting data from tests and producing reports. TestCafe doesn’t need browser plugins-it works out-of-the-box on all popular modern browsers.
The installation process for TestCafe is easy, but you might have problems getting stuff up and running locally. TestCafe depends on NodeJS to work. For example, you will need to set up at least one compatible browser on your device. In certain situations, for various purposes, such as having restricted access to your work machine’s user account, you might not be able to execute these instals easily.
There is also the scenario of running your end-to-end tests in various environments, such as continuous integration tools like Jenkins, for instance. The chances are that these systems would not have all the requisite dependencies you need to run your tests. Plus, you’ll need to ensure these systems remain up to date for each dependency.
One of the key advantages of using Docker is that you can use the same image on several operating systems and not worry about anything working on each tester’s device differently. It ensures a consistent image in all environments with everything you need, provided you use the same image version.
Ok, enough of theory. Let’s materialized it. Considering you already have Docker installed in your machine and you have git cloned the eShopOnWeb, the steps we explained with details in the first, second and third posts of a series of four, we are now prepared to test some scenarios using the official and stable TestCafe Docker image.
First, let me explain the scenarios I have prepared. We are going to test 5 scenarios. TestCafe calls each scenario as a fixture:
- eShopOnWeb Home page – verifies if the eShopOnWeb’s home page loads properly. Don’t forget to change the IP address of URL (https://192.168.2.217:5001/) shown below to your machine’s (i.e. the docker containers host)
- eShopOnWeb Login – checks if the user can log in to their account
- eShopOnWeb Use Case 01 – checks if the brand filter visual component exists
- eShopOnWeb Use Case 02 – checks if the brand filter is clickable and there is a “.NET” option
- eShopOnWeb Use Case 03 – checks if the filter returned the right number of items
Learning TestCafe is very straightforward and powerful, so take a chance and get to know it!
Before you go to the next step, make sure you have the MSSQL Server 2019 running. We explained how to do it here.
First, let’s run eShopOnWeb without debugging by selecting the menu Run and then Run Without Debugging as shown below and wait until it automatically opens the browser with the URL https://localhost:5001.
To run the 5 tests, open Powershell or Windows Terminal and run this command:
docker run --rm -v c:/sitk/eShopOnWeb/tests/e2eTests:/tests -it testcafe/testcafe chromium /tests/*_test.js . It will show something like that image below if you haven’t pulled the TestCafe image before.
As you can see, all tests have passed, which is great!!
The docker run command on a TestCafe image dissected
Let me explain the command:
docker run --rm -v c:/sitk/eShopOnWeb/tests/e2eTests:/tests -it testcafe/testcafe chromium /tests/*_test.js.
docker run tells Docker to run the image you have specified in its separate container.
The next parameter
--rm is an optional parameter that tells Docker to delete the container after it has been completed. By default, Docker maintains the file system of the container when the container commands are ended. It helps retain the file system for debugging or getting some data from the container after running. But it’s best to delete them for short-lived processes like running end-to-end tests to avoid cluttering your disc space.
-v tells Docker to mount a directory from your computer to a directory inside the container. The container will access your local test files in this way. In this case, the position of your TestCafe tests in your local environment is “c:\sitk\eShopOnWeb\tests\e2eTests”, and the directory in the Linux container is /tests. Don’t miss the separator colon (:) between the local directories and the container directories.
Next, the parameter
-it tells Docker to run the specified image in interactive mode. In short, it sets up an interactive shell to pipe our terminal window into what the container is doing. If you run this command without the
-it parameter, the command will still be executed, but its execution will not be correctly displayed on your terminal.
We need to specify the image we want to use for our container after configuring the parameters we need for the Docker. It’s the
testcafe/testcafe the official stable image we chose to run it.
Finally, we get to the commands inside the container that we want to execute. For the browser parameter, as these are the pre-installed browsers in the Docker image, you can use either
chromium. You can then use the test file parameter to specify which test files you want to run. In our example, we ran all files ending with *_test.js in the /tests mapped folder inside the container.
Notice that you will not see the browser open up and run your tests even in normal mode when running them into the Docker image. Additional configuration is needed to run graphical interfaces using Docker on your desktop, such as a web browser so that Docker can use your desktop’s display server. It’s not a straightforward solution and differs by the system.
I installed TestCafe on my machine to let you see how it worked when we said it is a user perspective test approach. See the video below and pay attention to the browser’s footer in the video so you can follow TestCafe going through all 5 scenarios.
Congratulations, you have a powerful software development environment, and now you know how to test the software you are developing using cutting-edge technology and methodology. Next, we will wrap it up by adding Jenkins to orchestrate all tests in an automated manner.
We also have pro bono projects just in case you are interested in learning more about them.
Be aware of why we started this project by clicking here.
Learn more about other posts here.
If you are a good reader like myself, I recommend the following readings:
- Docker Quick Start Guide: Learn Docker like a boss, and finally own your applications
- Docker for Developers: Develop and run your application with Docker containers using DevOps tools for continuous delivery
- C# and .NET Core Test-Driven Development: Dive into TDD to create flexible, maintainable, and production-ready .NET Core applications, Kindle Edition
See you in the next post!