Automated testing (AT) is a long, step-by-step process. First, you need to determine the scope of automation and select tools, create a strategy, and only then proceed to the execution of test cases. What’s next? The final stage is the construction of testing reports, which describe the completed process and record the results. Web tools such as Reportportal and Allure make working with large amounts of information easier. In this article, we’ll look at the features of both solutions, but we’ll start with allure report project the basics.
Table of Contents
What Is Test Automation Report?
Reporting is an essential step in the automation process. The test report generation tool summarizes the work results and contains information with which you can compare the current situation with the test plan and evaluate the volume and quality of the work performed. The data obtained helps to draw conclusions about the product development and make managerial decisions.
Such properties as informativeness, accuracy, and objectivity are essential for the report. After the test report and reading allure report graphs in it, there should be no questions about the quality of the project or any doubts. Results data are of interest to a wide range of people.
Who is interested in studying test results:
Project manager: studies data on the current situation and makes decisions.
Development team leader (dev-lead): forms an objective view of the project.
Test team leader (test lead): structures thoughts and collects data to contact the project manager.
Customer: views information about the project in which they invested money.
In automation, reporting is just as important as in manual testing. The viability of the automation environment may depend on how well the reporting mechanism is implemented. Test automation report is a blueprint of everyday execution that shows executed test scenarios and steps in full detail: time, checkpoints, actual results, and failure reasons.
Each QA team is interested in getting objective, detailed and complete testing data. It is crucial that this does not require a lot of time and does not create an unnecessary burden for team members. It is the reason for the popularity of simple but reliable web solutions Reportportal and Allure. They compete because they each have strengths and are worthy of attention.
Best Test Automation Reporting Tools
Reportportal
It is an open-source web service created by EPAM and the OSS community. Simply saying, it is designed to collect, combine, and analyze documents and results of various testing projects. This data becomes available to interested parties: both testers and IT specialists, as well as business customers.
Key features of Reportportal:
integration with various testing environments with the ability to collect and view tests in one place;
integration with bug tracking systems such as Jira, Rally;
collection and consolidation of all information (history, logs, attachments), which simplifies the search for the causes of failures in previous launches;
real-time test analytics, so you can start investigating the causes of failures immediately after completing a test;
access to all automatic tests within the project, convenient search for information about any elements;
classification of failed tests (e.g., a product error or a problem in the system) to visualize the structure of failures;
analysis of failure causes using Auto-Analyzer, which is based on ML algorithms.
The web-based solution improves the speed of analyzing results and compiling test automation reports. It is possible thanks to a wide range of built-in analytical functions. Reportportal is often chosen as an additional tool for continuous integration or CI (when developers regularly merge code changes) and constant testing.
With this tool, the QA team can:
Achieve complete automation transparency by aggregating all results and reports;
provide collaborative analysis of failed tests and identify problems and bugs;
take analytics and trend forecasting to the next level by creating custom metrics and dashboards;
unite team members in an automation environment for coordinated decision-making and improved delivery.
Reportportal handles heavy workloads and meets performance requirements because it’s built with cutting-edge technology. The official website lists 3 technologies: PostgreSQL, REST Web Service, and Mobile responsive UI. Working with the tool does not cause both experienced and novice users problems.
How to get started:
Set up and deploy in one of the following ways: Docker or Kubernetes.
Open in a new tab and log in.
Integrate with a test environment.
We recommend choosing a Linux-based environment for deployment. In the final step, it is proposed to select integration by language: Java, .NET, Javascript, Python, PHP, and others. If there is no integration, you can do it yourself.
Allure
Allure reports is a flexible multilingual reporting tool that provides maximum useful information to all stakeholders. The Qameta Software team is responsible for its support. Developers and QA value it for its ability to minimize the defect lifecycle. Project managers choose a tool to generate allure report to keep abreast of the general situation in the project: tested features, groups of defects, schedule, etc.
Allure key features:
general report statistics and statistics for each run;
custom classification of defects;
demonstration of statistics in graph format;
retrospective visualization with precise timings;
grouping results by Epic, Feature, Story, and packages;
test case page with detailed data about it;
support for different types of tests in one report;
integration with custom tools.
The main features are related to unreliable tests, test class and defect categories. So, tests (test classes) can be marked with a special mark to display them as unstable. As for the defect categories, they can be classified as product defects or test defects.
With this tool, the QA team can:
get important information about the progress of the tests;
maintain communication with CI and bug tracking systems;
ensure transparency of the process, results, and problems;
reduce the time spent analyzing and fixing defects.
The data source for the allure report is the adapter. For example, generate allure reports if tests are written in jUnit, the Allure jUnit adapter is included. During tests, it collects information and aggregates it into files. Based on these files, a report is built. Each framework has its adapter. Other data sources are also provided, e.g., CI server (test history), Jira (statuses, errors in the product), etc.
How to get started:
Download and install commandline (depending on your environment).
Run test to get data to build a report.
Generate a report from the extracted information.
The tool is called an absolute polyglot. It integrates with many languages and test frameworks: Java, Python, JavaScript, Ruby, Groovy, PHP, .NET, Scala, Go, and Cucumber. It also supports TDD and BDD approaches and is “loyal” to various CI systems (GitHub, GitLab, Jenkins, TeamCity, Bamboo, CircleCI). You can do the integration yourself by following the instructions and manuals.
Reportportal vs. Allure: What to Choose and When?
Both solutions have powerful features that can be useful in a particular situation. Let’s look at a few examples where they can be useful.
We recommend using Reportportal when necessary:
Save time. The results can be viewed within seconds of running a test. It reduces up to 90% of the time spent on reporting and analyzing results and, importantly, allows you to start debugging in real-time.
Merge data. The team has access to information about the test (logs, history, screenshots). Engineers can quickly sort tests and perform analysis, while managers can view statistics and graphs from the dashboard.
Set up KPI monitoring. The web solution allows you to customize the executive panel using widgets. It is convenient to track the key performance indicators used in the company.
We recommend using the Allure reporting tool when necessary:
Reduce the defect lifecycle. The tool allows you to divide defects into failed and broken tests, as well as steps, attachments, logs, fixtures, and timings. This is an opportunity for QA and DEV to save time and reduce workload.
Learn from experience. Adapters collect and organize timings of executed tests. This data is displayed in a timeline format (sequential, parallel). Visualization of the retrospective becomes the basis for further improvements.
Simplify decision-making. The tool is useful both for dev and QA and for the manager. A clear picture of what is going on in the project is always at hand: from the covered functions to test deadlines. It is easier to make decisions, plan, and make forecasts with this data.
As you can see, both tools deserve to be in your arsenal. Zappletech uses allure Report-portal when speed comes first. We value Allure’s flexibility, which is indispensable in multitasking, and its “polyglot” functions.
Do not underestimate the importance of test automation reporting. A lot depends on how well the process of allure results, creating and analyzing reports is set up: the team’s effectiveness, the timeliness of identifying problems in the product and the system path determining the causes of failures, the correctness of managerial decision-making. Want to get rid of the routine of reporting and processing results? Then use proven tools with a wide range of features, such as Reportportal and Allure.
