We created this whitepaper to provide an essential, independent guide to the testing apps in the Atlassian world. The Marketplace is a vast and rapidly changing, enabling organizations a multitude of choices for many solution areas – of which, testing is well represented. With the gradual demise of the old HP ALM platform, many organizations have realized that they already have and could utilize Jira, an excellent platform to support all of these testing functions.
The term “apps” is used throughout the article; Atlassian’s terms have included also “plugins” or “add-ons”. This paper provides a context for assessing testing apps, identifying the relevant ones for discussion, analyzing their strengths and weaknesses. We are not going to focus on feature matrices – these are easily manufactured and dependent on each app vendor’s own development efforts. Instead, we provide a logical framework to help you select the right tool for you.
Our analysis takes a deeper dive inside the selected apps, to understand how they address the “secret sauce” of testing – how together teams can identify problems and fix them quickly.
Although described in more detail below, it is important to distinguish Atlassian tools’ functionality from what additional apps have to offer. In this paper, we are not evaluating applications developed by Atlassian, but turning our attention to the add-ons that enhance testing capabilities in Jira.
The vast variety of Jira testing apps can, based on how they’re built, be divided into two categories:
Another means of distinguishing Jira apps, is to assess available hosting options: Cloud / Server / Data Center.
In this analysis, we have focused on the most common scenario, the Jira Server application. We will provide a Cloud vs Server comparison, and a more thorough analysis on standalone apps in our next review. Although constantly improving, today’s Cloud apps tend to be less feature rich than those on Server.
The standalone testing apps in the market have a rich set of capabilities, and are designed to seamlessly integrate with Jira, unlike some available testing platforms (e.g. the old HP ALM). We decided that most organizations will do their own very focused review on standalone testing apps so we remained focused on the Jira Server versions.
In this paper, we focused on three current testing apps, market leaders for Atlassian Jira (Core / Software):
All the above mentioned are excellent apps – but each comes at the testing space from different vectors. Contegix has experience implementing all three. We have practical real world observations of their strengths and weaknesses. It’s this we share.
In the diagram below the very basic model of how Jira works is explained. Jira calls all its record items “issues”. Essentially types of issues are created that really can be anything – not only epics and stories but also risks and other project-control records, and potentially tests.
The key is that once issues are created they can be managed in the various facilities that Jira provides such as Filters, Projects and Boards. When considering third party apps an important question is: “how does the application work with other Jira facilities, rather than as a standalone product?”
Conversely what does the new application provide that is not natively available in Jira. A further element is that reporting apps such as eazyBI can generate excellent reports if the items or issues in question conform to the basic Jira architecture displayed below.
The next aspect to understand is how the various core Atlassian applications work together to support the Software Development Lifecycle (SDLC). Testing is best thought of in the context of the SDLC main functions. As an example, when code is built using a build engine such as Bamboo, it will automatically detect and expose any breaks; this in itself is a form of initial basic testing. Moreover, build engines can trigger further automated testing which will create defects that need triage in the development backlog.
Best practices for using Atlassian tools for SDLC is generally as follows:
Thus traceability is ensured throughout the entire lifecycle, documentation gets centralized in a documentation platform and relationships are clear.
Notionally, Testing could be supported by creating an issue type called “test case” with more functional information carried in Confluence. These test cases could be excluded from the relevant scrum boards. The problem with this is that a test case can be run over and over and in different environments. Thus test in the traditional sense needs more attention.
So if we add testing into this mix – we essentially need some additional entities or objects. Pure agile calls for a fully integrated testing approach but in reality testing applications will typically support the following functional model described below. Fundamentally, defects certainly need to end up back in the Jira backlog for teams to work on.
They need to be a part of the release management process as well. When reviewing testing apps, how they deal with this matter is a major consideration. Testing specialists will look at the elements below and consider how well supported they are and how manageable the various aspects of the overall process.
Given the needs of Test Planning, Test Cases and Test Execution, the question is how various apps tie these elements together. To answer, we have to incorporate the above with the Atlassian overall model. This helps us truly appreciate how a testing app will work with the Atlassian stack. In the model below we have three stakeholder perspectives:
As can be seen below manual testing is going to generate defects but so obviously will DevOps initiated automated testing.
We appreciate that with agile best practices there should not really be a divide between the core Agile development process and testing but also it’s important to accept that for platforms and other major systems there are often many pre-existing test cases and automated testing scripts, especially for regression testing.
Any solution that enables visibility and reporting across this spectrum will be very advantageous because it will enable teams to determine where there are bottlenecks, and provide full traceability, deployment speed and other key performance metrics. It’s important to point out that the end result of any test is either a pass, or the creation of a Jira issue such as a defect or bug. This Jira issue creation forms the work items in the backlog of the development team.
Thus testing apps ultimately trigger the creation of defect issues that involve the larger whole of the project and team. Because of the highly “intermingled” nature of development and testing we believe that any world class app needs to be “intermeshed” as much as possible with core Jira functionality.
This is in addition to the straight race to “feature richness”, as well as integration with build engines and other related tools.
There is one more additional dimension to our analysis and that’s perceived market share. At the time of this review we are showing the relative number of downloads each product has below. In our experience this is indicative of the popularity of the app. We always encourage clients to inspect the marketplace download statistics – just to ensure that they appreciate the feedback and popularity of apps. It’s our belief that Zephyr has been traditionally the market leader and so has dominated the space.
This doesn’t mean it’s the richest testing app in the market. Rather it’s the first mover and current market leader by volume. There is so much feedback on these apps that it’s difficult to intuit anything from this data set. The results though are clear below – Zephyr is larger by 4* than Xray and Xray is 50% larger than TM4J. We know that downloads are not sales, but in this case we’re assuming that Xray and TM4J convert similarly, while Zephyr has a relatively poorer close rate (nevertheless leaving them clearly the market leader).
The testing space is a complicated arena because there are three, very well qualified solutions, competing for your business. Yet as will be seen they come at the challenge from varying directions which means that it’s possible to distill the true differences between them. We feel that it’s important to reflect a number of key considerations in how you view these offerings:
As an FYI we reviewed pricing and did not consider it a material consideration.
Below is our Jira Testing app excellence matrix:
What does this mean to you the buyer?
We think that it’s important to consider your own context.
It doesn’t surprise us that all three products are finding success, likely because of some of the factors identified above.
This is a short version of a longer research paper. To get the research version please use below opt-in to download.
Editor’s Note: This whitepaper is branded as Blended Perspectives as it was created prior Blended Perspectives being acquired by Contegix.