Nikolaj Tolkačiov – Don’t hate the cucumber, hate the salad

At the February Test Automation Meetup, Nikolaj Tolkaciov, head of Devbridge’s testing practice, talked about his experience with different test automation frameworks, the reasons for choosing Cucumber, and the results.

Cucumber is a testing framework where you can create scripts for automated tests. Some testers react exceptionally negatively to Cucumber. What they don’t like is this. It was promised that to switch to automated testing in Cucumber, you can simply copy and paste the old manual test cases, and everything should magically work. But this did not happen; expectations did not coincide with reality.As for the private opinion of Nikolaj Tolkaciov, Cucumber is a regular framework with its own advantages and disadvantages. His experience as a QA engineer includes experience with calaba.sh, SpecFlow and Protractor.

Alternatives to Cucumber

Calaba.sh has been used to automate mobile app testing. It connected to Apple and Android test devices, which allowed feedback on the quality of testing performance.The downside of using SpecFlow was different steps for different situations, up to 20 options for the same action. Each time you write a function file, it is not known in advance whether the chosen option will work. Also, in SpecFlow, it is problematic to reuse tests each time you need to write a new one.Protractor is a BDD framework used for AngularJS apps, similar in functionality to Cucumber. And if, after some time using business intelligence, project managers and designers abandoned the use of BDD and returned to simple requirements, testers decided to continue using this approach.

Experience with Cucumber

The main goal of using this framework is to achieve maximum reusability. It is essential to keep the number of step options as low as possible so that testers always know which step is next and be sure that it will work.Nikolai’s team was able to achieve the following:

• Feature step writing became easier as there are fewer step definitions you can cheese from.
• Lower maintenance.
• Step definitions became more flexible.

Speaking about whether this framework justified itself for 3 years of its active use, it is worth paying attention to the statistics:

• Total Step count: 12579 (+1811 including examples) in scripts.
• Total step definition count: 53.
• Average: 237(271) steps/step definition. If you have 53 step definitions, they will be used in 237 different places (buttons, drop-down lists, fields, etc.).
• Median ~ 500 steps/step definition. 10 of the 53 step definitions are still used once or twice, so the actual median will be about 500 steps/step definition.

For comparison, there were 3500 steps and 175 step definitions in the previous implementation of the specification flow. That’s four times the number of steps for the same number of scenarios. Accordingly, working with such a set of characteristics is 16 times more complex – in maintenance, determining the steps for work, and refactoring big data, such as the codebase. On average, each scenario was used only 19 times.

Summary

If we compare Gherkin and non-Gherkin approaches, non-Gherkin automation will be somewhat linear, i.e., complexity will increase over time, but this will not be a problem. In the case of Gherkin automation, there is some overhead. Sometimes this is a minus when you run into bottlenecks or function files. Because if you already have a step definition and it’s already in use, and you want to create a new one, you might be locked out. In such a situation, you will have to come up with something creative to add a new step definition.