At the February Test Automation Meetup, Nikolaj Tolkaciov, head of Devbridge’s testing practice, talked about his experience with different test automation frameworks, the reasons for choosing Cucumber, and the results.Cucumber is a testing framework where you can create scripts for automated tests. Some testers react exceptionally negatively to Cucumber. What they don’t like is this. It was promised that to switch to automated testing in Cucumber, you can simply copy and paste the old manual test cases, and everything should magically work. But this did not happen; expectations did not coincide with reality. As for the private opinion of Nikolaj Tolkaciov, Cucumber is a regular framework with its own advantages and disadvantages. His experience as a QA engineer includes experience with calaba.sh, SpecFlow and Protractor.
Alternatives to CucumberCalaba.sh has been used to automate mobile app testing. It connected to Apple and Android test devices, which allowed feedback on the quality of testing performance. The downside of using SpecFlow was different steps for different situations, up to 20 options for the same action. Each time you write a function file, it is not known in advance whether the chosen option will work. Also, in SpecFlow, it is problematic to reuse tests each time you need to write a new one. Protractor is a BDD framework used for AngularJS apps, similar in functionality to Cucumber. And if, after some time using business intelligence, project managers and designers abandoned the use of BDD and returned to simple requirements, testers decided to continue using this approach.
Experience with CucumberThe main goal of using this framework is to achieve maximum reusability. It is essential to keep the number of step options as low as possible so that testers always know which step is next and be sure that it will work. Nikolai’s team was able to achieve the following:
- Feature step writing became easier as there are fewer step definitions you can cheese from.
- Lower maintenance.
- Step definitions became more flexible.
- Total Step count: 12579 (+1811 including examples) in scripts.
- Total step definition count: 53.
- Average: 237(271) steps/step definition. If you have 53 step definitions, they will be used in 237 different places (buttons, drop-down lists, fields, etc.).
- Median ~ 500 steps/step definition. 10 of the 53 step definitions are still used once or twice, so the actual median will be about 500 steps/step definition.