AQA ProcessTest Automation Frameworks

Aaron Evans – Test Automation Frameworks: Putting together the puzzle

Published 24 August 2023. Updated with a short answer, key takeaways, sources and a FAQ; the talk text is unchanged. Published by Zapple.Tech, founded by the founders of CodeceptJS and Codeception, open-source testing frameworks.

Watch on YouTube
Short answer

In this talk, Aaron Evans, an independent consultant with over 20 years of testing experience, breaks a test automation framework into pieces: tests, automation code, execution, reporting and communication. He covers criteria for evaluating a framework, missing pieces such as flakiness, what sits outside it, and how to find the right fit.

Key takeaways
  • The main pieces of a test framework
  • Criteria for evaluating a framework
  • Missing pieces: flakiness, traceability, reproducibility, communication, timeliness
  • What lives outside the framework
  • Finding the right fit and steps to build or improve it

Speech bullet points:

1) The main pieces of a test framework: tests, automation code, execution, reporting, communication
2) Criteria for evaluating a test framework: maintainability, reliability, speed, complexity, coverage
3) Identifying the missing pieces: flakiness, traceability, reproducibility, communication, timeliness
4) What is outside the framework: tools, processes, skills, environments, data
5) Finding the right fit for your project: language & libraries, types of tests, execution strategy
6) Solving the puzzle: steps for building or improving your framework

About the speaker:

I’ve been testing software for over 20 years. I’m an independent consultant and have worked with over 100 companies building (or fixing) test automation code and frameworks. I have seen a lot of good and bad code and met a lot of cool people. Currently, a test automation architect for Massachusettes Medical Society, publishers of the New England Journal of Medicine.

This is part of the speech from the online Test Automation Meetup – August 16

Questions about this talk

What are the main pieces of a test automation framework?

In his talk, Aaron Evans lists five main pieces: tests, automation code, execution, reporting and communication. He then gives criteria for evaluating a framework: maintainability, reliability, speed, complexity and coverage, and identifies missing pieces such as flakiness, traceability, reproducibility, communication and timeliness.

How do you choose the right test automation framework?

Aaron Evans talks about finding the right fit for your project by looking at language and libraries, types of tests and execution strategy.

What lies outside a test automation framework?

According to Aaron Evans, tools, processes, skills, environments and data lie outside the framework. His talk also lists the main pieces of a framework and the criteria for evaluating one. He has worked with over 100 companies building or fixing test automation code and frameworks, according to his speaker bio.

Request a QA audit quote

Tell us what you ship and what worries you.

What happens after you send this

  1. You send the form: your situation and what you need tested, then your name and work email.
  2. Within one business day, we reply with a suggested first step, its scope, timeline and a quote.
  • Fixed scope
  • You decide after each step
  • The tests live in your repository
  • An NDA signed before access

“They helped us to close the positions and consult our team how to proceed further. High-quality quick feedbacks with the deep checking of the required tasks, professional approach, great understanding of the company's technologies and product, detailed reports.”

Kristina Lavrentieva, Manager at keepgo.com, Israel
  • Rated 4.9 out of 5 on Clutch, from 7 client reviews
  • 42 completed test automation and infrastructure projects
  • Founded by the founders of CodeceptJS and Codeception, open-source testing frameworks

Request a QA audit quote

    What is closest to your situation?
    Optional. The product, the platforms, how often you release, what broke last.

    You decide after our reply whether to go ahead.

    Prefer to talk first? Book a call

    One-week QA audit · written report

    Request a QA audit quote