A Briefing on Test Automation Frameworks: Keyword-Based, Data-Driven and Hybrid

Abstract

As software development teams and testers diverge for greater speed and efficiency, there has been a increasing need for stable and streamlined test automation frameworks. This brief examines three test automation frameworks that stand out for their flexibility and rapid deployment capabilities. We look at what each framework is built to accomplish, along with their high-level pros and cons.

The Role of Test Automation Frameworks

The growing need for continuous delivery and the uptime expectations for Rich Internet Applications (RIAs) have put increased pressure on software development teams. Most of the RIAs dominating the market today would be impossible without automation and regression testing. Not only does test automation cover more ground at a lower cost with a higher confidence interval, but also regression tests ascertain whether any aberrant or unexpected behavior arises from data changes or software modifications. Development teams, often incorporating insights from operations specialists, are moving automated test scripts earlier in the software development life cycle (SDLC).

At the same time, the functions of testers and developers have diverged, allowing companies to reallocate development resources and still maintain speed of deployment. This is where test automation frameworks come in, allowing testers with less experience to manage sophisticated testing on shorter development cycles. Frameworks incorporate components such as coding standards, test data handling and object repository treatments. There are many test automation frameworks, but all are designed to make code more portable, reusable and easier to maintain. Currently, three of the most common test automation frameworks are keyword-based, data-driven and hybrid. Each incorporates different testing methodologies.

The 3 Frameworks and Their Uses

There are a wide range of automation tools that can be adapted to take the best advantage of any framework. Many companies perform automation testing using platforms like Selenium WebDriver. For data-driven frameworks, Selenium pulls test data from an external database table such as an Excel spreadsheet, a table using comma-separated values (.CSV) or an XML table.. For keyword-based frameworks, the code calls external files with keywords relevant to executable test cases. Here’s a closer look at these two different frameworks usually operate, and the hybrid framework that combines elements from each.

Keyword-Based Automation Testing

Keyword-driven testing is a simple methodology for software testing that separates the documentation of test cases and test case data from the way the way test cases will be executed. The defining feature of a keyword-based or keyword-driven automation testing framework is that the test automation implementation can be developed in isolation from the test case design. Every action in the test case is assigned a reusable keyword so that a developer is not required to assemble, recombine or maintain testing projects. Many experienced development teams prefer the keyword-based framework because it shares similarities with traditional manual test cases. The full functionality of the RIA is tested in using step-by-step instructions.

One of the most valuable aspects of keyword-driven testing is that detailed test plans are laid out with expected inputs and verification data are contained in an easily shareable spreadsheet. With utility scripts in place, you can launch the automation testing tool using spreadsheets and data tables. Tester productivity is higher because you do not need extensive training on automation tools prior to run the testing scripts. The downside is that test plan development can be slow on the front end as scripts and keywords are assembled.

Data-Driven Automation Testing

Data-driven testing is a software testing methodology that pulls test inputs from a table of conditions and verifies outputs against that table. Nothing is hard-coded but added into the table as needed. No matter what input and output values are pulled from data files and compared, the automation tool does the data-driven testing. These data files can be objects or can be taken from an ODBC source. Hard-coded scripts load in variables from the source files or are captured from the RIA. Many teams prefer this as they can generate scripts on the fly during the early stages of development. There is no wasted work because scripts are created to test specific components of the RIA. Automated testing can run and return values without being closely monitored.

As a tester, you have less control with this method though, because developers are required to generate the scripts for the automation tools. In addition, the company must make a great deal of computing resources available for handling the data files. You will have to manage not only your test plan but also keep track of all the data files in various directories.

Hybrid Automation Testing

A combination of the keyword-based and data-driven frameworks is considered a hybrid approach. This is especially useful in cases like enterprise portal automation testing due to the vast amount of data variability involved. This is true for some of the most common portals such as portals for customers, suppliers, employees, business intelligence (BI) and e-commerce.

Many testing teams end up creating their own hybrid automation testing frameworks that are customized to their unique SDLC. A specially formulated mix of keyword-based and data-driven testing frameworks can capture the best aspects of both approaches.

Summary

How companies choose to use a keyword-based, data-driven or a hybrid automation testing framework is ultimately determined by the most efficient use of their resources for any given project. It also depends on matching the testing team’s expertise to the right automation tools. Companies can get started immediately with data-driven automation testing frameworks as long as testers have the data and other resources they need. Keyword-based automation testing frameworks take longer to set up initially but give testers more freedom during the development cycle. Hybrid often makes the most sense, especially in testing projects like portals, where a vast amount of data has to be handled in a short amount of time and the RIA functionality must be tested adequately against any possibility of failure.

Leave a comment