In software, tests are imperative for quality assurance as they ensure that your application is working as expected, irrespective of modifications in the source code to comply with changing requirements. However, you may have come across a specific category of flaky tests; these tests are unstable and unpredictable because they may pass or even be in identical conditions, i.e., have the same configuration without code, data or environmental changes. This article discusses flaky tests, why they occur and some best practices to avoid them.

An Overview of Flaky Tests

A flaky test is defined as a test that can pass or fail regardless of any changes to code, data or environment. This inconsistency in test results can stifle development by breeding disbelief, and delay production rollouts due to the need for unnecessary debugging. This phenomenon puzzles developers and undermines their trust in testing; hence, a systematic approach is needed to deal with this problem.

Step 1 of 7

14%

Which of the following best describes how you currently monitor and manage systems across your hybrid cloud environment?(Required)

Multiple separate tools for different environments

Multiple separate tools for different monitoring and management functions

Single consolidated dashboard

Custom built monitoring and management solution

No full monitoring and management solution

Don’t know

How valuable would it be for your monitoring and management solution to incorporate predictive analytics to help you identify and address system issues that are critical and not waste time or effort working through all the data and “noise” that is not critical?(Required)

Extremely valuable

Moderately valuable

Somewhat valuable

Slightly valuable

Not valuable

How do you most typically address potential system issues BEFORE they impact your operations?(Required)

Automated workflows and playbooks to remediate potential issues as soon as they are detected

An orchestrated response process that leverages automation, AI, and human expertise to rapidly respond to potential issues

Advanced monitoring tools such as predictive analytics and anomaly detection to proactively identify and address potential issues

Respond to issues as they arise, using manual processes and troubleshooting techniques

A combination of automated remediation, orchestrated response, and/or proactive monitoring to address potential system issues

Don’t know

Other

What methods does your organization use to control and monitor cloud/IT spending?(Required)

Centralized procurement processes

Pre-approved budget thresholds

Automated cost monitoring tools

Department-level spending controls

Regular cost audits and reviews

Cloud cost optimization platform

Approval workflows for resource provisioning

Native financial operations tools provided by the cloud provider

No formal controls in place

Don’t know

Other

Please specify:

Which methods do you rely on most to detect security vulnerabilities in your systems?(Required)

Automated scanning tools

Manual assessments

Vendor notifications

Threat modeling (for early detection)

None of these

Don’t know

Other

Please specify:

What are the TWO most common ways you prioritize vulnerabilities to address within your environment?(Required)

Based on potential impact

Based on exploitability

Regulatory compliance requirements

Based on CVSS scores

Risk-based analysis of multiple factors

First-in-first-out approach

No formal prioritization process

Don’t know

How do you typically handle system updates and security patches?(Required)

Automated updates and patches

Manually review and apply updates and patches on a regular schedule

A combination of automation and manual review

Apply updates and patches on an as-needed basis.

Outsourced update and patch management

No defined process

Don’t know

Other

Δ

Flaky tests are inconsistent since a test can fail on one occasion and then succeed on another while using a static test environment and codebase. There are many reasons behind this inconsistency including timing issues such as race conditions and reliance on external systems which might behave unpredictably, among others.

Impact on Software Development

Flaky tests can greatly affect the workflow in SDLC leading to a waste of time and resources when teams chase after errors that are nondeterministic, unreliable and hard to find inherently. The following are the key areas where flaky tests can impact in a typical SDLC process.
• Delayed software delivery
• Reduced trust in test results
• Increased maintenance costs
• Decreased test coverage
• Increased debugging effort
• Impediments to CI/CD

Common Causes of Flaky Tests

To determine a proper strategy for minimizing the impact, we must first understand why tests become flaky.

• Concurrency Issues: When tests are executed concurrently, i.e., multiple tests contend for shared resources at the same time, deadlocks, race conditions may occur and these in turn might lead to unstable outcomes.
• External Dependencies: When your tests need interaction with external systems such as APIs or databases, this introduces flakiness because of network latency as well as different response times.
• Non-Deterministic Logic: When your tests rely on random elements such as dates, times or user input, i.e., anything that cannot be predicted with certainty, they can have different outcomes.
• Unstable Environment: When the environment where the tests are conducted lacks proper isolation, there might be differences in application performance, configurations and availability rates, which might eventually cause a test to work once but fail another time under identical conditions.

How to Detect Flaky Tests

There are several strategies you can adopt to detect flaky tests, such as using different methods and tools to find inconsistency and unreliability patterns within the outputs generated by them.

Analyzing Historical Test Results
Examining historical test results is a reliable approach to uncovering unreliable tests. You can do this to identify irregularities or patterns within your data sets while comprehending the common trends or occurrences that may be useful when estimating the likelihood of a test failure.

Using CI Tools With Built-In Flaky Test Detection Capabilities
You can also leverage CI tools that have built-in flaky test detection capabilities to identify tests that have inconsistent behavior. By using such tools, you can detect flaky tests at an early stage in the SDLC process thereby saving you time and effort in identifying flaky tests manually.

Execute Tests Multiple Times
You can execute your tests multiple times using testing frameworks or as part of a CI/CD pipeline to determine variability in the test outcomes. If nothing was changed in the code or the environment, yet the test results indicated anomalies and inconsistencies, then that is a good indicator of flakiness.

Monitoring and Documenting
Another strategy is to keep an eye on test execution history to determine any anomalies in the test results over some time. By recording metrics such as, failure rates, you can understand how consistent, stable and trustworthy the tests are.

How to Fix Flaky Tests

Here are the key strategies you could adopt to fix flaky tests.

Isolate the Test
Firstly, you should identify and quarantine the flaky test. Then, replicate it to determine the root cause, i.e., why a test becomes unstable. Remember to reset or clean the state before and after every test.

Refrain From Using Random Data
Since random data can cause unpredictable behavior, you should ensure your test doesn’t depend on random or unexpected data. It would help if you used fixed or predefined values for user input and deterministic algorithms or methods to generate or process data. If you need random data, ensure you log the data appropriately to enable a rerun of the tests.

Make Your Tests Robust
Your tests should be equipped with retries, timeouts or waits for handling network or performance issues. They should also be adept at being executed under various environments and conditions. Create assertions meant to verify an expected result or behavior by checking ranges, patterns and other pertinent parts correctly designed for that purpose.

Eliminate External Dependencies
You can use mocks, stubs or fakes to reproduce external system behavior without using real dependencies. You must ensure that your tests don’t rely on external factors or affect other tests.

How to Maintain a Test Suite Free of Flaky Tests
To build a test suite devoid of flaky tests, you need to be proactive and follow some recommended practices.

Remove Duplicate Tests
While it is essential to review tests, it is equally important to get rid of repetitive tests, as they slow the testing process. Regularly maintaining the tests ensures accuracy and reliability, thus making the test results more precise and actionable.

Continuous Monitoring
Establish a monitoring system that continuously monitors test execution and output to help identify discrepancies quickly and eradicate them from the test process. You should also provide your team with the necessary tools (monitoring tools, reporting systems and necessary hardware) to run the tests.

Prioritize the Flaky Tests
Prioritize them based on the level of business risk, the amount of effort needed to fix them and test timing. There is a lot more value in focusing on tests that validate critical business workflows. Tests seldom used in a business workflow should be low priority and ignored. Additionally, specific flaky tests need much time and effort to fix, which should be removed during test maintenance.

Analyzing Test Metrics
You should maintain records of common issues to help your peers learn from past problems and solutions. To get insights, identify trends and patterns, and make the right decisions. It would help if you collected relevant metrics such as test execution times, failure rates and flakiness patterns. It would also help if you imparted training on writing reliable and robust tests, what flaky tests are and how to deal with them.

Conclusion

While eradicating flakiness is challenging, you can always trace its origin and handle it pronto. Managing flaky tests is an ongoing iterative approach where you identify, understand and fix intermittent or unpredictable tests. You can mitigate the challenges of flaky tests by adopting the strategies and best practices outlined in this article.