Software Testing Strategies and Types: A Complete Guide
Read about 20+ software testing strategies and types, their use cases, examples, and benefits. Learn how to choose the right testing approach for your product.
In 2012, Knight Capital Group updated the software on their trading platform. The system started acting strange, making trades that weren’t planned for within minutes. That bug cost them $440 million and almost put the company out of business in the 45 minutes it took them to find the kill switch. This failure was not caused by a single “missed test.” The software’s release and validation processes were the source of the breakdown.
This example now serves as a case study of what occurs when actual production risks are not taken into account during testing and release procedures. The reality is that most bugs won’t cost you anywhere near that much, but they will cost you something: revenue loss, customer trust, and development time.
There are dozens of testing types out there, and everyone has different opinions. While some people vouch for test-driven development, others find it impractical. Some teams automate aggressively, while others still rely on manual testing where it makes sense.
Instead of adding to that debate, this guide focuses on what actually matters: which testing strategies and types are useful in practice, what problems they’re good at catching, and when they’re probably not worth the effort.
What Is Software Testing
Software testing is the process of checking whether a system behaves a certain way under real conditions. It’s not just about finding bugs or proving that something works once. Testing looks at how software handles everyday use, edge cases, mistakes, and changes over time. In terms of practical application, testing matches requirements with reality. Testing allows teams to verify that they’ve built the right solution and that it works as intended. Good testing looks at both the technical side and how real users interact with the system in practice.
Types of Software Testing
A good software product is built after each element is tested for reliability. A feature can work perfectly on its own and still fail once it’s connected to other parts of the system. A change that looks harmless can quietly break something that already worked. And when you ship broken software, it’s worse than not shipping at all. Software testing exists to solve these problems before your product is pushed live and problems turn into user-facing failures.
Since most software products are complex and heavily integrated, there are various types of software testing that teams use to test different elements of a product.
On the surface level, there are two types of testing: manual and automated. But it’s not as simple. Manual testing breaks further into multiple branches, and it goes on.
A brief overview looks like this:
Now, there are different ways in which experts break software testing types down into different categories and classifications, and different visual representations of testing type “tree” may exist out there. But the image above shows the core strategies that are common in most software.
A key thing to remember is that there is a significant overlap between all these testing types in practice. A lot of testing types can be automated and potentially fall under automation testing, and a lot of software testers consider following the testing pyramid, which divides all testing into three core strategies: unit tests, integration tests, and end-to-end tests.
Regardless of the approach you follow in practice, a clear overview of different types helps you make sense of and labelize what you’re actually doing.
Automation Testing
Automation testing involves using specialized tools and scripts to execute test cases automatically, reducing the need for human intervention. It is especially useful for repetitive tasks like regression testing, where the same tests must be run frequently after updates. By leveraging frameworks such as Selenium, Cypress, or Playwright, teams can build robust test suites that run quickly and consistently. One of its biggest advantages is speed, as automated tests can execute far faster than manual ones, especially at scale. It also improves accuracy by eliminating human error in repetitive validations and calculations. Continuous Integration and Continuous Deployment (CI/CD) pipelines often integrate automated tests to ensure faster feedback during development. However, automation requires an upfront investment in tools, scripting knowledge, and maintenance of test scripts as the application evolves. Despite this, it becomes highly cost-effective in the long run, particularly for large and complex projects with frequent releases.
Manual Testing
Manual testing is the process of evaluating software by executing test cases without the use of automation tools. Testers interact with the application as end users would, checking for bugs, usability issues, and overall functionality. It is particularly valuable in exploratory testing, where human intuition and creativity help uncover unexpected issues. Unlike automation, manual testing requires no scripting knowledge, making it more accessible for beginners or non-technical stakeholders. It also allows testers to assess aspects like user experience, visual design, and ease of use, which are difficult to automate. However, manual testing can be time-consuming and prone to human error, especially when dealing with repetitive tasks. Despite these limitations, it remains essential for scenarios where human judgment and flexibility are required.
Manual testing is further broken down into the three types of testing:
White Box Testing: White box testing focuses on the application to verify how the code works. It checks logic paths, conditions, loops, and error handling to ensure all critical branches are exercised. These tests help uncover hidden issues like unreachable code, incorrect assumptions, or unhandled scenarios that may never surface through user-facing tests alone.
Black Box Testing: Black box testing focuses on what the system does, not how it’s built. Testers interact with the application by providing inputs and checking outputs against expected results, without any knowledge of the internal code. This approach mirrors real user behavior and is especially useful for validating requirements, workflows, and edge cases that developers may not anticipate.
Grey Box Testing: Grey box testing is a software testing approach that combines elements of both black-box and white-box testing. In this method, the tester has partial knowledge of the internal workings of the application but does not have full access to the source code. This limited insight allows testers to design more informed and effective test cases compared to purely black-box testing.
Black box testing is performed in two ways or two stages: functional testing and non-functional testing.
Functional Testing and Its Types
Functional testing is a type of software testing that verifies whether an application behaves according to its specified requirements. Instead of looking at how the code is written internally, it focuses on what the system is supposed to do from a user or business perspective. Testers provide inputs, execute specific actions, and then check if the outputs match the expected results defined in requirements or user stories. For example, if a login feature is being tested, functional testing ensures that valid credentials allow access and invalid ones are rejected correctly. It essentially answers the question: “Does this feature work as intended?” This makes it a core part of quality assurance across almost every software project. Because of its user-focused nature, it is often aligned closely with real-world use cases.
Functional testing includes several levels, such as unit testing, integration testing, system testing, and end-to-end testing.
Unit Testing: Unit tests are used to break down an application into the smallest level of testable pieces, such as a function or a method. Each unit will then be run in isolation in order to make sure the unit has the expected output. Unit tests are very quick-running tests and are used in order to ensure a stable application.
Integration Testing: Integration testing checks how different modules, services, or APIs interact once they are connected. Sometimes, even when individual components work correctly on their own, problems often arise at integration points, such as data mismatches or communication failures. These tests help identify issues that only appear when systems depend on each other. There are two ways to perform integration testing.
1. Incremental Testing: A testing approach where software is tested in parts as new modules are added and integrated step by step.
2. Non-incremental Testing: An approach where all modules of the software are combined at once and tested as a complete system.
System Testing: System testing validates the whole application in an environment that closely resembles production. It verifies that all components work together as expected if the system meets both functional and non-functional requirements. This testing helps catch issues that can only appear when the full system is in place.
End-to-end (E2E) Testing: End-to-end (E2E) testing is a type of testing that verifies an entire application workflow from start to finish, just as a real user would experience it. Instead of testing individual components in isolation, it checks whether all parts of the system work together correctly. E2E testing is typically slower, more complex, and more expensive compared to unit or integration testing. These tests often require a fully deployed environment and can be sensitive to small changes, making them harder to maintain.
Learn more about unit testing, integration testing, and end-to-end testing in the testing pyramid blog.
Some testing types are not necessarily categorized or classified as functional testing, but they have testing layers that support or include functional elements in practice.
Smoke Testing: Smoke testing is not a functional testing type but a test execution level / build verification type (BVT). It’s a broad, high-level test that includes functional checks to ensure the critical functionalities of a new build are stable.
Sanity Testing: Sanity testing is a narrow, deep check performed on a stable build to verify that specific bug fixes or code changes work correctly. It can also be classified as a narrowed regression test.
API Testing: API testing is functional testing at the service layer. It verifies request/response behavior, business logic in APIs, and data correctness
Database Testing: It’s also largely functional (backend functional validation) at a test data layer and checks data integrity, CRUD operations, stored procedures / queries, and data consistency with UI/API.
Non-Functional Testing and Its Types
Non-functional testing checks how well a system works, rather than whether specific features work. While functional testing asks, “Does the login button work?”, non-functional testing asks things like, “How fast does it respond?,” “Can it handle 10,000 users?” or “Is it secure and easy to use?” It focuses on qualities such as performance, usability, reliability, scalability, and security. It’s especially critical for real-world readiness, where user experience and system stability matter just as much as functionality.
Non-functional testing includes several types, such as performance testing, usability testing, compatibility testing, and security testing.
Performance Testing: Performance testing assesses how the system responds to varying loads. As usage rises, it considers response time, resource consumption, and overall stability. These tests prevent failures during demand spikes and help teams understand system limitations. Three common types of performance tests are load testing, stress testing, and stability testing.
Usability Testing: Usability testing evaluates how easy and intuitive a product is for real users to interact with. It focuses on user experience by observing how people navigate the interface, complete tasks, and respond to the design. Testers often look for issues like confusing layouts, unclear instructions, or unnecessary steps that slow users down.
Security Testing: Security testing focuses on protecting the system and its data from threats. It finds defects like exposed data, exploitable inputs, and poor access controls. This type of testing is critical for reducing risk and ensuring the application can withstand real-world attacks.
Compatibility Testing: Compatibility testing ensures an application works correctly across different environments, such as operating systems, browsers, devices, networks, and hardware configurations. Its main goal is to verify that the software delivers a consistent user experience regardless of where or how it is accessed.
Other Types of Software Testing
There are other types of testing that are not commonly included in charts because they can occur at different levels and intervals, depending on how your product is developed and shipped. These testing types include:
Acceptance Testing
Acceptance testing determines if the software is ready to be delivered to the users. It verifies the system from a business and a user perspective, and it often involves stakeholders and product owners. The focus is on confidence, verifying that the software meets expectations and supports real-world use.
There are various types of acceptance testing, which heavily vary on specific project needs and requirements.
User Acceptance Testing (UAT): User acceptance testing is performed by the end users or clients to ensure the software meets real-world business needs and works as expected in practical scenarios. It focuses on usability and business workflows rather than technical issues.
Business Acceptance Testing (BAT): This type is performed to verify whether the software aligns with business goals, processes, and requirements. It is usually carried out by business analysts or stakeholders.
Contract Acceptance Testing (CAT): This ensures the software meets the conditions and requirements specified in a contract between the client and the development team. It is important in outsourced or vendor-based projects.
Regulatory Acceptance Testing (RAT): This checks whether the software complies with legal, industry, or government regulations. It is critical in sectors like healthcare, finance, and aviation.
Alpha Testing: Conducted internally by the development or QA team before releasing the product to external users. It helps catch major bugs early.
Beta Testing: Done by a limited group of real users outside the organization in a real-world environment. It helps gather feedback before the final release
Regression testing verifies that the recent changes have not caused any new issues with existing functionality. As software evolves, even small updates can have unintended side effects. Regression testing acts as a safety net, helping teams move faster without constantly rechecking the same areas manually. It occurs whenever there’s a new change or update in the software product.
System Integration Testing
System integration testing (SIT) is a higher-level form of integration testing where multiple integrated systems or external systems are tested together as a complete ecosystem. It ensures that different systems, such as third-party services, databases, or external applications, work seamlessly with the main system. SIT focuses on end-to-end data flow and interaction between multiple systems rather than just internal modules. It is commonly used in enterprise software testing environments where software depends on multiple interconnected systems.
While testing types state what you test, a testing strategy explains how you approach testing overall. It is the thinking behind the work. A testing strategy helps the team in deciding where they need to focus more, what risks matter most, and which testing types would actually make sense for the product and stage they’re in.
The majority of teams don’t just use a single strategy. Rather, they combine multiple strategies based on the system, the risks, and how the software is built and released.
Below are some of the most common testing strategies and how they’re typically applied in practice.
Exploratory Testing Approach
Exploratory testing is not a structured testing type, but primarily an approach that emphasizes personal freedom and continuous learning to improve test quality. It involves simultaneous learning, test design, and execution rather than following predefined scripts. It is best described as a flexible, human-centric approach, often structured into sessions instead of test cases.
Ad Hoc Testing Approach
Ad hoc testing is considered an informal type or method of software testing. It is an unplanned, unstructured, and random approach aimed at finding defects quickly by breaking the system without using documented test cases, often relying on the tester’s intuition and experience.
Static Testing Strategy
Static testing is a type of software testing approach where the application is tested without executing the code. Instead of running the program, testers review and analyze documents, requirements, design specifications, or source code to find errors early. It focuses on preventing defects rather than detecting them during execution.
Common techniques include reviews, walkthroughs, inspections, and static code analysis. Static testing is usually performed in the early stages of the software testing lifecycle, even before the software is built. It helps identify issues like unclear requirements, coding standards violations, and design flaws at a very low cost.
Dynamic Testing Strategy
Dynamic testing is a type of software testing where the application is executed and tested by running the code. It involves providing inputs to the system and validating the outputs against expected results. This type of testing is used to find runtime errors, performance issues, and functional defects. Dynamic testing includes all most common testing types like unit testing, integration testing, system testing, and acceptance testing. It is performed after the code is developed and focuses on verifying actual system behavior. Unlike static testing, it ensures the software works correctly in real execution environments.
Structural Testing Strategy
A structural testing strategy focuses on the internal workings of the software. It looks at how the system is built rather than how it appears to users. This strategy is tied to the codebase, and it is usually applied in early stages and continuously during the development phase. Unit testing, code-level integration testing, and white box testing are examples of a structural testing strategy. These test types validate logic paths, data handling, error conditions, and interactions between internal components.
Behavioral Testing Strategy
A behavioral testing strategy is a software testing approach that focuses on verifying how a system behaves from the perspective of the end user or business requirements. Instead of looking at internal code structure, it verifies that the software delivers the expected outputs when users interact with it. It is commonly applied using techniques like black box testing, system testing, acceptance testing, and regression testing. Behavioral testing is especially important because it validates whether the software actually solves the problem it was built for.
Front-End Testing Approach
Front-end testing focuses on the user interface (UI) and everything users directly interact with. It checks things like:
Layout and design consistency
Buttons, forms, and navigation
Browser and device compatibility
User interactions (clicks, inputs, validations)
UI responsiveness
It overlaps heavily with functional testing, usability testing, and compatibility testing. Front-end testing is more of a UI-focused testing scope, not a standalone formal category.
Back-End Testing Approach
Back-end testing focuses on the server-side logic and data processing that users don’t see. It checks things like:
APIs and services
Database operations
Business logic
Data integrity
Server responses and performance
It overlaps with API testing, database testing, integration testing, security testing. In other words, back-end testing is a system-layer testing focus, not a separate testing type.
Using TestFiesta for Software Testing
Testing strategies only work if the tools supporting them don’t get in the way. That’s where TestFiesta fits in.
TestFiesta is a flexible test case management platform designed to support different testing strategies without forcing teams into a rigid structure or workflow. Whether you’re focusing on behavioral testing, structural coverage, or a mix of approaches, TestFiesta lets teams organize test cases in a way that reflects how they actually work.
It supports truly flexible test management with features like tags, reusable steps, native defect tracking, and custom fields make it easier to adapt testing as products evolve. Instead of rebuilding test suites and test plans every time priorities shift, teams can adjust how tests are grouped, executed, and reviewed. This flexibility supports both fast-moving teams and those working on more complex systems, without adding unnecessary overhead.
Conclusion
Software testing doesn’t have a universal formula. The most effective testing strategies are shaped by real constraints, product complexity, team skills, release pace, and risk.
Understanding the different types of testing and how they fit into broader strategies helps teams make better decisions about where to focus their effort.
When testing is intentional and aligned with how software is built and used, it becomes a strength rather than a bottleneck.
FAQs
What is a test strategy in software testing?
A test strategy is a high-level plan that explains how testing will be approached for a product. It outlines what will be tested first, where effort should be concentrated, and how different types of testing fit together. Instead of listing individual test cases, it focuses on priorities, risks, and practical constraints.
What is the 80/20 rule in testing?
The 80/20 rule in testing suggests that a large portion of issues usually come from a small part of the system. In practice, this means a few features, workflows, or components tend to cause most problems. Teams use this idea to focus their testing efforts on high-risk or high-usage areas instead of trying to test everything with equal measure.
What are some common software testing strategies?
Common testing strategies functional testing, white-box testing, black-box testing, system integration testing, user acceptance testing, smoke testing, and behavioral testing. Most teams don’t rely on just one strategy. They combine several approaches based on the type of product they’re building and how it’s delivered.
Which software testing strategy is good for my product?
The best strategy depends on your product’s risk, complexity, and pace of change. A fast-moving product with frequent releases may need strong regression and automation support, while a simpler or early-stage product might benefit more from focused manual and exploratory testing. Team skills, timelines, and user impact also matter. The right strategy is the one that helps you catch the most important problems without slowing development down.
Tool
Pricing
TestFiesta
Free user accounts available; $10 per active user per month for teams
TestRail
Professional: $40 per seat per month
Enterprise: $76 per seat per month (billed annually)
Xray
Free trial; Standard: $10 per month for the first 10 users (price increases after 10 users)
Advanced: $12 per month for the first 10 users (price increases after 10 users)
Zephyr
Free trial; Standard: ~$10 per month for first 10 users (price increases after 10 users)
Advanced: ~$15 per month for the first 10 users (price increases after 10 users)
Accessibility testing checks whether people with disabilities can use your website or app without barriers. This includes users who are blind or low-vision, deaf or hard of hearing, keyboard-only navigators, and those with cognitive or learning disabilities.
Accessibility testing checks whether people with disabilities can use your website or app without barriers. This includes users who are blind or low-vision, deaf or hard of hearing, keyboard-only navigators, and those with cognitive or learning disabilities.
While most teams value accessibility, many struggle to define what to test, select the right tools, or maintain checks in their release workflow. This guide covers the essential standards of accessibility testing, a three-layer testing methodology, key web and mobile differences, top tools, and answers to common team questions.
What Is Accessibility Testing
Accessibility testing is the process of checking that a digital product works for everyone, including people who rely on assistive technology or use their devices in different ways. It asks simple questions, such as:
Can someone reach every button using only a keyboard?
Does a screen reader announce what an image shows?
Is the text readable against its background?
Can someone who can’t hear follow a video?
Accessibility testing is continuous, and it works best when it runs alongside the rest of your QA process.
Why Accessibility Testing Matters
There are three good reasons to take accessibility testing seriously:
More people can use your product: Disabilities are common, and many more people deal with temporary or situational limits, like a broken arm, bright sunlight on a screen, or a noisy room.
Legal risk goes down: Many countries have laws that require digital products to be accessible, and most of them point to the same set of guidelines.
Quality goes up: Clear labels, logical structure, and good contrast make a product easier for everyone, not just users with disabilities.
What Are Web Content Accessibility Guidelines (WCAG)?
The Web Content Accessibility Guidelines, or WCAG, are the global standard for digital accessibility. They’re published by the World Wide Web Consortium (W3C), and most accessibility laws and audits measure products against them.
WCAG is built on four principles, often shortened to POUR:
Perceivable: Users can see or hear the content. Images have text alternatives, videos have captions, and text has enough contrast.
Operable: Users can use every control, whether they navigate with a mouse, keyboard, or voice.
Understandable: Content and behavior are clear and predictable. Forms explain errors, and navigation stays consistent.
Robust: Content works with a wide range of browsers and assistive technologies.
Under these principles sit testable rules called success criteria. Each one has a conformance level:
Level A (Minimum Compliance): Establishes the essential baseline for web accessibility. It addresses fundamental barriers that would otherwise prevent users with disabilities from accessing or interacting with content entirely (e.g., providing alt text for images, basic keyboard navigation, and avoiding content that causes seizures).
Level AA (Standard Compliance): Serves as the widely accepted global benchmark for web accessibility and legal compliance. It removes significant barriers for a broader population of users with diverse visual, hearing, cognitive, and motor impairments (e.g., ensuring adequate text contrast, visible focus states, consistent navigation, and clear error handling).
Level AAA (Enhanced Compliance): Represents the highest and most stringent level of accessibility. It incorporates specialized criteria designed to optimize usability for users with specific or severe disabilities (e.g., providing sign language interpretation for audio, extended color contrast ratios, and simplified reading levels). Note that full Level AAA compliance across an entire site is generally not required or feasible for all content types.
The Accessibility Testing Tools Worth Knowing
Instead of using dozens of accessibility tools and cluttering your QA process, focus on a core set that covers each layer of testing. Most QA teams rely on the following essentials.
axe DevTools: Scans pages for WCAG issues and explains how to fix each one.
WAVE: Places icons directly on the page to show errors, alerts, and structure.
Google Lighthouse: Gives each page an accessibility score and a list of issues.
Pa11y: Command-line tool that runs accessibility checks on any URL.
Accessibility Scanner (Android): A free app that scans any screen on an Android device and suggests fixes for issues.
Xcode Accessibility Inspector (iOS): A tool that comes with Xcode and lets you inspect the accessibility details of each element in an iOS app.
Learn more about accessibility testing tools.
TestFiesta Keeps Your Accessibility Testing From Falling Through the Cracks
Accessibility testing often fails due to fragmented workflows: automated scans stay in CI/CD, manual logs remain in personal notes, and screen reader findings get lost in chat. TestFiesta unifies all three testing layers into a single platform.
Reusable Test Cases: Convert keyboard, zoom, and screen reader checks into standardized test cases with reusable steps for repeated actions.
Test Execution and Tracking: Schedule accessibility passes per release to monitor pass/fail status and test coverage in real time.
Defect Management: File accessibility defects directly from test runs and sync them to Jira or GitHub with full contextual details.
AI Test Case Generation: Automatically generate accessibility test cases from requirements, user stories, or UI screenshots.
CI/CD Integration: Import automated test results via the TacoTruck CLI alongside manual testing reports.
Accessibility shouldn’t be an afterthought or a chaotic scramble before release. Take control of your accessibility testing with TestFiesta.
How is accessibility testing different from usability testing?
Usability testing checks whether a product is easy and pleasant to use for a general group of users. Accessibility testing checks whether people with disabilities can use it at all, often against WCAG criteria.
Is accessibility testing different from user acceptance testing?
Yes, User acceptance testing (UAT) confirms that a product meets business requirements before release, usually with end users or stakeholders. Accessibility testing focuses on whether people with disabilities can use the product.
Which WCAG level should my team be targeting, A, AA, or AAA?
Most teams should target WCAG 2.2 Level AA. It’s the level most laws and contracts ask for, and it covers the barriers users hit most often.
Can I pass an accessibility audit using only automated tools?
No, automated tools only catch part of the WCAG criteria, and they can’t judge things like whether alt text makes sense or whether a screen reader user can complete a task. Auditors test manually and with assistive technology, so you should too.
Testing guide
Best practices
October 13, 2026
Testing guide
Best practices
LLM Testing: A Guide for LLM QA Teams
Large language models are now a part of core enterprise infrastructure, which is why it’s critical to ensure they are safe, accurate, and high-performing at scale.
Large language models are now a part of core enterprise infrastructure, which is why it’s critical to ensure they are safe, accurate, and high-performing at scale.
Traditional software testing methods fall short in this landscape. As a result, QA teams are required to adopt entirely new frameworks, metrics, and red-teaming strategies to build truly production-ready AI applications.
What Is LLM Testing
LLM testing is the process of evaluating a large language model or LLM-powered application against defined expectations for quality, safety, performance, and security. In practice, most QA teams are not testing a foundation model from scratch. They are testing what sits on top of it, such as prompts, retrieval pipelines, tool calls, guardrails, and business logic that turn a general model into a product. That distinction shapes the work.
A chatbot that answers billing questions needs to pull the right account policy, stay on topic, refuse requests it should not fulfill, and respond fast enough that customers don’t give up on it. In this case, each function a chatbot performs is a testable behavior.
LLM testing usually combines three kinds of checks:
Deterministic assertions for anything with a clear right answer, such as valid JSON, required fields, or a banned phrase never appearing.
Model-graded evaluation, where a second model scores outputs against a rubric. This is often called LLM-as-a-judge.
Human review for judgment calls that automated scoring still gets wrong.
The result of LLM testing is a measured picture of how often the system behaves as intended, and whether it improves or degrades with each change.
The 4 Main Types of LLM Testing
Most LLM test plans fall into four categories, each targeting a different risk. A production-ready application needs coverage in all four.
1. Functional Testing
Functional testing checks whether the application does its job, like answering correctly, following instructions, and producing output in the expected format.
Typical test cases cover:
Task accuracy: Does the summary capture the key points? Does the extraction pull the right fields?
Instruction following: Does the model respect length limits, tone requirements, and output schemas?
Retrieval quality: In RAG systems, did the pipeline fetch the right documents, and did the answer actually use them?
Multi-turn behavior: Does the assistant keep track of context from earlier in the conversation?
Tool and agent behavior: Did the agent call the right tool, with valid arguments, in a sensible order?
Build functional tests from real user scenarios rather than generic prompts. A test set drawn from actual support tickets or product requirements catches far more than one built from templated questions. Every bug a user reports should also become a permanent regression case.
2. Performance and Reliability Testing
Performance testing measures how fast and how consistently the application responds under realistic load. LLM latency behaves differently from typical API latency because responses stream token by token and output length varies with every request.
Key areas to cover:
Latency: Time to first token, total response time, and tokens per second.
Load behavior: How latency and error rates change as concurrent users grow, and where provider rate limits kick in.
Consistency: Whether the same prompt produces stable quality across repeated runs.
Failure handling: Timeouts, provider outages, truncated responses, malformed output, and whether retries and fallbacks work.
Cost:Token usage per request, since a prompt change that doubles output length also doubles spend.
Reliability is where non-determinism hurts most. Run each critical test case several times and track the spread, not just the average. A feature that answers correctly 9 times out of 10 still fails visibly for a share of real users.
3. Safety and Content Testing
Safety testing checks whether the application produces harmful, biased, or off-policy output, including when users are not trying to break it.
Coverage usually includes:
Toxicity and harmful content: Hate speech, harassment, and violent or sexual content.
Bias: Whether answers or tone shift based on names, genders, or other demographic cues in the prompt.
Policy compliance: Staying within brand guidelines and avoiding medical, legal, or financial advice where the product shouldn't give it.
Refusal calibration: Refusing what it should without over-refusing legitimate requests.
Privacy: Not repeating personal data from context or logs in its responses.
Over-refusal deserves its own test cases. A model that declines half of valid customer questions passes every safety check and still fails the product. Track refusal rate and harmful output rate together.
4. LLM Penetration Testing
LLM penetration testing attacks the application the way a malicious user would. Safety testing asks whether the model says something harmful, whereas penetration testing asks whether an attacker can make the system do something it was never meant to do.
Common attack categories to test:
Direct prompt injection: Inputs that try to override system instructions, such as “ignore previous instructions and…”
Indirect prompt injection: Malicious instructions hidden in content the model reads, like a web page, uploaded file, or document in a RAG index.
System prompt leakage: Attempts to extract hidden instructions, internal rules, or credentials placed in prompts.
Sensitive data disclosure: Getting the model to reveal other users’ data or internal documents from its context.
Excessive agency: Tricking an agent into calling tools, sending messages, or changing data beyond what the user is authorized to do.
Insecure output handling: Model output passed unchecked into a browser, database query, or shell.
Resource abuse: Prompts designed to max out tokens, loop agents, or run up costs.
When a model can take actions, a successful injection can send emails, modify records, or expose data. Pair automated attack generation with manual testing, and retest after every prompt or tool change.
LLM Testing Frameworks and Tools Worth Knowing
No single tool covers every type of LLM testing. Most teams pair an evaluation framework for functional and RAG metrics with a dedicated tool for security scanning.
1. DeepEval
DeepEval is a Python LLM testing framework (Apache 2.0) integrated with Pytest. It provides metrics for RAG quality, agent behavior, conversation context, toxicity, custom evaluation, and synthetic dataset generation for CI/CD integration.
Best for: Teams that want LLM tests to live alongside their existing Python test suite.
2. Promptfoo
Promptfoo is a CLI tool and library (MIT) for declarative evaluation and red-teaming. It compares prompts and models across providers and generates adversarial tests for prompt injection, jailbreaks, and data exfiltration.
Best for: Prompt and model comparisons, and automated LLM penetration testing.
3. Giskard
Giskard is an open-source Python library (Apache 2.0) that automatically detects hallucinations, prompt injection, and bias. Its RAGET module generates test cases to evaluate retriever and generator performance separately.
Best for: Surfacing vulnerabilities you don’t typically test for, especially in RAG applications.
4. TruLens
TruLens (MIT) pairs OpenTelemetry tracing with model evaluations to measure latency, token usage, cost, and the RAG triad (context relevance, groundedness, and answer relevance).
Best for: Pinpointing where in a multi-step pipeline quality breaks down, and comparing app versions on quality, latency, and cost.
5. Ragas
Ragas (Apache 2.0) evaluates retrieval-augmented generation systems through faithfulness, answer relevancy, context precision, and recall metrics, alongside synthetic test dataset generation.
Best for: RAG evaluation and teams that need to build a test set from scratch.
Key LLM Testing Metrics
LLM testing differs from usual software testing metrics. Most quality metrics in LLM testing score from 0 to 1, and your team sets the threshold that counts as a pass.
1. Faithfulness
Faithfulness measures whether the claims in an answer are supported by the context the model was given. It is usually calculated by breaking the answer into individual statements and checking each one against the retrieved documents. A low score means the model is adding information its sources don't contain.
2. Answer Relevance
Answer relevance measures how directly a response addresses the question asked. An answer can be accurate and faithful but still miss the point, filling space with related facts instead of answering. This metric catches that.
3. Hallucination Rate
Hallucination rate is the share of outputs that contain fabricated or unsupported information. Faithfulness scores a single response, while hallucination rate is tracked across a full test set, so you can see whether a model or prompt change made the problem better or worse.
4. Toxicity Score
A toxicity score estimates how likely an output is to be offensive, abusive, or harmful, usually using a classifier or judge model. Track the share of responses above your threshold rather than the average score. One toxic reply matters more than a low mean.
5. Groundedness
Groundedness measures how well each statement in a response can be traced back to a provided source. It overlaps heavily with faithfulness, and frameworks define the two terms differently. Check how your tool calculates each before comparing numbers across tools.
6. Latency / TTFT (Time to First Token)
Time to first token is the delay between sending a request and receiving the first token of the response. It drives how fast a streaming interface feels. Track it alongside total response time and tokens per second, and report percentiles such as p95 instead of averages, because slow outliers are what users notice.
7. Pass@k
Pass@k is the probability that at least one of k generated attempts passes a test. It comes from code generation, where a task counts as solved if any of k samples passes its unit tests. For user-facing features, the stricter question is often whether all k attempts pass, since users see one response, not the best of several.
TestFiesta Was Built for the Complexity LLM Testing Brings
LLM testing creates a lot of moving parts, including functional, safety, and security test cases, results from automated eval runs, scores that shift with every prompt or model change, and defects that need a clear trail back to the input that caused them. TestFiesta gives QA teams one place to manage all of it.
AI Copilot for Test Design: Upload requirements, user stories, or screenshots, and the Copilot generates structured test cases from that context. Nothing enters your suite until you review and approve it, which matters when you are writing hundreds of prompt scenarios and adversarial cases.
Flexible Organization: Reusable templates, universal tagging, and shared steps let you group cases by test type, risk, or feature, so a safety suite or red-team suite can run on its own.
Reusable Configurations: Run the same test cases across environments, or treat model and prompt versions as configurations to track results per version.
Automated Results in One Place: The TacoTruck CLI and full API access push results from CI/CD runs into TestFiesta, so eval framework output sits next to manual and exploratory testing.
Built-In Bug Tracking: Log defects in the same platform where you manage test cases, or turn a failed test into a Jira or GitHub issue with full context attached.
Set up your first LLM test suite in TestFiesta and get a QA workflow that fits your team.
Yes, LLM testing can be largely automated. Deterministic checks, model-graded metrics, and adversarial attack generation can all run in a CI/CD pipeline on every prompt, model, or code change. That said, automation can’t fully replace human review of judgment-heavy outputs, or periodic checks that judge whether models are scoring accurately.
What is LLM red teaming?
LLM red teaming is adversarial testing where testers deliberately try to make a model or application misbehave through jailbreaks, prompt injection, data extraction, or harmful content requests. It overlaps with LLM penetration testing but is usually broader, covering safety and misuse as well as security flaws.
How do I test an LLM when I don’t have a ground truth dataset?
You can still test an LLM if you don’t have a ground truth dataset. Start with reference-free metrics such as faithfulness, answer relevance, and groundedness, which score outputs against the question and retrieved context instead of an expected answer. You can also generate a synthetic test set from your own documents and have domain experts review a sample. Over time, real user queries from production, labeled as you go, become your ground truth dataset.
Testing guide
Best practices
October 10, 2026
Testing guide
Metamorphic Testing: What It Is and How It Works
Testing efforts vary from software to software, and in some cases, there is no right answer to what the correct system output is. The perfect examples are climate models predicting weather 50 years from now, a machine learning model ranking millions of search results, or an autonomous vehicle detecting objects in real time.
How do you verify whether the system output is actually correct when nobody knows what the exact expected answer should be?
In software engineering, this challenge is known as the oracle problem; a bottleneck where traditional unit tests and pass/fail assertions hit a wall. When checking single outputs against fixed expected values is impossible or too expensive, metamorphic testing offers a powerful alternative: instead of checking whether a single output is correct on its own, you check how outputs relate across logical transformations of the input.
What Is Metamorphic Testing
Metamorphic testing is a way to test software without knowing the exact correct output for any single input. Instead of checking one result against an expected value, you run the program more than once with related inputs. Then you check that the outputs relate to each other the way they logically should.
Here is a simple illustration:
Say you are testing a payroll function. You may not know the exact total for a 400-person company with overtime, bonuses, and tax rules. But you do know one thing for sure: If every employee’s hourly rate doubles and nothing else changes, gross payroll should double too. Run both versions, compare, and if the second total isn’t twice the first, you can determine that the output is wrong. As a result, you found a problem without ever knowing the “right” answer.
Metamorphic testing has two main strengths:
It eases the oracle problem: You verify relationships between outputs, not the correctness of each output on its own.
It generates new tests: Every existing test case can spawn follow-up cases, so coverage grows from tests you already have.
What Is a Metamorphic Relation
A metamorphic relation (MR) is the rule that ties metamorphic test runs together. It states how a specific change to the input should affect the output. If the outputs break the rule, the program has a fault somewhere.
Every MR has two parts. One is the input transformation, which is how you change the original input to create a new one. The other is the output relation, which is what must hold between the original output and the new output.
Imagine you are testing a function that sorts a list of numbers. For a short list, you can check the result by looking at it. But for a list of 100,000 numbers, you need a function. When testing complex functions like sorting a massive list of numbers where you cannot manually verify every expected output, you can still rely on fundamental rules or properties that must always remain true. Each of these inherent properties serves as a Metamorphic Relation (MR) that can be used to catch software bugs without needing to know the exact final answer in advance.
In this specific case, the MRs would be:
MR 1: Shuffling the input should not change the output. If list B contains exactly the same numbers as list A in a different order, then sort(B) must equal sort(A). If they differ, the function is mishandling order somewhere.
MR 2: Adding a larger number should only extend the output. If you append a value bigger than everything in list A, the sorted result should be the original sorted list with that value at the end. Anything else means the new element disturbed items it never should have touched.
Neither rule tells you what the sorted list should look like. But both can still catch real bugs. Mathematical functions work the same way. You may not know sin(x) to fifteen decimal places, but you know sin(x) and sin(π − x) must be equal.
How Metamorphic Testing Works: A Step-by-Step Guide
Metamorphic testing follows the five steps, which remain the same regardless of testing a sort function or a self-driving model.
To make it simple, we’ll use the e-commerce search example.
Step 1: Identify metamorphic relations: Study what the system is supposed to do and write down properties that must hold across related inputs, like “adding a filter never adds results.” This step takes the most thought and depends on domain knowledge, so pull in the people who understand the system best.
Step 2: Create source test cases. Choose the original inputs the process starts from, such as a set of real search queries. These can be existing test cases, randomly generated inputs, or anonymized production data.
Step 3: Derive follow-up test cases. Apply each MR’s input transformation to every source case, such as by adding a size filter to each query. One source case can feed several MRs, so a small seed set grows quickly.
Step 4: Execute both sets. Run the program on the source inputs and the follow-up inputs, then record every output. Keep each follow-up paired with its source so you always know which results to compare.
Step 5: Verify the relation. Check whether each pair of outputs satisfies the output relation, such as “filtered results are a subset of unfiltered results.” A violation means at least one output is wrong, which signals a bug in the program or a flaw in the MR itself.
A Pro Tip: A passing metamorphic test does not prove the program is correct. A function could break in a way that still satisfies a given relation. That is why teams usually define several MRs per feature, each checking the behavior from a different angle.
Where Metamorphic Testing Matters
Metamorphic testing matters wherever expected outputs are unknown, expensive to compute, or open to interpretation. These five areas are where it shows up most include:
Scientific and mathematical software
Simulations and numerical programs that often have no known exact answer to compare against.
Mathematical properties like symmetry, scaling, and conservation laws.
Published studies have applied it to bioinformatics programs, epidemiological models, and software at NASA.
Machine learning and AI models
Models that produce outputs that no one can fully specify in advance, which is the oracle problem in its purest form.
Machine learning models where output should remain consistent or stable when minor, non-essential changes are made to the input. In this context, if you slightly darken a photo, an image classifier should still recognize the same object, because a minor change in lighting doesn’t change the underlying subject of the image.
Large language models where rephrasing a factual question should not change the factual answer.
Search engines and ranking systems
Systems where there is no objective, agreed-upon “correct” result set for most queries.
Subset and consistency relations, like the filtering examples.
Compilers and code translation tools
Tools where verifying the compiled output for every possible program is out of reach.
Translation tools where two programs that behave the same on a given input should still behave the same after compilation.
Web APIs
API responses that often depend on live, constantly changing data, so exact expected values are hard to pin down.
How TestFiesta Helps With Test Case Management for Metamorphic Testing
Metamorphic testing multiplies your test cases fast. Five MRs applied to 40 source cases give you 200 follow-up cases, and every one of them needs to stay linked to its source, its relation, and its result. Without structure, that turns into a cluttered dashboard or spreadsheet.
TestFiesta keeps everything around them organized and traceable.
Customizable Tags:TestFiesta offers unlimited customizable tags for grouping, filtering, and sorting test cases. Tag each case with its MR and its role as a source or follow-up case. When an MR starts failing, you can filter straight to every related case.
Custom Fields: Custom fields capture data at the test case, run, or result level. Use them to record the input transformation, the expected output relation, and the ID of the source case each follow-up came from.
Reusable Templates: Build one MR template with fields for the transformation and the relation. Every new MR then gets documented the same way, no matter who on the team writes it.
Shared Steps: Source and follow-up cases usually share setup, like loading a dataset or authenticating against an API. Write those steps once and reuse them everywhere, so a change in setup updates every case at once.
Configurations: Configurations let you reuse one test case across many setups. Run the same MR against different environments, browsers, or model versions without cloning cases.
Jira and GitHub Integrations: When a relation is violated, turn the failed test into a Jira or GitHub issue with full context, or log bugs natively in TestFiesta. Developers see both inputs and both outputs, which is exactly what they need to debug a metamorphic failure.
An API for Automated Results: Metamorphic tests are usually automated, and TestFiesta’s Automation API supports custom automations, so results from your test runs can flow into the platform instead of being copied by hand.
Don’t let the complexity of managing metamorphic relations and automated test suites slow down your release cycles.
Yes, metamorphic testing can be automated, and it is well-suited to automation. Once your MRs are defined, generating follow-up inputs, running both sets, and checking the relations can all be scripted.
What tools do I need for metamorphic testing?
You don’t need a dedicated tool for metamorphic testing. Most teams write MRs as ordinary tests in the framework they already use, such as pytest or JUnit, and property-based testing libraries like Hypothesis can help generate source inputs. A test case management platform keeps your MRs, source cases, follow-ups, and results connected as the suite grows.
Do I need to know the correct output to write metamorphic tests?
No, you don’t need to know the correct output to write metamorphic tests. That’s the entire point of metamorphic testing. You only need to know how outputs should relate to each other when the input changes in a specific way. Writing good relations still takes domain knowledge, but you never need the exact expected value for any single input.
Testing guide
Ready for a Platform that Works
The Way You Do?
Stop fighting your tools. Start shipping with confidence. TestFiesta adapts to your workflow, not the other way around.