AI Under the Hood: How to Build Reliable Automated Tests for Non-Deterministic Systems
- SQADays / 38
-
40 min
In this talk, I will show how to build reliable automated tests for non-deterministic AI features, what tools and approaches to use, and what limitations and challenges you will face in practice.
We will discuss:
• how to automate non-deterministic flows and which techniques help stabilise such scenarios;
• key principles for designing automated tests in systems with variable behaviour;
• how to validate results when fixed expectations no longer work (where assertions are sufficient and where eval-based approaches are needed);
• nuances of selecting test data for automated tests;
• cost aspects of AI testing: tokens, infrastructure, parallelisation, and maintenance.
This talk will be useful for QA automation engineers and SDETs working with non-deterministic systems and aiming to maintain control over quality in conditions of variability.