Introduction to Combinatorial Testing

This book gives a self-contained tutorial on using combinatorial testing for real-world software. It introduces the key concepts and methods of combinatorial testing, explains the use of software tools for generating combinatorial tests, and explores advanced topics. In each chapter, a section on costs and practical considerations examines tradeoffs and limitations that may impact resources or funding. Accessible to undergraduate students, the text includes an extensive set of references for further reading.

Combinatorial testing of software analyzes interactions among variables using a very small number of tests. This advanced approach has demonstrated success in providing strong, low-cost testing in real-world situations. Introduction to Combinatorial Testing presents a complete self-contained tutorial on advanced combinatorial testing methods for real-world software. The book introduces key concepts and procedures of combinatorial testing, explains how to use software tools for generating combinatorial tests, and shows how this approach can be integrated with existing practice. Detailed explanations and examples clarify how and why to use various techniques. Sections on cost and practical considerations describe tradeoffs and limitations that may impact resources or funding. While the authors introduce some of the theory and mathematics of combinatorial methods, readers can use the methods without in-depth knowledge of the underlying mathematics. Accessible to undergraduate students and researchers in computer science and engineering, this book illustrates the practical application of combinatorial methods in software testing. Giving pointers to freely available tools and offering resources on a supplementary website, the book encourages readers to apply these methods in their own testing projects.

Combinatorial Methods in TestingSoftware Failures and the Interaction Rule Two Forms of Combinatorial Testing Covering Arrays The Test Oracle Problem Quick Start: How to Use the Basics of Combinatorial Methods Right Away Combinatorial Testing Applied Document Object Model, Carmelo Montanez-Rivera, D. Richard Kuhn, Mary Brady, Rick Rivello, Jenise Reyes Rodriguez, and Michael PowersRich Web Applications, Chad M. Maughan Configuration Testing Runtime Environment ConfigurationsHighly Configurable Systems and Software Product Lines Invalid Combinations and Constraints Cost and Practical Considerations Input Testing Partitioning the Input Space Input Variables versus Test Parameters Fault Type and Detectability Building Tests to Match an Operational Profile Scaling Considerations Cost and Practical Considerations Test Parameter Analysis, Eduardo MirandaWhat Should Be Included as a Test Parameter Combination Anomalies Classification Tree Method Modeling Methodology Selecting the System under Test Combinatorial Testing and Boundary Value Analysis Managing System State, George SherwoodTest Fact or Partitions with StateTest Fact or SimplificationsSequence Unit Replay Model Single Region State Models Multiple Region State Models Measuring Combinatorial Coverage Software Test CoverageCombinatorial CoverageUsing Combinatorial Coverage Cost and Practical Considerations Analysis of (t + 1)-Way Coverage Test Suite Prioritization by Combinatorial Coverage, Renee Bryce and Sreedevi SampathCombinatorial Coverage for Test Suite Prioritization Algorithm Prioritization Criteria Review of Empirical StudiesTool: Combinatorial-Based Prioritization for User-Session-Based TestingOther Approaches to Test Suite Prioritization Using Combinatorial Interactions Cost and Practical Considerations Combinatorial Testing and Random Test Generation Coverage of Random Tests Adaptive Random Testing Tradeoffs: Covering Arrays and Random Generation Cost and Practical Considerations Sequence-Covering Arrays Sequence-Covering Array Definition Size and Construct ion of Sequence-Covering ArraysUsing Sequence-Covering Arrays Cost and Practical Considerations Assertion-Based TestingBasic Assertions for Testing Stronger Assertion-Based Testing Cost and Practical Considerations Model-Based Testing Overview Access Control System Example SMV Model Integrating Combinatorial Tests into the Model Generating Tests from Counterexamples Cost and Practical Considerations Fault Localization Fault Localization ProcessLocating Faults: Example Cost and Practical Considerations Evolution from Design of Experiments Background Pairwise (Two-Way) Testing of Software Systems Combinatorial t-Way Testing of Software Systems Algorithms for Covering Array Construction, Linbin Yu and Yu LeiOverview Algorithm AETG Algorithm IPOG Cost and Practical Considerations Appendix A: Mathematics ReviewAppendix B: Empirical Data on Software FailuresAppendix C: Resources for Combinatorial TestingAppendix D: Test Tools References Index A Summary and Review appear at the end of each chapter.

D. Richard Kuhn, Raghu N. Kacker, Yu Lei