We have to mention that mutation testing relies on two things:
- Competent Programmer: If a programmer is competent enough to write and test a small piece of code, s/he will be good at writing larger programs too.
- Coupling effect: Lots of developers often say, “I don’t write test like this, because it gets coupled to that.”, If you do not couple your tests to your production code, you are not testing your production code!
My advice here is: Be very intentional with what you are choosing to be coupled. Unit tests should couple to the method implementation, UI tests should couple to the navigation hierarch
Mutation Testing Tools
Lots of tools exists for nearly all modern programming languages. In the following, I will be citing the most known in the mobile app development field:
Dos and Don’ts of Mutation Testing
- Start step by step: mutation testing can take a long time, so if you are planning to apply it to an existing project, it is wise to incrementally mutation test your project.
Hint: some tools allow you to explicitly specify files to run mutation.
- Review test metrics: It is crucial to regularly review your test metrics. Use code coverage to see how much of your code you are testing, as well as to figure out if you need to write more unit tests, and use mutation score to see from where to start mutation test.
- Address all surviving mutants at once: This can lead to a lot of refactoring to your test suite as well as to your production code. Instead pick the logic that is the most important.
- Mutation test when code coverage is low: Mutation tests tell us how effective our tests interact with our app code, so with a low code coverage we will get a false sense of confidence from it.
Advantages & Disadvantages of Mutation Testing
- Powerful approach to attain high coverage of the source code
- Brings a good level of error detection into the program
- Very good at detecting hidden defects, which might be impossible to identify using the conventional testing techniques
- Discovers ambiguities in the source code
- Increase customer satisfaction by providing the most reliable and stable product
- Extremely costly and time-consuming, but it’s fair to say that this testing cannot be done without an automation tool.
- Each mutation will have the same number of test cases as the original program.
- Complex mutations are difficult to implement
- Testers need to have programming skills to do mutation testing
- Not applicable for Black Box Testing at all, as it involves a lot of source code changes
As human being you can’t always avoid making a mistake, but when it comes to software development, there are ways to catch that mistake before it goes to production. One way is mutation testing. Even though it is time-consuming and requires automation, mutation testing is still the undisputed best technique to test any program and highlights code that needs additional tests or better assertions.
I hope that I managed to give you a brief introduction on what mutation testing is about and how it works, and it’s over to you now to look into how to leverage it and put it into practice. Otherwise, if you are interested to dig deeper into the topic, we'll put mutation testing to good use and tackle some everyday scenarios in a future article.