Pattern-Based Quality Assessment of TTCN-3 Test Suites

The Testing and Test Control Notation (TTCN-3) is a language for test specification and test implementation which has been standardised by the European Telecommunications Standards Institute (ETSI) and the International Telecommunication Union (ITU). Experience has demonstrated that large industrial size TTCN-3 test suites suffer from the same quality problems like any ordinary source code of larger implementations does. For example, code duplication, unused variables, or high complexity lead to low quality in terms of readability, re-usability, and maintainability. While such internal quality issues have been investigated for source code of implementation languages, assessment of the internal quality of test suites has not been well studied, yet. One recent approach for the quality assessment of TTCN-3 test suites is the application of test metrics. However, such size and structural metrics are sometimes not powerful enough to detect certain issues. While metrics are able to locate e.g. unused definitions or complex code, they cannot detect advanced internal quality problems, e.g. code duplication or anomalies in control or data flow. In addition to our previous results based on test metrics, we present therefore our latest results on applying a more powerful pattern-based approach of quality assessment and issue detection for TTCN-3 test suites. To this aim, we have developed a catalogue of more than 30 so called TTCN-3 "smells" which are patterns of inappropriate usage of TTCN-3. Examples from this catalogue are: duplicate branches in alternative behaviour, usage of hard coded values, unreachable code, violation of naming conventions, not setting a test verdict in a test case, or asymmetric activation/deactivation of TTCN-3 defaults. By associating TTCN-3 refactorings to these "smells", we are not only able to detect issues, but even provide support for removing issues. For this, we have developed an accom-panying catalogue of TTCN-3 refactorings which includes instructions how to restructure test suites without changing their external behaviour. As a result, we are not only able to assess the internal quality of TTCN-3 test suites, but also to improve their internal quality. The effort for applying our approach is significantly reduced by providing corresponding tool support: TRex is an open-source TTCN-3 Refactoring and Metrics tool which is freely available as an Eclipse plug-in on its website. We present latest results from applying our TTCN-3 smell catalogue and the TRex tool to huge real-word TTCN-3 test suites which have been standardised by ETSI for testing implementations of the SIP and IPv6 protocol. Our approach of using pattern-based detection of issues in TTCN-3 test suites immediately helps users of TTCN-3 to assess the quality of their test suites and to locate and to avoid issues in TTCN-3 test suites. The overall approach which consists not only of pattern-based issue detection, but also of TTCN-3 metrics and TTCN-3 refactoring, has been put into practice by our TRex TTCN-3 Refactoring and Metrics Tool. TRex and the underlying concepts have been developed together with Motorola Labs UK, where this approach is used to assess and improve the quality of industrial size TTCN-3 test suites.