Efficient and Effective Testing of Automotive Software Product Lines

Abstract

Within the automotive industry, the clients’ high demand for individually customized products results in a growing number of product variants. In order to control the complexity of developing these variants, a product line approach is used that supports reuse of the common set of assets (e.g. requirements, software code, and test cases). The naive approach to assure quality of the variants derived from the product line would be to individually test each variant. However, due to the large number of variants it is virtually impossible to test all variants in detail and still release the product line on time. In this paper, we propose a new test approach based on requirements coverage and variant properties coverage (i.e. feature coverage) that leads to effective but also efficient test coverage of all variants of the product line. A set of variants is selected that is 1) as small as possible, 2) covers all requirements of the product line, and 3) covers all features of the product line. This small set of variants is then tested in detail. Reducing the number of tested variants allows for deeper testing and thus finding more defects. Because of the coverage achieved by the variant set, the quality of all other variants can be inferred from the test results. Finding the optimal set of variants in itself is a very hard problem, i.e. in an industry setting with huge numbers of possible variants it is practicably infeasible to strive for the optimum. Therefore, we developed two approaches which are not guaranteed to find the optimal set but get very near to it in little time. First, a greedy algorithm was created which produced very good results in very little time in all case studies. For example, for a system with one million variants a set of eight representing variants was selected within seconds. Second, a simulated annealing approach was evaluated in order to check for further potential of improvement. However, the case studies showed that the greedy algorithm is the better choice for practical results.