Testing framework: Only test specific parameter combinations

In an advanced parametrized test class, is it possible to omit certain combinations of parameters from the resulting test suite? For example, lets say I have:

properties (TestParameter)
   parameter1 = {'a','b'};
   parameter2 = {'x','y'}
end

I want to run a certain test for the parameter combinations ax, ay, and bx (omit by).

Answer 1

[Sean de Wolski]

You can use the TestSuite.selectIf with HasParameter and boolean logic with multiple HasParameters to remove them.

https://www.mathworks.com/help/releases/R2017a/matlab/ref/matlab.unittest.selectors.hasparameter-class.html

[Opt User]

It works, although I would have preferred something that can be used inside the class, so certain parameters combinations are not left undefined.

[Sean de Wolski]

It's not ideal but you could create a sequential parameter list and leave out the combinations you don't want. It would require duplication on your part.

From the doc https://www.mathworks.com/help/matlab/matlab_prog/create-advanced-parameterized-test.html#zmw57dd0e71143:

    methods (Test, ParameterCombination='sequential')
        function testSize(testCase,dim1,dim2,dim3)
            testCase.verifySize(rand(dim1,dim2,dim3),[dim1 dim2 dim3])
        end 
    end

[Steven Lord]

If you have many combinations that you want to accept and only a small number (one or two) that you want to filter, you could use one of the assume* methods on the testCase object. If the assumption fails, "The test framework then marks the test content as filtered and continues testing. Often, assumptions are used to ensure that the test is run only when certain preconditions are met. However, running the test without satisfying the preconditions does not produce a test failure."

testCase.assumeFalse(isequal(parameter1, 'b') && ...
                     isequal(parameter2, 'y'))

or

assumeFalse(testCase, isequal(parameter1, 'b') && ...
                      isequal(parameter2, 'y'))

or

% since they're both characters, we can concatenate the parameters
testCase.assumeNotEqual([parameter1, parameter2], 'by')

[Sean de Wolski]

Or you could verifyError or verifyWarning when those parameter sets are hit to ensure that your system under test does the right thing when provided with bad inputs.