VSTS/TFS Visual STudio Test Task – Filter Criteria – DOTNET CATCH

Do you know all the options available to use in the “Test Filter criteria” on the Visual Studio Test task? I didn’t but it sounded powerful so I did the research and am sharing it here.

A Bing search will have plenty of StackOverflow and blog posts about using Test Categories to filter by:

        [TestMethod]
        [TestCategory(&quot;Category1&quot;)]
        public void TestMethod1_Category1()
        {
        }

Surely there are more options!

vstest.console.exe

Many developers don’t realize all the new VSTS/TFS build and release tasks from Microsoft are open source on Github. So I dug into the VsTest task source to get more information. Here I found the task is calling the Invoke-VSTest PowerShell cmdlet which calls vstest.console.exe. So I cranked up my Developer Command Prompt for VS2015 and found the some helpful documention:

/TestCaseFilter:<Expression>
      Run tests that match the given expression.
      <Expression> is of the format <property>Operator<value>[|&amp;<Expression>]
         where Operator is one of =, != or ~  (Operator ~ has &#039;contains&#039;
         semantics and is applicable for string properties like DisplayName).
         Parenthesis () can be used to group sub-expressions.
      Examples: /TestCaseFilter:&quot;Priority=1&quot;
                /TestCaseFilter:&quot;(FullyQualifiedName~Nightly
                                  |Name=MyTestMethod)&quot;

So that was helpful, we can use multiple operators, sub-expressions and other test properties like Priority and FullyQualifiedName. Here are a few examples (No filter, TestCategory Equals, Name Contains, TestCategory Does Not Equal)

>vstest.console.exe UnitTestProject1.dll
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   TestMethod1_Category1
Passed   TestMethod2_Category1
Passed   TestMethod1_Category2
Passed   TestMethod1

Total tests: 4. Passed: 4. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.3628 Minutes

>vstest.console.exe UnitTestProject1.dll /TestCaseFilter:&quot;
TestCategory=Category1&quot;
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   TestMethod1_Category1
Passed   TestMethod2_Category1

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.3782 Minutes

>vstest.console.exe UnitTestProject1.dll /TestCaseFilter:&quot;
Name~Category1&quot;
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   TestMethod1_Category1
Passed   TestMethod2_Category1

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.4059 Minutes

>vstest.console.exe UnitTestProject1.dll /TestCaseFilter:&quot;
TestCategory!=Category1&quot;
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   TestMethod1_Category2
Passed   TestMethod1

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.4043 Minutes

The FullyQualifiedName property appears to be a good option to filter your tests by area. Most developers organize their tests into folders by area which is translated to the namespace and fully qualified name of the test. Here I filter by FullyQualifiedName and a contains search:

>vstest.console.exe UnitTestProject1.dll /TestCaseFilter:&quot;
FullyQualifiedName~Area2&quot;
Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Passed   TestMethod1

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.4657 Minutes

I did eventually find a blog post from the VS2012 RC release with these details and one more property to filter by (Name). This post looks really old but has some good info so I made a copy just in case.

Happy Testing!

Here is the sample unit test code I was testing with:

namespace UnitTestProject1.Area1
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        [TestCategory(&quot;Category1&quot;)]
        public void TestMethod1_Category1()
        {
        }
        [TestMethod]
        [TestCategory(&quot;Category1&quot;)]
        public void TestMethod2_Category1()
        {
        }
        [TestMethod]
        [TestCategory(&quot;Category2&quot;)]

        public void TestMethod1_Category2()
        {
        }
    }
}
namespace UnitTestProject1.Area2
    {
        [TestClass]
        public class UnitTest1
        {
            [TestMethod]
        public void TestMethod1()
        {
        }
    }
}

VSTS/TFS Visual STudio Test Task – Filter Criteria

vstest.console.exe

1 thought on “VSTS/TFS Visual STudio Test Task – Filter Criteria”

Leave a Reply Cancel reply