Category: Quality

SpecFlow Manual Testing

Sometimes there are scenarios that can only be tested manually. Maybe you are testing colors or placement of a picture or some other important feature that only human eyes can assert as right or wrong. When I have a manual test definition in my automated test framework I want to tell the framework to ignore it, but still report it so we don’t lose sight of it. I am building features into my test framework to handle ignoring manual testing, but I found this code below that does it easily in SpecFlow.

[Binding, Scope(Tag = “Manual”)]
public class ManualSteps
{
    [Given(“.*”), When(“.*”), Then(“.*”)]
    public void EmptyStep()
    {
    }

    [Given(“.*”), When(“.*”), Then(“.*”)]
    public void EmptyStep(string multiLineStringParam)
    {
    }

   [Given(“.*”), When(“.*”), Then(“.*”)]
    public void EmptyStep(Table tableParam)
    {
    }
}

From https://github.com/techtalk/SpecFlow/wiki/Scoped-Bindings

With this any scenario tagged with @Manual will be ignored, but they will still be reported in the test report. Sweet.

SpecFlow Ambiguous Step Definitions

It’s been a long time since I posted anything. I have a ton of material to post, just been too busy or lazy to post it.

Anyway, here is the problem. I use SpecFlow, Selenium WebDriver, and the Page Object Model pattern to implement UI tests. I want to scope my SpecFlow Step Definitions and I ran into this link that made me think twice about doing it. https://github.com/cucumber/cucumber/wiki/Feature-Coupled-Step-Definitions-%28Antipattern%29

The basic premise is that you shouldn’t tie your step definitions to features.

Feature-coupled step definitions are step definitions that can’t be used across features or scenarios. This is evil because it may lead to an explosion of step definitions, code duplication and high maintenance costs.

I can agree with this, but there should be a way to tie a step to a context. The best example of what I mean is when a step is only relevant to a page or control when doing UI tests. When you have a generic step definition, but the implementation can be specific to a page or control it makes sense to be able to scope the step to the page or control.  For example, if we take the scenario from the wiki page above

Scenario: add description
  Given I have a CV and I’m on the edit description page
  And I fill in “Description” with “Cucumber BDD tool”
  When I press Save
  Then I should see “Cucumber BDD tool” under “Descriptions”
(Note: Save is a regular expression on the wiki, but I rely on page object models so it isn’t necessary to pass the value to the step method as my actions are explicit.)

The “When I press Save” step is going to call a method in a page object to actually press the Save button. The step will use a specific page object to execute the step and this generic definition does not provide any context to say which page object to use. If I could scope the step definitions and implementations to a particular page or control, I could have various implementations to target various page objects to drive the page or control under test.

With this we are not coupling by feature, but by page or control. Is this bad or another anti-pattern? Time will tell, but I have had the hardest time trying to name steps with context identifiers to try to get around the problem of step definitions having a global scope in SpecFlow. If I had another scenario that used the “When I press Save” definition, but is implemented with a different page object we run into ambiguity issues because SpecFlow doesn’t know which implementation to use. Without a scoping mechanism I have to add context to the step definitions. Our simple definition would become, “When I press Save on the CV Edit Description page”. This usually makes defining steps and reading them a lot harder than it should be because I have to use more words.

As a general practice, I normally scope my features and scenarios with a tag indicating the page or control under test and this could easily be used in the step definitions to couple step implementations to specific pages and controls. With SpecFlow we can use a feature called Scoped bindings to achieve page/control scoped step definitions.

The Scope Attribute can be used to restrict the execution of step definitions by features, scenario, or by tag. Since scoping by feature is an anti-pattern we won’t use that one. The scenario is a viable restriction, but I believe tag will provide the most flexibility as we can restrict multiple scenarios across various features without limiting the step to a particular feature or scenario. Although, there is the problem of tags only being applied at the scenario level. We can not tag a scenario step in SpecFlow, i.e. tag the Given, When and Then separately. I am not sure if this would be necessary. I have to get more specs written with scoped bindings to see what troubles I run into.

You can look at the Scoped bindings link for usage, but in our scenario above we could use this technique by tagging the scenario with the page name:

@CVEditDescription
Scenario: add description
  Given I have a CV and I’m on the edit description page
  And I fill in “Description” with “Cucumber BDD tool”
  When I press Save
  Then I should see “Cucumber BDD tool” under “Descriptions”

Then the “When I press Save” step can be scoped to the CV Edit Description page like so:

[When (@”I press Save”, Scope(Tag = “CVEditDescription”)]
public void WhenIPressSave()
{
     //Call Save on the page object
     cvEditDescriptionPage.Save();
}

We also get the added benefit of being able to run tests just for this page (tag). So far I like it. How do you solve ambiguity issues in SpecFlow?

Automated Testing in Visual Studio

NUnit in Visual Studio 2012/2013

I normally us MSTest as I like the integration with Visual Studio. At work we do a lot of NUnit and it just feels dirty having to run my tests outside of Visual Studio (we don’t have a runner for VS). Well we finally got a VS upgrade and I was able to install the NUnit Test Adapter for Visual Studio 2012 and 2013, http://nunit.org/index.php?p=vsTestAdapter&r=2.6.2.

I just installed it through the Extension Manager with Nuget, easy peasy. Now I can run NUnit tests without having to run the NUnit GUI or command line manually. Tests will appear in the VS Test Explorer so organizing, running and debugging tests is incredibly simplified.

Dev Life

This was a blast from the past from a draft post from January 2013 when unit tests in Visual Studio was becoming easier to deal with. Below we review the current state of automated testing in Visual Studio.

Modern Automated Testing in Visual Studio: A Seamless Experience

As developers, we thrive on tools that simplify our workflows and integrate seamlessly into our development environments. Back in the day, running NUnit tests without a dedicated Visual Studio runner felt clunky and disjointed. Fast forward to today, Visual Studio has matured into an incredibly robust IDE for automated testing, offering rich support for various frameworks like MSTest, NUnit, and xUnit, all within the IDE itself. Let’s explore how modern Visual Studio takes automated testing to the next level.

Integrated Test Adapters

One of the major advancements in Visual Studio since the olden days is the seamless integration of test adapters via the NuGet package manager. For example, installing the NUnit Test Adapter or xUnit.net Test Adapter is as simple as adding the package to your test project. Once installed, tests are automatically discovered by the Visual Studio Test Explorer. No additional configuration is required, and the days of running separate GUIs or command-line tools are long behind us.

Test Explorer: The Nerve Center

The Visual Studio Test Explorer has become the central hub for managing and executing tests. It’s packed with features:

  • Test Discovery: Automatically detects and organizes tests from all supported frameworks in your solution.
  • Grouping and Filtering: Group tests by project, namespace, or custom traits for easy navigation. Filters allow you to focus on failing tests or specific categories.
  • Real-Time Feedback: See pass/fail results instantly, with options to debug failing tests directly from the Test Explorer.
  • Continuous Testing: Enable live unit testing (available in Visual Studio Enterprise) to automatically run tests impacted by your code changes in real-time.

Debugging Tests

Debugging unit tests is now as straightforward as debugging application code. Simply set breakpoints in your test or application code and run the test in debug mode from the Test Explorer. Visual Studio’s rich debugging tools, including watch variables, call stacks, and IntelliTrace (in Enterprise), make diagnosing issues a breeze.

Code Coverage and Test Impact Analysis

Understanding how well your tests cover your codebase is critical. Visual Studio provides built-in tools for:

  • Code Coverage Analysis: Highlighting which parts of your code are tested and which are not.
  • Test Impact Analysis: Identifying which tests are affected by your recent code changes, optimizing the tests you need to run.

Cross-Platform and CI/CD Integration

With .NET Core and .NET 6+, Visual Studio supports cross-platform testing, allowing you to run and debug tests on Windows, macOS, and Linux. Additionally, automated tests integrate seamlessly with CI/CD pipelines using Azure DevOps, GitHub Actions, or other CI platforms. Test results can be published as part of your build and deployment workflows, ensuring quality gates are met.

Choosing the Right Framework

While MSTest remains the native framework for Visual Studio, NUnit and xUnit are popular for their flexibility and extensive ecosystem. All three frameworks are first-class citizens in Visual Studio, making it easy to choose one based on your project’s needs or team preferences.

Getting Started

Here’s a quick guide to setting up automated testing in Visual Studio:

  1. Install the Framework: Use NuGet Package Manager to add MSTest, NUnit, or xUnit to your test project.
  2. Install the Adapter: Add the corresponding Test Adapter package for NUnit or xUnit.
  3. Write Your Tests: Create test classes and methods following the chosen framework’s conventions.
  4. Run Your Tests: Open Test Explorer and run or debug your tests directly from within Visual Studio.
  5. Analyze Results: Use Test Explorer’s grouping, filtering, and debugging tools to refine your tests.

Dev Life Today

Automated testing in Visual Studio has come a long way since the early days. Whether you’re building enterprise-grade software or experimenting with side projects, Visual Studio’s modern testing tools streamline the process, reduce context switching, and let you focus on delivering quality code.

Gone are the days of manually running tests in external tools. With Visual Studio, everything you need for automated testing is right at your fingertips, making it an indispensable tool for modern developers. We can also mimic a lot of this functionality in VS Code so .NET developers have options. What are your favorite testing features in Visual Studio or VS Code?