SpecFlow Manual Testing

Sometimes there are scenarios that can only be tested manually. Maybe you are testing colors or placement of a picture or some other important feature that only human eyes can assert as right or wrong. When I have a manual test definition in my automated test framework I want to tell the framework to ignore it, but still report it so we don’t lose sight of it. I am building features into my test framework to handle ignoring manual testing, but I found this code below that does it easily in SpecFlow.

[Binding, Scope(Tag = “Manual”)]
public class ManualSteps
{
    [Given(“.*”), When(“.*”), Then(“.*”)]
    public void EmptyStep()
    {
    }

    [Given(“.*”), When(“.*”), Then(“.*”)]
    public void EmptyStep(string multiLineStringParam)
    {
    }

   [Given(“.*”), When(“.*”), Then(“.*”)]
    public void EmptyStep(Table tableParam)
    {
    }
}

From https://github.com/techtalk/SpecFlow/wiki/Scoped-Bindings

With this any scenario tagged with @Manual will be ignored, but they will still be reported in the test report. Sweet.

SpecFlow Ambiguous Step Definitions

It’s been a long time since I posted anything. I have a ton of material to post, just been too busy or lazy to post it.

Anyway, here is the problem. I use SpecFlowSelenium WebDriver, and the Page Object Model pattern to implement UI tests. I want to scope my SpecFlow Step Definitions and I ran into this link that made me think twice about doing it. https://github.com/cucumber/cucumber/wiki/Feature-Coupled-Step-Definitions-%28Antipattern%29

The basic premise is that you shouldn’t tie your step definitions to features.

Feature-coupled step definitions are step definitions that can’t be used across features or scenarios. This is evil because it may lead to an explosion of step definitions, code duplication and high maintenance costs.

I can agree with this, but there should be a way to tie a step to a context. The best example of what I mean is when a step is only relevant to a page or control when doing UI tests. When you have a generic step definition, but the implementation can be specific to a page or control it makes sense to be able to scope the step to the page or control.  For example, if we take the scenario from the wiki page above

Scenario: add description
  Given I have a CV and I’m on the edit description page
  And I fill in “Description” with “Cucumber BDD tool”
  When I press Save
  Then I should see “Cucumber BDD tool” under “Descriptions”
(Note: Save is a regular expression on the wiki, but I rely on page object models so it isn’t necessary to pass the value to the step method as my actions are explicit.)

The “When I press Save” step is going to call a method in a page object to actually press the Save button. The step will use a specific page object to execute the step and this generic definition does not provide any context to say which page object to use. If I could scope the step definitions and implementations to a particular page or control, I could have various implementations to target various page objects to drive the page or control under test.

With this we are not coupling by feature, but by page or control. Is this bad or another anti-pattern? Time will tell, but I have had the hardest time trying to name steps with context identifiers to try to get around the problem of step definitions having a global scope in SpecFlow. If I had another scenario that used the “When I press Save” definition, but is implemented with a different page object we run into ambiguity issues because SpecFlow doesn’t know which implementation to use. Without a scoping mechanism I have to add context to the step definitions. Our simple definition would become, “When I press Save on the CV Edit Description page”. This usually makes defining steps and reading them a lot harder than it should be because I have to use more words.

As a general practice, I normally scope my features and scenarios with a tag indicating the page or control under test and this could easily be used in the step definitions to couple step implementations to specific pages and controls. With SpecFlow we can use a feature called Scoped bindings to achieve page/control scoped step definitions.

The Scope Attribute can be used to restrict the execution of step definitions by features, scenario, or by tag. Since scoping by feature is an anti-pattern we won’t use that one. The scenario is a viable restriction, but I believe tag will provide the most flexibility as we can restrict multiple scenarios across various features without limiting the step to a particular feature or scenario. Although, there is the problem of tags only being applied at the scenario level. We can not tag a scenario step in SpecFlow, i.e. tag the Given, When and Then separately. I am not sure if this would be necessary. I have to get more specs written with scoped bindings to see what troubles I run into.

You can look at the Scoped bindings link for usage, but in our scenario above we could use this technique by tagging the scenario with the page name:

@CVEditDescription
Scenario: add description
  Given I have a CV and I’m on the edit description page
  And I fill in “Description” with “Cucumber BDD tool”
  When I press Save
  Then I should see “Cucumber BDD tool” under “Descriptions”

Then the “When I press Save” step can be scoped to the CV Edit Description page like so:

[When (@”I press Save”, Scope(Tag = “CVEditDescription”)]
public void WhenIPressSave()
{
     //Call Save on the page object
     cvEditDescriptionPage.Save();
}

We also get the added benefit of being able to run tests just for this page (tag). So far I like it. How do you solve ambiguity issues in SpecFlow?

Formalized Versioning

Phil Haack did a post about SemVer 2.0, a formal specification for public API versioning. I think it’s a novel idea as it puts a name and formal structure to what a lot of dev shops I know have been doing. It also allows packaging tools to standardize, like the work Phil talks about with NuGet.

I have added “integrating SemVer and NuGet into CI” to my very long list of things to do for my main side project. That list just keeps on growing.

I Can’t Reproduce That

Sorry, I can’t reproduce that.

I hate saying those words. It seems like a cop out, like I am giving up on the problem. How do you tell someone that you can’t reproduce their issue? Do you ask for more details? Do you just leave it up to the user because you think it is an isolated user issue?

I always ask for more info. The problem is sometimes I don’t know what to ask for. On a web app, if I don’t get an error log I have to ask for browser info from the user and this is an exercise in and of itself. I need to be aware of browser plugins, state of cache (last time cleared), the user and steps used to trigger the issue… and more. After I get the basics necessary for most browser based issues, I start asking more questions specifically related to the scenario that is bombing. Here in lies the problem. Asking users for info usually results in surface data that don’t help get down to the nuts and bolts of the problem. Also, I don’t have a test lab readily available to me so spinning up a browser configured like the user is not possible.

It is hard to get the info you need without stepping through a debug session to see what is going on.  I sometimes have to result to adding additional instrumentation to code to hopefully get better error messages or state info, but this requires a push to production and the user having to wait until all of this is done. So, how can I analyze an issue without this information?

I know there are solutions out there that will record live user sessions in production and even record the stack trace down to the method that caused the issue. Unfortunately, I don’t have access to any of these tools. I guess I will have to just get better at asking questions that even tech illiterate users can follow.

IIS HTTP Error 503. The service is unavailable.

Today I received an error while trying to browse an ASP.NET website application locally, “HTTP Error 503. The service is unavailable.”

Fortunately, I found an excellent post on troubleshooting IIS issues, Troubleshooting IIS7 503 “Service unavailable” errors with startup debugging by Mike Volodarsky. I am just documenting some of the things that were a little different for me. I am running Windows 7 and IIS 7. I downloaded Windows Software Development Kit (SDK) for Windows 8 from http://msdn.microsoft.com/en-US/windows/hardware/hh852363 and only installed the debugger to the default location. This made the path to the debugger executable  C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\ntsd.exe. I used regedit to add the w3wp.exe Registry Key (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe) with the Debugger string value of C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\ntsd.exe -server npipe:pipe=w3wp -g. I also added C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\windbg.exe to my Path to make it easy to start in command line.

Unfortunately, I didn’t need to go through all of that. I never got the debugger to attach because the Worker Process wouldn’t start. After some experimenting I decided to change the Application Pool for my web application and the app came up. I looked at the old Application Pool and noticed something that didn’t hit me before. I spent 2 hours researching and poking around and my real error in this instance was the Application Pool I created for my web application was assigned to my local Windows account. When I was forced to update my password the password did not update on the Application Pool. So, updating the password fixed my issue. I have since created a new Windows account specifically for this web app that doesn’t have a password that expires.

Anyway, it was a nice exercise getting windbg up and running. Maybe I will really need it one day.

Fun Stuff!

*Safety Note: Use the Windows Debugger Install instructions above at your own risk. It may affect your Visual Studio Debugger.

Specifications?

This is more of a “what I’m thinking” post than something full of good information. I am trying to frame my idea of feature specifications and I need to answer some questions. This seemed like as good a place as any to store my questions, but you are more than welcome to provide answers and opinions. Continue reading

The Phoenix Project – A Novel about IT

We had to do a restore of a production database and it made me remember a book I read that you may enjoy. It’s called The Phoenix Project. It’s not the best written book, a little unbelievable and hokey at times, but I honestly couldn’t put it down (I’m a tech geek). I have never read a book like it.

It’s about DevOps yet, it’s a novel not a tech book. It leans more on the Ops perspective, is a retelling of The Goal, and is rooted in Kanban and Manufacturing Operations. Yet, it’s something I believe anyone working in Enterprise IT can relate to.

The Kindle version is only $10. If you read it let me know what you think as DevOps is something I am passionate about and I’d like to hear your perspective.

http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592

http://itrevolution.com/ – Author’s website

Open Office Delima

I worked in an open office (non-cubicle) floor plan once as a call-in Tech Support Rep for AOL. It was noisy, but we all had headsets on with noise cancelling microphones. This wasn’t a job where concentration really matter. We just followed on screen prompts and basically guessed at what may be wrong with someone’s issue. Well my current employer is moving to a new building with an open floor plan. I heard someone talking about concerns over potential noise levels in our new space. Thinking about this made me do a little research. They mentioned that there may be noise generators or other noise cancelling techniques employed, but I have my doubts on their effectiveness in an open space when sound sources are right next to you.

In my research, I also found it interesting that someone put a number, although subjective, on the effect of open floor plan noise to productivity.

https://www.facebook.com/TED/posts/10152742192960652
http://www.ted.com/talks/julian treasure the 4 ways sound affects us.html

I wonder how accurate any of this is?

Continue reading

Production Updates with a Big Red Button

Imagine if you would, a modified game of Jeopardy hosted by Alex Trebec of course. This game is all about getting changes to production and a smart Dev is in the hot seat.

Dev: Can I have Daily Production Pushes for $200?

Alex: You just deployed a new code change to production, it is causing major issues with performance and you need to roll it back. How do you roll back the troublesome change without having to roll back the entire release? (Jeopardy theme music playing)…

Dev: What is a Feature Toggle.

Alex: Correct.

Dev: Can I have Daily Production Pushes for $500?

Alex: You just started work on a new feature that is no where near ready and you check in your changes to source control. The powers that be want to push to production the branch you’ve checked in the new feature… and they want to do it NOW! How do you push the branch and not expose the unfinished feature changes without having to revert the feature and other work mixed in between your feature changes?

Dev: What is a Feature Flag.

Alex: Correct.

Dev: Can I have Daily Production Pushes for $1,000?

Alex: Daily Double

Alex: Marketing wants to be able to turn a feature on and off for customers depending on their transaction volume. How do you accomplish it?

Dev: What is a Ticket Flag.

Alex: Correct.

Alex: You are our Production Jeopardy Champion?

Continue reading

Temporary Home

Welcome to my temporary home. Normally, I blog at charlesbryant.com, but because my former host basically SUCKS!!!!! I have to start all over again and I don’t feel like the hunt for a host, WordPress setup, config and repost of all of my blog posts manually. So, here I am. Hopefully, I can easily get these posts to their permanent home without a fight.

Thanks for stopping by and I hope that I helped you in some way. Drop me a line in the comments if you would like to discuss anything I have posted.