Category: Test Automation Tips

Do We Test the Ticket or the Feature in the Ticket?

This question is more for change tickets (e.g. bug fixes, feature enhancements…not new features). I am still fairly new at testing, so I have been asking a lot of questions and this was one of them. We can spend time exploring every aspect of a feature that may have been affected by a change, but does that provide the best use of my time. We only have a limited amount of time to do testing so every minute counts. To insure we are providing value to the team we need to focus on getting the best bang for those precious minutes. I believe we should test the ticket by focusing on the most risky aspects of a change. After reporting the initial risk assessment then we can focus on additional manual testing as time permits. After doing the initial manual testing, decide if any of the manual tests are valuable enough to be a candidate for automation. If so, add them to the list of tests to automate.

Test the Ticket

By testing the ticket I mean a manual exploration of the changes to a feature to insure the intent of the ticket was satisfied and its highest risks evaluated. Yes, manual. Even though I am responsible for automated testing (or checking), I first have to understand how to test manually. The automation are just running checks like a line of robots in an assembly line, it runs the same checks over and over, no real variation, just what is coded in the test.

Unfortunately, our users aren’t robots and they don’t follow the test script when using the application so manual testing is absolutely necessary to add variability like our users will. Yet, automation is a time saver for mundane tests like regression. So after we are done with manual testing, we evaluate the tests to decide if any of them are valuable enough to automate and add them to an automated test suite (e.g. regression, smoke, performance…).

As the ticket progresses through the delivery pipeline, at each stage we run additional test. This way additional feedback can be had at each stage and we also increase test coverage the closer the ticket gets to delivery. By not trying to rely only on automation, we also benefit from early feedback from the manually exploration of the ticket changes. I believe we can manually explore them than automate them (IMHO). So, as soon as a ticket is ready for testing we begin exploration. A ticket is only ready after it has passed a manual code review, automated build, automated unit test, and automated code analysis. So, we have multiple gates that a ticket must pass and fast feedback at each step.

Conclusion

That was probably an obvious answer to seasoned testers, but it may not be pondered by new testers.The point is to focus at first on the change in the ticket as this will be the fastest way to provide feedback on the risks in delivering the ticket. Then move on to running longer running tests as the ticket moves closer to being deployed to production. Fast feedback is the name of the game.

Selenium WebDriver Custom Table Element

One thing I like to do is to create wrappers that simplify usage of complex APIs. I also like wrappers that allow me to add functionality to an API. And when I can use a wrapper to get rid of duplicate code, I’m down right ecstatic.

Well today I’d like to highlight a little wrapper magic that helps simplify testing HTML tables. The approach isn’t new, but it was introduced in TestPipe by a junior developer with no prior test automation experience (very smart guy).

First some explanation before code as it may be a little hard to understand. TestPipe actually provides a wrapper around Selenium Web Driver. This includes wrappers around the Driver, SearchContext, and Elements. TestPipe has a BaseControl that is basically a wrapper around WebDriver’s IWebElement. This not only allows us to bend WebDriver to our will, the biggest benefit, but it also gives us the ability to swap browser drivers without changing our test code. So, we could opt to use WatiN or some custom System.Net and helpers and not have to do much, if anything, in terms of adjusting our test code. (Side Note: this isn’t a common use case, almost as useless as wrapping the database just so you can change the database later…doesn’t happen often, if ever, but its nice that its there just in case)

GridControl

Like I said, we have a BaseControl. To provide some nice custom functionality for working with tables we will extend BaseControl.

public class GridControl : BaseControl
{
 public GridControl(IBrowser browser)
   : base(browser, null, null)
 {
 }
 //...Grid methods
}

Breaking this down, we create a new class, GridControl, that inherits BaseControl. In the constructor we inject an IBrowser, which is the interface implemented by our WebDriver wrapper. So, outside of TestPipe, this would be like injecting IWebDriver. The BaseControl handles state for Driver and exposes it to our GridControl class through a public property. This same concept can be used to wrap text boxes, drop down lists, radio button…you get the idea.

GetCell

I won’t show all of the code, but it’s on GitHub for your viewing pleasure. Instead, I will break down a method that provide some functionality to help finding cells in tables. GetCell, this method provides a simple way to get any cell in a table by the row and column.

public IControl GetCell(int row, int column)
{
 StringBuilder xpathCell = new StringBuilder();
 xpathCell.Append("//*[@id='");
 xpathCell.Append(this.SelectId.ToString());
 xpathCell.Append("']/tbody/tr[");
 xpathCell.Append(row);
 xpathCell.Append("]/td[");
 xpathCell.Append(column);
 xpathCell.Append("]");
 Select selector = new Select(FindByEnum.XPath, xpathCell.ToString());
 IControl control = this.Find(selector);
 return control;
}

This isn’t the actual TestPipe code, but enough to get the gist of how this is handled in TestPipe. We are building up an XPath query to find the tr at the specified row index and the td at the specified column index. Then we pass the XPath to a Selector class that is used in the Find method to return an IControl. IControl is the interface for the wrapper around WebDriver’s IWebElement. The Find method uses our wrapper around WebDrivers ISearchContext to find the element…who’s on first?

One problem with this approach is the string “/tbody/tr”. What if your table doesn’t have a tbody? Exactly, it won’t find the cell. This bug was found by another junior developer that recently joined the project, another very smart guy. Anyway, this is a problem we are solving as we speak and it looks like we will default to “/tr” and allow for injection of a different structure like “/tbody/tr” as a method argument. Alternatively, we could search for “/tr” then if not found search for “/tbody/tr”. We may do both search for the 2 structures and allow injection. These solutions are pretty messy, but it’s better than having to write this code every time you want to find a table cell. The point is we are encapsulating this messy code so we don’t have to look at it and getting a cell is as simple as passing the row and column we want to get from the table.

Usage in Tests

In our page object class we could use this by assigning GridControl to a property that represents a table on our page. (If you aren’t using page objects in your automated browser tests, get with the times)

//In our Page Object class we have this property
public GridControl MyGrid
{
    get
    {
        return new GridControl(this.Browser);
    }
}

Then it is pretty easy to use it in tests.

//Here we use myPage, an instance of our Page Object,
//to access MyGrid to get the cell we want to test
string cell = myPage.MyGrid.GetCell(2,5).Text;
Assert.AreEqual("hello", cell);

Boom! A ton of evil duplicate code eradicated and some sweet functionality added on top of WebDriver.

Conclusion

The point here is that you should try to find ways to wrap complex APIs to find simpler uses of the API, cut down on duplication, and extend the functionality provided by the API. This is especially true in test automation where duplication causes maintenance nightmares.

Test Automation Tips: 3

#3 Don’t become complacent in testing when a change is simple.

The Heartbleed bug was caused by a somewhat simple change that was not properly tested before it was released to the public. Maybe that is a overreaching generalization, but the fact is, the change and the eventual fix to Heartbleed were trivial, but incredibly damaging. I have found time and time again that many bugs are introduced through simple changes that are not properly tested before releasing them because they are so simple. I’m just inverting some logic, doing a simple refactor, updating how a variable is updated, adding a column to a query…I can create a long list of bugs that jump in my mind that were thought to be simple changes, but leaked risk and chaos into a release.

Quality Starts with Development

When I say test, I’m not talking about QA testing, I’m talking about developer testing. Quality is a team sport and the developers have the ball most of the game. The fact is developers have the largest share in insuring quality. There are usually more developers than any other role on the product team. The developers usually eat up most of the clock in the release cycle.

QA just got stuck with quality in their title because developers where a custom to tossing their code across the fence for others to worry about it working or not. QA was enlisted to sit on the other side of the fence and catch the bugs. OK, that is not the true story of how the QA role came to be, but that’s how I imagine it. I have seen this happen so many times: developers deliver code at last the last minute to QA, QA has to rush to test it, a bug is found that could have easily been caught with a more thorough inspection of quality by the developer, then QA has to test it again and pray that the developer tested it as they deal with their backed up schedule with no time to thoroughly test. I don’t look at QA as quality managers, but risk managers. They measure, analyze, and report the risk of a release. They don’t build quality into the product developers do. They don’t even insure that the team ships a quality product. They only point out risks in a release version. It’s up to the team to insure a certain level of quality is maintained and the analysis by QA is a part of defining what level of quality is expected. In the end, the quality buck stops with development, but the team insures quality of the product. Yet, if something breaks in the code, its not QA or the BA that will code the fix.

Test It

Simple changes from the database to the UI have been found to be culprits of pesky, hard to find bugs. The one thing they all have in common is that they were not tested or the tests were not good enough to cover the change. I am not going to say that “every change has to have a unit test…roar!”, but you need to find a way to test the fix that is better than visual inspection of an awesomely simple code change.

To me the best test is from the perspective of the user. In the end its the user who has to suffer from crappy code. So, if you are hacking on a web app, open a browser and test your change. Developing a service, open Fiddler or WireShark and send a request to exercise your change. If you are into it, go ahead and write a unit test. Learn about testing, just like you learn about new technologies to make your coding life easier.

Developing is more than coming up with elegant algorithms and simple solutions to complex problems. Do what you have to in order to find a meaningful way to exercise your changes. Don’t stop with one test, find more than one way to test your change. Your goal should be to commit the code better than you checked it out. If not, your QA will waste time and money rejecting your change or worse your users will reject your change and you will risk the reputation and brand of your company on something you thought was simple.

Conclusion

Well, if you have been a developer long, you have heard most of this before. This was more of a rant against myself for being asleep at the wheel on a simple change, I am my biggest critic. I believe all good developers share my sentiments in some way. Yet, simple changes have a habit of biting us because of our complacency. To validate our assumptions about a change, no matter how simple, we must try to test the change through the UI or public API. Even a simple smoke test to verify the UI still works is better than no test at all. Running a stored procedure after a simple column change is better than just checking it in because it was so easy to add that misspelled column name. You can take the time now to test or later, but later may cost you and your team in the end.

As a Software Developer in Test, I will ask a developer how they tested their change just to measure their due diligence. If I don’t hear something about the UI or API, unit test or some kind of substantive test, I have an urge to release the hounds. When I release the hounds, the cats run up a tree. When the cats run up a tree they scare the birds and they poop on your mom’s car. Don’t get your mom’s car pooped on 🙂

View more tips.

When I am writing and maintaining large functional UI tests I often realize somethings that would make my life easier. I decided to write this series of posts to describe some of the tips I have for myself in hopes that they prove to be helpful to someone else. What are some of your tips?

Test Automation Tips: 2

#2 If you update or delete test data in a test, you need to isolate the data so it isn’t used by any other tests.

A major cause of flaky tests (tests that fail for indeterminate reasons) is bad test data. This is often caused by a previous test altering the test data in a way that causes subsequent tests to fail. It is best to isolate test data for scenarios that change the test data. You should have an easy way to isolate read only data so that you can insure it isn’t corrupted up by your tests.

Delta Seed

One way I like to do this is having what I call a delta test data seed. This delta seed loads all read only test data at the start of a test suite run. Any test data that needs to be updated or deleted is created per test. Mutable test data seeds are ran after the delta seed. So, in addition to the delta seed I will have a suite, feature or scenario seed.

Suite Seed

The suite seed is ran right after the delta seed, usually with the same process that runs the delta seed. Because the suite seed data is available to all tests being ran, it is the most riskiest seed and the least efficient unless you are running all of your tests as you may not need all of the data being loaded. I say risky, because it opens up the scenario where someone writes a test against mutable data when it should only be used by the test that will be changing the data.

Feature Seed

The feature seed would run at the beginning of a feature test during text fixture setup. This basically loads all the data used by tests in the feature. This has some of the same issues as the suite seed. All of the data is available for all tests in the feature and if someone gets lazy and writs a test against the mutable data instead of creating new data specifically for the test may result in flaky tests.

Scenario Seed

The scenario seed runs at the beginning of an individual test in the feature test fixture. This is the safest in terms of risk as the data is loaded for the test and deleted after the test so no other tests can use it. The problem I have with this is when you have a lot of tests having to create hundreds of database connections and dealing with seeding in the test can have an impact on overall and individual test time. If not implemented properly this type of data seeding can also create a maintenance nightmare. I like to use test timing as an indicator of issues with tests. If you can’t separate the time to seed the data from the time to run the test, having the seed factored into the test can affect the timing in a way that has nothing to do with what is being tested. So, you have to be careful not to pollute your test timing with seeding.

Which Seed?

Which seed to use depends on multiple factors. How you run your tests? If you are running all tests, it may be efficient to use a suite seed. If you run features in parallel, you may want a feature seed to quickly load all feature data at one time. If you run tests based on dependencies in a committed change, you may want to keep seeding granular with a scenario seed. There are many other factors that you can take into account and trial and error is a good way to go as you optimize test data seeding.

Conclusion

The thing to take away is you need a strategy to manage the isolation of test data by desired immutability of the data. Tests that don’t alter test data should use read only data. Tests that alter test data should use test data specifically created for the test. If you allow a test to use data altered by another test, you open yourself up to flaky test syndrome.

View more tips.

When I am writing and maintaining large functional UI tests I often realize somethings that would make my life easier. I decided to write this series of posts to describe some of the tips I have for myself in hopes that they prove to be helpful to someone else. What are some of your tips?

Test Automation Tips: 1

#1 Don’t test links unless you are testing links.

Unless you are specifically testing menus or navigation links, don’t automate the clicking of links to navigate to the actual page under test. Instead take the shortest route to set the test up.

Let’s say I have a test that starts by clicking the product catalog link, then clicks on a product to get to product details, just so I can test that a coupon appears in product details. This test is testing too many concepts that are out of the scope of the intent of the test. I am trying to test the coupon visibility, not links. Instead, I should navigate directly to the product details page and make my assertion.

If you use the link method and fool yourself into believing that your test is acting like a user, when links are broken your test will fail for a reason other than the reason you are testing. If you use the same broken link in multiple tests, you will have multiple failures and will have to fix and rerun the tests to get feedback on the features you are really trying to test. If the broken tests are in a nightly test, you just lost a lot of test coverage and and you don’t know if the features are passing or failing because they never got tested. Using the link method can cause a dramatic loss of test coverage. So, test links in navigation tests only when the links are front and center in the purpose of the test. For all other tests, take the shortest route to the page under.

View more tips.

When I am writing and maintaining large functional UI tests I often realize somethings that would make my life easier. I decided to write this series of posts to describe some of the tips I have for myself in hopes that they prove to be helpful to someone else. What are some of your tips?

Test Automation Tips

When I am writing and maintaining large functional UI tests I often realize somethings that would make my life easier. I decided to write this series of posts to describe some of the tips I have for myself in hopes that they prove to be helpful to someone else. What are some of your tips?

#1 Don’t test links unless you are testing links.

#2 If you update or delete test data in a test, you need to isolate the data so it isn’t used by any other tests.

#3 Don’t become complacent in testing when a change is simple.