An Agile Transformation

I wrote this a few years ago, but I’m going through a similar agile transformation right now. Although, every agile transformation is different, this still makes sense to me although it is just a draft post. I figured I’d just post it because I never search my drafts for nuggets of knowledge :).

If we are going to do Kanban we shouldn’t waste time formally planning sprints. Just like we don’t want to do huge up front specifications because of waste cased by unknowns, we don’t want to spend time planning a sprint because the work being done in the sprint can change anytime the customer wants to reprioritize. We should have a backlog of prioritized features. The backlog is regularly prioritized (daily, weekly…) to keep features available to work. If we want to deliver a specific set of features or features in two weeks, prioritize them and the team will do those features next.

There is a limit on the number of features the team can have in progress (work in progress or WIP). Features are considered WIP until they pass UAT. Production would be a better target, but saying a feature is WIP until production is a little far fetched if you aren’t practicing continuous delivery. So, for our system production is considered passing UAT. When the team is under their WIP limit they are free to pull the next feature from highest priority features in the backlog.

This is going to most likely reduce resource utilization, but will increase throughput and improve quality. Managers may take issue at developers not being used at full capacity, but there is a reason for this madness and hopefully I can explain it.

Having features pulled into the pipeline from a prioritized backlog instead of a planned sprint allows decisions on what features to be worked to be deferred until the last possible moment. This provides more agility in the flow of work in the pipeline and the product owner is able to respond quickly to optimize the product in production. Isn’t agile what we’re going for? Pulling work with WIP limits also gives greater risk management. Since batch sizes are smaller, problems will only affect a limited amount of work in progress and risk can be mitigated as new work is introduced in the pipeline. Focusing on a limited amount of work improves the speed at which work is done. There is no context switching and there is a single focus on moving a limited amount work through the system at one time. This increases the flow of work even though there may be times when a developer is idle. The truth is the system can only flow as fast as its slowest link, the constraint. Having one part of the system run at full capacity and overload the constraint introduces a lot of potential waste in the system.

On my current team, we have constraints that determine how quickly we can turn around a feature. Currently, code review and QA are constraints. QA is the largest constraint that limits faster deployment cycles, but more on that later. If we follow the five basic steps outline in the TOC from the book The Goal, we would:

  1. Identify the constraint(s) – in this instance it’s code review and manual testing
  2. Exploit the constraint to maximize productivity –
  3. Subordinate all other steps or processes to speed up or reduce capacity of the constraint – no new work may enter as WIP until the constraint has WIP available.
  4. Elevate the constraint – for us we will prioritize work that helps remove to remove these work centers as constraints.
  5. Repeat

The plan is to have developers do code reviews any time WIP stops the movement of work. Also, developers should create automated tests to help lessen the work that QA has to do. The reason we don’t first focus on optimizing QA processes is because focusing on simply optimizing QA processes would actually increases the capacity for QA without increasing the speed at which we can flow work to production. We don’t want to increase the number of features that QA can handle. We also don’t want to speed up manual testing because it is important to take the proper time in testing. What we want to do is remove manual regression as work for QA to open us up to deliver new features to QA faster and get QA to deliver the feature to production faster. QA can focus on what they do best, test. Not running mundane scripted checks.

Normally, we would have to wait for a manual regression test cycle to occur and couldn’t introduce new work because it would invalidate the regression test. With automation handling +80% of regression QA can move faster, actually test more, and we can not only increase throughput through the entire system, but the overall quality of the product.

Monitoring Delivery Pipeline

We track work through the delivery pipeline as features. A feature in this sense is any change, new function, change existing function, or to fix a defect. Features are requested on features kept in a central database. We monitor the delivery pipeline by measuring:

  • Inventory
  • Lead Time
  • Quantity: Unit of Production
  • Production Rate


Inventory (V) is any work that has not been delivered to the customer. This counts all work from the backlog to a release awaiting production deployment. Whenever there is undelivered work that is considered invalid it becomes an Operational Expense. Invalid meaning it won’t be delivered at all or there are issues like defect or doesn’t match spec. Invalid work is wasted effort and in the case of a defect causes expensive un-budgeted rework. In traditional cost accounting inventory is seen as an asset, but in TOC it is a “potential” Operational Expense if it is not eventually delivered to customer so turning inventory as fast as possible without injecting defects is a goal.

Quantity: Unit of Production

Quantity: Unit of Production (Q) is the total number of units of work (feature) that have moved through our delivery pipeline to date. Our unit of production is a feature. When a feature is ready to be deployed to production we can increase Q one unit, but the feature is still considered inventory until it has been delivered to customer. If a customer decides they don’t want the feature or some other reason to stop the deployment of the feature, it is counted as an Operational Expense and Q is reduced one unit.

Lead Time

Lead time (LT) is the time it takes to move a feature, one Q, from submission to the backlog to deployed to a customer in production.

Production Rate

Production rate (PR) is the number of Q delivered during a time period. 3 features per month, 2 features per week…

Optimize Delivery Pipeline for Lead Time

We should strive to optimize the delivery pipeline for lead time instead of production rate. The Theory Of Constraints – Productivity Metrics in Software Development posted on explains this well.

Let’s say our current lead time (LT) is 1 unit (Q) in a week or a production rate (PR) of 4 Q per month. If we optimize LT to 1 Q in 3 days, we will see a jump in PR to 6.67 Q per month or a 59% increase.

If we focus on optimizing PR, we may still see improvement in LT, but it can also lead to only an increase in inventory. The PR optimization may increase Q that is undeliverable because of some bottleneck in our system so the Q sits as inventory. The longer a feature sits in inventory the more it costs to move it through the pipeline and address any issues found in later stages of the pipeline.

So, to make sure we are optimizing for LT we focus on reducing waste or inventory in the pipeline. The delivery team keeps a single purposed focused on a limited amount of work in progress to deliver what the customer needs right now, based on priority in the backlog. Reducing inventory reduces Operation Expense. (Excuse me if I am allowing some lean thinking into this TOC explanation)



Investment (I) is the total cost invested in the pipeline. In our case we will count this as hours invested.

Operating Expense

Operating expense (OE) is the cost of taking an idea and developing it to a deliverable. Any fixed overhead is considered OE. We will just use salaries of not only developers, but BA, QA, IT as our OE. Not sure how we will divide up our fixed salaries (still learning).


Throughput (T) is the amount earned per Q. It is calculated by taking the amount of features delivered to production minus the cost of delivering the feature.

Throughput Accounting

To maximize ROI and net profit (NP) we need to increase T while decreasing I and OE.

NP = (T – OE)


Average Cost Per Feature

Average cost per feature (ACPF) is the average amount spent in the pipeline to create a feature.


There are more metrics that we can gather, monitor, and analyze; but we will keep it simple for now and learn to crawl first.

Average Lead Time Per Feature

The average time it takes to move a feature from the backlog to production. We also calculate the standard deviation to get a sense on how varying work sizes in the pipeline affects lead time.

Bonus: Estimating Becomes Easier

When we begin to monitor our pipeline with these metrics estimating becomes simpler. Instead of estimating based on time we switch to estimating based on size of feature. Since we are tracking work, we have a history to base our future size estimates on.


Issues in Transformation

Our current Q is a release, a group of features that have been grouped together for a deployment. We will build up an inventory of features over a month at times before they are delivered to production. This causes an increase in inventory. It would be better to use a feature instead of a release as our Q. When a feature is ready, deliver it. This reduces inventory and increase the speed at which we get feedback.

To change our unit, Q, to feature we have to attack our largest constraint, QA. Currently, we have to sit on features or build up inventory to get enough to justify a QA test cycle. We don’t want to force a two week regression on one feature that took a couple days to complete. So, reducing the test cycle is paramount with this approach.


The Goal: A Process of Ongoing Improvement, by Eliyahu M. Goldratt

The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, by Gene Kim, Kevin Behr, and George Spafford.

The Metrics in TOC: Productivity Metrics In Software Development, by Derick Bailey,

Agile Management for Software Engineering, by David J. Anderson

Reaching The Goal, by John Arthur Ricketts

Applying Theory of Constraints to Manage Bottlenecks, by Kamran Khan,




Adding Report to Existing TFS 2017 Project

I had an issue where I couldn’t see reports for my TFS projects because they weren’t installed. I knew this because I opened SQL Reporting Services and I didn’t see a folder for my project under the TFS collection’s folder. I did a little digging and found a command that I could run to install the reports:

  1. Open administrator command prompt on server hosting TFS.
  2. Change directory to C:\Program Files\Microsoft Team Foundation Server 15.0\Tools
    Note: 64bit would be Program Files (x86)
  3. Run TFSConfig command to add project reports

TFSConfig addprojectreports /collection:”https://{TFSServerName}/{TFSCollectionName}” /teamproject:{TFSProjectName} /template:”Scrum”

You should replace the tokens with names that fit your context (remove the brackets). The template will be the template for your project:

  • Scrum – you will have backlog items under features
  • Agile – you will have stories under features

There’s another one, CMMI, but I’ve never used it. You should see a requirements work item, but I’m not sure if this template has a feature item.

Once you run the command, the reports will be added and you will be able to see how your team is doing by viewing the reports in SQL Reporting Services.

Visual Studio, DotNet Core, Windows and Docker, a Match Made in Heaven


If you haven’t heard about Docker, catch up. Out the gate, the best reasons for me to use Docker is being able to run a production like environment locally and being able to instantly create new test environments without having to go through the tiring dance of manual configuration. So, each test deployment gets a brand new fresh environment and the environment can be thrown away after testing is done. So, Docker delivers on “infrastructure as code” and allows you to save your Docker container configuration in source control and iterate it along with the source code. There are other reasons, but they had me at “create new test environments without having to go through the tiring dance…”

With Windows Server 2016 having Docker support, it’s time to get on board. It has been possible to run ASP.NET/.NET Core on Linux, but I have been waiting to be able to do this on Windows. So, here we go.


Create New .NET Core Solution

  • Create ASP.NET Core Web Project (.NET Core)
  • Use the ASP.NET Core 1.1 Template for WebAPI
  • Opt to Enable Docker Support

Use Existing .NET Core Solution

  • Open an existing .NET Core solution
  • Right click on the Web Project in the Solution Explorer and click “Add Docker Support”

Docker Support

When you Enable Docker Support a Docker Compose project is created. The Docker Compose project is Visual Studio’s tooling to manage the creation of Docker containers. It has two YAML files:

  • – configures how the image is built and ran.
  • docker-compose.yml – configures the image to be built and ran. It also has nested files that allow you to override the configuration for debug and release (similar to web.config transforms).


Debugging is as simple as debugging traditional web projects. To give a little background on what’s happening behind the scenes, when you F5/debug your Docker containerizes the application (is that a word)

  • Visual Studio runs docker compose
  • The image defined in the is downloaded if not in cache
  • ASPNETCORE_ENVIRONMENT is set to Development inside the container
  • Port 80 is explosed and mapped to a dynamically assigned port for localhost. The Docker host controls the dynamic port. You can see the container port with the Docker CLI command to list containers:
docker ps
  • The application is copied from the build output folder into the continer
  • The default browser is launched with the debugger attached to the container.

The application is now running and you can run docker ps to see some of the properties of the running container. The dev image that was built does not contain the application, rather it is mounted from drive we shared with the containers during install. This is to allow you to iterate and make changes while developing without having to to go through the expense of writing the application in the image.

If you make changes to a static file, they are automatically updated without having to recompile. You do have to refresh the browser, so it’s not as lovely as client side development, but better than stopping and starting the container.

If you make changes to a compiled file, hit Ctrl-F5 and the application is compiled and the Kestrel web server is restarted in the container. The container doesn’t have to be rebuilt or stopped, because we don’t have to rebuild the application in the image (hence the reason for the empty dev image).


When you are ready to release the application to Docker Hub or your private hub, you create a production image of the application. When you build in release mode the application is actually copied into the image and unlike dev images, future changes have to be re-imaged.


This was a very pleasing development experience. I had no issues at all (knock on wood) and I was debugging a running .NET Core application in a Docker container locally on Windows 10  in less than 15 minutes (not including install time). I’m still very new at this new tooling and Windows support, so I hope I will get to write more about it as I hit road blocks. There are always road blocks and I’m sure that I will hit them when I answer some of my current questions:

  • How to automate and integrate in continuous delivery pipeline
    • Container build and publish
    • Deploying container and running application to Windows Server 2016 on premise and in Azure
  • How to run multiple load balanced containers
  • How to monitor containers
  • How to deploy more containers to handle increase load or failover
  • How to use as base of Microservices architecture
  • And the questions keep coming…


TransactionScope Async Thread Fail

I updated some data access code to wrap some operations in a TransactionScope. The operations are async methods running some Dapper execute statements to write data to a SQL Server database. Something like:

public async Task InserData(SourceData sourceData)
  using (var transactionScope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
    using (IDbConnection connection = new SqlConnection(this.ConnectionString))

      await InsertSomeData(sourceData.Registers, connection);
      await InsertMoreData(sourceData.Deposits, connection);


Anyway, I wire up a unit test to this method and it fails with this message:

Result Message:
Test method ExtractSourceDataTest.CanStart threw exception:
System.InvalidOperationException: A TransactionScope must be disposed on the same thread that it was created.

As usual, Google to the rescue. I found a nice blog post that explains the issue, Basically, TransactionScope was not made to operate asynchronously across threads, but there is a work around for that. Microsoft released a fix, TransactionScopeAsyncFlowOption.Enabled. I went from a zero

using (var transactionScope = new TransactionScope())

to a hero

using (var transactionScope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))

Now, if this would have been turned on by default I wouldn’t of had this little problem… talking to you Microsoft. I’m sure there is some backward compatibility issue or other quirk that makes default enable difficult, but I’m ranting anyway.


This is awesome, but I basically just enabled a distributed transaction and that scares me. You do not know the trouble I have seen with distributed transactions. Hopefully, its not that bad since we are distributing across processes on the same machine and not over the network, but scary none the least.

AWS Device Farm vs Microsoft Mobile Center

Someone at worked asked if I have ever used AWS Device Farm. I have never used it, but testing mobile apps in the cloud against hundreds of device profiles on real devices sounds like the way to go. It would be hard for us to build and manage a device farm on premise.

AWS Device Farm

After reading up on AWS Device Farm, I discovered that it is a mobile app testing service. It allows you to run tests on Android and iOS devices in the cloud with automated tests against multiple devices at once. It also allows you to run a manual test on a real devices in real time (awesome!!!). You can view video, screenshots, logs, and performance data on your tests to get deep insights into your app.

Microsoft Mobile Center

I went to a Xamarin meetup and learned that Microsoft Mobile Center is basically the same thing as AWS Device Farm, but it covers the entire continuous delivery pipeline for iOS, Android and Windows mobile devices:

  • Build
  • Test
  • Dstribute to app store
  • Monitor crashes and analytics

The build part was compelling for me because it allows me to build iOS in the cloud without having to own a Mac.

It also provides integration with:

  • Git repositories on GitHub or Visual Studio Online (Bitbucket coming soon).
  • Azure Table Data Storage for online/offline data storage and sync.
  • Azure Identity for app user identity management.


The major difference between them is scope. Device Farm is concerned only with testing in the cloud. Mobile Center is concerned with hosting your entire continuous delivery pipeline in the cloud. So, this is like comparing React to Angular, two different levels of abstraction.

Device Farm doesn’t provide Windows platform testing, but I don’t think that is a deal breaker for many people right now. It also doesn’t support any other continuous delivery automation outside of the test stage. So, you will have to find other services for build, distribute, and monitor or script your own automation.

Mobile Center doesn’t have Remote Access like Device Farm, but you could always write an automated test for the manual actions you’d like to reproduce.


I don’t have any real world experience with either solution. This is a surface level comparison based on docs and demos. I’m a little biased towards Microsoft because I am primarily a .Net developer, so yell at me if I was too unfair to Device Farm.

Build a .Net Core WebAPI using Visual Studio Code

So, we have an intern and she is helping us build an internal tool. She is good on the client side, but very light in experience on the back-end. So, I wanted to give her a challenge, Build a .Net Core WebAPI using Visual Studio Code. I wrote up these instructions and she had the API up and a basic understanding of how to iterate it forward in less than an hour. I thought I’d share it in hopes it helps someone else.

Check out Cmder,, as an alternative to Windows command prompt.

  • Make a directory for the application. I am creating my application in an “api” folder inside my _projects folder. Run
mkdir c:\_projects\api
  • Change to your new directory. Run
cd c:\_projects\api
  • Create a .Net Core application. Run
dotnet new
  • Restore dependencies that are listed in your project.json. Run
dotnet restore
  • Open Visual Studio Code and open your application folder. Run
  • You may see a warning, “Required assets to build and debug are missing from ‘api’. Add them?”, click yes.
  • Open the Quick Open (Ctrl+P)
  • Run this command “ext install charp”.
  • Back in the console you should be able to run the application and see “Hello World!” printed to the console. Run
dotnet run

The project.json currently looks like:

  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable",
    "emitEntryPoint": true
  "dependencies": {
  "frameworks": {
    "netcoreapp1.1": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.1.0"
      "imports": "dnxcore50"

We need to update this to run ASP.Net MVC:

  "version": "1.0.0-*",
  "buildOptions": {
    "debugType": "portable",
    "emitEntryPoint": true
  "dependencies": {
  "frameworks": {
    "netcoreapp1.1": {
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.1.0"
        "Microsoft.AspNetCore.Server.Kestrel": "1.1.0",
        "Microsoft.AspNetCore.Mvc": "1.1.1",
        "Microsoft.AspNetCore.Mvc.Core": "1.1.1"
      "imports": "dnxcore50"

Under frameworks, you will notice that we are running .Net Core 1.1, the current version when this was written. Also, we added some additional dependencies:

  • Kestrel – a web server that will serve up your API endpoints to clients
  • Mvc – The base ASP.Net Core 1.1.1 dependency
  • Mvc.Core – The core ASP.Net Core 1.1.1 depencency

These dependencies will allow us to write and serve our API using ASP.Net Core MVC.

Once you save the project.json Visual Studio Code will let you know “There are unresolved dependencies from ‘project.json’. Please execute the restore command to continue.” You can click “Restore” and you can open the console and run

dotnet restore

This will install the new dependencies that were added to project.json.

Now we need to configure our application to serve our API. We need to update Program.cs from:

using System;

namespace ConsoleApplication
    public class Program
        public static void Main(string[] args)
            Console.WriteLine("Hello World!");


using System;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.DependencyInjection;

namespace BInteractive.StoryTeller.Api
    public class Program
        public static void Main(string[] args)
            var host = new WebHostBuilder()

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app)

Here we added new using statements at the top of the class to reference the dependencies we want to use. I changed the namespace to match my application, you can customize the name space to match you application. Normally, I like to have my namespace with MyCompanyName.MyApplicationName.{If the class is in a folder Under my root folder, MyFolderName}.

Now we update the Main method, the entry into the application, to run our API instead of printing “Hello World”. We wire up a host using the Kestrel web server, using this Program class as the start up class, then we build and call run on the host. This starts the server listening and will route based on the configured routes and handle them through the MVC service.

The ConfigureServices method allows you to configure the services you want to use with your API. Right now we only have MVC configured.

The Configure method allows you to inject middle wear into the HTTP pipeline to enhance HTTP request and response handling. You can add things like logging and error pages handling that would work across every request/response.

Now that we are wired up for ASP.Net MVC lets build an API. We are going to build an API that collects and serves questions. So, let define what a question is. Create a new folder under your root folder named “models”. Then create a file name questionmodel.cs.

using System;

namespace BInteractive.StoryTeller.Api.Models
    public class Question
        public string Id { get; set; }
        public string Title { get; set; }

This is a plain old CSharp object that has properties to get and set the question Id and Title.

With this we can create a controller that allows clients to work with this model through our API. Create a new folder under your root folder named “controllers”. Then create a file named questioncontroller.cs.

using System.Collections.Generic;
using Microsoft.AspNetCore.Mvc;
using StoryTeller.Api.Models;

namespace BInteractive.StoryTeller.Api.Controllers
    public class QuestionController : Controller
        private static List<Question> _questions;

        static QuestionController()
            _questions = new List<Question>();

            Question question = new Question();
            question.Id = "1";
            question.Title = "Hello World?";


        public IEnumerable<Question> GetAll()
            return _questions.AsReadOnly();

        [HttpGet("{id}", Name = "GetQuestion")]
        public IActionResult GetById(string id)
            var item = _questions.Find(x => x.Id == id);

            if (item == null)
                return NotFound();

            return new ObjectResult(item);

        public IActionResult Create([FromBody] Question item)
            if (item == null)
                return BadRequest();

            item.Id = (_questions.Count + 1).ToString();


            return CreatedAtRoute("GetQuestion", new { controller = "Question", id = item.Id }, item);

        public void Delete(string id)
            _questions.RemoveAll(n => n.Id == id);

There is a lot here, but the gist is we are setting up an endpoint route for our question API and we are adding methods to get, post, and delete questions. You can dive more into what this is doing by reading up on ASP.Net Core,

You should be able to Ctrl-Shift-B to build the application and if everything is good you won’t see any errors. If you are all good you should be able to run the application. In the console go to the application root directory and run

dotnet run

Then you should be able to browse the API at http://localhost:5000/api/question and see a JSON response with the default question of “Hello World?”.

Modular MicroSPAs

Warning – this is just an unstructured thesis and a challenge for myself to find a solution for building applications with microSPAs. There is no real substance here, just me brainstorming and recording thoughts.

I recently had to bring many microSPAs under the control of one application. A microSPA in this context is just a SPA (single page application) that is meant to coexist with other SPAs in a single application. Each SPA is focused on a discrete domain of the application, maybe a decomposition something like microservices.

I only say micro because I have been through exercises to break up server side monolithic APIs into microservices. Now the break up was client side. Take a client side massive SPA or monolith and break out functionality to smaller SPAs then combine them with new SPAs to form a new modular application. This is nothing new, but it is new to me.

MEAN.js has a wonderful structure for discrete modular AngularJS microSPAs.

The idea is to have a folder containing all of your microSPAs. Each microSPA get’s its own folder. Each microSPA gets its own repository and development life cycle. An example is below, borrowing heavily from MEAN.js. I can’t go into the particulars because this is just a thought from a problem we had with microSPAs, but something I will be involved in solving.

  • app
    • myapp.core <— this is a microSPA
      • client
        • config
        • controllers
        • css
        • directives
        • images
        • models
        • services
          • interceptors
          • socket
        • views
      • server
        • config
        • controllers
        • data
        • models
        • policies
        • routes
        • templates
        • views
      • tests
        • client
          • small
          • medium
          • large
        • server
          • small
          • medium
          • large
      • myapp.core.client.js
    • myapp.stories
    • myapp.users
    • myapp.admin
    • myapp.other_micro_spa

Now the question is, how do you stitch the microSPAs together under one domain name, client context, user session… and manage the entire application across composed micro-SPAs? We need to think about problems areas like:

  • Authentication
  • Root Application and microSPA Level
    • Authorization
    • Routes
    • Menu
    • Layout Templates
    • Static Assets
      • Styles
      • Images
  • Sharing Across MicroSPAs
    • State
    • Components/Modules
    • Dependencies
  • Debugging
  • Testing
  • Delivery Automation (Build, Package, Test, Release)
  • Monitoring and Analytics

How to solve this with AngularJS 1 & 2, React, Vue.js…?

Why am I thinking about this? I just failed gloriously at breaking apart a monolithic SPA and stitching it back together with other SPAs and ran into issues in all of the problem areas above. I didn’t use the MEAN.js architecture or even the structured modular file layout above. The project was done fast and dirty with the only goal of getting the app working again with the new architecture and new SPAs as fast as possible (a few days fast).

The team finished the task, but I was embarrassed by the resulting structure and by many of the hacks we employed to over come issues in the problem areas above. Why we had to accomplish it so fast is another story, so is how we are going to use lessons learned to refactor and address the problem areas above. It’s been a long time since I blogged regularly, but I am hoping to journal our journey and how we solve the issues we faced with microSPAs.

If you have worked with combining multiple SPAs please share, I’m sure there are solutions all over the interwebs.