Category: Debugging

Build Once, Deploy Everywhere

We were faced with an interesting problem. We want to build once and deploy the build to multiple environments for testing, staging and ultimately consumption in production. Well in addition to build once, we also want to allow remote debugging. We need to build in debug mode to get pdb generated and have other configurations that will allow debugging. Yet, we don’t want to deploy a debug build to staging or production. What do we do?

The thought right now is to do a debug build and create to packages, one for debug and one for release. To do this we would have to strip out the pdb and turn on debugging in the release package. So, we still have one build, but we have two flavors of packages.

I am not yet sure if this is viable, hence the reason I am blogging this out. First I need to fully understand the difference between a debug and release build in MSBuild. I have an idea, but I need to verify my assumptions.

Difference Between Debug and Release Build

What I have found is the main difference between debug and release building are:

  1. Debug build generates pdb files.
  2. Release build instructs the compiler to use JIT optimizations.

PDB files are symbol databases that allow a debugger to map machine code to source code (actually MSIL) so you can set break points in source code and the debugger can halt execution of the machine code. This is probably a terrible explanation, but you should get the gist.

JIT optimizations are things the compiler does to speed up the execution of your code. It may reorganize loops to make them run faster and other little magic tricks that happen under the cover that we usually never have to worry about.

Scott Hanselman has an interesting post on this, http://www.hanselman.com/blog/DebugVsReleaseTheBestOfBothWorlds.aspx. This posts suggest that you could do a release build and configure the runtime with an ini file that would determine if JIT optimizations are performed or tracking information is generated.

http://msdn.microsoft.com/en-us/library/9dd8z24x(v=vs.110).aspx this post explains more about the ini.

Now What?

After doing this research I learned a lot about building .Net code, but I also realized that I am taking this a little to far. My primary goal is that we build our application once and use that build in multiple environments to get it tested and deployed to production. When we need to do a remote debug we are using researching an issue and there is no reason that we couldn’t flip a switch on one particular build so that it builds in debug mode, deploy it to a test environment, debug it, make some fixes after finding the cause of the issue we are debugging, then flip the switch back to release and build again, this time allowing the build to go all the way to production.

Issues

The problem here is that we need to make sure that we do not allow debug builds to make it into production. My initial thought is to mark debug builds with a version that is tagged with DEBUG. Then I can have logic in the production deploy that checks for the DEBUG tag and fail the deploy if it is present. We can do the same for pdb files and web.config. Specifically, check for inclusion of pdb (we shouldn’t have pdb file in production). We can also have logic that checks for debug=true and other configurations that we don’t want leaking into production.

We would have to alter our deployment pipeline to add a job that will do these checks based on the environment being deployed. We would have to also look at maybe putting debug builds in a different artifact repository to keep them segregated from release candidates. This would also cause another change to the deployment pipeline where we check the release candidate or debug repository based on some setting.

Conclusion

This would be a lot of changing to our pipeline, but I believe it is worth it in the long run. It also prevents us from leaking manual processes into how we build and deploy the app.

Haunted by the ASP.NET White Screen of Death

You ever have a bug that you just can’t put your finger on. Well this incomplete post is a tale of such a bug. I never had a chance to finish the post or the bug hunt. The post stares at me every time I log into my blog, haunting me. So, I had to post it to release it from its torment.

Early one evening I get the “White Screen of Death” in an ASP.NET application. Basically, it is an empty web page, just empty HTML tag when it should be a full blown data driven web page. You scared yet? This isn’t the first time I have seen this in this application. I am pretty sure it is related to other errors as I remember seeing errors in the application log when ever I see the screen in certain scenarios. The problem is, on this evening it happened in a scenarios without logged errors, but that doesn’t mean there are no errors…right?

Next to see if the error may have been captured elsewhere I set off to check the server logs. I check the server event logs for issues. Nothing jumps out at me.  Then I want to check the IIS logs for the page request so I need to turn on failed request logging on the server. I haven’t done this in a while so I Binged it and got a good hit on this post,  http://www.iis.net/configreference/system.applicationhost/sites/sitedefaults/tracefailedrequestslogging. Now I have it installed and configured, I just don’t know how to inspect the resulting traces. Another Bing and I found the answer at http://www.trainsignal.com/blog/iis-7-troubleshooting. OK, tracing is working and I can view the trace files, now I can’t reproduce the error, oh the horror… figures 😦

Finally, I found a failing scenario and I get nothing of any value in the trace. So I run the scenario again, I check the server event logs again, you guessed it, nothing jumps out at me. So far nothing, I don’t see anything obvious, well nothing I’m noticing as I could just be burnt out and missing the obvious (It happens).

I do a little more Binging and the pickings are slim. I get hits on white page issues involving SSLAlwaysNegoClientCert. Seems some people were having an issue with a 413 – Request Entity Too Large error causing the white page (https://communities.bmc.com/docs/DOC-6259), but there is no way this could be my issue because I’m not uploading anything…right? There is no way ViewState is incredibly large…nah…I’ll check anyway. Better safe than sorry and maybe something will finally jump out at me.

So, I need to:

  • Check ViewState, mainly the size
  • Check Network Traffic, view what is being sent and recieved
  • See if I can repro the scenario locally so I can step through it in a debugger

And this ends our post. Sorry for the abrupt cliff hanger with no solution. I never finished the exploration on this as it was a very low priority edge case bug. It does provide some links on IIS tracing and a little insight into my thought process at the time on trying to discover the source of the bug. One thing I have always admired about many of the smart developers I work with is there thought processes and tool sets used in investigating issues. I have always felt that I came up short in my ability to quickly discover the root cause of issues so I have great respect for the software Sherlock Holmes of the world.

Well hindsight is 20/20 and the biggest mistake that I see is I didn’t automate the scenario. If I would have captured the scenario in a test, I could open it right now and continue where I left off, but now I have no idea where to even start to find the scenario that triggered this issue. So, the issue may still be lurking deep in the bowels of the application. This is the real horror in this post. Oh well, I have a lesson learned. Automate my bug hunt scenarios.

Actually, I wrote this some time ago and remember spending about a hour or so running through this drill in vain. So, in an effort not waste something that may be of value later I decided to just post it to stop it from haunting my post list.

WinDbg a Real Developers Debugger

WinDbg is something I have never really used, actually I just ran through a couple demos a few years ago. I always see serious engineers using this as their debugger and I wan’t to grow up to be a serious engineer so I set out to learn WinDbg once again. Actually, if you can’t hear the trauma in my voice I am still recovering from the mother of all bugs and I am prepping myself for the next time I get a crazy issue.

I can’t really give you a reason to use WinDbg yet, but if you want a legitimate reason, you can check out this answer on Stack, http://stackoverflow.com/questions/105130/why-use-windbg-vs-the-visual-studio-vs-debugger. I have had my share of ugly debug problems and I want to know if WinDbg will give me more insight. So, I will learn now and the next time one of those debug problems rears its ugly head I will hit it with WinDbg and see what I get.

Install

Try this at your own risk and don’t attempt this at home and all that legal stuff. For me this was Difficult! (with a capital D)

First there isn’t a stand alone version that I could find. So, I had to install the Windows SDK. Then I had to find a version compatible with my environment, Windows 7 and .Net 4. Most links to the SDK redirect to the newest version. After many install, uninstall, Google, install, uninstall Google loops the correct process for my computer was this (it may be different for you).

  1. Uninstall Microsoft Visual C++ 2010 x64 Redistributable and Microsoft Visual C++ 2010 x86 Redistributable
  2. Uninstall Microsoft Visual C++ Compilers 2010 SP1Standard x64 and Microsoft Visual C++ Compilers 2010 SP1Standard x86
  3. Install the Windows 7.1 SDK – http://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=8279
  4. Your installation will only be partially complete after the compiler error. Install the Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1 – http://www.microsoft.com/en-us/download/confirmation.aspx?id=4422.
  5. Re-run your Windows SDK web installer. Choose the first option to add features to the existing installation.
  6. Re-choose (either under redistributables or common) the desired features, including the Debugger Tools.

If you have errors, I recommend you open the log in notepad and look for the reason for failure and plug it into Google. I was able to resolve a few issues like that and I don’t care to rehash what I went through as its boring as hell and painful to talk about.

On my setup I find WinDbg here – C:\Program Files\Debugging Tools for Windows (x64)

Specify Symbol Location

As you probably know the Visual Studio debugger works with symbols to provide you with information about the source code you are debugging. WinDbg is no different and you should tell it where to find your symbols.

_NT_SYMBOL_PATH
C:\symbols; SRV*C:\symbols*http://msdl.microsoft.com/download/symbols

*Note: This is a global setting and will affect symbol loading in Visual Studio. Found out about this the hard way and got some good info on issues it can cause here, http://blogs.msdn.com/b/mahuja/archive/2008/07/08/resolving-very-slow-symbol-loading-with-vs-2008-during-debugging.aspx.

This tells WinDbg where to look for code symbols. In this example WinDbg would first check the symbols folder, then if not found it would check the CachedSymbols folder, then if it is till not found it would try to download from Microsoft symbols server and store it in the CachedSymbols folder.

Basic Commands

If you want some basic WinDbg commands and a quick start you can check out http://mtaulty.com/communityserver/blogs/mike_taultys_blog/archive/2004/08/03/4656.aspx. I am too lazy to blog about my experience with my basic WinDbg walk through. I will update the blog when I get to really flex WinDbg’s muscles.

Anyway, if you are a brave sole or one of those coding genius types and you actually try WinDbg, I hope it provides you with some extra fire power in your debug arsenal.

Generate Test Service from WSDL

I had the genius idea to create a test service implementation to test our client services. I would use the test service as a fake stand-in for the real services as the real services are managed by third parties. Imagine wanting to test your client calls to the Twitter API without having to get burned by the firewall trying to make a call to an external service.

As usual, it ends up that I am not a genius and my idea is not unique. There isn’t a lot of info on how to do it in the .Net stack, but I did find some discussions on a few Bings. I ended up using wsdl.exe to extract a service interface from the WSDL and implementing the WSDL with simple ASMX files. I won’t go into the details, but this is basically what I did:

    1. Get a copy of the WSDL and XSD from the actual service.
    2. Tell wsdl.exe where you stored these files and what you want to do, which is generate a service interface 
      wsdl.exe yourFile.wsdl yourfile.xsd /l:CS /serverInterface
    3.  Then implement the interface as an ASMX (you could do WCF, but I was in a hurry)
    4. Lastly, point your client to your fake service

In your implementation, you can return whatever you are expecting in your tests. You can also capture the HTTP Messages. Actually, the main reason I wanted to do this was to figure out if my SOAP message was properly formatted and I didn’t want to go through all of the trace listener diagnostics configuration with a Fiddler port capture crap. There may be easier ways to do this and even best practices around testing service clients, but this was simple, fast, and easy, IMHO, and it actually opened up more testing opportunities for me as its all code and automatable (is that a word).

New Laptop, New Problems…Mind Your Install Order

I recently moved my local development to a new laptop. Actually, a pretty sweet laptop, solid state drive and everything. My IT department set the laptop up and IIS was not installed, but .Net and Visual Studio were. So there were a few things I had to do to get .Net properly registered in IIS. Here are some symptoms and solutions, just incase you run into these problems or I forget what I did.

SqlException

When trying to use a web app that connects to a database I would get an SqlException,

A connection was successfully established with the server, but then an error occurred during the login process. (provider: SSL Provider, error: 0 – The message received was unexpected or badly formatted.).

A simple command cleared this up.

netsh winsock reset

Actually, I got this fix from a co-worker and he was even good enough to provide some research on the possible cause of the issue, but I didn’t read any of it. I already have too much reading to do. As I understand it, it had something to do with my upgrade to Visual Studio 2013, but don’t quote me.

http://technet.microsoft.com/en-us/library/cc753591(v=WS.10).aspx

http://www.techsupportforum.com/forums/f31/netsh-int-ip-reset-and-netsh-winsock-reset-467970.html

http://support.microsoft.com/kb/299357

WCF Service Issue

When trying to run a service locally I kept getting a 405 error in my UI. Under the covers I was actually getting a 404.3, The page you are requesting cannot be served because of the extension configuration. If the page is a script, add a handler. If the file should be downloaded, add a MIME map.

You can check the reference on this issue here: http://msdn.microsoft.com/en-us/library/ms752252(v=vs.90).aspx

As it turns out, in my situation it was another artifact of the botched install order. I had to register WCF:

  1. Run Visual Studio Command Prompt as “Administrator”
  2. Change directory to C:\Windows\Microsoft.NET\Framework\v3.0\Windows Communication Foundation
  3. Run
ServiceModelReg –i

For good measure I also registered ASP.NET with IIS by running these commands in the same command prompt (you don’t have to change directory):

aspnet_regiis -I

then reset IIS

iisreset

Actually, I had to do even more. The above actions registered everything on the server, but the settings weren’t copied down to my websites and services. This is another install order issue as I installed the sites before everything was registered. There may be an easier way to fix this, but I updated my DefaultWebsite (this is were I place all my sites and services). In feature view I opened Handler Mapping and in the Actions pane I clicked Revert to Parent. Then I had to do this in my sites and services. Actually, because there are so many and I have some handy dandy automated install/uninstall scripts, I just uninstalled everything and reinstalled all the sites which picked up the configuration change.

Now that order has been restored to my world I can get back to enjoying this new laptop.

Multiple PostBacks in ASP.NET WebForms

Here are some, not all, reasons why there would be multiple PostBacks on a page:

  • Handling an event multiple times
    • AutoEventWireup = true in Page declaration and manual wire up event in Page Init. Check your codebehind for the old ASP.Net 1 style of registering events in system generate code.
    • Having control’s AutoPostBack = true while also doing an explicit PostBack in another control event.
  • Custom event JavaScript that doesn’t cancel the click event allowing a double post.

Debugging tips:

  • Do an HTTP trace to view the double PostBack requests.
  • Debug and add watch for Request.Form[“__EVENTTARGET”] to find control initiating PostBack.
  • HTML Validate your page, PostBack could be caused by bad markup

Lastly, this is a little hack that may help turn off double PostBacks temporarily.

     <form id=”Form1″ runat=”server” onsubmit=”Q++; if(Q==1){return true;} else { return false;}”>

I Can’t Reproduce That

Sorry, I can’t reproduce that.

I hate saying those words. It seems like a cop out, like I am giving up on the problem. How do you tell someone that you can’t reproduce their issue? Do you ask for more details? Do you just leave it up to the user because you think it is an isolated user issue?

I always ask for more info. The problem is sometimes I don’t know what to ask for. On a web app, if I don’t get an error log I have to ask for browser info from the user and this is an exercise in and of itself. I need to be aware of browser plugins, state of cache (last time cleared), the user and steps used to trigger the issue… and more. After I get the basics necessary for most browser based issues, I start asking more questions specifically related to the scenario that is bombing. Here in lies the problem. Asking users for info usually results in surface data that don’t help get down to the nuts and bolts of the problem. Also, I don’t have a test lab readily available to me so spinning up a browser configured like the user is not possible.

It is hard to get the info you need without stepping through a debug session to see what is going on.  I sometimes have to result to adding additional instrumentation to code to hopefully get better error messages or state info, but this requires a push to production and the user having to wait until all of this is done. So, how can I analyze an issue without this information?

I know there are solutions out there that will record live user sessions in production and even record the stack trace down to the method that caused the issue. Unfortunately, I don’t have access to any of these tools. I guess I will have to just get better at asking questions that even tech illiterate users can follow.

IIS HTTP Error 503. The service is unavailable.

Today I received an error while trying to browse an ASP.NET website application locally, “HTTP Error 503. The service is unavailable.”

Fortunately, I found an excellent post on troubleshooting IIS issues, Troubleshooting IIS7 503 “Service unavailable” errors with startup debugging by Mike Volodarsky. I am just documenting some of the things that were a little different for me. I am running Windows 7 and IIS 7. I downloaded Windows Software Development Kit (SDK) for Windows 8 from http://msdn.microsoft.com/en-US/windows/hardware/hh852363 and only installed the debugger to the default location. This made the path to the debugger executable  C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\ntsd.exe. I used regedit to add the w3wp.exe Registry Key (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe) with the Debugger string value of C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\ntsd.exe -server npipe:pipe=w3wp -g. I also added C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\windbg.exe to my Path to make it easy to start in command line.

Unfortunately, I didn’t need to go through all of that. I never got the debugger to attach because the Worker Process wouldn’t start. After some experimenting I decided to change the Application Pool for my web application and the app came up. I looked at the old Application Pool and noticed something that didn’t hit me before. I spent 2 hours researching and poking around and my real error in this instance was the Application Pool I created for my web application was assigned to my local Windows account. When I was forced to update my password the password did not update on the Application Pool. So, updating the password fixed my issue. I have since created a new Windows account specifically for this web app that doesn’t have a password that expires.

Anyway, it was a nice exercise getting windbg up and running. Maybe I will really need it one day.

Fun Stuff!

*Safety Note: Use the Windows Debugger Install instructions above at your own risk. It may affect your Visual Studio Debugger.