I should have posted this days ago, but hack.summit starts today. If you don’t know, hack.summit is now the largest developer conference in the world. It features programming language creators, open-source contributors, and thought leaders from around the world. The conference is free, if you share it on social media, but giving a donation is the best way to give back to the development community.
Anyway, check it out, register, donate, and I’ll see you online.
I was on a project to improve an application deployment process that used zip files for packaging the applications. Zips are good. They allow you to package and compress files into one bit sized file, but there is so much more to be had with a dedicated package solution. Maven, gem, wheel, npm, cpan, rpm, deb, nuget, chocolatey, yum… the list goes on and with so many options to provide an improved package for deployment its hard to justify using plain old zips.
Since this was a .Net project I focused on NuGet. NuGet is itself a zip file, but a zip on steroids. Zip provides the compression and NuGet adds additional meta data and functionality.
Enough of the sales pitch. Actually, one problem that I had with using NuGet alone was no easy way to validate the package through checksum. So, in addition to NuGet, using a dedicated artifact repository solution like Artifactory gives an added layer of comfort. A good paper, although biased, on Artifactory can be found here.
Happy Packaging!
We were faced with an interesting problem. We want to build once and deploy the build to multiple environments for testing, staging and ultimately consumption in production. Well in addition to build once, we also want to allow remote debugging. We need to build in debug mode to get pdb generated and have other configurations that will allow debugging. Yet, we don’t want to deploy a debug build to staging or production. What do we do?
The thought right now is to do a debug build and create to packages, one for debug and one for release. To do this we would have to strip out the pdb and turn on debugging in the release package. So, we still have one build, but we have two flavors of packages.
I am not yet sure if this is viable, hence the reason I am blogging this out. First I need to fully understand the difference between a debug and release build in MSBuild. I have an idea, but I need to verify my assumptions.
What I have found is the main difference between debug and release building are:
PDB files are symbol databases that allow a debugger to map machine code to source code (actually MSIL) so you can set break points in source code and the debugger can halt execution of the machine code. This is probably a terrible explanation, but you should get the gist.
JIT optimizations are things the compiler does to speed up the execution of your code. It may reorganize loops to make them run faster and other little magic tricks that happen under the cover that we usually never have to worry about.
Scott Hanselman has an interesting post on this, http://www.hanselman.com/blog/DebugVsReleaseTheBestOfBothWorlds.aspx. This posts suggest that you could do a release build and configure the runtime with an ini file that would determine if JIT optimizations are performed or tracking information is generated.
http://msdn.microsoft.com/en-us/library/9dd8z24x(v=vs.110).aspx this post explains more about the ini.
After doing this research I learned a lot about building .Net code, but I also realized that I am taking this a little to far. My primary goal is that we build our application once and use that build in multiple environments to get it tested and deployed to production. When we need to do a remote debug we are using researching an issue and there is no reason that we couldn’t flip a switch on one particular build so that it builds in debug mode, deploy it to a test environment, debug it, make some fixes after finding the cause of the issue we are debugging, then flip the switch back to release and build again, this time allowing the build to go all the way to production.
The problem here is that we need to make sure that we do not allow debug builds to make it into production. My initial thought is to mark debug builds with a version that is tagged with DEBUG. Then I can have logic in the production deploy that checks for the DEBUG tag and fail the deploy if it is present. We can do the same for pdb files and web.config. Specifically, check for inclusion of pdb (we shouldn’t have pdb file in production). We can also have logic that checks for debug=true and other configurations that we don’t want leaking into production.
We would have to alter our deployment pipeline to add a job that will do these checks based on the environment being deployed. We would have to also look at maybe putting debug builds in a different artifact repository to keep them segregated from release candidates. This would also cause another change to the deployment pipeline where we check the release candidate or debug repository based on some setting.
This would be a lot of changing to our pipeline, but I believe it is worth it in the long run. It also prevents us from leaking manual processes into how we build and deploy the app.
I have been setting up build servers and I have been exploring automating the process. So, I have been scripting every step I take to stand the servers up. In this post I will sharing some of the commands I use to create GoCD Agents. If you decide to go down this road, you should think about creating reusable scripts and parameterize the things that change (I didn’t want to do all the work for your :). Also, it would make sense to use a configuration manager like DSC, Puppet or Chef to actually run the scripts.
I am using PowerShell remotely on the build servers, which is indicated by [winbuildserver1]: PS> in the command prompt. Check out my previous post to learn more about configuring remote servers with PowerShell.
The first thing I do is copy the install files from the artifact repository to the server.
[winbuildserver1]: Copy-Item -Path \\artifactserver\d$\repository\Go-Agent\go-agent-14.1.0-18882\go-agent-14.1.0-18882-setup.exe -Destination "D:\install-temp\" -Recurse -Force
[winbuildserver1]: PS>([WMICLASS]"Win32_Process").Create("D:\install-temp\go-agent-14.1.0-18882-setup.exe /S /SERVERIP=<ip of go server> /GO_AGENT_JAVA_HOME=<path to JRE> /D=D:\Go Agents\Internal\1\")
Here we are getting a reference to the static WMI class “Win32_Process”, call the create method passing the command line to install an agent (http://www.thoughtworks.com/products/docs/go/current/help/installing_go_agent.html). In the command line we have
If I want to run multiple agents on the same server I do a little extra work to get the other agents installed.
[winbuildserver1]: PS> Copy-Item "D:\Go Agents\Internal\1\*" -Destination "D:\Go Agents\PCI\1"
[winbuildserver1]: PS> Remove-Item "D:\Go Agents\PCI\1\config\guid.txt"
[winbuildserver1]: PS> Remove-Item "D:\Go Agents\PCI\1\.agent-bootstrapper.running"
Here we are just copying an installed agent to a new location and removing a couple files to force the agent to recreate and register itself with the server.
Lastly, I create a service for the agent.
[winbuildserver1]: PS> New-Service -Name "Go Agent PCI 1" -Description "Go Agent PCI 1" -BinaryPathName "`"D:\Go Agents\PCI\1\cruisewrapper.exe`" -s `"D:\Go Agents\PCI\1\config\wrapper-agent.conf`""
Get more on using PowerShell to configure services in my previous post.
I use similar commands to install the server, plug-ins, and other tools and services (e.g. Git, SVN, NuGet…) that I need on the build server. I have to admit that this isn’t totally automated yet. I still have to manually update the service account, credentials and manually accept a certificate to get SVN working with the agent, but this got me 90% done. I don’t have to worry about my silly mistakes because the scripts will do most of the work for me.
I wanted to use some of the GoCD Environment Variables in my build scripts, unfortunately finding info on how to do that was limited or my search skills lacking.
Anyway, to use a Pipeline Parameter you would tokenize the parameter like so:
#{ParameterName}
To use a GoCD Environment Variable you would tokenize the variable like this:
%VariableName%
[go] Could not fetch artifact https://127.0.0.1:8154/go/remoting/files/pne.test.build/127/Build/1/Build/cruise-output/PreTest.PreTest.nant.log.xml?sha1=8899RvS5mElcpqSju5FdfoYPUQU%3D. Pausing 19 seconds to retry. Error was : Unsuccessful response '404' from the server
This error stumped me for a while. The reason it stumped me is because of the IP addrdess and port, my Go Server is not located there. The Go Agent is not on the same server as the Go Server, so it shouldn’t be using a local IP. The agent configuration is properly pointed to the Go Server’s IP and port. I assumed that the 404 was because of the incorrect IP and port and I did a lot of research and digging trying to correct it.
I finally figured out that this error is simply stating that the file was not found.
I am not sure why the wrong IP and port is reported, but when the file in the error was added to the artifacts on the server, the error went away.
The world is a change’n.
From the horses mouth, http://blogs.msdn.com/b/dotnet/archive/2014/11/12/net-core-is-open-source.aspx. They also have a GitHub repo, but it currently doesn’t include everything – https://github.com/dotnet/corefx.
This is very exciting news for .Net developers as this opens up our .Net skills to be eventually applied on other platforms (Linux, MacOS, iOS, Android…). Also, getting the opportunity to browse the code base and learn from it is appealing. Hopefully, I get a chance to contribute.
I have a Git material setup that uses a public git repository (private repos are another animal).
<materials> <git url="http://mydomain/MyRepo.git" materialName="myrepo.git" /> </materials>
When I trigger the pipeline that uses this material it results in an error.
[go] Start to prepare build/5/Build/1/Build on mygoserver [D:\Go Agents\Internal\1] at Fri Oct 24 08:44:49 EDT 2014 [go] Start updating files at revision b0b18a838a108a208003178fb17e8769edf9587c from http://mydomain/MyRepo.git Error while executing [git config remote.origin.url] Make sure this command can execute manually. [go] Job completed build/5/Build/1/Build on mygoserver [D:\Go Agents\Internal\1] at Fri Oct 24 08:44:49 EDT 2014
Looking at go-agent.log I see that there was a problem executing the git command.
…blah blah 2014-10-24 08:27:48,616 [loopThread] ERROR thoughtworks.go.work.DefaultGoPublisher:142 - Error while executing [git config remote.origin.url] Make sure this command can execute manually. java.lang.RuntimeException: Error while executing [git config remote.origin.url] Make sure this command can execute manually. …blah blah Caused by: com.thoughtworks.go.util.command.CommandLineException: Error while executing [git config remote.origin.url] Make sure this command can execute manually. …blah blah Caused by: java.io.IOException: Cannot run program "git" (in directory "pipelines\build"): CreateProcess error=2, The system cannot find the file specified …blah blah
When I set the material up I got a successful test connection. So an error saying that it can’t find the git command is somewhat perplexing. Does the Git connection test use a different git than the pipeline material processor. When I installed msysgit I manually added the git bin folder to the PATH. I could run git in the command window and on git bash.
After some hair pulling I decided to re-install msysgit and this time to use the evil option that has the red danger sign.
Notice that it says it will override Windows tools, like find.exe and sort.exe. Now, I have to remember that these tools are busted when the server ever has to run a script that needs these. I am not sure of any other changes, but it looks like it is a PATH change. Instead of just having {gitinstall}\bin it also adds {gitinstall}\cmd.
When I restart the Go Server and Agent and try again… IT WORKED!!!
If you are using a Windows Server with Go and you want to use Git Materials, you may need to allow git to override some of your Windows tools and remember that you allowed Git to break said tools when problems arise…and this will arise.
This question is more for change tickets (e.g. bug fixes, feature enhancements…not new features). I am still fairly new at testing, so I have been asking a lot of questions and this was one of them. We can spend time exploring every aspect of a feature that may have been affected by a change, but does that provide the best use of my time. We only have a limited amount of time to do testing so every minute counts. To insure we are providing value to the team we need to focus on getting the best bang for those precious minutes. I believe we should test the ticket by focusing on the most risky aspects of a change. After reporting the initial risk assessment then we can focus on additional manual testing as time permits. After doing the initial manual testing, decide if any of the manual tests are valuable enough to be a candidate for automation. If so, add them to the list of tests to automate.
By testing the ticket I mean a manual exploration of the changes to a feature to insure the intent of the ticket was satisfied and its highest risks evaluated. Yes, manual. Even though I am responsible for automated testing (or checking), I first have to understand how to test manually. The automation are just running checks like a line of robots in an assembly line, it runs the same checks over and over, no real variation, just what is coded in the test.
Unfortunately, our users aren’t robots and they don’t follow the test script when using the application so manual testing is absolutely necessary to add variability like our users will. Yet, automation is a time saver for mundane tests like regression. So after we are done with manual testing, we evaluate the tests to decide if any of them are valuable enough to automate and add them to an automated test suite (e.g. regression, smoke, performance…).
As the ticket progresses through the delivery pipeline, at each stage we run additional test. This way additional feedback can be had at each stage and we also increase test coverage the closer the ticket gets to delivery. By not trying to rely only on automation, we also benefit from early feedback from the manually exploration of the ticket changes. I believe we can manually explore them than automate them (IMHO). So, as soon as a ticket is ready for testing we begin exploration. A ticket is only ready after it has passed a manual code review, automated build, automated unit test, and automated code analysis. So, we have multiple gates that a ticket must pass and fast feedback at each step.
That was probably an obvious answer to seasoned testers, but it may not be pondered by new testers.The point is to focus at first on the change in the ticket as this will be the fastest way to provide feedback on the risks in delivering the ticket. Then move on to running longer running tests as the ticket moves closer to being deployed to production. Fast feedback is the name of the game.
I just received my certification as a NationBuilder Expert and I wanted to give something to the NationBuilder community to celebrate :). As I thought about something I could do, I remembered that I have a customer that wants to sell products in their nation. So, I will share that solution. I warn you, it is not for the technically challenged or the faint of heart.
At the moment there isn’t an easy way to sell products and services on NationBuilder websites. So, here is how I approached this problem, but the instructions may not be applicable in the future once NationBuilder makes changes to the system. Please note that charging tax and shipping is an issue with this approach, so you may want to rethink your usage of it. If you need help setting this up, leave a comment or find me in the NationBuilder Expert directory.
We will use the following page types to mimic a shopping cart experience:
We want to be able to track purchases and alert certain people about purchases so we have to configure certain items to allow this.
Create a tracking code for purchases made through your shop (e.g. products).
Create a new Contact Type (e.g.“Purchase”). This will be used to create a custom way to alert someone to trigger product shipping or service provisioning.
Create a Thank You page as a Basic Page. You can make this an empty page for now. We just need it ready when we want to point the Shopping Cart to it.
Create a new Donation page. Set the page for “one time payment” unless you want to allow people to make installment payments.
Note: Installment payments may be hard to track, but I haven’t tried. If you offer installment payments and you only ship once the final payment is made, you are effectively offering layaway. If you ship immediately and allow your customer to make installments, you are basically offering them credit. In both instances there is a lot to think about and track and I am not sure if NationBuilder in its current form would help you do either effectively.
The rest of the options are up to you. This is NationBuilder so there are many more options you can set, but this should get you started with an OK shopping cart page.
Next, create an Event page for each production you want to sell.
In the product catalog and product page templates, you can add a link to your shopping cart page. Alternatively, this can be added to the website template or you can add the page to the navigation. There should be a way for your visitors to get back to the shopping cart page.
<div><a href="/shopping_cart">View Cart</a>
Of course you should add a class so you can add some flair. If you add the link to the individual product page templates, as you create new product pages you may want to clone them so you don’t have to manually add the link.
There is a message that displays when a product is added to the cart. It references tickets so I also replace that message. This is the message, “Thanks for coming! Now please complete your order, your tickets have been automatically added to the page below.”
I use a technique discussed on this post, . Basically, in your _flash.html template you override the default message that mentions tickets to one that talks about items (more generic).
Replace
{{ flash.message }}
with
{{ flash.message | replace: 'Thanks for coming! ', '' | replace: 'Now please complete your order, your tickets have been automatically added to the page below.','Thank you. Your items have been added.' }}
I had to do the nasty double replace for some reason or it wouldn’t work.
I can’t give a public demo, because I’m lazy, but below are some screen shots of an example I worked through as I wrote this post. You will have to pretend that I took the time to properly layout and style the pages :).
Product Catalog
Imagine product pictures and descriptions with a view details or buy now buttons instead of just text links.
Imagination is needed here too.
Imagine the items laid out in a table and other niceties. Again, charging tax and shipping is an issue so you may want to rethink your usage of this. This could also use a subtotal (a little JavaScript magic may be in order). Once user is ready for purchase, they would follow the normal workflow for donation payments.
This is not a robust system for product and service sales, but it works. There is a lot to optimize and make better if sales is your primary on your site, you need another solution. You can’t use it to manage a large catalog of products as maintaining the products and the associated pages would be a nightmare. Every time you need to add, edit or remove a group of products you have to do it on each individual Product (Event) page. Yet, it gives you a way to offer a few products for sale on your Nation until the new feature for product sales is ready on NationBuilder.
Decoupled Logic 