Tagged: GoCD

Why Would You Have a Go.cd Pipeline with No Materials?

So, I need to run a Go pipeline that isn’t dependent on a material, meaning Go’s out the box implementation of a material. What I’m talking about is a dumb hack to get a production deployment working in an environment with stringent security policies. We have two Go servers. One orchestrates pipelines in our pre-production environments. Because of the security concerns we have a second Go server just to deploy production. When a package has gone through all of the quality gates in pre-production it is ready to deploy to production and the deployable packages are placed in an artifact repository (right now a file share) that is accessible by the production Go server.

When we want to deploy production, we manually trigger a pipeline on the production Go server that kicks off a job to get the deploy package from the package repository and place it in the production Go server’s artifact repository. With the package in the production, all of the production deploy agents have access to deploy it. Normally, this “get files from a share” business would be handled by a material setup in Go, but I couldn’t find one that could poll a file share and sticking these binaries in a source repo that was accessible to the preproduction and production domains sounded like overkill at the time.

What would be even better and make me fall in love with Go even more would be if two server could talk to each other then I could just have the production Go server poll the preprod server to see if it has a new package in some pipeline. I guess I could

  • do a little Java code and do some type of file share repository poller
  • setup Artifactory or NuGet and use the pollers already build for them
  • explore the Go API and see if I could do a service to have the servers talk to each other through the API
  • spend a little more time on Google and Github to see if this has already been solved

Because I’m not fluent in Java, much of this felt like a trip down the rabbit hole and I was already shaving a Yak with trying to get this supposedly quick hack up. Yet, what I did was the simplest thing I could think of to get going.

Anyway, I finally set up a fake git repo and ignored that material I setup to point to it. First setting up Git: initializing a repo, creating a bare repo, cloning the initialized repo into the bare repo and configuring the bare repo as a material in Go. Then on the Stage config there is an option for “Fetch Materials”. Go will only do material updates and checkouts if this is checked, so I unchecked it. On the Material config there is an option for “Poll for new changes”. This will stop Go from polling the material for changes, but you still have to set one up. It has to be a valid material. You can’t just put “myfakegitrepo” in the URL, I tried… it failed.

So, although you can’t get away with not using a material you can make the material insignificant after it is setup. I hope this doesn’t help anyone, if it does you are probably doing something complex and wrong like me.

GoCD: Install Multiple Agents with Powershell, Take 2

I wrote about how to Automate Agent Install with PowerShell and thought I would provide the script I am using now since I recently had to deploy some new agents. The script is below and it is pretty self explanatory and generally follows my previous blog post and the Go.cd documentation.

We basically, copy an existing agent to a new location, remove some files that are agent specific, and create a Windows service to run the agent. Until I feel the pain of having to do it, I set the service account/password and start the service manually. Also, I configure the agent on the server manually through the Go.cd UI. When I have to install more agents I will probably automate it then.

$currentAgentPath = "D:\Go Agents\Internal\1";
$newAgentName = "Go Agent Internal 3";
$newAgentPath = "D:\Go Agents\Internal\3\";

Write-Host "Copying Files"
Copy-Item "$currentAgentPath\" -Destination $newAgentPath -Recurse;

Write-Host "Deleting Agent Specific Files"
$guidText = "$newAgentPath\config\guid.txt";

if (Test-Path $guidText)
{
 Remove-Item $guidText;
}

Remove-Item "$newAgentPath\.agent-bootstrapper.running";

Write-Host "Create Agent Service"
New-Service -Name $newAgentName -Description $newAgentName -BinaryPathName "`"$newAgentPath\cruisewrapper.exe`" -s `"$newAgentPath\config\wrapper-agent.conf`"";

#$credential = Get-Credential;
#Eventually, we will write a function to set the service account and password and start the service would be nice to have a way to automatically configure the agent on the server too.

I guess I decided to do the work for you 🙂

Enjoy

GoCD: Agent Running Web Driver Test Hangs After Test Failure [SOLVED]

Problem

I had a nagging issue were some tests were failing the build on our GoCD server, but the agent was not reporting the failure to the server. We are using NAnt to run NUnit tests that in turn call Web Driver to exercise web pages. There were some test failures that correctly returned a non-zero value that failed the build in NAnt. Also, the failure is captured in the log and saved in a text file. Yet, the agent didn’t report the build failure or send the artifacts to the server.

Issue

After a 2 day search for answers and a deep dive into the bowels of GoCD I discovered that a Web Driver process was kept open after the test fails the build. Specifically, the process is IEDriverServer.exe. This process was being orphaned by improper cleanup in the tests that resulted in the Web Driver and browsers staying open after the test failure.

When I ran the tests again, I watched for the failure then manually killed Web Driver and the agent magically reported to the server. I am still unsure why Web Driver would prevent the GoCD agent from reporting to the server. They are both Java processes, maybe there is something going on in the JVM or something… not sure.

Solution

My work around at the moment is to run a task killer on failure in the test script. Here is the relevant portion of the nant script that drives the tests:

<property name="nant.onfailure" value="test.taskkiller" />
<target name="test.taskkiller">
 <exec program="taskkiller.bat" failonerror="false">
 </exec>
 </target>

The taskkiller.bat is just a simple bat file that will kill Web Driver and open browsers.

taskkill /IM IEDriverServer.exe /F
taskkill /IM iexplore.exe /F

Now this is just a band-aid. We will be updating our test framework to handle this. Additionally, killing all the processes like this isn’t good if we happen to be running tests in parallel on the agent, which may be a possibility in the future.

GoCD: Automate Agent Install with PowerShell

I have been setting up build servers and I have been exploring automating the process. So, I have been scripting every step I take to stand the servers up. In this post I will sharing some of the commands I use to create GoCD Agents. If you decide to go down this road, you should think about creating reusable scripts and parameterize the things that change (I didn’t want to do all the work for your :). Also, it would make sense to use a configuration manager like DSC, Puppet or Chef to actually run the scripts.

I am using PowerShell remotely on the build servers, which is indicated by [winbuildserver1]: PS> in the command prompt. Check out my previous post to learn more about configuring remote servers with PowerShell.

Copy Install Files

The first thing I do is copy the install files from the artifact repository to the server.

[winbuildserver1]: Copy-Item -Path \\artifactserver\d$\repository\Go-Agent\go-agent-14.1.0-18882\go-agent-14.1.0-18882-setup.exe -Destination "D:\install-temp\" -Recurse -Force

Install Agent

[winbuildserver1]: PS>([WMICLASS]"Win32_Process").Create("D:\install-temp\go-agent-14.1.0-18882-setup.exe /S /SERVERIP=<ip of go server> /GO_AGENT_JAVA_HOME=<path to JRE> /D=D:\Go Agents\Internal\1\")

Here we are getting a reference to the static WMI class “Win32_Process”, call the create method passing the command line to install an agent (http://www.thoughtworks.com/products/docs/go/current/help/installing_go_agent.html). In the command line we have

  • the path to the install file
  • /S switch for silent install (no user prompts)
  • /SERVERIP switch for the IP of the Go Server (this is optional)
  • /GO_AGENT_JAVA_HOME switch for the path to the JRE (this is optional)
  • /D switch is the path to location you want to install the agent.

Run Multiple Agents on Same Server

If I want to run multiple agents on the same server I do a little extra work to get the other agents installed.

[winbuildserver1]: PS> Copy-Item "D:\Go Agents\Internal\1\*" -Destination "D:\Go Agents\PCI\1"
[winbuildserver1]: PS> Remove-Item "D:\Go Agents\PCI\1\config\guid.txt"
[winbuildserver1]: PS> Remove-Item "D:\Go Agents\PCI\1\.agent-bootstrapper.running"

Here we are just copying an installed agent to a new location and removing a couple files to force the agent to recreate and register itself with the server.

Create Agent Service

Lastly, I create a service for the agent.

[winbuildserver1]: PS> New-Service -Name "Go Agent PCI 1" -Description "Go Agent PCI 1" -BinaryPathName "`"D:\Go Agents\PCI\1\cruisewrapper.exe`" -s `"D:\Go Agents\PCI\1\config\wrapper-agent.conf`""

Get more on using PowerShell to configure services in my previous post.

Conclusion

I use similar commands to install the server, plug-ins, and other tools and services (e.g. Git, SVN, NuGet…) that I need on the build server. I have to admit that this isn’t totally automated yet. I still have to manually update the service account, credentials and manually accept a certificate to get SVN working with the agent, but this got me 90% done. I don’t have to worry about my silly mistakes because the scripts will do most of the work for me.

GoCD: Environment Variables in Build Scripts

I wanted to use some of the GoCD Environment Variables in my build scripts, unfortunately finding info on how to do that was limited or my search skills lacking.

Anyway, to use a Pipeline Parameter you would tokenize the parameter like so:

#{ParameterName}

To use a GoCD Environment Variable you would tokenize the variable like this:

%VariableName%

GoCD: 404 Error Fetching Artifact [SOLVED]

Problem

[go] Could not fetch artifact https://127.0.0.1:8154/go/remoting/files/pne.test.build/127/Build/1/Build/cruise-output/PreTest.PreTest.nant.log.xml?sha1=8899RvS5mElcpqSju5FdfoYPUQU%3D. Pausing 19 seconds to retry. Error was : Unsuccessful response '404' from the server

This error stumped me for a while. The reason it stumped me is because of the IP addrdess and port, my Go Server is not located there. The Go Agent is not on the same server as the Go Server, so it shouldn’t be using a local IP. The agent configuration is properly pointed to the Go Server’s IP and port. I assumed that the 404 was because of the incorrect IP and port and I did a lot of research and digging trying to correct it.

Issue

I finally figured out that this error is simply stating that the file was not found.

Solution

I am not sure why the wrong IP and port is reported, but when the file in the error was added to the artifacts on the server, the error went away.

GoCD: Error updating Git material on Windows Server [SOLVED]

Problem

I have a Git material setup that uses a public git repository (private repos are another animal).

<materials>
  <git url="http://mydomain/MyRepo.git" materialName="myrepo.git" />
</materials>

When I trigger the pipeline that uses this material it results in an error.

[go] Start to prepare build/5/Build/1/Build on mygoserver [D:\Go Agents\Internal\1] at Fri Oct 24 08:44:49 EDT 2014
[go] Start updating files at revision b0b18a838a108a208003178fb17e8769edf9587c from
http://mydomain/MyRepo.git
Error while executing [git config remote.origin.url]
 Make sure this command can execute manually.
[go] Job completed build/5/Build/1/Build on mygoserver [D:\Go Agents\Internal\1] at Fri Oct 24 08:44:49 EDT 2014

Error

Looking at go-agent.log I see that there was a problem executing the git command.

…blah blah
2014-10-24 08:27:48,616 [loopThread] ERROR thoughtworks.go.work.DefaultGoPublisher:142 - Error while executing [git config remote.origin.url]
Make sure this command can execute manually.
java.lang.RuntimeException: Error while executing [git config remote.origin.url]
 Make sure this command can execute manually.
  …blah blah
Caused by: com.thoughtworks.go.util.command.CommandLineException: Error while executing [git config remote.origin.url]
 Make sure this command can execute manually.
  …blah blah
Caused by: java.io.IOException: Cannot run program "git" (in directory "pipelines\build"): CreateProcess error=2, The system cannot find the file specified
…blah blah

When I set the material up I got a successful test connection. So an error saying that it can’t find the git command is somewhat perplexing. Does the Git connection test use a different git than the pipeline material processor. When I installed msysgit I manually added the git bin folder to the PATH. I could run git in the command window and on git bash.

Solution

After some hair pulling I decided to re-install msysgit and this time to use the evil option that has the red danger sign.

GitInstallOption

Notice that it says it will override Windows tools, like find.exe and sort.exe. Now, I have to remember that these tools are busted when the server ever has to run a script that needs these. I am not sure of any other changes, but it looks like it is a PATH change. Instead of just having {gitinstall}\bin it also adds {gitinstall}\cmd.

When I restart the Go Server and Agent and try again… IT WORKED!!!

Conclusion

If you are using a Windows Server with Go and you want to use Git Materials, you may need to allow git to override some of your Windows tools and remember that you allowed Git to break said tools when problems arise…and this will arise.

GoCD: Integrating Bug Tracking

Go.cd allows you to integrate with your tracking tools. You can define how to handle ticket numbers in your source control commit messages. Go.cd parses the message looking for a specific pattern and transforms matches into links to your tracking tool.

We use AxSoft OnTime and integration wasn’t as straight forward as I envisioned. OnTime uses two query strings to identify the ticket number and the type of ticket (defect, feature, incident…).

View Defect: https://name.ontimenow.com/viewitem.aspx?id=102578&type=defects

Create Defect: https://name.ontimenow.com/edititem.aspx?type=defects

From what I can tell Go.cd only allows use of one parameter and no facility to expand the regex and parameter to work with the various patterns for the type of ticket. Example: we may have a defect ticket OTD 102578 and a feature OTF 87984. When commits are related to either one of these tickets, the ticket number, including the prefix, is added to the commit message. To turn this into a link in Go.cd, we have to parse the OT* and correlate that to defect or feature, depending on the value of *, and add that to the type query string parameter. Next we have to grab the numbers after the space after the ticket type prefix and add that to the id query string parameter.

I am not really sure how to get this to work, aside from messing around with the source code. Did I say source code? Solved! Well maybe…tune in.

GoCD: Pipeline Parameters

I am currently getting a Go Continuous Delivery Server stood up and I am reusing scripts from our CCNET Continuous Integration server to save time getting the new server up. In doing this, I am also reviewing the scripts for improvements.When defining builds for my shinny new Go server I noticed that I have duplicate properties across tasks when calling Nant targets.

I have a property that defines the server environment the task should work with. This basically, sets properties in the script so that they target the correct environment. If I set the property to dev, the script will set the server names, paths and more to point to the dev environment.

There is a property that tells the task what source code repository branch to use. In the context of a task, this mainly has to do with paths to the branch already checked out and updated on the Go server and not controlling the actual branch updates. The branch name is concatenated in a common path so the task knows where to get and save source files.

There are more duplicated properties, but it got me wondering if there is a better way to do this than repeating the same value over and over for each task. When I need a new task or a new pipeline with different values this can become a maintenance nightmare. Not to mention, this duplication will make it hard to take advantage of Go Pipeline Templates so I need to solve this, if I want to gain the ease in creating new pipelines that templates afford.

Go has the concept of Environment Variable that is a common value saved on an agent. That is a nice feature as it lets me define values common to every pipeline running on an agent targeting a specific environment. To fix my issue I need to be able to set values at the job level. The agent level is to broad as I can have multiple pipelines target specific agents with different values for the properties I want to abstract. I wonder if we can set up a common variable at the job or pipeline level that the tasks can use?

Go Pipeline Parameters

Oh look, there is something called a Parameter in the pipeline configuration.

Parameters help reduce repetition within your configurations and combined with templates allow you to setup complex configurations. More..

Well let’s look into that and see what we come up with. The “More” link above gives the details of how to use parameters in Go. Basically, you define a parameter then you can use them in your tasks. To use the parameter in your task, just wrap it with #{}. Example: I have a parameter named ENV and I would use it in my script by tokenizing it like so, #{ENV}.

use_go_pipeline_parameter_in_task

This gets rid of a bunch of unnecessary duplication, improves the maintainability of build scripts, and opens up the possibility of creating templates.

Conclusion

There is so much more to understand about Go. One thing that was reinforced in this exercise was to remember to constantly look for pain points in solutions and processes and search for ways to cure the pain.

ThoughtWorks Go Continuous Delivery, Now In Open Source Flavor

If you haven’t heard the ThoughtWorks Go Continuous Delivery server is now open source. The source code is located on GitHub, https://github.com/gocd/gocd. I decided to give it a test drive and I was pleased. Since I am primarily a Windows developer my point of reference is CCNET (which is based on ThoughtWorks Cruise Control Continuous Build server), TFS Team Build and TeamCity. I don’t have a lot of TeamCity experience, but I can say that I can easily see automating many scenarios in Go that I was having a hard time conceiving in CCNET. Adding the concept of Environments, Pipelines, Stages, and User Roles opened an easier path to automated production deployment for me.

Install

Install was pretty simple. Go is cross platform, but I have a Windows server. I downloaded the Windows packages from http://www.go.cd/download/. I installed the server and agent on my server, opened it up in a browser, and it was there ready to go. Very easy, only a few minutes of clicking and I was ready to start. Before I started building pipelines, I made a few customization for my environment. I want to use Git and NAnt in my build, test, deploy process so I added the path to their exe’s to the Path (Windows system environment variable). This makes it less painless to run them from Go.

Server User Authentication

I am eventually going to use LDAP for user authorization, but for now I setup an htpasswd file with usernames and SHA-1 hashed passwords. Then I entered the name of the file, htpasswd.txt, in the server configuration (Admin > Server Configuration > User Management > Password File Settings). I generated the contents for the SHA-1 hashed password file on http://aspirine.org/htpasswd_en.html, but I could have easily just used a Crypto library to hash the passwords myself. Usernames are not case sensitive, but you shouldn’t have colon, spaces or equal sign in the username unless you escape them with backslash.

Configuration

The Go configuration is stored in an XML file, like Jenkins and CCNET. I know many people have a disdain for XML, but it doesn’t bother me and it makes Go portable. I can deploy it to another server or even a developer workstation, use a common config file, and its ready to start processing pipelines. You can use the UI to configure most of what you want to do, but I enjoy the fine grain control in editing the XML directly. There is an XML validator so when my error prone fingers type the wrong character it will automatically reject the change and continue using the current configuration. Since the configuration is XML, I decided to put the file under source control. The reason for this is to have a backup of the config and to allow the ability to config the server from XML and automatically push the changes to the Go server with the Go server (sweet). This doesn’t work both ways, so if there are changes made through the UI they won’t be pushed to source control (although I can envision some convoluted solution for this). For now, I am the only person managing the server and I will configure through the XML file and not the UI.

Pipelines

Pipelines are a unit of organization in Go. A pipeline is comprised of stages which is comprised of jobs which is comprised of tasks. Tasks are the basic unit of work in Go. In my instance most of my tasks are NAnt tasks that call targets in nant build scripts. There are all kinds of ways to create chains of actions and dependencies. This is going to probably be where I focus a lot of attention as this is where the power of the system lies, IMHO. Being able to customize the pipelines and wire up various dependencies is huge for me. Granted I could do this in CCNET to a certain degree, but Go just made it plain to envision and implement.

NAnt Problems

Working with Nant was a pain. Actually, this was the only major hurdle I had to cross. I couldn’t figure out how to pass properties to the nant build file. Then I decided to try to pass the properties through the target argument of the Go nant task, like this

<nant buildfile=”testcode\test\this-is-my-buildfile.xml” target=”-D:this-is-a-nant-property=&quot;dev&quot; -D:another-nant-property=&quot;jun&quot; ThisIsMyNantTarget” />

Note: Paths are relative to your Agent pipeline working directory.

This worked great, but a more intuitive way of doing this would have been good. Maybe an arguments property so there is no confusion between nant property and nant target.

Conclusion

I know this post is lite on details, but I just wanted to get a quick brain dump of my experience with Go. Go has pretty good documentation on the Go.cd website and posting questions to support elicited pretty fast feedback for a free product. I am excited to get involved with the Go and the Go Community. Overall, it was very easy to get a powerful Continuous Delivery server up and running in no time. You should check it out.