Tagged: ThoughtWorks Go

Why Would You Have a Go.cd Pipeline with No Materials?

So, I need to run a Go pipeline that isn’t dependent on a material, meaning Go’s out the box implementation of a material. What I’m talking about is a dumb hack to get a production deployment working in an environment with stringent security policies. We have two Go servers. One orchestrates pipelines in our pre-production environments. Because of the security concerns we have a second Go server just to deploy production. When a package has gone through all of the quality gates in pre-production it is ready to deploy to production and the deployable packages are placed in an artifact repository (right now a file share) that is accessible by the production Go server.

When we want to deploy production, we manually trigger a pipeline on the production Go server that kicks off a job to get the deploy package from the package repository and place it in the production Go server’s artifact repository. With the package in the production, all of the production deploy agents have access to deploy it. Normally, this “get files from a share” business would be handled by a material setup in Go, but I couldn’t find one that could poll a file share and sticking these binaries in a source repo that was accessible to the preproduction and production domains sounded like overkill at the time.

What would be even better and make me fall in love with Go even more would be if two server could talk to each other then I could just have the production Go server poll the preprod server to see if it has a new package in some pipeline. I guess I could

  • do a little Java code and do some type of file share repository poller
  • setup Artifactory or NuGet and use the pollers already build for them
  • explore the Go API and see if I could do a service to have the servers talk to each other through the API
  • spend a little more time on Google and Github to see if this has already been solved

Because I’m not fluent in Java, much of this felt like a trip down the rabbit hole and I was already shaving a Yak with trying to get this supposedly quick hack up. Yet, what I did was the simplest thing I could think of to get going.

Anyway, I finally set up a fake git repo and ignored that material I setup to point to it. First setting up Git: initializing a repo, creating a bare repo, cloning the initialized repo into the bare repo and configuring the bare repo as a material in Go. Then on the Stage config there is an option for “Fetch Materials”. Go will only do material updates and checkouts if this is checked, so I unchecked it. On the Material config there is an option for “Poll for new changes”. This will stop Go from polling the material for changes, but you still have to set one up. It has to be a valid material. You can’t just put “myfakegitrepo” in the URL, I tried… it failed.

So, although you can’t get away with not using a material you can make the material insignificant after it is setup. I hope this doesn’t help anyone, if it does you are probably doing something complex and wrong like me.

GoCD: Integrating Bug Tracking

Go.cd allows you to integrate with your tracking tools. You can define how to handle ticket numbers in your source control commit messages. Go.cd parses the message looking for a specific pattern and transforms matches into links to your tracking tool.

We use AxSoft OnTime and integration wasn’t as straight forward as I envisioned. OnTime uses two query strings to identify the ticket number and the type of ticket (defect, feature, incident…).

View Defect: https://name.ontimenow.com/viewitem.aspx?id=102578&type=defects

Create Defect: https://name.ontimenow.com/edititem.aspx?type=defects

From what I can tell Go.cd only allows use of one parameter and no facility to expand the regex and parameter to work with the various patterns for the type of ticket. Example: we may have a defect ticket OTD 102578 and a feature OTF 87984. When commits are related to either one of these tickets, the ticket number, including the prefix, is added to the commit message. To turn this into a link in Go.cd, we have to parse the OT* and correlate that to defect or feature, depending on the value of *, and add that to the type query string parameter. Next we have to grab the numbers after the space after the ticket type prefix and add that to the id query string parameter.

I am not really sure how to get this to work, aside from messing around with the source code. Did I say source code? Solved! Well maybe…tune in.

GoCD: Pipeline Parameters

I am currently getting a Go Continuous Delivery Server stood up and I am reusing scripts from our CCNET Continuous Integration server to save time getting the new server up. In doing this, I am also reviewing the scripts for improvements.When defining builds for my shinny new Go server I noticed that I have duplicate properties across tasks when calling Nant targets.

I have a property that defines the server environment the task should work with. This basically, sets properties in the script so that they target the correct environment. If I set the property to dev, the script will set the server names, paths and more to point to the dev environment.

There is a property that tells the task what source code repository branch to use. In the context of a task, this mainly has to do with paths to the branch already checked out and updated on the Go server and not controlling the actual branch updates. The branch name is concatenated in a common path so the task knows where to get and save source files.

There are more duplicated properties, but it got me wondering if there is a better way to do this than repeating the same value over and over for each task. When I need a new task or a new pipeline with different values this can become a maintenance nightmare. Not to mention, this duplication will make it hard to take advantage of Go Pipeline Templates so I need to solve this, if I want to gain the ease in creating new pipelines that templates afford.

Go has the concept of Environment Variable that is a common value saved on an agent. That is a nice feature as it lets me define values common to every pipeline running on an agent targeting a specific environment. To fix my issue I need to be able to set values at the job level. The agent level is to broad as I can have multiple pipelines target specific agents with different values for the properties I want to abstract. I wonder if we can set up a common variable at the job or pipeline level that the tasks can use?

Go Pipeline Parameters

Oh look, there is something called a Parameter in the pipeline configuration.

Parameters help reduce repetition within your configurations and combined with templates allow you to setup complex configurations. More..

Well let’s look into that and see what we come up with. The “More” link above gives the details of how to use parameters in Go. Basically, you define a parameter then you can use them in your tasks. To use the parameter in your task, just wrap it with #{}. Example: I have a parameter named ENV and I would use it in my script by tokenizing it like so, #{ENV}.

use_go_pipeline_parameter_in_task

This gets rid of a bunch of unnecessary duplication, improves the maintainability of build scripts, and opens up the possibility of creating templates.

Conclusion

There is so much more to understand about Go. One thing that was reinforced in this exercise was to remember to constantly look for pain points in solutions and processes and search for ways to cure the pain.

ThoughtWorks Go Continuous Delivery, Now In Open Source Flavor

If you haven’t heard the ThoughtWorks Go Continuous Delivery server is now open source. The source code is located on GitHub, https://github.com/gocd/gocd. I decided to give it a test drive and I was pleased. Since I am primarily a Windows developer my point of reference is CCNET (which is based on ThoughtWorks Cruise Control Continuous Build server), TFS Team Build and TeamCity. I don’t have a lot of TeamCity experience, but I can say that I can easily see automating many scenarios in Go that I was having a hard time conceiving in CCNET. Adding the concept of Environments, Pipelines, Stages, and User Roles opened an easier path to automated production deployment for me.

Install

Install was pretty simple. Go is cross platform, but I have a Windows server. I downloaded the Windows packages from http://www.go.cd/download/. I installed the server and agent on my server, opened it up in a browser, and it was there ready to go. Very easy, only a few minutes of clicking and I was ready to start. Before I started building pipelines, I made a few customization for my environment. I want to use Git and NAnt in my build, test, deploy process so I added the path to their exe’s to the Path (Windows system environment variable). This makes it less painless to run them from Go.

Server User Authentication

I am eventually going to use LDAP for user authorization, but for now I setup an htpasswd file with usernames and SHA-1 hashed passwords. Then I entered the name of the file, htpasswd.txt, in the server configuration (Admin > Server Configuration > User Management > Password File Settings). I generated the contents for the SHA-1 hashed password file on http://aspirine.org/htpasswd_en.html, but I could have easily just used a Crypto library to hash the passwords myself. Usernames are not case sensitive, but you shouldn’t have colon, spaces or equal sign in the username unless you escape them with backslash.

Configuration

The Go configuration is stored in an XML file, like Jenkins and CCNET. I know many people have a disdain for XML, but it doesn’t bother me and it makes Go portable. I can deploy it to another server or even a developer workstation, use a common config file, and its ready to start processing pipelines. You can use the UI to configure most of what you want to do, but I enjoy the fine grain control in editing the XML directly. There is an XML validator so when my error prone fingers type the wrong character it will automatically reject the change and continue using the current configuration. Since the configuration is XML, I decided to put the file under source control. The reason for this is to have a backup of the config and to allow the ability to config the server from XML and automatically push the changes to the Go server with the Go server (sweet). This doesn’t work both ways, so if there are changes made through the UI they won’t be pushed to source control (although I can envision some convoluted solution for this). For now, I am the only person managing the server and I will configure through the XML file and not the UI.

Pipelines

Pipelines are a unit of organization in Go. A pipeline is comprised of stages which is comprised of jobs which is comprised of tasks. Tasks are the basic unit of work in Go. In my instance most of my tasks are NAnt tasks that call targets in nant build scripts. There are all kinds of ways to create chains of actions and dependencies. This is going to probably be where I focus a lot of attention as this is where the power of the system lies, IMHO. Being able to customize the pipelines and wire up various dependencies is huge for me. Granted I could do this in CCNET to a certain degree, but Go just made it plain to envision and implement.

NAnt Problems

Working with Nant was a pain. Actually, this was the only major hurdle I had to cross. I couldn’t figure out how to pass properties to the nant build file. Then I decided to try to pass the properties through the target argument of the Go nant task, like this

<nant buildfile=”testcode\test\this-is-my-buildfile.xml” target=”-D:this-is-a-nant-property=&quot;dev&quot; -D:another-nant-property=&quot;jun&quot; ThisIsMyNantTarget” />

Note: Paths are relative to your Agent pipeline working directory.

This worked great, but a more intuitive way of doing this would have been good. Maybe an arguments property so there is no confusion between nant property and nant target.

Conclusion

I know this post is lite on details, but I just wanted to get a quick brain dump of my experience with Go. Go has pretty good documentation on the Go.cd website and posting questions to support elicited pretty fast feedback for a free product. I am excited to get involved with the Go and the Go Community. Overall, it was very easy to get a powerful Continuous Delivery server up and running in no time. You should check it out.