Automation + Vso + Octopus = Love

The triangle of love between an Automation Test Pack, VSO and Octopus
By Filipas Ciprian Posted 04 July 2016

Automation + Vso + Octopus = Love

Automation + VSO + Octopus = Love

The dream

A while ago a dream was born. A dream forged from the deepest fears of the software development world. The dream of delivering pure, clean, quality code on TIME.

As the community was growing and the idea got seeded in the hearts of the people, a movement arose and so the CONTINUOUS DELIVERY concepts took shape.

Now, in all seriousness, as an aspirant software developer, getting you code to market in the shortest period of time possible, must be one of your main goals.

It may seem simple at first, you automate as many processes as necessary in order to reduce the time you wait for your colleagues to validate the "masterpiece" you wrote and to decide whether or not it will see the light of production.

Well, it may become a hard and frustrating piece of work if it's not done properly from the beginning.

The story

Keeping our mind on the goal and with our hearts filled with enthusiasm we started the race towards a better future. A process for Octopus was created in order to automate the deployment. With that one out of our way we wanted to tackle the challenge of running our Selenium based automated end-to-end test suite automatically.

Solution Diagram

The solution

Octopus Level

In order to achieve the desired solution a new step was created in our current deployment process (in Octopus) for triggering a new VSO Build.

Octopus Powershell

The template consists in a parametrized PowerShell script that triggers the build.

The parameters are:

  • BuildDefinitionId – your build definition Id. In order to obtain this ID go on the build definition in VSO, hit edit, check the variables tab and you should find it under "system.definitionId"
  • BuildUsername – the username for queueing the build. You must use alternate credentials for this
  • BuildPassword – the password for the BuildUsername
  • ProjectName – the name of the VSO project
  • BranchName – the name of the branch used for building

Using these parameters, the script is creating a JSON object for the body and a header. These are then used for a REST Call towards the VSO API.

The REST Call is done via the Invoke-RestMethod cmdlet.

VSO Level

The VSO Build definition looks like this.

VSO Build Overview

Step1. We build the testing solution source code

Step2. We start the setup for the agent

Step2.1. Install Chocolatey using a Command Line step using the following script:

Installing Choco Powershell

Step2.2. Install Firefox via an inline Powershell script

Installing Firefox Powershell

Step2.3. Install NUnit via an inline Powershell script

Installing NUnit Powershell

Step3 Running the tests using NUnit command

Running Tests with NUnit from Powershell

For this we implemented a parametrized Powershell script that calls the proper command with parameters provided at build time via VSO.

The parameters are:

  • NUnitTestAssemblies – The assemblies for NUnit to take into account
  • Tags – The desired tags for NUnit to take into account when selecting the tests to run

Step4. Publish the Test results via a Public Test Results VSO Step

This step will generate a summary report similar with this one.

Publish Test Results

Publish Test Results

Step5. Publishing various artifacts

Step6. Sending an E-mail Notification

Send Email Notification

This is achieved via the following parametrized Powershell script

The parameters are:

  • DistributionList – The list of emails to witch the e-mail is sent
  • BuildNumber – The build number
  • SmtpUser – The Smtp User
  • StmpPword – The password of the SmtpUser
  • SmtpServer – The Smtp Server addressEnvironment
  • SmtpPort – The Smtp Port


This solution is far from being perfect, however it solved some of our issues.

Thanks to this approach we do not need dedicated machines for test running. This results in a cost reduction and a more efficient use of resources since multiple agents can be hosted on the same machine and their management is governed by the VSO instance.

By using agents, we have a more consistent environment while the build definition is easy to configure and it's extremely flexible.

In terms of improvements, we want to notify the Octopus server when the build was done and based on the status we should either promote the deploy or revert it.

Filipas Ciprian
Filipas Ciprian
Senior Software Developer