In my previous post, Testing ARM Templates with Pester #1 - PowerShell, I showed how to test behaviours ARM template deployment without actual deployment. At the end of the post, I also briefly mentioned how we can integrate this testing in our CI/CD pipeline. However, I was actually asked many times how I did it on Visual Studio Team Service (VSTS). To be honest, it's not that intuitive for this, especially we need to run test before the resource deployment. This post, Test Azure deployments in your VSTS Release Pipeline, talks about after resource deployment in the release pipeline, but not in the build pipeline. In this post, I'm going to show how to test ARM templates in the build pipeline.
The code sample can be found at Testing ARM Templates
Azure CLI
Pester is a PowerShell-based test framework. So, it's easy to think we can use PowerShell for testing, with this cmdlet Test-AzureRmResourceGroupDeployment. This is somewhat true, but as you know, PowerShell is not as good as error handling. On the other hand, Azure CLI returns an error object, even if there is an error, which we can gracefully handle errors. In addition to that, Azure CLI can also run on PowerShell! So, this time, we're using Azure CLI for ARM template testing.
In order to validate ARM template in Azure CLI, here's the command:
| az group deployment validate ` | |
| -g [RESOURCE_GROUP_NAME] ` | |
| --template-file [TEMPLATE_FILE] ` | |
| --parameters `@[TEMPLATE_PARAMETER_FILE] |
When we run this command, it returns a JSON object that contains the validation result. Here's the actual result that the command returns.
| { | |
| "error": null, | |
| ... | |
| "properties": { | |
| ... | |
| "validatedResources": [ | |
| { | |
| "apiVersion": "2016-06-01", | |
| "id": "...", | |
| ... | |
| } | |
| ] | |
| }, | |
| ... | |
| } |
As all necessary information resides in the validatedResources property, we can simply parse this JSON string to JSON object and utilise it.
Test with Pester
Here's the sample test script using Pester:
| # Extract of LogicApp.Tests.with.CLI.ps1 | |
| ... | |
| Describe "Logic App Deployment Tests" { | |
| # Init | |
| BeforeAll { | |
| az login --service-principal -u $Username -p $Password -t $TenantId | |
| } | |
| # Teardown | |
| AfterAll { | |
| } | |
| # Tests whether the cmdlet returns value or not. | |
| Context "When Logic App deployed with parameters" { | |
| $output = az group deployment validate ` | |
| -g $ResourceGroupName ` | |
| --template-file $SrcDirectory\LogicApp.json ` | |
| --parameters `@$SrcDirectory\LogicApp.parameters.json ` | |
| | ConvertFrom-Json | |
| $result = $output.properties | |
| It "Should be deployed successfully" { | |
| $result.provisioningState | Should -Be "Succeeded" | |
| } | |
| It "Should have name of" { | |
| $expected = "log-app" | |
| $resource = $result.validatedResources[0] | |
| $resource.name | Should -Be $expected | |
| } | |
| } | |
| } |
As you can see, the $output variable stores the validation result converted into a JSON object, then check whether the deployment behaviour is expected or not. It looks simple because I use it for Logic App deployment, but if you deploy an Azure VM or App Service instance, or something else, it would be more complex and there would be more behaviours to test. Now, run the test script and you will see the result like this:
Looks all good. Now, this needs to be integrated into VSTS. Let's move on.
VSTS Integration
There are third-party tasks for Pester, but they are not for our purpose. One is not designed for Azure resources, and the other is only useful for the release pipeline. Also, the current version of Azure CLI task can't install additional PowerShell module, that is Pester for our case. Therefore, we need to install Pester and Azure CLI in a separate PowerShell task.
Install Pester
First of all, we need to install Pester. VSTS has a built-in Pester, but its version is 3.4.0, which is quite old. According to the official Pester repository, it's always a good idea to install the newest version of it. Therefore, install Pester from PowerShell Gallery.
Install Azure CLI
As VSTS has already got Python runtime, we can easily install Azure CLI with the following command.
Invoke Pester
Now, we've got Pester and Azure CLI. It's time to run the test script. Here in this post, I have two PowerShell scripts – the test script itself and the Pester runner. In the Azure PowerShell task, it calls the Pester runner and the runner calls the test script.
It looks messy in the arguments field because it needs several parameters. The actual runner is actually very simple.
| # Extract of Run-Pester.ps1 | |
| ... | |
| $segment = $TestFilePath.Split("\") | |
| $testFile = $segment[$segment.Length - 1].Replace(".ps1", ""); | |
| $outputFile = "$OutputDirectory\TEST-$testFile-$BuildNumber.xml" | |
| $parameters = @{ ResourceGroupName = $ResourceGroupName; SrcDirectory = $SrcDirectory; Username = $Username; Password = $Password; TenantId = $TenantId } | |
| $script = @{ Path = $TestFilePath; Parameters = $parameters } | |
| Invoke-Pester -Script $script -OutputFile $outputFile -OutputFormat NUnitXml |
Publish Test Results
Pester also supports test results output. Currently it only supports the NUnit format, but that's fine anyway because VSTS supports it. In the Pester runner script above, it has already set up the output file and output format. Therefore, another task, Publish Test Results can capture those results.
Activate Build Pipeline
Now, we're all set on VSTS! Let's run the build. Once the build is successful, the build result screen might look like this:
All build tasks has passed and tests results are successfully integrated into VSTS dashboard.
So far, I have worked through how to integrate ARM template testing in VSTS, with Pester and Azure CLI. Currently, the build time takes around 10 mins because it includes Peter and Azure CLI installation, which is a caveat of this approach. If there is a task integrated with Azure CLI and Pester, the user experience would be a lot better.
This has been originally posted at Mexia.