You may have heard of the term, “Walking Skeleton” if you work with the agile methodology. Alistair Cockburn defines the term “Walking Skeleton” in his article:
A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.
This concept is very important, from the DevOps or SRE (Site Reliability Engineering) point of view, because we experience fail-fast and fail-often while building the system and testing it, before going live. Throughout this, we can also secure the system’s stability and reliability.
Let me interpret the quote above in a different way. The concept of “Walking Skeleton” is to build a system/application first in a working condition, no matter it is all hard-coded or not. When the Walking Skeleton is ready, it has to be running in the entire ALM process. And it includes unit tests, integration tests and end-to-end tests with CI/CD pipelines. Once everything is OK in the pipelines, then the Walking Skeleton gets more “Flesh” by “Continuous Improvement”. It sounds straightforward. In fact, it doesn’t go well unless other supporting parties have everything ready for us. For example, setting up CI/CD pipelines requires system access permissions, service principal impersonations, cloud resource access permissions, etc. Although we have our system/application developed and ready, without those non-functional requirements, our delivery can’t be done. Therefore, to minimise these hassles, all the DevOps/SRE related requirements have to be sorted out with the Walking Skeleton at the very early stage of the delivery.
As stated above, the first step of running the Walking Skeleton is to run different testing environments consistently in the CI/CD pipelines. This post shows how to build the Walking Skeleton of Azure Functions app with all testing scenarios, set up CI/CD pipelines on Azure DevOps, and complete the first cut of SRE requirements.
Sample codes used in this post can be downloaded from this GitHub repository.
System High-level Architecture
In my previous post, we’ve already developed an Azure Function app and performed unit tests and integration tests. Here’s the high-level architecture diagram of the Azure Functions app.
Writing Unit Tests and Integration Tests
I showed both unit tests and integration tests in my previous post. Unit testing uses the dependency injection feature from Azure Functions library and integration testing uses Mountebank to perform tests. I’m not going to repeat here. Let’s move onto the end-to-end (E2E) testing.
Writing End-to-End Tests
Generally speaking, E2E testing comes with functional testing. Functional testing validates the acceptance criteria by manually running the application. If we can capture the functional testing scenarios in a coded way, they become a part of E2E testing, and we run them in the CI/CD pipelines. If we can mock external API dependencies, this E2E testing scenario can be a part of integration testing, too.
Now, here’s the question. It sounds we re-use the same code base for both integration testing and E2E testing without modifying them. How can we achieve this? Let’s have a look at the code below.
LocalhostServerFixture for integration testing.
Let’s have a look at the
As you can see the code above, it only works in the integration testing scenario. If we want to re-use this code for both integration and E2E testing, we need to refactor the
I use a simple factory method pattern for the refactoring exercise. Let’s create a
ServerFixture class. It declares the method,
CreateInstance(serverName), to create an instance, based on the server name passed.
Let’s refactor the existing
LocalhostServerFixture now inherits
ServerFixture. The existing
GetHealthCheckUrl() method now has the
We’ve just completed refactoring
LocalhostServerFixture. It’s time to create another fixture class for the E2E tests.
Let’s create the
FunctionAppServerFixture class that inherits
ServerFixture, and implement the
GetHealthCheckUrl() method to only return the endpoint URL. As you can see, the relevant information to compose the endpoint URL comes from the environment variables.
Now, we’ve got
FunctionAppServerFixture for E2E testing. Refactor the test code!
First of all, we need to modify the
Init() method of the
HealthCheckHttpTriggerTests class. It gets the server name from the environment variable, which decides to create either
LocalhostServerFixture for integration testing or
FunctionAppServerFixture for E2E testing. In addition to that, we add another decorator,
TestCategory("E2E"), on the test method for E2E testing.
Now, we’ve got the test code that is used for both integration tests and E2E tests. Let’s run the tests in our local development environment.
Run Integration Tests
Based on the instruction from my previous post, run the Mountebank server and the Azure Functions runtime locally, then execute the command below for integration tests.
Our refactored integration works just fine and here’s the result.
Run End-to-End Tests
This time, we’re running the E2E tests from our local machine. To do this, we assume that the Azure Functions app has already been deployed to Azure. Set up the environment like below. Don’t get bothered of these key and name as they are not real.
The following command is to run the E2E tests.
As we implemented above, the E2E tests use
FunctionAppServerFixture and here’s the result.
Now, we’ve got the
LocalhostServerFixture class refactored, and both integration tests and E2E tests successfully run on our local machine.
Compose Azure DevOps CI/CD Pipelines
Based on our successful local test runs, we’re going to compose Azure CI/CD pipelines, as the last step of building the Walking Skeleton. Both unit tests and integration tests are placed within the build stage, and the E2E tests are placed within the release stage. The pipeline written in YAML uses the Azure DevOps Multi-Stage Pipelines feature. You can see the whole pipeline structure from the source code. I’m extracting some bits and pieces here for discussion.
This is the extraction of the unit tests steps from
Unit Test Function Apptask looks overwhelming. In overall, it’s not that different from the test command above. Instead, it comes with a few more options.
--filter: This option filters out all tests not having either
E2E. In other words, with this filter, this task only performs unit tests.
--logger: This option has a value of
trx, which exports the test results in the
.trxformat, which is used in Visual Studio.
--results-directory: This option sets the output directory of the test result.
/p:CollectCoverage: This option enables the code coverage analysis.
/p:CoverletOutputFormat: This option has a value of
cobertura, which defines the output format of the code coverage.
/p:CoverletOutput: This option sets the output directory of the code coverage analysis result.
In addition to this, this task has another attribute of
continueOnErrorand its value of
true. It forces the pipeline to continue, although this task fails (test fails).
Save Unit Test Run Statustask stores the previous task status, whether it
Publish Unit Test Resultstask uploads the test result. As
trxis used for the test result format, this task should select
Publish Code Coverage Resultstask uploads the code coverage analysis report. As it uses the
coberturaas its value.
That’s it for the unit test pipeline setup. Let’s move on to the integration test jobs on the pipeline.
This is the extraction of the integration test steps from
Integration Test Function Apptask is the same as the one for unit tests, except all the code coverage analysis options. It also has the filter of
TestCategory=Integrationso that this task only takes care of integration test methods.
Save Integration Test Run Statustask stores the integration test status to
Publish Integration Test Resultstask uploads the test result to the pipeline.
Cancel Pipeline on Test Run Failuretask is important. It looks for the value of
IntegrationTestRunStatus. If both unit tests and integration tests are successful, it lets the pipeline continue so that all artifacts are generated for release. If either unit tests or integration tests fail, it lets the pipeline stop and mark the pipeline as
Now we’ve got the integration test pipeline setup. E2E test is coming up.
E2E tests are performed after the application is deployed to Azure. Here are the steps for the E2E testing extracted from the pipeline. To use YAML for the release in the pipeline, Multi-Stage Pipelines feature MUST be turned on. If you’re interested in this feature, please have a look at my the other post.
Run E2E Teststask is similar to the other two test steps. Within the build stage, as tests are performed against the
.csprojprojects, we use
dotnet test ...command, while in the release stage, we test against
.dllfiles. Therefore, we should use the
dotnet vstest ...command. Because of the command change, the options are also changed, even though they do the same thing.
--testCaseFilter: It’s the same option as
--filter. Set up the value to
TestCategory=E2Eso that only E2E tests are performed.
--resultsDirectory: It’s the same option as
Let’s have a look at the environment variables part. For E2E testing, extra environment variables are required. Therefore, this task includes a few attributes under the
Save Test Run Statustask stores the test run status to the
Publish E2E Test Resultstask uploads the E2E test results to the pipeline.
Cancel Pipeline on Test Run Failuretasks checks the
TestResultStatusvalue to determine the release stage has succeeded or not. If this value is
Canceled, the pipeline itself is failed or cancelled respectively.
Now we’ve got all the pipeline details, including three different test runs. Let’s run the pipeline. Once it’s done, you can find out the high-level result view with each stage on the Summary tab.
This Tests tab articulates the test results from unit tests, integration tests and E2E tests.
This Code Coverage tab shows the code coverage analysis results. From this analysis, more unit tests need to be written (oops).
So far, we’ve composed the Walking Skeleton for Azure Functions API through Azure DevOps. As mentioned at the beginning of this post, the Walking Skeleton is the minimal set as the working condition. In addition to this, in the automated CI/CD pipeline, all testing scenarios are running. Therefore, when we add more “Flesh” onto the Walking Skeleton over time, it only requires minimal efforts for the growth of the system/application, with extra testing scenarios. From the SRE perspective, automation is essential, and that automation should comprise almost everything. Now, our Walking Skeleton has got everything.
In fact, SRE is not only about automation, but also about the broader practice including monitoring, scaling and resiliency. But they are beyond this post. Once I have another chance, I’ll discuss those topics too. I hope this post would help you start thinking of SRE experiences.
- Site Reliability Engineering
- Book Review: Site Reliability Engineering
- What is Walking Skeleton?
- Factory Method Pattern
- .NET Core CLI – Test
- .NET Core CLI – Test Configuration
- .NET Core CLI – Test Filtering
- .NET Core CLI – Test Reporting
- Azure DevOps Multi-Stage Pipelines
- Azure Pipelines Conditions
- Azure Pipelines Expressions