Scheduling Posts with GitOps, Azure Durable Functions and GitHub Actions

This post is over a year old, some of this information may be out of date.

When you use a static website generator, like Gridsome, for your blog posts, you might have found that it's tricky to set a scheduled publish on your post. Like Wordpress, it's a blogging specific tool and offers the scheduled post feature. But mostly static website generator doesn't have such feature. There are two options to achieve this. 1) Give up and publish each post on the day you desire, or 2) Build a tool for schedule post. Throughout this post, I'm going to take the second option using GitOps, Azure Durable Functions and GitHub Actions.

You can download the source code from https://github.com/devkimchi/GitHub-Repository-Event-Scheduler.

Acknowledgements

I got a basic idea from PublishTo.Dev using Durable Functions, which is developed by one of my colleagues Todd. If you don't like this idea, blame Burke. 😉

About Durable Functions

One of the characteristics of the serverless architecture is "stateless". This statement is correct from one perspective - building a serverless API application. On the other hand, if you encompass the aspect to "event-driven architecture", this statement is not always correct as most events are "stateful". For example, there is a timer function that runs every hour. Where does the "every hour" come from? There must be storage that keeps the timer information. This stored timer information is called "state" in this context, and the application that relies on the "state" is "stateful".

One of the "stateful" serverless applications on Azure is Logic Apps. It manages workflow, and each trigger and action in the workflow has its "state" which can be used by the following actions. What if Azure Functions can manage the workflow like what Logic Apps does? How can Azure Functions manage "state" to manage workflows? Azure Durable Functions has implemented this idea, which is "stateful".

Then, what sort of "stateful" workflow do we need for this post?

Designing Workflow

Let me describe the workflow briefly.

First of all, a schedule is sent to the given function endpoint. This function is nothing special but works as a gateway that takes the payload and passes it to the Durable Functions orchestrator. The actual orchestration is managed in the second function. It checks the schedule and calls the timer that sends a queue message to Azure Queue Storage. At the same time, it stores the "state" to Azure Table Storage. When the scheduled time arrives, the queue is triggered, and the third (and the last) function is executed based on the "state" from the Table Storage.

The second function takes care of all orchestration workflows, and the last function takes care of the business logic, which is to call an API on GitHub in the context of this post.

Implementing Workflow

Endpoint Function

This function publicly opens the endpoint, takes the payload and pass it to the orchestration function. Here's how the payload looks like:

	{
	"owner": "devkimchi",
	"repository": "blog",
	"issueId": 3,
	"schedule": "2020-03-25T07:00:00+09:00"
	}

view raw 01-payload-dk.json hosted with ❤ by GitHub

You can see the GitHub repository name and its owner/organisation name, PR number and schedule for publish. Let's have a look at the code below. It takes the payload (line #8) and calls the orchestration function with the payload (line #9). Finally, it returns the metadata that can check the orchestration status (line #13).

	[FunctionName("SetSchedule")]
	public async Task<HttpResponseMessage> SetSchedule(
	[HttpTrigger(AuthorizationLevel.Function, "post", Route = "orchestrators/{orchestratorName}")] HttpRequestMessage req,
	[DurableClient] IDurableOrchestrationClient starter,
	string orchestratorName,
	ILogger log)
	{
	var input = await req.Content.ReadAsAsync<EventSchedulingRequest>();
	var instanceId = await starter.StartNewAsync<EventSchedulingRequest>(orchestratorName, instanceId: null, input: input);

	log.LogInformation($"Started orchestration with ID = '{instanceId}'.");

	return starter.CreateCheckStatusResponse(req, instanceId);
	}

view raw 02-entrypoint.cs hosted with ❤ by GitHub

Orchestration Function

The orchestration function consists of the followings.

Take the payload from the context (line #6).
Check the maximum duration of the schedule (line #9-13). Due to the limitation of Azure Queue Storage of 7 days, Durable Functions timer also has a lifespan of 7 days. The maximum duration is configurable, and I set it up to 6.5 days.

You can set the more extended scheduling than seven days, but it's beyond this post.

Check the input schedule is longer than the maximum duration (line #25-28).
Run the timer for scheduling (line #30). At this time, the function stops running and goes into sleep until the timer expires.
Once the timer expires the orchestration function continues where it stops and calls the third (activity) function (line #32).

	[FunctionName("schedule-event")]
	public async Task<object> RunOrchestrator(
	[OrchestrationTrigger] IDurableOrchestrationContext context,
	ILogger log)
	{
	var input = context.GetInput<EventSchedulingRequest>();

	// Set the maximum duration. Max duration can't exceed 7 days.
	var maxDuration = TimeSpan.Parse(Environment.GetEnvironmentVariable("Duration__Max"), CultureInfo.InvariantCulture);
	if (maxDuration > threshold)
	{
	return "Now allowed";
	}

	// Get the scheduled time
	var scheduled = input.Schedule.UtcDateTime;

	// Get the function initiated time.
	var initiated = context.CurrentUtcDateTime;

	// Get the difference between now and schedule
	var datediff = (TimeSpan)(scheduled - initiated);

	// Complete if datediff is longer than the max duration
	if (datediff >= maxDuration)
	{
	return "Too far away";
	}

	await context.CreateTimer(scheduled, CancellationToken.None);

	var output = await context.CallActivityAsync<object>("CallRepositoryDispatchEvent", input);

	return output;
	}

view raw 03-orchestrator.cs hosted with ❤ by GitHub

Activity Function

This function actually calls the GitHub API to raise an event. Let's take a look at the code below. It calls the repository_dispatch API defined in the GitHub API document. Octokit makes it really easy, but there's no implementation on this API yet. Therefore, in the meantime, you should directly call the API (line #18-19).

	[FunctionName("CallMergePrRepositoryDispatchEvent")]
	public async Task<object> MergePr(
	[ActivityTrigger] EventSchedulingRequest input,
	ILogger log)
	{
	var authKey = Environment.GetEnvironmentVariable("GitHub__AuthKey");
	var requestUri = $"{Environment.GetEnvironmentVariable("GitHub__BaseUri").TrimEnd('/')}/repos/{input.Owner}/{input.Repository}/{Environment.GetEnvironmentVariable("GitHub__Endpoints__Dispatches").TrimStart('/')}";
	var accept = Environment.GetEnvironmentVariable("GitHub__Headers__Accept");
	var userAgent = Environment.GetEnvironmentVariable("GitHub__Headers__UserAgent");

	this._client.DefaultRequestHeaders.Clear();
	this._client.DefaultRequestHeaders.Add("Authorization", authKey);
	this._client.DefaultRequestHeaders.Add("Accept", accept);
	this._client.DefaultRequestHeaders.Add("User-Agent", userAgent);

	var payload = new RepositoryDispatchEventRequest<EventSchedulingRequest>("merge-pr", input);

	using (var content = new ObjectContent<RepositoryDispatchEventRequest<EventSchedulingRequest>>(payload, this._formatter, "application/json"))
	using (var response = await this._client.PostAsync(requestUri, content).ConfigureAwait(false))
	{
	response.EnsureSuccessStatusCode();
	}

	return payload;
	}

view raw 04-activity.cs hosted with ❤ by GitHub

The payload passed from the orchestration function is wrapped with another object for the repository_dispatch API (line #16). Once the activity function calls the API, it triggers the GitHub Actions workflow.

Webhook Function

This function is almost identical to the activity function. The only difference is that it sets the event type of publish (line #18). I'll discuss this later in this post.

	[FunctionName("CallPublishRepositoryDispatchEvent")]
	public async Task<IActionResult> Publish(
	[HttpTrigger(AuthorizationLevel.Function, "post", Route = "events/publish")] HttpRequest req,
	ILogger log)
	{
	var input = JsonConvert.DeserializeObject<EventRequest>(await new StreamReader(req.Body).ReadToEndAsync());

	var authKey = Environment.GetEnvironmentVariable("GitHub__AuthKey");
	var requestUri = $"{Environment.GetEnvironmentVariable("GitHub__BaseUri").TrimEnd('/')}/repos/{input.Owner}/{input.Repository}/{Environment.GetEnvironmentVariable("GitHub__Endpoints__Dispatches").TrimStart('/')}";
	var accept = Environment.GetEnvironmentVariable("GitHub__Headers__Accept");
	var userAgent = Environment.GetEnvironmentVariable("GitHub__Headers__UserAgent");

	this._client.DefaultRequestHeaders.Clear();
	this._client.DefaultRequestHeaders.Add("Authorization", authKey);
	this._client.DefaultRequestHeaders.Add("Accept", accept);
	this._client.DefaultRequestHeaders.Add("User-Agent", userAgent);

	var payload = new RepositoryDispatchEventRequest<EventSchedulingRequest>("publish", input);

	using (var content = new ObjectContent<RepositoryDispatchEventRequest<EventSchedulingRequest>>(payload, this._formatter, "application/json"))
	using (var response = await this._client.PostAsync(requestUri, content).ConfigureAwait(false))
	{
	response.EnsureSuccessStatusCode();
	}

	return payload;
	}

view raw 05-webhook.cs hosted with ❤ by GitHub

Designing GitHub Actions

So, we got the workflow on the Durable Functions side, which sends a scheduled event to GitHub. Now, the GitHub Actions workflow takes the event and runs its own pipeline workflow. Let's have a look at the picture below that describes the end-to-end workflow.

Once a new post is ready, create a PR for it.
When the PR number with the publishing schedule is ready, send an HTTP request to Durable Functions endpoint.
The Durable Functions puts the timer and raises the event on the scheduled day, that calls the GitHub API.
GitHub Actions is triggered to merge the PR.
Once the PR is merged, it triggers another GitHub Actions to deploy (publish) the new post.
New post is published.

You got the Durable Functions covered above. The second GitHub Actions was handled by the other post. This section takes the first GitHub Actions using the repository dispatch event. Let's take a look at the following YAML definition. This workflow is only triggered by the repository_dispatch event (line #3). In addition to the trigger, it runs the workflow only if the if statement meets – the event type MUST match with merge-pr (line #8). The workflow itself is pretty straightforward. We've previously got the PR, and the workflow uses the github-pr-merge-action action to merge the PR (line #14).

	name: Merge PR

	on: repository_dispatch

	jobs:
	merge_pr:
	name: Merge PR
	if: github.event.action == 'merge-pr'

	runs-on: ubuntu-latest

	steps:
	- name: Merge PR
	uses: justinyoo/github-pr-merge-action@v0.8.0
	with:
	authToken: ${{ secrets.GITHUB_TOKEN }}
	owner: ${{ github.event.client_payload.owner }}
	repository: ${{ github.event.client_payload.repository }}
	issueId: '${{ github.event.client_payload.issueId }}'
	mergeMethod: Squash
	commitTitle: ''
	commitDescription: ''
	deleteBranch: 'true'
	- name: Send dispatch event for publish
	shell: bash
	run: \|
	curl -X POST 'https://${{ secrets.AZURE_FUNCTIONS_NAME }}.azurewebsites.net/api/events/publish' -d '{ "owner": "${{ secrets.OWNER }}", "repository": "${{ secrets.REPOSITORY }}" }' -H "x-functions-key: ${{ secrets.AZURE_FUNCTIONS_KEY }}" -H "Content-Type: application/json"

	build_and_publish:
	name: Build and publish
	if: github.event.action == 'publish'

	runs-on: ubuntu-latest

	steps:
	...

view raw 06-workflow.yaml hosted with ❤ by GitHub

NOTE: The GitHub PR Merge action is that I contribute. 🙈

Please note. The next workflow should have been automatically triggered for deployment. But this is not the case. Therefore, you should manually execute the deployment workflow. However, GitHub Action doesn't support manual trigger at the time of this writing. Instead, you can use the repository_dispatch event that triggers the deployment workflow as a workaround (line #24-27). As we wrote the webhook function earlier, this workflow will call the webhook to raise the publish event.

Once the merge succeeds, it triggers the next GitHub Actions workflow, and the new post is published!

If you've completed by this far now, you're all set. Build the function app and deploy it to Azure. Send an HTTP request to the function endpoint, and wait. Then a new post will be published on your scheduled day. Actually, this post is published by this scheduler!

Where's GitOps?

The idea of GitOps that Weaveworks introduced is roughly "to deploy applications based on changes detected by PR". The workflow used in this post is not exactly the same as GitOps, but the concept, PR-based application deployment, is similar.

Let's give an example with this post:

A new post is ready to publish.
A new PR is created to publish the post.
Set the publish schedule using Azure Durable Functions.
Durable Functions sends an event to GitHub based on the schedule.
The event captured by GitHub repository runs the GitHub Actions to merge the PR.
The merged PR eventually builds and deploys the application, which is the static website with the new post.

How do you feel like? Is it similar to GitOps?

So far, we've walked through how we used Azure Durable Functions and GitHub Actions in the context of GitOps to schedule blog posts. It's like a very comprehensive example that uses Durable Functions and GitHub Actions. If you've been using a GitHub repository to host your blog, now you can schedule your new posts!