Using CircleCI is a fun trip, but there are some things which do not come easy when using this system. Sure, you get Windows, Linux and macOS build support in a SaaS service which is nice, but sometimes you might want to do something not readily supported by CircleCI that you got used to when using another CI system. I know I did.

This post is a companion to freshly created GitHub repository - circleci-playground, which is going to contain some of the hacks I'm proud(-ish) of and I think that might come of use to others. Feel free to reuse it!

Before we get to the first showcase, which is triggerring a scheduled workflow manually, bear with me for some introduction to triggers.

Running workflows in CircleCI

So, basic method of running a workflow in CircleCI is to assign it to a specific branch (or even all of them), optionally limit triggering to open Pull Requests and master branch if you don't want to run too many jobs. There is even an API for that, which allows you to tell to trigger a workflow assigned to a specific branch. It's still labeled as preview, but don't worry, we're going to do something much crazier soon.

The way this API works, is that it mimics a webhook sent by github to CircleCI (I think it's an PushEvent webhook, but don't quote me on that :)). In return, CircleCI evaluates repository's config and triggers any workflows attached to the branch we specified. Voilà!

But there is more than one way to skin a cat. At some point, you've most likely used or wanted to use a scheduled workflow, which can be used to run some long tests each night or once a week, for example. There is a chance that this periodic workflow contains some specific jobs layout, sometimes really convoluted. You create a cron schedule for it and are done with it, it runs accordingly and you're happy.

Unless you want to run it without waiting for the scheduled time.

Hero of the day - on-demand run of a scheduled workflow

Disclaimer: What I'm describing here is risky since we're going to use a API which, at the time of writing, is in preview state, so it might break at any time. But it may work like this for a long time still, so measure the risks you can take and be mindful when following my footsteps :).

If you define a scheduled workflow, you are setting it to trigger on a specific time and from a specific branch like this:

workflows:
  version: 2
  periodic:
    triggers:
      - schedule:
          cron: "0 10 * * 5"
          filters:
            branches:
              only:
                - master
    jobs:
      - say_hi
      - say_bye:
          requires:
            - say_hi

Now, I don't know how exactly this is handled by CircleCI but I can tell that, for sure, this is not going to be triggered by a push to GitHub nor a call to the aforementioned API. This would mean that there is no other way to run it that to wait for the time to come.

But here comes the part when features from experimental API v2 come into play.

Pipeline parameters

In an effort to allow more dynamic changes to the pipelines, CircleCI introduces yet another set of parameters - pipeline parameters - which can be used to alter the config in runtime. Let's start with defining a parameter we're going to use to control our pipelines:

version: 2.1

parameters:
  run_workflow:
    type: boolean
    default: false

Note that we have to use version: 2.1 here, which in itself is breaking ability to test your jobs before pushing the config (OK, at least making it harder...) while enabling reusable configs. Everything comes with a price, I guess...

Now, let's use our newly added parameter for more fun and profit.

Conditional workflows

Here we're cheating a little bit by creating an additional workflow, but I'll try to make it as painless as possible.

In order to actually manipulate our config, we're going to use when: clause for conditional workflow scheduling based on our run_workflow parameter:

workflows:
  version: 2

  periodic_manual:
    when: << pipeline.parameters.run_workflow >>
    jobs: *periodic

So now our periodic_manual workflow will be considered to be run on every commit but will only be actually scheduled when run_workflow is set to true. Which for now isn't - remember that the default value is false and we're not overriding it (yet). But before we get to do that... Did you notice something unusual in the YAML snippet above?

*periodic is a dereferenced YAML anchor. It's the cheating part. We're using it here to follow the DRY rule by reusing our job definitions from periodic workflow, defined like this:

workflows:
  periodic:
    triggers:
      - schedule:
        <snip>

    jobs: &periodic
      - say_hi
      - say_bye:
          requires:
            - say_hi

To read more about using YAML anchors, you can go to learnxinyminutes, for example. They're a part of the YAML spec!

So the state of things for now is as follows:

1. parameter run_workflow exists to control whether we run our periodic-like workflow
2. we have a periodic_manual (heh) workflow, which reuses the jobs chain from periodic
3. we were forced to use YAML anchors, which make reading the YAML file harder :(
4. we don't know how to trigger our new pipeline

Let's tackle the last step :).

Trigger pipeline API

Last element of our puzzle is an API call to a new endpoint, available in API v2, which is trigger pipeline. Now you might be confused - I've talked about abother endpoint, which can be used for running pipelines from a branch! Well, I can't blame you for that, those endpoints seem to do a very similar job, but the difference is that this new v2 endpoint allows us to specify pipeline parameters as well as a target branch!

So, we can call this endpoint using a command such as:

curl -u ${CIRCLECI_TOKEN}: -X POST --header "Content-Type: application/json" -d '{
  "branch": "dev",
  "parameters": {
    "run_workflow": true
  }
}' https://circleci.com/api/v2/project/github/wurbanski/circleci-playground/pipeline

Like that, we're going to set our workflow-defining parameter run_workflow to true and trigger our periodic_manual workflow on a branch named dev. (Note the v2 part of the endpoint URL.)

We got it!

Thus, by using three not-yet-officially-released features, we got a manual trigger of a scheduled workflow to work in CircleCI. Kind of. If you don't mind a little bit of cheating using anchors. And an additional workflow. Anyway...

Noice.

If this writeup does not explain the solution well enough - let me know! But also have a look at this shiny repository, which contains all of the code from above in its natural habitat. Hope this helps (and the API does not break while we're using it!).

It's only fair for me to mention that I wouldn't have been able to come up with this solution without reading a blog post by labs42 regarding conditional workflows and how they can be used to mitigate another of non-supported use cases for CircleCI - handling monorepos. If you use monorepos with CircleCI - this is a must-read.