There are a number of problems we’re running into:
Travis’s lack of resources and limited build times lead to
unreliability of tests, which are frequently hitting Travis’s 50
minute time limit even with all dependencies cached. I don’t think
we’ve ever gotten the macOS build to complete. We’ve also seen a
lot of network instability which leads to spurious build
failures.
Running integration tests with Gitlab CI gets us past the
resourcing/timeout problems, but Gitlab CI doesn’t integrate well
with Github (e.g., showing test status for pull requests), which
rules out moving all tests to it.
For Gitlab CI, we have to manage a Gitlab runner on an EC2
instance, and pay costs of the EC2 instance that hosts it.
Using multiple CI services increases complexity and maintenance
burden.
Back in September, Microsoft announced Azure Pipelines, a new CI/CD service
that integrates with Github, provides unlimited free build minutes,
a 20 hour time limit for jobs, and supports all three major
platforms (Linux, macOS, and Windows). This looked very intriguing,
and we’ve been trying it out with success.
Most builds complete in less than 15 minutes.
All three platforms are building reliably (when there are test
failures, they are legitimate).
Unit tests are run on every push.
Integration tests are run nightly on the master and
stable branches, and nightly binaries are published as
artifacts.
There is now
one place to see the status of all automated
tests.
Converting all the existing pipelines was not trivial, but also
not overly difficult. UPDATE We’ve since moved this CI over to Github Actions, and have removed the files referenced. See the YAML configurations for unit tests,
nightly integration tests, and emplates
they reference.
The Azure Pipelines UI is not very intuitive. In particular,
adding a second pipeline (for nightlies) was confusing, and it’s
hard to find settings and options.While we are generally happy with
Azure Pipelines, that is not to say everything is perfect:
Parts of the CI configuration (such as scheduled builds) are not
controlled by the YAML files and must be configured through the
UI.
Azure Pipelines does not have built-in support for caching. For
Stack, this isn’t a big deal since built-in caching solutions are
rarely a great fit for large Haskell projects, so we use the
cache-s3 tool to
manage a cache in an S3 bucket.
This ties us a bit more closely to Microsoft, and it’s unknown
whether they’ll offer all the features we use for free
indefinitely. That said, we’re already tied to Microsoft by using
Github (which
Microsoft has acquired), and moving to a different CI platform
in the future isn’t terribly difficult should it turn out to be
necessary.
Any contributions to move other commercialhaskell or fpco repos over to
Azure Pipelines would be welcome (we don’t necessarily plan to move
them all over immediately if Travis is working well enough, but
happy for help any time).
Email subscriptions come from our Atom feed and are handled by Blogtrottr. You will only receive notifications of blog posts, and can unsubscribe any time.
