CI for Open Source Dotnet

7 min readApr 22, 2022

I have an open source dotnet library that I keep on Github. Normally I use TeamCity for CI but I wanted to have a look at some of the cloud options that are free for Open Source. I came up with a shortlist which consisted of CircleCi, GitHub, Travis, AppVeyor and Azure Devops. The results were mixed and some were unexpected.

For the sake of consistency I aimed to use t least an Ubuntu image (others if readily available), and build a Release configuration, then run tests. Some of the results were due to the use of the .Net Framework as a target. This is now removed in favour of .NET Standard and .NET 6 so I have decided to review the situation and see what the differences are. To try to make this fairer I limited the time spent on each system to what I could do in an hour or two.

Azure Devops

First time around I abandoned this after spending far too much time meddling with it, it may just be me but I didn’t like it and the whole debugging cycle felt slow and clumky. Second time around the process seems even more clunky than I remember it and it’s not always obvious how to get back to where you just were. In addition the error messages are completely unhelpful. Thankfully most of mine were syntax errors or lower/upper case errors and they got resolved after a few attempts.

The configuration that worked for me was a minor variation of the Starter Pipeline

trigger:
- masterpool:
  vmImage: ubuntu-latestvariables:
  buildConfiguration: 'Release'steps:
- script: dotnet build -c $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'- script: dotnet test -c $(buildConfiguration)

I tried to add a displayname for the test step but it wouldn’t work so I abandoned that — I’m sure it used to do it so I probably overlooked something minor — other than that minor detail the script now works. The only other thing to note is I get an email every time the build completes with the results. It would be nice to just get this when there was a change in the result. The settings for this (under User Notifications) are per user and can only be on or off. The only options you get with a build are when it completes or when it fails.

The main issue with this one is that it’s not free but many devs will have access which is why I included it. I’ve also got a couple of tests failing when they don’t on any other CI system, while I am sure this is an oversight on my part with such a relatively simple looking config file, it’s a little concerning

AppVeyor

This was the one I was most interested in as it allowed the CI to run on a Windows VM. Unfortunately I struggled with this one first time round, in particular as it doesn’t integrate well with Github in that we couldn’t see build/test progress.

Second time around it looks really nice. One thing I particularly like is the configuration setup, you choose all the options you want (and there are a lot) and it generates the YAML for you, this is much nicer and less error prone than some of the other options. It even does badges. The only minor downside was that each settings page needs to be saved separately, but I can get used to that. It can also only send notifications when your build status changes and a notification can take the form of a GitHub PR. It’s integration with Github appears to have significantly improved too

The downside was that for me at least, it only ran on the master branch no matter what I put into appveyor.yml to tell it to build another branch. It also seemed to ignore the target frameworks in my proj files. It looked like it was using old settings. Unsuprisingly this turned out to be my mistake, my file was called Appveyor.yml when it should have been appveyor.yml. There is an awful lot to like about AppVeyor.

version: 1.0.{build}
 image: Visual Studio 2022
 configuration: Release
 before_build:
 - cmd: dotnet restore
 build:
   project: NLC.Library.sln
   verbosity: minimal

This is the resultant configuration though some of it isn’t needed. The main point to note is the dotnet restore which was needed to use .Net 6 in my case. Also there is nothing needed in order for it to find and run the tests, though the tests can be limited to specific categories.

Travis

Travis used to have two sites, one for open source and one for paid customers. First time round I only looked at the open source on. Now they both use the same site. The free plan is limited to what appears to be a generous amount of credit and allows unlimited users! Unfortunately it appears that Travis has been subject to abuse so it now requires a credit card which I don’t have. Unfortunately this meant that I couldn’t return to the system.

What I can say is that first time around, this is the one that worked for me. The config is more complex than many but that just shows the power of it.

language: csharp
# solution is optional and results in a restore
solution: NLC.Library.sln# mono only needed for paket
mono: latest# attempt to use linux and mac
os:
- linux
- osxosx_image: latest# dotnet version can be more specific but that is handled by global.json# specific version of sdk to prevent mac issuesdotnet: 3.1.403 # 404 not on travis yetbefore_install:
- if [ "$TRAVIS_OS_NAME" = "osx" ]; then brew update          ; fi# needed for tests
# dotnet: 2.1script:
- dotnet build src/NLC.Library.csproj -c Release
- dotnet test tests/NLC.Library.tests.csproj -c Release

As you can see it let me install mono into the VM and it made Linux and Mac images available, it can also facilitate some quite complex logic

CircleCI

The first thing to note here is that Free and Open Source project are allowed an extremely generous amount of credits per month. I really struggled with this one. It uses some terminology that’s new to me e.g. Orbs — which are actually some modular YAML which is quite a nice idea as it can be re used, though the other side of that is that it conceals complexity. The first issue I came up against was YAML — my spacing was slightly out in a few places, after a few goes that was fixed. Then my library refused to build whatever I tried. Further investigation indicated that .NET 6 wasn’t available.

version: 2.1orbs:
  windows: circleci/windows@2.4jobs:
   test:
     description: 'Setup and run tests'
     executor: 
       name: windows/default     
     steps:
       - checkout
       - run: 
           name: 'Install dependencies'
           command: dotnet.exe restore
       - run:
            name: 'Run tests'
            command: dotnet test
   build:
     description: 'Build app'
     executor: 
       name: windows/default
     steps:
       - checkout
       - run:
           name: 'Build app'
           command: dotnet.exe build -c Release
 workflows:
   test_and_build:
     jobs:
       - test
       - build:
            requires:
              - test

This config is quite a complex one. The main advantage is the modularity, not just through orbs but that in the workflow you can define what steps in what order along with dependencies. Unfortunately not having .NET 6 available was a showstopper for me.

Github Workflows

In some ways this is a no brainer as the code is hosted on Github. For me a nice feature of the original was that it was simple to add a Mac OS pipeline as I also develop on a Mac. This is one of those that was relatively simple to setup the first time around as it is quite well documented for simple cases.

On return I was reminded of some of the nice features, it’s easy to setup a workflow to run on all the main OS’s. One really nice feature is the ability to setup starter workflows which are then accessible to users in your organization, there are even docs on migrating to Github Actions from many of the other CI systems discussed here, including Azure Pipelines

Somewhat disappointingly I couldn’t get this working on Windows in the time available, getting the MSBUILD : error MSB1009: Project file does not exist. error. It did however work on Ubuntu and Mac and as that is the same OS as most of the others, I’m broadly considering this a success.

# this is our main Windows based build
name: .NET Coreon:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]jobs:
  build:runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ windows-latest, macos-latest, ubuntu-latest ]
        include:
             -   dotnet-version: '6.0.101'
                 tfm: 'net6.0'steps:
    
    - uses: actions/checkout@v3
    - name: Setup .NET Core ${{ matrix.os }}
      uses: actions/setup-dotnet@v1.7.2
      with:
        dotnet-version: ${{ matrix.dotnet-version }}
    - name: Display dotnet version
      run: dotnet --version
    - name: Install dependencies
      run: dotnet restore -p:TargetFramework=${{ matrix.tfm }}
    - name: Build
      run: dotnet build --configuration Release --no-restore
    - name: Test
      run: dotnet test --no-restore --verbosity normal -f=${{ matrix.tfm }}

This config can be simplified, but for my purposes it did the job, to get it to work without errors I just removed Windows-Latest from the OS part of the matrix.

Conclusions

I was a little surprised by the results here. I didn’t have good memories of Azure Devops but this time around it was much better. I was disappointed about Travis as this had worked for some time. For me the standout was AppVeyor — the configuration was simple, you don’t need to worry about spaces in the right place or whether you have used upper or lower case — for someone who rarely uses this stuff, that can be quite significant.

I also discovered that Rider has quite a nice YAML editor, the only one I have tried that I liked. Additionally the CI systems were sometimes picky about the exact version of dotnet core they would use. I tend to restrict it in Global.json to prevent issues I have had in the past and this caused me a few issues during this exercise.

Internally I mostly use TeamCity but that’s a cost option on the cloud so it’s not really feasible for this project. I’m likely to be using AppVeyor for any open source projects including my main one from now on but GitHub actions came in a close second.