We use a combination of AWS autoscaling and Nix to make our CI pipeline bearable.
For autoscaling we use terraform-aws-github-runner which will bring up ephemeral AWS machines if there are CI jobs queued on GitHub. Machines are then destroyed after 15 minutes of inactivity so they are always fresh and clean.
For defining build pipelines we use Nix. It is used both for building various components (C++, Go, JS, etc) as well as for running tests. This helps to make sure that any developer on the team can do exactly the same thing that the CI is doing. It also utilizes caching on an S3 bucket so components that don't change between PRs don't get rebuilt and re-tested.
It was a bit of a pain to set up (and occasionally a pain to maintain), but overall it's worth it.
For autoscaling we use terraform-aws-github-runner which will bring up ephemeral AWS machines if there are CI jobs queued on GitHub. Machines are then destroyed after 15 minutes of inactivity so they are always fresh and clean.
For defining build pipelines we use Nix. It is used both for building various components (C++, Go, JS, etc) as well as for running tests. This helps to make sure that any developer on the team can do exactly the same thing that the CI is doing. It also utilizes caching on an S3 bucket so components that don't change between PRs don't get rebuilt and re-tested.
It was a bit of a pain to set up (and occasionally a pain to maintain), but overall it's worth it.