GitHub Actions Self Hosted Runner (Autoscaling with Kubernetes)

Подписаться 48 тыс.

Просмотров 21 тыс.

50% 1

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com
▬▬▬▬▬ Experience & Location 💼 ▬▬▬▬▬
► I’m a Senior Software Engineer at Juniper Networks (12+ years of experience)
► Located in San Francisco Bay Area, CA (US citizen)
▬▬▬▬▬▬ Connect with me 👋 ▬▬▬▬▬▬
► LinkedIn: / anton-putra
► Twitter/X: / antonvputra
► GitHub: github.com/antonputra
► Email: me@antonputra.com
▬▬▬▬▬▬ Related videos 👨‍🏫 ▬▬▬▬▬▬
👉 [Playlist] Kubernetes Tutorials: • Kubernetes Tutorials
👉 [Playlist] Terraform Tutorials: • Terraform Tutorials fo...
👉 [Playlist] Network Tutorials: • Network Tutorials
👉 [Playlist] Apache Kafka Tutorials: • Apache Kafka Tutorials
👉 [Playlist] Performance Benchmarks: • Performance Benchmarks
👉 [Playlist] Database Tutorials: • Database Tutorials
=========
⏱️TIMESTAMPS⏱️
0:00 Intro
1:10 Create GitHub Repository
1:42 Create EKS Cluster with Terraform
3:38 Install Cert-Manager on Kubernetes
4:40 Install actions-runner-controller
8:46 Create Single GitHub Actions Self Hosted Runner
12:35 Create Multiple GitHub Actions Self Hosted Runners
15:17 Autoscaling with Kubernetes
=========
Source Code
📚 - Tutorial: github.com/antonputra/tutoria...
Music by Vincent Rubinetti
Download the music on Bandcamp:
vincerubinetti.bandcamp.com/a...
Stream the music on Spotify:
open.spotify.com/album/1dVyjw...
#GitHub #Kubernetes #GitHubActions

Наука

Опубликовано:

7 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 84

@AntonPutra 9 месяцев назад

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com

@smokzpolski 2 года назад

More and more useful videos! Thank you!

@AntonPutra 2 года назад

Thanks Alex:)

@SuvodeepPyne Год назад

Awesome work! Thank you so much for sharing this!

@AntonPutra Год назад

Thanks Suvodeep!

@valour.se47 2 года назад

Keep up the good work, as always to the point.

@AntonPutra 2 года назад

Thanks Valour!

@saadullahkhanwarsi5853 Год назад

Amazing man Keep sharing that level of content

@AntonPutra Год назад

Thanks SAADULLAH! Will do!

@Han-ws8he 20 дней назад

Very nice! Thanks!

@AntonPutra 20 дней назад

thanks! I'll refresh series soon..

@elibukin4840 7 месяцев назад

super stuff! thank you my good dude, that helped alot!

@AntonPutra 7 месяцев назад

Thanks, Eli!

@siamak.hatami Год назад

It is more than great. Thanks.

@AntonPutra Год назад

Thanks siamak!

@sacardillas 2 года назад

amazing job, thank you so much for this material

@AntonPutra 2 года назад

Glad it was helpful!

@jonathanhle Год назад

Thanks!

@AntonPutra Год назад

Welcome!

@mondo1926 Год назад

Clear, concise and to the point. Nice job on this video Anton. I just have a general question about the custom resource definition for k8s self hosted runners. Did you create it? Is there an officially supported CRDs for Github self hosted runners?

@AntonPutra Год назад

Sorry for the late response, here is a reference - github.com/actions-runner-controller/actions-runner-controller/tree/master/config/crd

@haseebahmad857 Год назад

Hello Anton. Thank you very much, it was straight to the point and a very good explanation of the topic. I have one question, I am just curious if we can provide custom docker image to our runner pods having all the dependencies for the applicaton build preinstalled. Any article/video will be much appreciated. Thanks in advance

@AntonPutra Год назад

Welsome, you can build your own image github.com/machine-learning-apps/self-hosted-k8s-runner/blob/master/README.md#optional-customize-your-self-hosted-runner

@csvidzz 2 года назад

Hello Anton. Thank you very much, it was straight to the point and a very good explanation of the topic. I have one question though: Since pods don't store the state, I sometimes want to cache the docker image after the first build to make the actions execute faster. Is there a way with this approach to cache the built docker images, so I have faster builds in the future? Thank you very much in advance.

@AntonPutra 2 года назад

I think I already sent you a link to this - github.com/marketplace/actions/docker-layer-caching maybe help

@kirbyliu8965 Год назад

Great video Anton! I had a question regarding installing cert-manager, if you aren't handling SSL/TLS for DNS, did you need to install cert-manager? You mentioned the webbook as an option which I'm sure is where the cert-manager would have been supported/used.

@AntonPutra Год назад

Thanks, you need cert-manager only if you want to expose your service from kubernetes to the internet and encrypt the traffic with SSL/TLS (HTTPS). If you just want to use plain http, you don't need cert-manager.

@mohammadkarimi9386 2 года назад

Awesome tutorial. Have learned so much. Would you create another video on how to create this with webhook and using OIDC or similar

@AntonPutra 2 года назад

Propably in the future, combining with org level runners.

@jimbrannlund4677 2 года назад

I just did this (webhook), it was pretty straightforward. But then we have a sweet setup with Contour and Envoy which makes ingresses a breeze.

@jishaashok Год назад

@@jimbrannlund4677 Can you please share the entire steps here?

@jimbrannlund4677 Год назад

@@jishaashok I can try

@LunaSicilian Месяц назад

This is legacy version. It would be greate if you make an other video about GitHub Actions Self Hosted Runner with runner-scale-set chart and runner-scale-set-controller chart.

@AntonPutra Месяц назад

I was thinking about refreshing. Maybe I'll create a mini playlist for GitHub Actions.

@kayoutube690 3 месяца назад

Do you have module how to build image via tags, pr and branch using github actions?

@hrushie Год назад

Hi Anton, Great video! but I have a question, what if we want to use our own docker image for the runners? I did not see any configuration for custom docker image. Thank you!

@AntonPutra Год назад

Thanks! You can then configure the runner to use a custom docker image by configuring the image field of a Runner or RunnerDeployment. Doc - github.com/actions-runner-controller/actions-runner-controller/blob/master/docs/detailed-docs.md#software-installed-in-the-runner-image

@amalberty Год назад

Hello Anton. Thank you for the video. Re: AWS EKS, are you using Fargate or EC2 instance for autoscaling?

@AntonPutra Год назад

Hi Alejandro, in that video as far as i remember i was using ec2 instances.

@amalberty Год назад

@@AntonPutra Thank you

@armandoleon9901 Год назад

Awesome video!!! Curious, is it possible to configure the images of the runners? Especially via the runner deployment?

@armandoleon9901 Год назад

whoops, ignore, I read the docs and found my answer

@AntonPutra Год назад

yeah, you should be able to create custom image

@InigoMontoyaGAMING 2 года назад

Thanks a lot Anton! Can you please create a video on how to host Self Hosted Runner with Fargate profile? or any tips about it?

@AntonPutra 2 года назад

Sure, I already have a video for AWS Fargate

@dtwilliams Год назад

Hi. Could you let me know where your Fargate video is? I couldn’t find it. I wasn’t sure if I could use Fargate to host self hosted runners as Fargate doesn’t allow privileged containers.

@spasham74 Год назад

Great video. Question. I have a self hosted runner deployed in AWS eks. I am able to build most the stages in GitHub actions. I understand docker is not longer supported on k8s so how do you build images on self hosted runner on k8s

@AntonPutra Год назад

You don't have to use docker to build docker images, for example you can use kaniko

@spasham74 Год назад

@@AntonPutra do you have any example snippets that you can share? There isn’t any office GitHub actions for kaniko

@OutOfDevOps 2 года назад

Awesome tutorial, I implemented something similar in GCP but using VMs, this allows me to scale down to zero when idle. Do you think would be possible to implement something similar in AWS? I didn’t put the link but you should be able to find it in my channel.

@AntonPutra 2 года назад

Thanks! You can defiantly scale down to 0 Kubernetes instance group.

@OutOfDevOps 2 года назад

@@AntonPutra do you have to pay for the master nodes even if you have zero nodes?

@AntonPutra 2 года назад

@@OutOfDevOps well, not even masters, you still need to run DNS and some other components. Even autoscaller has to be deployed somewhere. It's a good solution when you already have a cluster and want to share it with ci/cd.

@jimbrannlund4677 2 года назад

Execellent video, thank you! What's the point of RunnerDeployment, I don't understand the difference with custom runners.

@AntonPutra 2 года назад

It's a Custom Resource Definition provided by actions-runner-controller, that watches the repository and creates pods to run your workflows. Here is a link - github.com/actions-runner-controller/actions-runner-controller#runnerdeployments

@jimbrannlund4677 2 года назад

@@AntonPutra Thanks for the fast reply!

@aromaticcut6898 Год назад

Made my day!!

@AntonPutra Год назад

Thanks =)

@vovaripetsky6885 2 года назад

what image are those containers using?

@AntonPutra 2 года назад

this one - Docker: github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/Dockerfile, image - github.com/actions-runner-controller/actions-runner-controller/blob/master/charts/actions-runner-controller/values.yaml#L40-L42

@vinayhegde9888 Год назад

Can we setup this on EKS-Fargate!!?? Plz respond. I’m getting an error during this implementation on Fargate.

@AntonPutra Год назад

I haven't tried it yet, but my guess you can. You can run controller on the regular instance group and workers on fargate

@swathiashok2586 2 года назад

Can we not configure same runner for 2 repos? in k8s infrastructure

@AntonPutra 2 года назад

You can configure runner on org level which is more convenient actually.

@mrdeshapiro Год назад

for horizontal run autoscalar you specify repositoryNames:. What if you want to allow all repositories in an org?

@AntonPutra Год назад

Take a look at this "Adding ARC runners to a repository, organization, or enterprise" github.com/actions/actions-runner-controller/blob/master/docs/choosing-runner-destination.md

@mrdeshapiro Год назад

@@AntonPutra It sounds like metrics requires you to list repositories if I am reading it correctly. Note that I set this up, but I never seem to see scaldown happening. I even had cases where I was left with many sitting offline. I ended up manually deleting those in the Github console.

@jimbrannlund4677 2 года назад

Why does the app (the runners) need read & write for Administration?

@AntonPutra 2 года назад

I found it in the docs "Administration (read / write)". Here is a link - github.com/actions-runner-controller/actions-runner-controller#deploying-using-github-app-authentication

@shohidurrahmanshaju5798 11 месяцев назад

How can I have the autoscale runner pod to gave some linux dependency packages installed?

@AntonPutra 11 месяцев назад

You need to build custom docker image

@shohidurrahmanshaju5798 11 месяцев назад

@@AntonPutra thanks for the reply

@shohidurrahmanshaju5798 11 месяцев назад

@@AntonPutra I have tried that, but I get pod status CrashLoopBackOff for the actions-runner-controller pod.

@vamshikrishna-th2ly 2 года назад

I have couple of questions related to self hosted runners. 1. I have configured self hosted runner and i always wanted to be active so that my other team members can use it. But when my laptop is not turned on, other people who are running the pipeline, it is going to queue. Any idea on how to overcome that? 2. Lets say I create 3 individual self hosted runners in 3 of my team mates machines. I need github pipeline to pick the one which is online not the one which is offline. Is it possible and how do we do that.?

@AntonPutra 2 года назад

i never ran self hosted runners for prod environments on local machine. I would assume it should automatically run on online runners, or you can try to use labels/tags.

@Kruziikrel13 Год назад

Gee, Spamming everyone with this query.

@shohidurrahmanshaju5798 11 месяцев назад

Currently if I want to run cypress test on selfhosted runner, I get error saying dependency is not installed

@AntonPutra 11 месяцев назад

You can build your custom Docker image that you can use as the runner with all your dependencies.

@shohidurrahmanshaju5798 11 месяцев назад

@@AntonPutra Okay, thanks for the reply.

@user-or5mt2pc1f Год назад

How to add lablel to runner ?

@AntonPutra Год назад

it's been a while, next time when i refresh this tutorial I'll point it out

@AntonPutra 2 года назад

Build a Docker Image and Publish It to AWS ECR using Github Actions - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-Hv5UcBYseus.html Build a Docker Image and Publish It to GCP GCR & Artifact Registry using Github Actions - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-6dLHcnlPi_U.html

@AntonPutra Год назад

👉 How to Manage Secrets in Terraform - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-3N0tGKwvBdA.html 👉 Terraform Tips & Tricks - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-7S94oUTy2z4.html 👉 ArgoCD Tutorial - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zGndgdGa1Tc.html

@user-yd9sn5hw4h 11 месяцев назад

Anton hi, thanks for a great tutorial. Can you pls suggest how to define image for self hosted K8s github actions runner? Let's assume I have automation tests written on Java and to run those tests I need Chrome browser, for time saving purpose - it could be great to have already installed on my runner Java and Chrome , so I will justly run the tests and not perform any installations from workflows (within github actions).