Persistent Volumes on Kubernetes for beginners

Подписаться 77 тыс.

Просмотров 51 тыс.

50% 1

Subscribe to show your support! goo.gl/1Ty1Q2 .
Patreon 👉🏽 / marceldempers
In today's video, we're going to be talking about persistent volumes and how they help us to run workloads that require persistence. These can include databases, caches, message queues and anything that requires data to be stored.
But remember applications that require persistence can be complex. To help run these complex applications, Kubernetes uses StatefulSets instead of deployments.
Remember to checkout my video on StatefulSets here - • Understanding Stateful...
Also if you want to support the channel further, become a member 😎
marceldempers.dev/join
Checkout "That DevOps Community" too
marceldempers.dev/community
If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)
Checkout the source code below and follow along :)
Kubernetes Guide for Beginners:
---------------------------------------------------
• Kubernetes development...
Kubernetes Monitoring Guide:
-----------------------------------------------
• Kubernetes Monitoring ...
Kubernetes Secret Management Guide:
--------------------------------------------------------------
• Kubernetes Secret Mana...
Like and Subscribe for more :)
Source Code
--------------------------------------------------------------
github.com/marcel-dempers/doc...
Follow me on socials!
marceldempers.dev
Twitter | / marceldempers
GitHub | github.com/marcel-dempers
Facebook | thatdevopsguy
LinkedIn | / marceldempers
Instagram | / thatdevopsguy
Music:
Track: Fox Beat 2 - Jeff Kalee - Pillow Talk - Royalty Free Vlog Music [BUY=FREE] | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / jeff-kalee-pillow-talk...
Track: Amine Maxwell - Le Soir | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / le-soir
Track: Sappheiros - Spirit | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / spirit
Track: n0tavailable - mourner - sad trap beat | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / mourner
Track: souKo - souKo - Parallel | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / parallel

Наука

Опубликовано:

30 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 95

@MarcelDempers 4 года назад

Understanding StatefulSets for beginners👉🏽 ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zj6r_EEhv6s.html

@modenie 4 года назад

You're an excellent teacher Marcel, thank you for making these videos. Your vlog format is groundbreaking in tech. While there are plenty of others on youtube offering fantastic content, you've introduced (at least to me) an element that has made it so much easier to take the information in. Well done man!

@shanef1867 3 года назад

Possibly the best tutorial on the topic!!

@fxylxy 3 года назад

Hi Marcel Thank you so much for this simple explanation with a REAL WORLD example of persistent volumes. I have been wracking my brains trying to tie together pv, pvc, and statefulsets. You explained and demonstrated in 20 minutes what other people couldn't get across in hours of demos. Love your channel, keep it up mate 👌👍🙂

@ourjamie 4 года назад

Marcel, your stateful and persistance videos demos are brilliant, thank you for putting in the work to share your knowledge, your delivery is brilliant

@yovangrbovich3577 4 года назад

Thanks Marcel! Would really love to see one on Argo Workflows/Events for setting up and triggering pipelines in kubernetes!

@aufkeinsten7883 2 года назад

You're on another level mate, so concise, so clever, so well spoken, love your videos

@krishnamohanyerrabilli4040 Год назад

Man this is just awesome, I'm beating myself for not understanding, after watching this I can assure myself I got the idea I remember this quote, You can't connect the dots forward - only backward - Steve Jobs Thank you for the lecture marcel, You made my day.

@manideep6814 2 года назад

You are too good at this, your videos have helped me understand and breakdown those heavy jargons on K8s doc pages. Thank you.

@FuhQshane Год назад

Thank you so much for this. I’ve been in class all week and studying my ass off and just have NOT understood the pv claim until now. Now I can get some sleep. Lol

@PeterMehr 9 месяцев назад

Great explaination!! After watching at least 10 other tutorials, yours was the first one that really worked!

@karatefutzi Год назад

Bro out here making every DevOps engineers life so much easier, thanks the school system around the globe should take you as an example on how to transfer knowledge....

@wihannel 2 года назад

This video is almost as good as you triceps. Respect

@rolandvarga3912 3 года назад

this devops guy LIFTS!!

@MarcelDempers 3 года назад

💪🏽

@vincenzocalia6552 Год назад

Marcel you're a good man. You always explain things very clearly and dough you type as fast as the light you repo in GitHub shows everything. So many many thanks. I was stuck trying to bind a local PV to a mongo db StatefulSet , looking at your code(dough for Postgres ) made me find a way to debug my pv.yaml.. it results that was a typo in the nodeAffinity key.. thank you so much again. Vinny

@joaosakai 3 года назад

Thanks a lot, Marcel this is the best video about the concept that I've found on RU-vid, your explanation coming from docker volumes creates a solid ground to easily understand the kubernetes part :smile:

@mrpoeky2527 2 года назад

Thanks for this amazing video, I was really struggling with this part in Kubernetes. It helped me a lot :)

@maiezbenromdhane8233 2 месяца назад

excellent and well detailed! thank you for the explanation dude!

@bnssoftware3292 Год назад

You sir are amazing! Love your style and your editing and your teaching ability. You're like the Kubernetes god! Keep up the great work!

@bass9454 3 года назад

Your channel is a godsend 🙏🙏🙏

@syedmuhammadzaeemhasankazm7740 3 года назад

Thanks alot Marcel, I finally understood the concepts of PV, PVC and how to use them with Postgres thanks to you. Massive thanks! Also I would like to request if you could teach how to do authentication and authorizations from nginx-ingress using Keycloak, as I am sure many people struggle with it... And there arent any RU-vidrs like you, who teach from the basics and build up concepts upon that, rather they usually just assume that we just know stuff. Also, since KeyCloak is really popular Auth as a Service, so I am sure it would be very helpful for others too, so please consider doing a video on it, please. Thanks a lot in advance.

@marcziel5424 2 года назад

Good video generally, but there's a misconception about containers. A container is not a process, it's a sandboxed runtime environment in which you can run one or more processes. You can stop a container (`docker stop `) which terminate all processes inside of a container and then run it again (`docker start `) and the files will still be there. Kubernetes never bothers to restart a container - when it restarts a pod it recreates containers with the same IP address assigned, etc. so you might think that containers are completely stateless (and they're typically treated as such) but a container keeps its filesystem state after all of its processes terminate which is pretty useful when it comes to retrieving logs and inspecting what caused the termination.

@semibenhsan9551 4 месяца назад

Just Amazing ! Thank you very much man ! Keep Going !

@ravindramahajan2685 3 года назад

Excellent video It helped me to clear my concept Thanks.

@nikolai.kolosov 3 месяца назад

Thanks for the great content Marcel

@evgenisakhatsky612 Год назад

Great explaination!

@ghauri001 3 года назад

Thank you for this video, please make a video on the production-ready Postgres database.

@NIKHILVERMA-xr7lc 3 года назад

Thanks for sharing. Gives me better understading than k8s doc.

@karthickkarthi.3342 2 года назад

Every sec has information in your videos.

@marwanghubein7879 3 года назад

thanks a lot , very informative and useful

@carlosgoncalves56 2 года назад

very good video, tks Marcel.

@dapkman9354 3 года назад

You're great. Good job!

@Toemo1990 2 года назад

Great video!

@shubhamtomar676 2 года назад

Very informative

@martinpetersson6979 2 года назад

Best in class 🙌✌️

@maheshkariya 3 года назад

Thanks brother

@hamidullahmuslih6301 2 года назад

awesome !!

@nikhilverma4976 2 года назад

You are really good at explaining things. Can you do a tutorial on tricep extensions? Thanks!

@kingjameszz Год назад

super underrated comment lol

@bashir26 2 года назад

great video thank you

@magnomaximo2471 2 года назад

Greate, thanks a lot for this video it cant be better

@jaysistar2711 3 года назад

Thanks! I have a single tennet app that will have multiple copies of the same pod running in different namespaces with different persistant volume claims. Many of these resources have the same name, but are just deployed in different namespaces. It would be great to see a video about how a single pod .yaml could be used in that way to create different volumes.

@stefano8936 2 года назад

Thank you, the guide was very clear and concise. Do you have a guide on how to handle a "real world" project in the different phases? For example I imagine one would develop in local on minikube, then deploy on the cloud (AWS, GCP, etc), maybe having TEST/PRODUCTION as different clusters. How do you manage all this? Does it make sense to have in the same git repo different kubernetes folders, one for each environment, containing the appropriate yml? Or would you rather have different branches? A guide on this would be great.

@evgenisakhatsky612 Год назад

Thank you so much

@janituma9769 4 года назад

Thx Great Video 👍👍👍👍

@c0nsaw 3 года назад

Excellent

@ribalaladeeb8310 2 года назад

Great Video, breaking down the content in ingestible bytes. One improvement I'd suggest is timestamping your video so that people can skip ahead to the exact part they are looking for. Keep doing you man your content is great

@bAbApersianwale 2 года назад

Thank you

@hurolinci5986 3 года назад

Suffering for the past week figuring out how to persist data on a database container has finalldy led me here to find the solution...

@laxmankamble580 Год назад

Hi Gaurav, is this end of K8s series, please add if few more topics pending. and i must saym your tutorial made my concep clear and handson experianced.

@wesleyvandermeer9313 Год назад

HERO!

@haneefmustafa1600 2 года назад

Perfect thanks! Super .......

@xxdivinexx90 3 года назад

Hey Marcel, love the videos! I think there is a small mistake with your explanation on PVCs claiming storage from PVs. You cannot have several PVCs bound to one PV, once a PVC is bound to a PV the PV will not be able to bind to another PVC but many pods can use the same PVC. You create PVs to configure a certain way to claim storage and to use it you bind a PVC to it. If you aren't using PVs the storage class dynamically creates and provisions a PV for the requested PVC. Keep up the great content!

@weitanglau162 3 года назад

Great video Marcel! However, I do have a question. I am able to persist data through the lifetime of the container / pod, but not able to do so if I restart my cluster. How do I achieve this? I am planning to use local storage class

@davidchoy6220 3 года назад

The local file system path you specified, you would see files in there after creating the table right?

@abd.alharbi Год назад

Thank you kindly for this video. One question though. Why would a pvc specifies a storage class? isn't it already defined in a pv?

@AnandKumar-dc2bf 3 года назад

That Brilliant Guy who is Awesomeeeeee.....

@A_Dudkin Год назад

Hi Marcel and thanks you for the great job you do! Can you give a hint or direction please. I've deployed wp with pvc using nfs-provisioner on bare-metal. nfs share is located on the neighbouring vm host in the same subnet and basically both k8s nodes and nfs node use the same physical disk. What I see is that i can easily store anything on this share manually from any pod but WP + mysql is very slow when using this nfs (e.g. it take up to 15 sec to read a few KB files). perms on the share are 777. Access mode readwritemany if it makes any difference. I've googled for this issues and it turns out quite common but there is no specific solution or reason that I could find. Any advices please?

@syedshamshami5250 3 года назад

What is the best way to setup storage server and its configuration for kubernetes cluster persistence volume for production system? NFS or ISCASI any hardware brand specific. Thanks !

@linuxistul 3 года назад

Hi Marcel, I've created a PersistentVolume using the hostpath StorageClassName and I mounted it in a mysql pod. The Persistent Volume hostpath is "/var/folders/data", and it's mounted in "/var/lib/mysql" in the Container. What's strange is that when I list the PV hostpath "/var/folders/data" it's empty. But when I create a test database and then delete and recreate the deployment the test database is still there. You know where the container "/var/lib/mysql" files could be found? I was expecting to find them in the PV hostpath "/var/folders/data". I'm using docker for mac.

@NiteshKumar-do4en 4 года назад

Please a video on storage class for aws

@tanchwa3740 9 месяцев назад

How does it work, then, if I have multiple apps that need to share the same physical storage. From what it sounds, I still need to create a pv for all of them, but can mount them to the same physical path in the storage class, nfs for example. But then how does the storage amount come into play? How can a pv reliably tell a claim “hey, I have this much storage available, use me.” If there is overlap in the physical storage?

@anthonypham7518 4 года назад

Hey Marcel, once again thanks for another excellent video. One question though. Would the Postgres data survive a Docker Desktop restart?

@MarcelDempers 4 года назад

Technically, it should since the example used host path persistence so unless the host is destroyed, it should persist yes

@griet535 3 года назад

@@MarcelDempers Hey Macel, thanks for the video. unfortunately the data being lost when docker restarts.

@ezhisaivallabanr3189 2 года назад

@@MarcelDempers data loss after restarting the docker desktop

@rajump4276 Год назад

@13:40, How is the PersistentVolumeClaim referencing the PeristentVolume (example-volume) ? How does the pvc know which PV to use?

@georgelza 2 года назад

how would I add nodeselectos and affinity rules for statefulsets?

@braxtonl3319 3 года назад

Hey Marcel, great videos. After I altered the ConfigMap to use a different POSTGRES_USER and deleted/recreated everything including the PV, the old database user & table existed. Can you explain why that happened even after I deleted the PV? Is there more to "scrub" here? I assumed recreating everything would result in a fresh database with the new user credentials. Thanks!

@MarcelDempers 3 года назад

This highly depends on the storage class you use. In this demo I use host path which means data is persisted on the node at a path (which can be changed). The get a fresh storage, you either have to change the path to get a new folder on the host. Remember your old data would remain. Or SSH to the node, clean it up, Or run another container with a host mount and carefully run "rm" command to delete the content

@braxtonl3319 3 года назад

@@MarcelDempers Thank you for such a quick response! I used hostPath/your tutorial code too. If you have any advice on ssh-ing to the docker-desktop (Mac) node to clean up, I'm all ears. Thanks again for these videos, very helpful.

@jezzadiaz5972 3 года назад

Hello. I have a question. What if you would want to save and mount the logs of a server application? Should you still use deployment or statefulset?

@MarcelDempers 3 года назад

For logging, I have a separate video in the pipeline, You would rather export logs from the app and not store it in a volume. Take a look at the EFK\ELK concepts, where basically every pod in K8s writes logs to stdout. It is best practise not to write logs to file and rather use stdout\stderr. These logs will automatically end up in the host docker logs which you can push out of the system using Fluentd\FluentBit. This allows you to centralise logs for all deployments\statefulsets running in your cluster under something like Elastic Search or 3rd party application.

@jezzadiaz5972 3 года назад

@@MarcelDempers oh. interesting. I'll look it up! Thank youuu so much. been watching your videos. so much helpful for me. great videos btw :)))

@sushil1200 3 года назад

where is /var/lib/postgresql/data located? Is this location inside pod?

@turalaliyev6288 3 года назад

Hi Thanks you for great video.! I did the same and tried to deploy jenkins but when i create pv and pvc . mnt/data is empty. and i used volumemount to var/jenkins_home.

@MarcelDempers 3 года назад

If you need help with Jenkins checkout my CI/CD playlist, there are two Jenkins guides with source code for you 💪🏽👊🏽

@user-fi9hn4vd5s 2 года назад

Hi, i got a qusetion, if my volume is stored application's log file, this volume should be oversized, how could i logrotate in this volume to manage the log file size? do u have any suggestion? thank u so much

@MarcelDempers 2 года назад

I would recommend looking into Fluentd for shipping logs and not store logs in a volume. K8s has pod logs go to a container runtime folder and Fluentd mounts that directory to forward logs out. The container runtime is designed to rotate logs automatically. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-6kmHvXdAzIM.html

@user-fi9hn4vd5s 2 года назад

@@MarcelDempers thank u so much! its useful to a kubernetes beginner user.

@derencevoo9146 3 года назад

Hi Marcel, my setup is a bit unique, I have created a K8S cluster on a few VMs in our own private cloud and all the nodes are mounted with a single NFS drive (say /data), can I define a storageclass of hostpath type using /data? Does PV has to be mounted to all the nodes?

@MarcelDempers 3 года назад

That depends on the app you're planning to run and how it uses the storage. If you plan on using a statefulset its recommended that each pod has its own storage. StatefulSet will use a storage class to provision a PV per pod and not share the same storage. Hostpath will work, but if you mount a shared storage to all the nodes, they may override each other's data. So unless you mount unique storage to each node, it may not be ideal. I'm new to NFS, but i would see if its possible to provision NFS dynamically using a storage class and see if the statefulset can issue PV + PVCs per pods using NFS dynamic provisioning. That way each pod will get its own NFS storage provisioned by k8s when you scale up and down

@MarcelDempers 3 года назад

Checkout NFS provisioning using Rook rook.io/docs/rook/v1.4/nfs.html Seems interesting. If you have not worked with Statefulsets I'd highly recommend to ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zj6r_EEhv6s.html

@derencevoo9146 3 года назад

@@MarcelDempers thanks Marcel, I have deployed Rook Ceph RBD in my cluster and now the applications now works like charm

@marwanghubein7879 3 года назад

how to backup and restore database in this case ?

@devopskey6251 3 года назад

but If we have a three node cluster one is master other two are worker first time our Postgres pod is schedule on worker1 now its PV on worker1 and second time after deleting the pod k8s schedule it on Worker2 now pod create its PV on Worker2 but now data is not persistence. because data on worker1 can not allocate to worker2 pod? What's a solution?

@MarcelDempers 3 года назад

This is because you are using a deployment with PVs which is not meant for stateful workloads. You'll need to learn about StatefulSets ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-zj6r_EEhv6s.html