Тёмный

django Architecture - Connection Management 

Hussein Nasser
Подписаться 431 тыс.
Просмотров 48 тыс.
50% 1

In this video I explore Django’s architecture especially related to backend connection management and persistent connections.
docs.djangopro...
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
network.hussei...
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
database.husse...
Introduction to NGINX (link redirects to udemy with coupon)
nginx.husseinn...
Python on the Backend (link redirects to udemy with coupon)
python.hussein...
Become a Member on RU-vid
/ @hnasr
Buy me a coffee if you liked this
www.buymeacoff...
Arabic Software Engineering Channel
/ @husseinnasser
🔥 Members Only Content
• Members-only videos
🏭 Backend Engineering Videos in Order
backend.hussei...
💾 Database Engineering Videos
• Database Engineering
🎙️Listen to the Backend Engineering Podcast
husseinnasser....
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
partner.canva....
Stay Awesome,
Hussein

Опубликовано:

 

28 сен 2024

Поделиться:

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист
Посмотреть позже
Комментарии : 81   
@imanmokwena1593
@imanmokwena1593 2 месяца назад
Thanks to watching this guy last year, I now understand everything in this video lol. Last year, it felt so foreign. Even before he mentioned cores, i already knew where he was heading. Hussein is an excellent teacher.
@icns01
@icns01 Месяц назад
Only a few minutes in, I can tell this is a good instructor...Good clear explanations backed by nice diagrams to further illustrate the concepts...Subscribed..👍
@phantomrex1348
@phantomrex1348 Год назад
I feel like Hussain is one of the only RU-vidrs that actually puts hours into his work. Keep up the good job
@muhammadahsan5676
@muhammadahsan5676 Год назад
I agree
@MartinsTalbergs
@MartinsTalbergs Год назад
Feel ya, men must do work to be men
@luuc
@luuc Год назад
'hours' is peanuts when it comes to content creation lol
@fadious_padious2711
@fadious_padious2711 Год назад
I feel like he is also one of the few tech RU-vidrs who knows his stuff inside and out. There are definitely others but the majority of tech channels are new grads reading snippets out of the documentation.
@SsaliJonathan
@SsaliJonathan Год назад
Thanks for making this video Hussein.
@MakeItStik
@MakeItStik Год назад
Really informative video. Make more like these !! We love it.
@swapniljena8684
@swapniljena8684 Год назад
I'm going to post this here as well as on Stackoverflow. So here it goes: I am running Django app using the WSGI server (gunicorn) on Linux. I am using the local memory for caching inside the application. I am using 3 workers (workers are altogether different forked processes that shared different memory spaces) and multi-threads using g-event under each worker. As the memory space is the same for each worker I'm safe having one single cache in one worker. But the load to the workers is balanced by the OS kernel so if the workers have different memory space so, the application cache 'set' by one worker can be made a 'get' by another worker or will it turn into an issue. I can solve this using unified caching using Redis or Memcached but wanted to clear this doubt before implementing another new service.
@RajaKhan-dc4vs
@RajaKhan-dc4vs 8 месяцев назад
you actually are optimising reads. By having a cache pool and pluging it in with your application leverages the cache to fetch responses faster. at some extent mongo or mysql also uses a lru buffer to bring in the recent read pages into memory and thus saving a lot of disk reads as ordered. As per your specification you want to do it in the previous hop by putting a cache layer in between your database layer and your api layer. But you always have to trade latency with cost.
@Dinesh-babu
@Dinesh-babu Год назад
Greatly informative like every other video. What do you think about netty and its use cases??
@Namjoonsbonsai
@Namjoonsbonsai 3 месяца назад
Can I learn backend without having any knowledge on front-end
@isegofficial
@isegofficial 3 месяца назад
Lol of course. I know just Django,I download free frontend templates, then chatgpt helps me write js fetch requests to query my backend api endpoints until I starting doing that myself too
@Namjoonsbonsai
@Namjoonsbonsai 3 месяца назад
@@isegofficial oh Oka. I'm learning django currently. Just started the class for django and I only know basic python by now so I was really scared 😭 I'll learn the other languages once I'm done with this
@Namjoonsbonsai
@Namjoonsbonsai 3 месяца назад
@@isegofficial also ,is there a possibility of getting job after I have full django knowledge?
@isegofficial
@isegofficial 3 месяца назад
@@Namjoonsbonsai of course. Don’t let the internet deceive you. Just build stuff. Like 2 good projects and upload on your GitHub and linkdln
@DhavalAhir10
@DhavalAhir10 8 дней назад
​@@NamjoonsbonsaiThere is no possibility, nowdays companies expected Fronted, Backend, Docker, Kubernetes, Apache, nginx, FastAPI bla bla bla
@SB-qm5wg
@SB-qm5wg Год назад
django have sqlite3 api?
@abiiranathan
@abiiranathan Год назад
I believe the problem is not with Django per say but python's threading model. Yes you guessed right, the GIL. Also to avoid data races, Django has to execute the queries synchronously. Even using async frameworks like django channel, you must wrap your async coroutines with synchronous helpers. Hello GIL
@pleli
@pleli Год назад
I'm guessing multithearding in python deserve a video of its own. Django usually used behind a wsgi server (nginx, gunicorn etc.) In a way you will have multiple processes with single thread on each process.
@samydriss5223
@samydriss5223 Год назад
Hussein, I love and appreciate all the content you put out. You truly are a rare breed of teacher. I can't express my gratitude enough as well as how much you have taught and have helped me throughout my career. I always learn something new when I watch your videos, take your courses, listen to your podcasts or read your articles. You have this great ability to go very deep into the technology stack without confusing people. Really, not all heroes wear capes. Also, are we seeing a teaser for a Udemy course here? 👀 Salam from 🇩🇿
@hxxzxtf
@hxxzxtf 6 месяцев назад
🎯 Key Takeaways for quick navigation: 00:00 *🐍 Django simplifies database connections by using objects instead of writing SQL queries.* 01:22 *💻 HTTP requests to Django connect to the database to execute queries and build responses.* 02:56 *📊 Django establishes database connections upon receiving requests, minimizing connection overhead.* 05:37 *🔄 Persistent connections are favored in modern architectures to minimize connection overhead.* 06:19 *🛠️ Django allows configuring persistent connections with 'connection max age=none' for efficient handling.* 07:00 *📄 Django follows a one connection per thread model, limiting connections to the number of threads.* 09:35 *⛔ Django's single connection per thread model can lead to blocking when multiple clients request database connections simultaneously.* 11:57 *🔀 Scaling Django with multiple threads can improve performance but should align with CPU core recommendations to avoid context switching overhead.* Made with HARPA AI
@vfryhn2
@vfryhn2 Год назад
Hussein could you please explain how WSGI servers work or why are they necessary for production with the python frameworks and not with other languages like node js or go? I just can't understand why they need that intermediate layer between the python server and the clients
@MaulikParmar210
@MaulikParmar210 Год назад
Your inbuilt http server isn't a process manager, what wsgi does is it taps into process management and separates application execution. Allows separating environments, application context and much more stuff that is required to invoke application efficiently and securely while keeping resource usage minimum. Implementation may vary depending on what server you are using but simple idea is, it's webservers responsibility to sandbox request response lifecycle and wsgi provides that barebones for any application and allows it to be build on top of already solved problems when you are writing web applications. This is same as (but not equal to i.e. similar to) using express in node, php-fpm, tomcat / websphere or any other application server, they are server appa helping in management and hosting of application itself while keeping them isolated from host system in one wya or other and providing specific standardised API at your application layer where you actually write business logic.
@vfryhn2
@vfryhn2 Год назад
@@MaulikParmar210 thank you!
@alexsmart2612
@alexsmart2612 Год назад
Hi Hussein, I don't quite agree with your discussion regarding context-switching near the end of the video. Your typical web-server is IO bound and like you explained the worker threads are just mostly sitting idly waiting for a response from the database. If that is the case, why would context-switching be a significant concern? Even if you have a hundred threads, the context switch only happens when there is an actual response from the db server, right? And when it does happen, the cost of the context switch (< 10 μs) is insignificant compared to typical db response times (at least a few milliseconds).
@syedazzamzafar596
@syedazzamzafar596 Год назад
I am beginner django developer. What course from you should I take first to improve my backend knowledge?
@dipjul
@dipjul Год назад
Very interesting! Can you please make a video on spring/springboot backend architecture?
@illichoskypiotr
@illichoskypiotr Год назад
Got confused. Isn't WSGI spinning a new forked subprocess every time a new request arrives? How does that relates with this one connection per thread?
@NavidRashik
@NavidRashik Год назад
On another note is that from 4.1 it support async orm and views so yah async is supported there as well
@zeyadmoustafakamal
@zeyadmoustafakamal 4 месяца назад
Well I am here from 2024 and I heard that in python 3.13 you might don't have to use the GIL if you precompiled the library. So will this affect django or not ?? and also can we just use some rust code to improve this. by (for example) rewrite the template engine and maybe the ORM in rust ??
@DanielTateNZ
@DanielTateNZ Год назад
Very interesting, I would be interested to hear your opinions on Laravel a popular PHP framework.
@TheLolEdits
@TheLolEdits Год назад
fast api next?!
@CamaradaArdi
@CamaradaArdi Год назад
This is my Well actually comment: you have two CPU threads per physical core. Also it's the kernel who does the context switching, not the CPU Also a thread that's waiting for network can safely be idle. As long as it's not doing work. The kernel is smart enough to give it a lower priority. Since read()ing a socket blocks (unless you're doing epoll) you can safely have 100 threads waiting for responses without a significant performance degradation. The problem will be of course if all of those threads are doing work at the same time, where you're absolutely correct.
@yashvarshney8651
@yashvarshney8651 5 месяцев назад
yep it is the OS's work to schedule and context switch processes
@aashayamballi
@aashayamballi Год назад
Django is ❤ Thank you Hussein for talking about Django and covering about this topic...
@matrixtoogood5601
@matrixtoogood5601 Год назад
Thank you for this amazing video! I would love to see a video on frontend listening models.
@isurucumaranathunga
@isurucumaranathunga Год назад
Hussain I don't hv words to say, these stuff are absolutely important and these are the actual engineering stuff. I love this content and really appreciate your effort on these videos. There stuff are really important for everyone specially for fresh graduates like me. Again thank you so much. Watching and noting down your videos is my new hobby. Thanks again.
@Fido1hn
@Fido1hn 4 месяца назад
had to speed it up to 1.75 to make it to the end. A lot could be said quicker
@pythoninarabic3341
@pythoninarabic3341 4 месяца назад
Great content, but you forget about GIL in python.
@noriking1467
@noriking1467 Год назад
Is it possible to do spring boot architecture and how project reactor works ? love the storytelling here !
@riadalimammedov7209
@riadalimammedov7209 Год назад
Hello Hussein,please record fastApi Architecture analysis.
@Tekecthegoat
@Tekecthegoat Год назад
so a single thread only handles on cuncurrent user? Am I missing something?
@andresgutierrez1804
@andresgutierrez1804 Год назад
What about asgi with one thread how the Django will handle the transactions?
@mehdi-vl5nn
@mehdi-vl5nn Год назад
what about async orm (django4+)?
@S--xc4rv
@S--xc4rv Год назад
Qns??? I'm downloading something in my laptop from mobile's network so very interesting thing is happening all the time while downloading. That is , whenever in every 2-3 sec there is spam of 1 sec in which the Mobile network speed goes to less than 0 kb . Why is that happening? And and there is no drop-down speed in laptop. It's normal
@nucleusnode23
@nucleusnode23 Месяц назад
really good man!
@tobidegnon4409
@tobidegnon4409 Год назад
I'm sot sure but the way you talk about threads here make me feel that you are actually talking about worker process, not threads, each process having it's own memory, I'm I missing something here?
@frozeneye100
@frozeneye100 Год назад
Lol. Love way you say the bad kid dB. Really the orm. I am beginner Dev level but crap. I am forcing myself to do ORM way coz want to learn but just SQL seems so much easier than rubbing ORM into compliance. Really ORM is the hardest and doing Django way. But at times the SQL is way quicker than orm... But yeah learning is cool and your vids even better.
@venkatavineelyalamarthy6958
Is this video added in Udemy course as well. ? On another note, I have a question, if Python is not concurrent (can't run multiple threads at the same time due to GIL), how does Django handle concurrent API requests? Does it use multiple processes.
@9899722801
@9899722801 Год назад
Hussein, Can we do data replication in django? I am very confused about it. Some says yes, some say it cant be
@1412-kaito
@1412-kaito Год назад
So, if in Django there is like 1 client, 1 thread, 1 query, then how does it tackle several (10k like) clients?
@fartzy
@fartzy Год назад
So is it not good to use multiple threads? Even though there is context switching wouldn’t still go much faster than blocking I/o? Isn’t that why it exists? GIL or not?
@rizwanfirdous
@rizwanfirdous Год назад
I highly appreciate your video and the hardwork behind it. One request to you could you please make a same kind of video for hibernate and tomcat. Please...
@KostasPanagias
@KostasPanagias 8 месяцев назад
Great content! Very well explained! Thanks for all the effort you put into this.
@charank7852
@charank7852 Год назад
Hi Hussein, Thanks for sharing quality content. Can we expect a course on LInux OS fundamentals on udemy as u created for networking ? Please reply. Thanks
@vishal240993
@vishal240993 8 месяцев назад
Any solution to this or we need to use db proxy like pgboncer ?
@hnasr
@hnasr 8 месяцев назад
correct pgbouncer proxy helps this.
@yashvarshney8651
@yashvarshney8651 5 месяцев назад
use starlette?
@bibekjoshi3522
@bibekjoshi3522 Год назад
Was looking at something similar and notification for this video came up
@MohdImran-uq5iv
@MohdImran-uq5iv Год назад
Thank you Hussain.. I am waiting for your 5th udemy course 😅
@adityajoardar578
@adityajoardar578 Год назад
I think this is one of the major drawbacks of doing backend processing in django, because if we have to update databases continuously in a background thread, that means we can't use persistent connection as it will keep one of the connections in idle state always
@mikopiko
@mikopiko Год назад
Could you do a similar video but for Rails? Would be hella interesting to compare.
@amirhmahmoodi4607
@amirhmahmoodi4607 Год назад
Hussein, what does happen if you put PgBouncer into this?
@alexsmart2612
@alexsmart2612 Год назад
Nothing happens. You are still limited by the number of threads, and since django only uses one connection per thread, your thread is still going to sit idly by waiting for a response from the db server. (PgBouncer would have helped with the first problem mentioned in the video, that is django by default creating a new connection for each request if django didn't in fact have a setting to enable persistent connections, but does nothing to help with the second one.)
@malekalhourani5930
@malekalhourani5930 Год назад
Hi Hussein, thanks for such content, it is really helpful and help me to evolve. I feel like using your old way of explaining stuff (using a PowerPoint presentation) is much easier to follow and make the topic you are trying to explain much simpler, I hope you return back to use it again, Thanks.
@buildmore9829
@buildmore9829 Год назад
You remind me of Firas Zahabi.. related?
@holthuizenoemoet591
@holthuizenoemoet591 Год назад
I would love to see a video on a good multi client per thread model
@mustaphab32
@mustaphab32 Год назад
Great stuff! it think it worthes mentioning that ORM only lately started being async in a very recent version, which means the thread will keep the request waiting and any other calculation as-well! we d love to see how to do parallel processing in python and especially Js and Python!
@_overide
@_overide Год назад
As always, great video. Thanks you so much, learned few new things!
@LOLxUnique
@LOLxUnique Год назад
Great video from one Hussein to another
@Ysh.CS_guy
@Ysh.CS_guy Год назад
Great video 📹 👍 👏
@SachinDolta
@SachinDolta Год назад
Much love
@ujjwalbansal5649
@ujjwalbansal5649 Год назад
Awsome content.
@mlsandreas
@mlsandreas Год назад
Thank you sir!! Awesome video!!
@101kawsar
@101kawsar Год назад
Thank you 🥺❤️
@arcstur
@arcstur Год назад
Awesome video!
@uEisenhower
@uEisenhower 7 месяцев назад
Explaining before you criticise, i get you're making point but I'm learning nothing except disadvantages of django, which isn't the title
@himanshukandpal5585
@himanshukandpal5585 7 месяцев назад
The video did explain the architecture to you.
Далее
Pinterest moves to HTTP/3
25:06
Просмотров 35 тыс.
Threads and Connections | The Backend Engineering Show
49:30
titan tvman's plan (skibidi toilet 77)
01:00
Просмотров 5 млн
Django Interview Questions (Junior Developer)
30:26
Просмотров 124 тыс.
Node.js is a serious thing now… (2023)
8:18
Просмотров 650 тыс.
NGINX Internal Architecture - Workers
15:07
Просмотров 45 тыс.
FastAPI, Flask or Django - Which Should You Use?
9:49
Dan Palmer - Scaling Django to 500 apps
29:10
Просмотров 10 тыс.
NodeJS Architecture - I/O
25:09
Просмотров 47 тыс.