Designing my Server Architecture for Scalable Web Applications

Подписаться 424 тыс.

Просмотров 440 тыс.

50% 1

brilliant.org/DevonCrawford
^ This video is sponsored by Brilliant! To master computer science and support my channel use my link :)
In this video I decided to share my research over the past few weeks since I have encountered many roadblocks when designing my server architecture. My current design is just a single node js + express application which handles all routes for my front end, back end + any other web apps I want to integrate. This is design is okay for small websites but as soon as you want to do multiple apps it becomes problematic. Pretty sure my new design is called a "micro-services" approach. and lately I've just done a lot of research on what tech would best suit my needs. A few things I didn't get the chance to mention in this video.. 1) I don't actually know JavaScript that well tbh.. any real algorithmic processing I will do in c++ or c and then integrate that code using the node ffi library. 2) I only want to use JavaScript, node js and express to act as the glue between my programs and the website. I prefer languages like C, C++ or Java since the compiler will yell at you for anything you've done wrong. I'll do all my real processing in the languages which I know best. 3) See below for a bunch of resources that I found helpful when learning all of this stuff!
And also, I would really like to hear everyone's feedback and possible improvements or ideas you may think of when designing such a system.
Technology used in this video
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Server: Node js + Express (Javascript)
Front End: Next js + React (HTML, CSS, JS)
Reverse Proxy & Load Balancer: Nginx
Website Host: Digital Ocean Cloud computing platform
Data: JSON, AJAX (asynchronous calls from client to server)
Next Video:
Previous Video: • Nerds go Clubbing..
Research / Resources
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Production best practices (express): expressjs.com/en/advanced/bes...
Client vs Server side rendering: medium.freecodecamp.org/what-...
Next js: nextjs.org/
Nginx config explained: www.digitalocean.com/communit...
Social
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Website: devoncrawford.io
Twitter: / devoncrawford13
Snapchat: / devoncrawfordyt
Instagram: / devoncrawford_
Github: github.com/devoncrawford
Discord: / discord
Gear
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
- Video Editing Software (Premiere Pro CC): prf.hn/l/BOomWo3
- All Adobe Apps (I use this plan): prf.hn/l/Xv5qk5Q
- Keyboard (Velocifire TKL01): amzn.to/2AQTjJQ
- Monitors (Dell U2518D) : amzn.to/2AQQmco
- Webcam (Logitech C920): amzn.to/2Cw588c
- Desk Microphone (Blue Yeti): amzn.to/2FPFmzi
- DSLR (Canon T7i): amzn.to/2T54mpZ
- Wide Lens (Tokina 11-16mm F2.8): amzn.to/2MlpvcO
Music
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
swad beach - aruarian dance (swadmix)
Link: / aruarian-dance-swadmix
Inteus - I Pull up in a What is That
Link: / i-pull-up-in-a-what-is...
SwuM. - NEW DRUM KIT OUT NOW!
Link: / lofi-trap-drum-kit-out...
lzr - goin up
Link: / goin-up

Наука

Опубликовано:

31 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 732

@jamesallen74 6 лет назад

Devon, I am a 44 year-old software developer in the Enterprise world. I have to say your videos are some of the best I've seen for what you seem to want to accomplish. You're showing the people who watch this video the fact that you have no fear of digging through documentation, putting in lots of hours, overcoming frustration. So many people wanting to learn how to code just assume that "experts" just go on their keyboard and just start typing away without frustration. But you seem to show the reality of it. I would love to see you talk more about just the mental process you go through from square one to finish, the habits/practices you follow and the mental challenges you have to overcome to convince yourself that you can do this stuff. I think that's the biggest hurdle for a lot of people is that feeling of imposter syndrome. You were going to do really well in your career. Keep up the good work man.

@XenoContact 5 лет назад

I watch devon's videos too Can I get a job please? *.*

@lolo101100 5 лет назад

@@XenoContact im confused now.

@Ultimime 5 лет назад

I want you to know as a 31 year old developer currently struggling with doing new things out of my comfort zone this line "people wanting to learn how to code just assume that "experts" just go on their keyboard and just start typing away without frustration. But you seem to show the reality of it." is what I needed to hear right now, thanks

@martinaarhusgregersen935 5 лет назад

@@germanhoyos4422 You're never too old to learn something new - go get 'em :-)

@JtagSheep 5 лет назад

I feel your comment is very important for people who know nothing in programming mainly this part "You're showing the people who watch this video the fact that you have no fear of digging through documentation, putting in lots of hours, overcoming frustration" many people fail to realize even the most competent of programmers sit reading through documentation to make stuff work. You may know every piece of syntax and code for a given language but making it work how you need is not always going to be easy without documentation :) I feel there is often too much feeling of imposter syndrome for new coders and sometimes are afraid to read documentation feeling they are cheating however it could not be much further from the truth :) the documentation is there for people to learn from, the original library creators give the documentation to aid others in working with their library and such. Do not be afraid to read documentation and watch endless videos it will make you more competent in your goals and project. I myself am stuck deciding on what project to even embark on to expand my basic c# knowledge after learning from unity making games I now want to expand into software and other things ! The key here is to read and learn and read some more :) happy learning !

@squadzero_ 6 лет назад

> have photoshop in the taskbar > launches Paint to draw a workflow

@supercoolmunkee 6 лет назад

kzm 夢を I probably do that same kind of stuff in microsoft word and it is easier because in word, you can snap the boxes to be in line with each other and make sure that the boxes are in the same dimensions. Also switch from portrait to landscape 🤣

@squadzero_ 6 лет назад

Doug Cannon man you can do the same in photoshop once you're used to it and it's easier imo just a matter of habit i guess

@coolfrisbee 6 лет назад

this is galaxy brain

@DevonCrawford 6 лет назад

Hahahaa

@mulanszechuansauceisthemeaning 6 лет назад

first thing I did when I installed my first Linux distro - googled "alternative to MSPaint". Pinta is great btw :D

@teodorbajenaru8952 6 лет назад

Screw Python, programming in paint is the real deal in 2018

@fossforever512 6 лет назад

Teodor Bajenaru but... “Python’s where it’s at” (According to ads I get at least)

@eddiejaoude 6 лет назад

Its_ Me lol haha, I get that ad too 😂

@PainDive1 6 лет назад

So funny! I see that add every single day, it pisses me off lol

@h33p 6 лет назад

Someone has built a working program by making a bitmap through paint.

@brandonfleming7118 6 лет назад

I also get the Unity C# course too and I did end up buying it. It is actually pretty great.

@duduken20 6 лет назад

Amazing video with an amazing ad, just amazing! You don't know how much you help me with these videos each time you upload, not just because of the knowlegde, but also the motivation to graduate and learn so much more! Thanks a lot man.

@CloudGamesHD 6 лет назад

yoo i watched you since minecraft and it’s crazy how you grew this channel so fast and so big! I remember when this channel was at 1K :D. I’m learning so much from these video’s and this is the kind of stuff i wanna do later so this is really fun to watch and you make it even more entertaining. You really motivate me to do this kind of stuff like coding. Thanks for this amazing content! :)

@coolumar335 6 лет назад

Great stuff, Devon. Your coding channel always delves into new tech and I want to check out next.js after this. Cheers for another great vid!

@12gorman34 5 лет назад

I'm a final year computer science student this year, and this is the most informative channel for me on RU-vid. This is exactly the same topics i'm a struggling with and learning to myself at the moment. Keep up with the good videos dude, there are people out there who really benefit from them.

@prateekgogia8310 6 лет назад

I like how you shoot your videos and edit them, inspires more developers like me to do the same with more creative ways. Keep up the good work!

@EnerGieprodesign 6 лет назад

Dude.. your videos are so unique and fun, please keep doing what you doing! Great vid!

@fredericochopin 6 лет назад

I like your approach to programming related topics. It's fluid. You sketch first then dive deeper later. Seems very productive

@TheOisannNetwork 6 лет назад

I would do it just like this, but put each of the services in a docker container. That way you can have the container restart itself if the service crashes. Load balancing between containers instead of ports.

@666Makkura666 6 лет назад

I also vote for this idea. It really is a step up. More work eventually ends up in more convenience.

@Scottstimo 6 лет назад

And run the containers with Kubernetes!

@augusto1997 6 лет назад

Another thing that might consider, is to use javascript in asynchronous way.

@nicmarier 6 лет назад

Also, docker swarms could be used if you happen to have more machines available to run your apps at some point.

@thesnarkydev5600 6 лет назад

heads up, docker containers don't automagically restart, there's some further configuration and setup needed to catch process ending errors and restart (pm2, forever, etc).

@fawazaljohani8447 6 лет назад

You make me get excited to get back to programming and learning new stuff Such an inspiration , these new type of youtube channels are amazing and you are on the top of them in my opinion KEEP IT UP

@silvanverhoeven 6 лет назад

Ohh, how awesome! I am also getting started with NodeJS etc, you are basically doing exactly what I want as well. But haven't got the time to actually get my head wrapped around it due to work. Looking forward to the next videos, I hope I can learn a lot of stuff from that already!

@spacet1me 6 лет назад

Digging, hope you continue to architect and build it and take us along the journey. Loved this and the CS degree video.

@anshchawla9721 5 лет назад

Man, you're videos are just awesome! You inspire me to start a project without knowing anything ✌🏼

@BGroothedde 5 лет назад

All developers should do this every so often, in any field. This not only gives other people insight in your way of thinking, but it also serves the 'rubber ducky' purpose. Because you're explaining it to others, your mindset changes about the problem and you get additional different views as well. Good practise man, awesome!

@mizell8894 6 лет назад

Awesome videos where you give a great overview of what you need to do/learn to implement something! Keep doing these videos!

@AdolfoHernandez-gt8yj 6 лет назад

I’m a photographer and have no knowledge what so ever in coding, I just enjoy learning from you man. Keep it up.

@derekhiggs1513 6 лет назад

I honestly watch these having no idea what your talking about but somehow am just genuinely interested in what your talking about 🤷🏽‍♂️😂. It’s like I’m in a class that I don’t need for my degree but your a really cool professor so I automatically get sucked in to paying attention. Idk if that makes sense but keep up the good work!!! Been subbed to you for a good min.!

@zakariabahbaz5348 5 лет назад

I'm learning more and more programming right now, and I'm building up to deploy a website. I keep coming back to this video every 2 days or so to see how much more I understand of what's going on.

@duckchip95 5 лет назад

man I really appreciate your work in making these videos. love watching them! Maybe you can even add some caching tools as well like Redis or Varnish to make it even cooler! keep up to good work :)

@artsiom 6 лет назад

Congrats on such a cool sponsor, keep it going man! You do inspire me, спасибо

@williamgaze1262 6 лет назад

Hey Devon. I really appreciate posts like this. Thank you.

@LarryBordo 6 лет назад

This is a pretty good start. As long as you have your ports in order, and you're only working with one server, this is a pretty solid plan. Though I would just call this application organization, scalability kinda refers to multiple servers, load balancing, high availability, and managing requests with idempotency

@killorbekilledstudio 5 лет назад

I am beyond happy that I was able to keep up with this. Thank you!

@tavin136 6 лет назад

dude, you are insane! don't stop doing ur vids

@tavin136 6 лет назад

oh jeeeeeeeesus he liked ;0

@AMPTechGrade 6 лет назад

This was awesome. I've been pissed that I could not find a video like this sooner. I wasn't even sure where to begin.

@katarinazivkovic3102 6 лет назад

This is so interesting. I would watch your videos every day! Thank you for sharing :)

@bufmouse13 5 лет назад

I love seeing other people working through this kind of thing. Good luck.

@ryokaproduction2920 6 лет назад

dude love this kind of video of yours no one does this except you , i get a grasp (somehow lol) on some web apps .

@carloseustaquio7259 6 лет назад

Wow, awesome man. I wish you did more videos, I'm starting to study Computer Science this semester and I like your videos very much! (although I hardly ever understand what you're doing) hahaha keep doing them! hugs from Brazil! ;)

@KOOWALLLLLLLLL 5 лет назад

It's amazing how I spent a full week researching this. My goal is the same, to separate my project into multiple servers. One serves the frontend, another serves the backend. From there every application can be dedicated and/or bundled up somewhere, ideally organized to keep maintenance seamless when interacting with applications. I'm going to look into what you recommended to do and I thank you for finding out all of this. I would love to see your project files (excluding your applications of course) or if you made a boiler template out this, but that's a lot to ask for. Regardless. Thank you.

@konstantinh9412 6 лет назад

This is literally what I’m trying to learn rn thank you so much

@KellenChase 5 лет назад

Don’t know you at all, but this type of video is so valuable. It’s a piece that is missing from the “self taught” coding world. It’s like there are the basics courses and then there are completed project courses and then there are “watch expert live coders do amazing things”. This right here is the messy middle of “how do you Think about a project”. Thank you for this “design meeting”. I’ll be following you.

@ziebist4464 6 лет назад

I've seen a few others mention this already but definitely look into Docker and with it Docker Compose. Using Docker Compose you can not only automate the process of booting all of these services at once but also save yourself work in the future when deploying individual services becomes as easy as running a simple docker command. I use Docker for work and my personal server which is set up much like your proposal, and it works beautifully. Keep up the good work though man, love seeing how others approach the same problem.

@ironspider9280 6 лет назад

You are just amazing. I really love your videos man. You are my inspiration.

@LF-ro7nk 6 лет назад

I've checking everyday ur channel and ur videos makes my day better Thanks!!

@Retnuh1974 6 лет назад

Keep this style of videos coming, they ROCK!

@BSUGoliath 5 лет назад

Can't wait to see progress on this!

@DoakSteezy 6 лет назад

Every time I watch one of your videos I get so amped on programming and geeky computer stuff. So amped that I actually do my homework

@supersloth1667 5 лет назад

I dont understand much as Im just at the start of my CS schooling, but this kind of stuff is so interesting to see how it's actually done behind the scenes. Also 10/10 music selection at the paint part

@someguy8347 6 лет назад

Dude I'm been watching for a while and been using the same frameworks as you even before you did and now you're using next.js and react and I'm using them as well, what a coincidence lol

@sabihahmed760 6 лет назад

Devon you are doing a wonderful job by making such videos that are inspiring as well as motivating for our young programmers. You are an inspiration and motivating factor for me. Wish you best of luck for your future hope you have a wonderful future ahead. Keep up the good work!

@aseemlalfakawma5084 6 лет назад

Love these videos dude.

@solveit1304 5 лет назад

Dude you really go crazy with that Paint workflow xD

@kafanakodtome 6 лет назад

Glad you switched the topic to programming. Looking up to more videos!

@Joshimite 6 лет назад

Really like these vids dude keep them coming :)

@pencilmein6646 6 лет назад

I love when you upload these kinds of vids!!! 👍👍👍👍

@rosenoire9670 6 лет назад

you made me take on a whole new perspective on programming... that it should be a fun learning experience :)

@LuCkKyStEpHeN 6 лет назад

love the vids, please dont stop motivating me through college :) engineering student loans are insane

@xires1 5 лет назад

Jesus Christ Devon! What the heck happened?! You grew so much! Hope you remember me haha, your first fan you have ever met, sort of by accident. But I’m happy for you man. You grew more than I thought and I’m extremely impressed. Love the content and I’m gonna binge to catch up - nick, that weird dude from X15

@kkevincerdaa 6 лет назад

Love your videos! I just finished building a web app with scalability in mind as part of a capstone project. Looks like you're taking the right approach at tackling this and I'm excited to see the progression of your solution. I'm not sure if you mentioned it but look into docker. Containerization definitely helps with scalability and easy/fast deployments. What are you using to host your server? You can look into launching each app as independent microservices on their own AWS instances which would allow you to future scale them independently as they grow. Amazon offers great pricing on server hosting as you only pay for what you use.

@kkevincerdaa 6 лет назад

Btw I used the same exact tech stack as you, so if you have any questions feel free to ask!

@9xtryhx230 6 лет назад

Yeeeeey he's back!!! Hyped :)

@dylantweedy9452 5 лет назад

I don't really have any experience with what you're doing, but I think I understand the basics. I probably don't know what I'm talking about, but I really like the idea of trying out IPFS and neo4j, no idea what I could do with them until I try though! I do really like what you're doing, maybe I should dive in and try out a bit of everything, see what I like! Thanks for insiring me to try new things! :D

@CharlesTMunger1 6 лет назад

Best Computer Science channel in all of RU-vid!

@TheBcB2012 6 лет назад

Honestly this may make my current project work even better, but I do recommend using docker with the same setup you beautifully drew in paint.

@THE4ECHOK Год назад

wow! again! You know i was thinking about some programm which visualise the music from comp to a LED /RGB tape. And i thought it is easy. Bro. Love you that you showed us that none of the products are easy to do. Hope for new vids

@nolan6895 6 лет назад

dude these vids are soo great

@theman4426 3 года назад

i have no idea what hes saying but he presents all this info in such an entertaining way.

@h4shcoded56 5 лет назад

Really like this type of video dude, it's nice to see someone else go through their thought process. This seems like the best way to host on a single server. What would be the process if an app crashes or something?

@ThNostalgia 6 лет назад

Thanks for making this video! I'm a pretty junior programmer, and I get intimidated and overwhelmed easily with all the stuff that's available out there. (Do I use restify? express? what's the difference? Apply that to all the different things out there... ) Your videos really help me remember about how programming is really fun and useful, and makes me more motivated when I get back to the cubicle again tomorrow. :)

@shaneg457 6 лет назад

I know bits and pieces of what you are trying to accomplish, but I too would be stuck on a few things that I don't know how to do. I look forward to seeing your journey with this project.

@TruthAndLoyalty 6 лет назад

This is exactly how I do it. I'd suggest vue over react . I've used react, angular, and vue and vue is really great, especially if you're new. I've never tried next.

@aminullll 6 лет назад

I am so excited.. Please be more regular with videos. It is soooo awesome !!! Please try to be regular... Thanks !!!

@amentitm7420 6 лет назад

Where have you been?? I'm excited for this video

@ijusterik5384 6 лет назад

Same

@funy0n583 3 года назад

bruh

@bhargavsaikia3580 4 года назад

Exciting stuff !!

@kr4yder747 6 лет назад

OMG man, I have same trouble and you make me think a lot. Thank's bro!

@ksankararaman4941 5 лет назад

Awesome stuff man.

@ToeKneeTran 6 лет назад

I'm currently interning at a bank in Australia and what you said about the complexity is true. We also use Nginx to route our different applications/API's we build.

@jakewalker1802 6 лет назад

Nice content explaining the process! More systems

@jasontruter7239 6 лет назад

I've just completed my AWS Developer Certification, AWS does this all for you. Use an Auto scaling group with a min and max amout of instances/servers, use an Elastic Load Balancer to assist with routing and load of incoming traffic. You could also go serverless and use AWS lambda, API Gateway and Dynamo for your backend calls, then use S3 to host your static files.

@lestertarranza 6 лет назад

I laughed so hard when you captioned "Some High Quality Documentation", then I know what's going to happen next. Legendary MS Paint drawings.

@ivansostarec6793 6 лет назад

Go on with this :3 can't wait to see updates

@vDaviiiD 5 лет назад

You need to make more videos brother, i'm really digging them right now

@Guitrafani 6 лет назад

Hi, I from Brazil, I doing software engineering, I love watching your videos, it always inspires me to study more

@user-fj7bo8gh8e 6 лет назад

This is the pattern that I follow for my apps (Nginx). It's perfect.

@LEZPEZ1 5 лет назад

Dude this is not boring at all I've been trying to find a full stack app architectire but suck at this planning on my own.. Very cool I am def. Gonna try next on a new project

@fcogr9485 5 лет назад

awesome video when you say beginning of scalability, you look like a singer.

@ynotbutcher1891 4 года назад

sweet this is just what I needed fam!

@richardseecharan5996 6 лет назад

Didn’t even realize you crossed 100k congrats 🎉🎊

@iStudLion 6 лет назад

Holy Shit, same dude, last time I checked, he was at 14k subscribers WAW

@richardseecharan5996 6 лет назад

Same I swear the he was at 34k like less hen a month ago

@dealloc 5 лет назад

Another step to make your API scalable is versioning. If you have multiple apps talking to the same endpoints, changing the endpoints (either response/request or route) without versioning would require you to update all applications to understand the changes. With versioning, however you can create a new version of an endpoint and avoid breaking other apps the uses the older version. This allows for incremental updates without the need to break anything.

@cristianoaprigliano9712 6 лет назад

You should do some "what's inside my windows and linux" distributions video.. like what you prefer using for programming etc etc..

@amentitm7420 6 лет назад

I always learn some more in each of your video so thanks Crawford

@evanlken 5 лет назад

I setup something similiar with docker containers and I used haproxy to send traffic to the correct applications. It's pretty sweet

@mx0c 5 лет назад

Definetly look into docker (compose), gonna make your life way easier, especially when it comes to the deployment of the whole system!

@psjuillet 6 лет назад

Half video it's me actually understanding your code, the other half it's me looking your beautiful face, admiring how smart you are.

@daksh6752 6 лет назад

Oh you finally hit the microservice idea boy?

@kzakaria91 6 лет назад

nice upload.. nextjs is so simple to use it's amazing, i think you will have a hard time in react if you've never used it before :)

@shivamsingh1988 5 лет назад

i love your channel you are doing great work.that's inspiring for student like me thanx bro.

@rsurjrsurj6657 6 лет назад

Thank you Devon, I've subscribed to Brilliant.org :D and get an annual subscription, again thank you!

@paoloposo 6 лет назад

You might want to have a look at Docker + jwilder/nginx-proxy. It's a reverse proxy as well, but takes a domain-name-based approach instead of a route-based approach and I've come to appreciate it a lot since it requires virtually no configuration (that is, if your Docker environment is already up and running).

@manish-mk 6 лет назад

Awesome video Devon Make more this kind of video

@julezz3634 6 лет назад

Man 124k! So happy to see you grow, remember the minecraft channel days 😂 keep it up

@normanblakely2418 5 лет назад

@Scott I agree, but also it would beneficial to leverage a cloud provider. Google Cloud has a very good Kubernetes Service if you want to go the container route. I personally would use AWS API Gateway and EC2 for scalable backend servers as well as their Application load balancers instead of Nginx for setting up routing paths. Static resources can be stored in AWS S3 and Cloudfront for CDN. Frontend can also be hosted on EC2.

@toastman6847 6 лет назад

I know fuck all about programming but I just love to watch this dude mainly cuz he is just so damn enthusiastic about what he does

@BrendanMetcalfe 4 года назад

This is super helpful thanks

@_sudosu 5 лет назад

Docker container per app + Nginx on the main machine to route to each one. If anything grows big and consumes a lot of resources, you can extract it to separate server without any efforts.

@asdasddas100 6 лет назад

Sounds like a docker problem

@secularph8424 4 года назад

Exactly

@Jonathan-zq8vs 6 лет назад

Dude. Just wow. Keep it up.

@elcocodriloazul 6 лет назад

i love how he has a sock over his microphone. When you are a student, you have to be practical. Reminds me of the good old days of been in college but poor.