Serverless was a big mistake... says Amazon 

Brilliant.

They became serverless, but you became servermore.

On Point 👌

more, less = less, more

Heavy driver..... Bro

Its an established marketeering pattern. Every ten years or so, when a fresh crop of executives have seized the reins of corporations, they get sold on whatever buzzword for "da cloud" is handy because they don't listen to the ol' SYSADMINs who have seen this pattern over and over.

We've been in another "back to the mainframe" period and are moving toward computing at the edge, PC, etc. again. That will last longer than last time because the network is better, containers make deployment easier, and the cloud companies have absolutely shit on their customers' privacy.

@@artiefischel2579 also I think one of the reason is that the hardware price is getting cheaper and cheaper. So for a small to medium company, the cost for infra will break even in the long run

Well you should know the answer to it by now? It depends…

That does my head in. Just like the new marketing manager that comes with a new logo every 4 years.

🤔

Sad history of humanity :D

Please do a video on Tamagui

As Attorney Tom says, it depends. If the monolith has a part that is a resource hog, the whole system needs to be scaled for that part. HTTP is also not the the only protocol out there, but sure on board is faster than network communication.

​@@NuncNuncNuncNunc If there's specialized physical resources and processing that need to be optimized, then sure, you will probably build an appliance specific app and server. I/O limitations have to be considered. Enterprise ethernet is up to 10Gbps. NVMe is up to 32Gbps. The fastest RAM is now 69Gbps. Nice.

It is great. Though I'd reword it slightly: "When it comes to software architecture, there are always multiple solutions, with an array of trade-offs that are often hard to measure in advance." (The trade-off here being that over-engineering the pithy slogan makes it cumbersome.)

so how do I approach this problem, kind of annoy when there's no solutions

Software systems are like piles of dirt. You can hide it under this carpet or that carpet. Or divide it among multiple carpets. There is no way to completely get rid of the dirt.

That was my take away too

that is what i thought i read the article before coming here and seeing Jeff made a video. When he said only horizontal scaling only i was like.....wait a minute.....as you said there was too much overhead communicating to different lambda/step functions so grouped the highly commincating parts on one server to make processing one video way cheaper. Doesnt mean every video has to go through that one container.......

Couldn't they even run this single container as a serverless function?

Yeah, I don't understand how they decided that their original architecture would even work at all on paper. The whole article just screams "we created problems and then we solved them. Give us a promotion"

Yup seems like Amazon hires all the dumb folks, regardless if they get paid well or not

exactly, and this is where microservice purists aka naive newbs lose their minds and we all justly ignore them.

Exactly this, for this reason my biggest shock was to learn that prime video ran on serverless 🤯 it’s not just about performance though but the high demand and predictable traffic, which I guess they were unsure about when first setting off

@@AJ-le3yh That's the thing... you can't optimize for speed until you have some predictable pattern that you can optimize against. But all solutions are a two edge sword. When you optimize for speed (as we did at AOL), it's hard to pivot when the pattern changes and you need a more flexible solution to deal with the increased rate of change and a higher level of uncertainty. But when a new pattern emerges, you then are losing efficiency until you can retool your system for efficiency and speed. As they say, there is no free lunch. Personally, I prefer a flexible system that's highly modular. So as new patterns emerge, you can optimize specific key processes (including consolidating multiple modules into a single monolithic system) while keeping the old flexible modules on deck. Then you have a 'conductor' to help determine which solution to use and when. Yes, you lose some performance by doing this, and you could increase the cost slightly, but long term, you are better off, as you are better able to adjust to new, unexpected changes.

@@AJ-le3yh You know what would make sense would be if you could provision a certain amount of serverless capacity for only slightly more than the cost of the same servers

​@@AJ-le3yh , Prime Video is still running on serverless. They are talking only about one microservice here which changed its architecture to be more performant. Prime Video consists of *a lot* of microservices.

The same thing happens in software. You build the code in one giant lump, then that's hard to maintain so the next time you build it in lots of little classes, and that's hard to understand so you go back to building it in giant lumps again. Eventually you get "object oriented" were each class is a singleton with 300 instance variables pointing to other singletons.

Depends on the application, of course. I've been in offices with home-built server closets, and I've felt the wrath of customers when someone kicks a plug or pulls the wrong network cable or cuts the DS3 line to the building while they're doing utility work down the street (or when a fan dies, or when a NIC starts dropping packets). If service uptime is important, you're probably paying for rack space in a tier 3 datacenter. If it's critical, you're probably paying for space in multiple geographically distributed datacenters, with multiple ISP connections to each one. This complicates your network infrastructure, so you need network engineers capable of architecting and managing interconnectivity across multiple datacenters, and you need sysadmins to select and maintain the server hardware & software your application is running on. The cost of managing your own servers can be massively more than just electricity + the up-front hardware cost.

@@braindawg True, those data centers have specialized setups to deal with blackouts. I think this particular company was using their server for R&D, not for anything their clients were relying on. Actually another place I worked was a hospital, and they had an entire server room. Hospitals are mandated to have backup generators, so setting up a server might be easier for them.

Your price for electricity is insignificant compared to the cost of installation, initial configuration, and maintenance. And good luck with uptime.

Renting servers that are already in a data center is likely to be cheaper and the cost is fixed. Scaling is a bit slower and you have to deal with hardware failures, but you likely have more software-related outages at early stages. We contemplated migrating services to cloud at a small startup and at that point nobody could come up with any slightest idea how much it's going to cost (few thousand a month were big deal for us back then). So we just kept paying some $100 per month for a physical machine in a DC.

You need the cost to be exclusively electricity actually. If you ever idle more than a certain percentage you are better off renting the server. If the cost of staff is high you are better of renting. When you buy a server you are looking at a 5-10 year investment in most economics. Lot of people hate this actually because it hurts sales in hardware and job security. Some even moan about the power savings. More than likely they were interested in a offline server. If you have workstation that does a lot of computation for multiple people then own a central server may make more sense. The equivalent networking and cyber costs would be on top of hardware, staff, electricity, etc. The idea behind renting is to get away from operating cost and ownership. People that can afford ownership will do so. However for most who cannot will not without renting. The idea here is to allow them to, not take advantage of them.

Improper use of OOP, that's all.

you need to be pragmatic, not idealistic. I will Tatoo this myself.

The simpler the better 👍

Basically every good microservice system begins life as a successful monolith. If you build for microservices right away, the added complexity will ensure that you never reach V1.

​@@Keisuki golden words, I can't agree more

This is exists in every question of life, it not something only happening in CS lol

​@@jordixboy what is life, if not CS?

Wow such insight, how did you come across this rare wisdom? Surely it couldn't be an asinine summary of the video we all just watched on this very page?

Yep. And it's not just technical trade-offs, either. There are UX constraints to going distributed, cuz now you have to deal with the CAP theorem. If your UX demands that changes be atomic, you're entering a world of pain. But there are also implications to running steps 1-13 of a task within the same process, without any persistent storage mediating the hand-off from step to step. If the process dies, you can't easily resume at step #5. If your UX demands that the task be resumable, you're entering a world of pain.

@@sfulibarri a lot of ppl summarize videos in their own words in comment sections i think that’s normal

Indeed, they went with serverless to test the idea and deploy it fast to the market and then figured out the optimal architecture, reusing algorithms and code from the serverless when switching to monoliths while also lowering the costs and increasing the data analyzed It's only tradeoffs and nothing is perfect or terribly bad

Yeah, but money for startups is 99% dev hours cost, and 1% arch cost. I’ve once been to a meeting where 20 devs were discussing a giant CI/CD bill for an hour. And that meeting time alone was worth 2x the monthly bill from CircleCI.

Whenever someone says that their solution has no disadvantages, I always tell them that their solution does have them, it's only that they haven't discovered them yet. Life and software is all about trade-offs.

Non tech members of my company take the piss out of me for saying this. e.g. what would you like for lunch? “there are no solutions only trade offs”

Agreed

That's how Thomas Sowell characterises economics, that's why there was a video of him behind the quote

Yeah it sounds like just less micro microservices. They'd broken things down too far, now they're logically grouped into a more reasonable service.

@@LeonHazen That was my understaning also, there is nothing wrong with serverless or microservices, but there is often something wrong in how companies use them. Neither is a magic wand that just solves the problem, first you need to find out IF SL or MS is a viable solution to the problem, and if it is, then you decide the size depending very much on how interconnected the processing is and if there are other requirements. For my use MS is more about domain and process isolation where most services will not need to talk to each other. That way we avoid adding to much communication complexity while we make clear borders between domains, making testing easier. But his also means that we are not going to have hundreds of services but maybe a dozen or two ;)

Not quite as simple for video. It's not sufficient to process individual frames. You often need to process the delta between them, particularly as modern compression codecs only encode deltas from one frame to the next instead of the whole frame. You could probably split along keyframes or something but that will require a pre-process step. The trouble with what they did is that it doesn't really scale horizontally for a single input so if they ever need to scale for, say 8K video, they better hope that it works vertically. My guess is they've done their due diligence and expect the available h/w underpinning ECS to be fast enough for heavier video files by the time they start streaming those out. Scaling horizontally for serving out multiple files should be a no-brainer and I'd be willing to bet that their horizontal scaling issues with step functions and S3 became a real problem when trying to do this for multiple files viewed by multiple customers. Seems like they just figured out their bottleneck and cleared it up.

They even mentioned in their original post, they do horizontally scale

finally someone said it it's a microservice not a monolith who said monolith is serverless ??? or have to have separate physical servers they still have separate services each is a micro of the entire service they provide

i like that term... also ive realized am spending 90% of my work time solving unplanned issues related to this shit, send halp

Even worse than this is “lift and shift” of traditional infrastructure designs into the cloud, and somehow expecting it to be better and cheaper.

Cloud services push micro services trend to benefit themselves. You are just a puppet. I begin to think that software architecture is just full of scam.

Microservices can be built on-prem too. There's nothing cloud-specific about them.

@@roblreid Putting them on the cloud achieves a higher buzzword compliance score. In the last two companies, I have seen projects burn a lot of engineering resources for "beautiful" microservices architecture that should scale really really well...when there is not possible they will that kind of scaling for >5 years. This is an insane level of premature optimization, and calling it a psychosis seems justified.

yes almost nobody needs to use Step Functions

And the data transfer/synchronization.

Exactly, all this is so case specific... There are a lot things to be considered... hell if it's a small team things like personal taste might triumph over a 10% more efficient architecture...I would happily trade some on paper performance if it means a happier dev team any day

Man I agree, this is case-specific as hell. And the video is misleading, one can't judge an architecture just based on it's faulty for one use case. Plus monolith or serverless either of them are not a silver bullet for all the problem. We need to decide based per use case basis rather than thinking like lets do it and think later.

"(which can't be helped)" --- yes yes, just lower the video quality for the users, they will *never* notice :P

@@cheesebusiness One does not simply pick up and move an entire mission critical application to another cloud provider! 😆

A couple years back we were running route calculations for mapping on lambda, the lifetime was too short so the design was to cache the work and pick it up again on the next start, ran out of contorl one day and spawned 10,000 lambda instances for the same job, CFO was none to happy about that.

*MariaDB

@爱国程序员小葛 php extensions, libraries haven’t kept up with new ai buzz, python and others got more community support on new projects. It’s a trend, doesn’t much about language popularity or what skills u need to be valuable in the market

@爱国程序员小葛 In the UK it seems to be dying too, it's often seen as 'old tech' and has turned into a bit of a joke

​@爱国程序员小葛 Ah, yes the old "PHP is dying anyway" topic ^^ That one hasn't changed in the last 10+ years. In the last study about that topic that I've seen it was found out that there are less PHP developers in comparison to the other languages, but the total number is still growing. That is in general though, and not specific to any country. PHP isn't a bad language, at least not the more recent versions. People saying "PHP is bad" usually haven't looked at proper PHP code for the last 10 years. It has it good and bad sides like any other language, but it can be very powerful, especially in combination with a good framework like Symfony or Laravel. It all depends on the specific use case of the application and the skills of the devs that have to work with it. I would not recommend it for everything, but for quite a lot of stuff :)

Future: stop hiring people for very specific roles that box everyone in to a specific architecture. Couch technological goals in terms of user value and use the tools that you believe will best get you there with people who are able to learn and adapt. Instrument what you build so you know what’s going on with your system and can properly evaluate trade-offs when useful new technology inevitably comes along in future+1.

I don't think the writer knows what a monolith application actually is.

And to answer my question after searching: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-N6-Q2dgodLs.html (thanks for making this!)

Respect. +1

Problem with monoliths is mainly build times and complexity as the component grows bigger. And this is something that’s an issue when you have multiple teams working on the same piece of software, something that already for 99% of companies is not the case. People like to work with „their“ monolith but be sure they don’t like it anymore if Jim from the other team made the ship their feature 2 weeks late because he broke the build.

I have seen some AWS bills that just boggle the mind for 100 concurrent user products. Stuff that could totally run on a decent monolith plus a video streaming BaaS product

It's already out there named as 'Azure cloud service'. But Microsoft is gonna scrap it soon..

@@unity001 Can you explain more what you mean?

AWS Beanstalk Docker platform is working like this.

@@petercai3493 Thank you

Resume-driven development... I am stealing this acronym 😂

😂

@@chpsilvaUh, how old are you? That term appeared sometime in the late 90s.

@@bb5242 well I am old enough to be working on software development by the nineties but since I'm also not a native English speaker it's not exactly surprising either.

microservices can be good depending on what you do. examples can be random connections to weird external systems, to transform into a common interface. it doesn't really need to be a full-fledged MiCrOseRViCE ArCHITeCturRE

1000 times this; people want the "rush" of handling 100+ microservices and "feel" like a "great engineer/hax0r" for keeping it running, instead of just thinking a bit more and build something sustainable instead

shilling microservices nonstop is a great way to pretend you're smarter than you are to people that don't understand tech not that microservices are always bad but they should be implemented sparingly and only for things where it actually makes sense

Not really. Microservices come in handy in sales, because you can sell just specific parts instead of a monolith. But your second point is true, as it is true that probably no one ever actually replaced a microservice with a new microservice, which adheres to the old one's endpoints.

Yup you got it. And there are ways to make serverless monolith architectures. You just need to think out of the box. Now if it makes sense economically is a different question entirely but it can be done.

The most correct answer of all !!!

Amen 🌟

I came here to say exactly this lol

facts

Fargate is still considered to be serverless right? But with containers

Exactly, people seem to confuse microservices for serverless. You could very well have one monolith application running serverless on a lambda function or a non-serverless microservices architecture

@@timmeehan2365 what is an example of a monolithic application running on a lamdba?

I don't think he understands these concepts very well because he kept implying microservices and serverless are bound to each other and that monoliths can't be deployed on serverless architecture.

@@rumfordc A lot of them actually ! Any monolith application that doesn't need to keep some state (Sockets, sessions) or work with the filesystem. All basic CRUD apps could be deployed in a serverless fashion.

Well, but in this case they didn't build a monolith. The Prime video search, watchlist, player, billing, etc. isn't all in one piece of software. Ist just one specific fuctionality. It's one bounded context and all other bounded contexts are spreaded to other programs, so it's still a microservice. It's maybe a question of scale, so of course you can define the video analysis as the whole program and the sub steps as the microservices. If you have a markdown parser that generates html, you can build all in one AWS Lambda, or you can split it into multiple AWS Lambda functions, doing the processing of lists into a separate Lambda that maybe calls another lambda to add the html tags. You can do that, but it doesn't make sense. The service to scans a video for quality issues, is a very specific component, that makes sense to be put together into a bounded context, even though, it does different things to achieve that goal. So it is still a microservice running in a docker container that together with a lot of other pieces builds the Prime Video platform.

the issue is that this is not an example of moving to monolith they moved to one container but still are indeed microservices article even mentions horizontal scaling his video si wrong

@@ko-Daegu You are correct, but you missed that the title is about "serverless" vs monolith, not "microservices" vs monolith. You probably already know the difference, but just in case, instead of presenting my own rambling comparison, I did a search for "serverless vs microservices". A great result was one penned by Sumo Logic. I post the initial snippet here, but do search for the phrase and select the Sumo Logic result for the complete, well-explained discussion. A snippet: "Microservices and serverless are both important topics in the world of cloud-native computing. Yet, although serverless functions and microservices architectures often go hand-in-hand, they’re distinct technologies that fill different roles in modern software environments. [...] Serverless is one way to host microservices, but it’s not the only way. Nor do you deploy and manage microservices in exactly the same way as serverless functions. Both technologies offer important advantages for cloud-native computing, but they solve different sorts of problems."

Even Stack Exchange is partially microservice. They have separate servers for search and separate servers for tagging. They just offload things from the main monolith when it makes sense instead of trying to offload everything no matter what. Stack Exchange Data Explorer is its own thing. I'd be surprised if they didn't separate login from the rest of the system as well. And recommendations. They just don't try to make a questions service and an answers service and torture the design around that.

@@thewhitefalcon8539 When the number of replies to a comment are very short, be sure to read them quickly - your thought may already be addressed. In this case, it was. If you care to, open the full list of replies on my comment and read my response to @ko-117. Quickly though, the tl;dr version: the video title is about "serverless" vs monolith, not "microservices" vs monolith. I detail the differences of "serverless" vs "microservices" in my response to @ko-117, and I think it is information you'll appreciate.

I'm fairly convinced I'd be happier if I'd gone a different route in my career and knew what none of these words meant...

@@GSBarlev To really get ahead, you have to be autistic. The field is just too damn big. Just to be an expert in "Windows Server" would be a career on its own *if* you actually did know everything about it inside out. But nobody does. And we're expected to know: Virtualization, Containerization, Storage, Security, Authentication, Databases, and all the tools in those fields (vSphere, Azure, AWS, EMC, Hitachi, K8s, Linux. The 100 or so different SQL's, etc.). Hell just knowing AWS on its own is impossible for one single person. Just *that* is literally hundreds of services. I haven't even touched coding yet: C++, Python, Go, Ruby, JavaScript, C#, blah blah Even the most seasoned Senior Engineer know maybe 2 or 3 of these at an intermediate level, and another 10-15 at a foundational level. The rest is Google-flu and StackOverFlow, lol. Which is why I'm not worried about ChatGPT yet. It will simply weed out the L1, and provide nuclear pistols to the Senior types............................if they leverage and embrace it now.

@@GSBarlev nah it’s all shit, unless you’re talking about being a priest, this is something I can get behind

Thomas Sowell (quote source; the guy the gif is of) is full of gems like that

I feel like this is a sentence about programming in general, not only cloud.

The language is iffy. Anything we do to implement is a solution. IMO Would be better as something more like "There are no 'correct' solutions, only tradeoffs"

Based Thomas Sowell

Don't things like vendor lock-in make this kind of decision process more difficult?

​@@shadamethyst1258 Then use infrastructure as code tool like Terraform

In data processing systems, first consider the data, then organize your processes around what is most efficient for your needs with that data. Don't start with a disconnected cloud solution and then ham-fist shove the data into it.

That's why you are not simple

Same here, we are men of culture😅

I click I sits & learn more tips.

Good sir

same here, this is the only youtube alert that I have.

+

Thanks for sharing, I'm gonna look into it.

So basically, Its better to communicate over pcie instead of over ethernet.

@@SahilP2648 I'm using cloud functions to run a node script that formats a pdf, that doesn't require a $3000 server I think

@@Jiftoo any script can be run on that server using docker containers. You only pay for electricity as you aren't renting anything. Even network costs are based on your company's network. The only thing different will be the latency between your on premise server vs a server running in that particular region say Asia vs US. But that can be something which can be reduced because CDNs can cache your assets like images and video and offer them in that region. But I am only writing this for actual companies with multiple developers, not for personal projects.

@@SahilP2648 Even better don't use crappy tech which requires "builds".

So glad to see on this random video related to stuff I watch, that T Sowell shows up and all of you people know about him and appreciate the cameo! Wow 🎉

how about a group of monolith and microservices making up a multi-service network instead of either one? that's better than doing strictly either way right?

Actually, even with monolith only those functions that utilize the additional resources scale. Nobody really cares if you deploy the remaining 80% of your code that does not need to scale along because it's never invoked. Disk space is cheap. CPU/RAM are what's causing the cost.

The actual goal was to move clients from a commodity based rent-a-server system like EC2 and into a proprietary system more subject to vendor lock.

@@steveb7600 meanwhile binstuffing works at hashicorp

100%. Loved the video but the way to differentiate between microservices and monoliths using that aspect was way off.

Or peeling paint / old wallpaper

Moving video frames is expensive on any network or process.

I think the bigger issue was long running Lambdas under heavy compute from those payloads but perhaps the data consumption was super costly as well

@@AI-xi4jk True, I am not sure that would have been cheaper by moving it to a virtual instance

Thomas Sowell is the goat

allude*

aws is good for companies with too much money, solves the money overflow problem

Jeff is always dropping hints that he’s based, he knows the average lefty is too stupid to notice.

TRUTH!!!

Pretty sure he already talked about this

Shhhh, that is secret knowledge only to be shared in secret neckbeard meetings in the basement.

David Parnas? Fred Brooks? Michael Jackson?

@@user-si7gh3gw5e tell me more

well his son happens to be one, he's worked as a developer since the 1990s.

A total charlatan who couldn't hack it in academia and so became a think tank shill? Doesn't sound an anything like Knuth.

Dijkstra

I'm curious now. What are you referring to?

@@allesdurchprobiert the last clip used by fireship (or a clip of Sowell near the last portion of the video).

@@farqueueman Thanks! I missed it because I was reading comments while the video still played. A bad habit.

But but if you have a hammer all you see is nail everywhere - CTO of series A funded startup who just switched to Aws because its cool 😂

Who could've guessed - the lawyer's favorite phrase "it depends" makes a return in the programming field

nah, your summary is an oversimplification that loses the nuances of micro/macro service architecture and how considering performance when deciding boundaries is really important.

The decision barely has anything to do with the geography.

I mean clearly it can be done as Amazon just showed their flawed architecture (but still can be done). But the question is at what cost lol. 90% performance wastage is huge.

Did you watch the whole video? Near the end he argues why someone might really want micro service