I'm filming a video tomorrow about all of the dumbest things people have said about UploadThing. Reply with some good ones here and you might get featured ;)
I've been using uploadthing for a long time now. I know how a S3 bucket works but honestly I got screwed up on handling the permissions of S3 initially. Uploadthing is faster, smoother to configure & clean in it's operations. I hope uploadthing becomes a norm for all the businesses. It's really good. Wishing good luck to Theo and Julias.
"Would rather use the much more stable and simpler Amazon S3, and does speed even matter? The user should be fine waiting a few more seconds." - Some guy in discord
I mean even if you didn't try to do that consciously, it would happen - you don't write everything perfectly the first time, specially when working on an MVP. It has to work before it can be optimized
@@bastianventura dude, it's not a nuclear fusion equation analysing app! it's a freakin S3 uploader! you could have it up and running in 1 prompt! but i guarantee you 90% of its code is to limit your ability to upload based on your tier... you should 1. PLAN 2. CODE 3. Optimize, i guessed he missed 1.
the web dev world is slowly reverting. soon we will get "we used literally zero npm packages and just vanilla JS, and our product shipped 10x faster, and the average API response time is 0.0001ms"
Curiously recently I discovered a way with vanilla navigation api and view transition to make an app like nextjs, with all features, faster and don't need build step
while the tooling has a few npm packages for sure Astro is great for that you can ship zero JS if you want, proper grid layout with a few lines of CSS (as much as I love tailwind it adds pages and pages of CSS), and the (optional) SSR features like Astro Actions are specifically designed to work without JS.
More and more, I'm coming around to the idea that all these microservices, serverless, edge networks, etc. create way more complexity than is needed for the vast majority of use cases. We devs do love to complicate things.
One day we will figure out how to cut out the middleman entirely and upload straight to our own servers, which can then transcode files, upload them to S3, etc. Oh wait, we actually had that figured out in 2005...
I can make it even simpler by removing your server and just by using free and open source Uppy to upload directly to S3/R2 or wherever, it has resumability and other plugins for free too
I think the difference is that Uppy's "ingest server" can be run by you (using Tus) or by Transloadit. You still need a server if you wish to have resumability and no ghost files, though
Let us know about the costs difference later down the line, because serverless tends to be very expensive, but your new infrastructure uses a lot more bandwidth on the application side
This can be simply solved by client notifying the server once the file upload done. This is just over engineering at its finest. His reasoning was there will be ghost files if the client didn't notify the server. Solution to that is client always upload to a temp location and move the file to actual location when client notified the file has been uploaded. And you setup a s3 lifecycle to delete files based on the update date.
does it allow for resume and improve the time for smaller uploads? Still, they made their changes and going back to S3 isn't feasible for their marketing too. Plus they now support other types of "buckets" so I guess it isn't just S3 being inefficiently use, instead it gives them marketing leverage to be more independent and agnostic
This is just meant to be educational to show what things can be slow and how to resolve them for unexperienced developers who haven't reached or considered these steps on their journey.
@@macchiato_1881 That was my 2nd guess but I was seeing a lot of comments in bad faith so I really couldn't tell without any tone indicators lol. In a way my reply speaks to them too
@@sanjaux why do you need tone indicators? People like you need to handle negative comments better. I get not all criticism is good. But are you just going to whine at every valid negative criticism or joke you get?
@@macchiato_1881 Well the actual jokes no I'd ignore those, but criticism is best resolved through talking it out. Since this isn't criticism, more signs would have helped differentiate your joke from something actually worth discussing. Handle them better? I'm just trying to understand the thought process behind some comments (the serious ones)
Did you do any load/performance tests for your UT Ingest Server? Would be really nice to have a video just on that :) Also scaling of this server is an interesting topic...
@@theairaccumulator7144It's marketing. Just like the cloud and hypervisor. It sounds more cool than using someone else's server or dividing your computer's resources.
This puts a limit on the bandwidth available as you are proxying the file uploads to s3, if you have a ton of concurrent uploads you will also need to scale your own servers.
Not if the "Ingress Server" is on AWS EC2. Instead of paying S3 traffic coming from internet, they are paying S3 traffic from inside AWS (Which may be even cheaper). Incoming traffic to the server from internet is free (Well, let's say included on the per-hour price)
@@framegrace1 I am not talking about pricing, but about bandwidth, S3 has distributed endpoints for content delivery and you can have 100s of people upload simulateniously at high mbps, on the other hand your one ec2 instance is limited to whatever mbps amazon has to it, and if you try to upload 4-5 big files at the same time (from different users with good bandwidth) it will bottleneck it for everyone
@@framegrace1 And what is your point exactly? What i said is that to handle more concurrent users they will need to scale the number of instances they run. Then they need to use load balancing to distribute the content across the ec2 instances. And what is more you lose the advantages of the distributed infrastructure of s3 that amazon has built.
@@danhorus oh right, I got the impression it was clients own authentication and direct upload to S3. I obviously don't understand what this solution provides.
@@dancarter5595 An easier way to upload things? They also add some code to the process so you don't need to do it yourself. I mean it's like using Vercel so you don't have to set your infra.
Yeah, that's the thing here that sort of defeats using it for anything production that is user-data sensitive. In EU at least, cause the us ofc doesn't care for user data. Because you are going to be in breach of GDPR. Since you are the administrator of the data, you cannot share it with 3rd parties without consent.
1) S3 does support resumability 2) File sizes can be checked using `content-length-range` 3) S3 can reject on file extension and mime types 3) You could have ditched Lambda and done a webhook back to the server
Wow this is massive reduction in complexity! I hope though one day we'll have technology advanced enough to use this thing called "Your server" to store a file. Sure hope we would be able to achieve even less arrows on the graph then...
uploading should really just be a single chunked transfer http request with a single response. the server can easily athenticate that and save the partial data to get resumability, and more
Congrats. Owning your own infrastructure is something I always found important. Can we expect video titles like "We stopped using the cloud" with details about how you manage your own bare-metal Linux servers soon?
I'd love to see Theo work on some Remix projects. Remix offers a great deal of built-in type safety, eliminating the need for extra implementation effort.
Well that is nothing surprising, everyone should know that each serverless our cloud computation application always has an overhead. It is like saying, the new built file upload in rust is 10x faster than in javascript lol
This just in. Serverless proven to be a buzzword to keep you purchasing overpriced subscription model technology. In other news, paint is wet when applied.
Great success! It's also quite cute that, even after so many live-streams and videos that you have done, you end up sounding a bit like a school kid presenting their project the first-time in front of the class, when you are talking about something that you are really proud of.
its almost like vercel preys on new devs who don't know any better like supplement companies do on people new to working out. They push an unneeded product on people who don't know any better. Nearly the lowest form of ethics for a business model.
its almost like the vercel sponsorship was motivating his decision making, despite stating otherwise. hmmm.......... money changing people's views on something? Never heard of that before.... I respect him cutting off the vercel sponsorship honestly, like I really do, but I can't take these types of videos where he feigns discovering something groundbreaking, when in actuality, if he had listened to his critics who had be pointing out these problems with serverless for years, and instead of just hand waving them away under the guise of DX, he wouldn't have had to go around this whole cricle. Web dev and CRUD apps are solved problems. They've been solved since like 04 with wordpress, yet new devs always let their ego in the way and have to learn the hard way, instead of just listening to their elders that solved the problem. It's like first learning about parsers and then having the ego to be like "i'mma write my own!!!" when parsers have been solved for decades.
one year later. "we made file reading 10x faster and lowered our cloud cost 10x by going bare metal server. " it i always nice to see people gets excited when they re invent the wheel
It's kind of sad that resumable file transfer is a big feature now, because I remember it being a standard thing when I was a kid. It was lost somewhere along the way, and I'm glad to see someone is paying attention.
I wonder how pricing would work with "bring your own bucket". But we're very excited for it since our organisation has rules on what geolocation a bucket can exist in. And even just using local infrastructure.
@@Itsneil17 you know that this is like saying "just make you WordPress"? I guess Upload Thing is simpler, but getting right is really hard. That's why we use abstractions that hide the real complexity
Really cool stuff! Now that the infra is more flexible, something I'd love to see in UploadThing in the future is Cloudinary-like image transformations. UT would become a viable Cloudinary competitor with that! Could also be part of PicThing if you plan on doing more with it than just background removal :)
Gonna brag because I did this years ago. I also thought PWA was great when everyone was obsessed with native. Here's the next protip: Vanilla JS has everything you need. Frameworks are bloat
Nice, we have a similar architecture. We built a file upload service for our healthcare application to allow clinicians to upload patient documents, which we also used for other clients. We never touch serveless. The system is deployed in Kubernetes and uses MinIO for the object store. Seeing uploadthing have some commercial success, I wonder if I should compete with you guys? Haha, nah too busy.
Legitimate question, but isn’t 1.5s to upload 3.2MB still really slow? I don’t know what kind of internet you have, but a 50mbps upload would’ve sent the data in 500ms, what is taking the extra second?
When we trigger S3 uploads/copies through various means, rather than having our API state update the front end we allow our client to hit a headObject presigned url to assert that the object has successfully landed. Requires some ugly polling but it’s cheap polling
Fwiw, Lambdas are not the only way to have serveless compute in AWS. ECS Fargate also offers the benefits of serverless (scale to zero, pay for what you use, etc) without the limitations of Lambda.
I love the update. Arguably, before, you didn’t really have a meaningful product when you were serverless (the value-add above using S3 was small), but now you really do.
Infra matters more than what frontend/client could ever achieve. Because on frontend you can only show the loader nothing else because client has limited internet bandwidth.
It might have been worth measuring and removing the upload time from the benchmark, considering it has more variables than the speed of the ingest server itself. That or upload images that can be sent in a single chunk.
Congrats on the launch, less complex and faster, net win 👍 I imagine you went with the previous architecture first because it let you bootstrap more quickly, without committing yet to the upfront cost of rolling & maintaining your own ingest server. is that right?
1 to 2 years and we're gone full circle. "new" web devs already "discovering" PHP again. Not long before people uploading HTML files to a nginx/Apache server again and calling it "zero dependency websites". This will be the new big thing.
Huge improvements! It's great that you feel ready to tackle enterprises, but I can assure you - it's not easy, not at all. Data privacy standards are more looked at than ever, so I'd first go for SOC, HIPAA and EU variants of those to have certificates you can shield yourself against quick-shot enterprise questions :)
Yes, frontend life is better when I can literally write everything in a single HTML file. Vanilla JavaScript + sometimes Web Components for the win. I'm a Helix IDE user so keep that in mind since that may also influence why I don't mind making web apps in that way.
Now it's faster - but it also cost more money - you need to run server, you need to pay for the bandwidth and so on. So it's a trade off - you will pay more for your infra - you will get better user experience. It's the same as with Auth, you can use 3rd party auth system, which saves you ton of work but you can't control the user experience to the very details.
Turns out serverless isnt actually serverless, and making less API calls to do the same thing means you have a lot faster of a server and a break in one system doesnt turn off the economy for two days.
Soooo, uploads are not faster. Why would it matter if the client uploads to your ingest server or directly to s3? It's the calls before and after the actual upload that you removed/optimised. I bet uploading a 100MB file takes pretty much the same time as before. If you had 4 seconds of overhead (for 4 files) before, that sounds more like a crappy implementation :)
As someone who works with AWS for 3rd party security reviews, those enterprise features sound nice. Still, there’s a LOT of config settings that AWS requires (that are not always cheap and is constantly changing) to be meet the bar. Still, this is very cool infra design change and breakdown. I really appreciate this, folks who don’t work with AWS/cloud don’t understand.
I remember the whole serverless is designed to be short time, lightweight, infrequent requests for particular functionalities of your application. Hence the server doesn't need to run all the time and save your cost, and you don't need to maintaining the server. Lately, it was abused massively for all kinds of heavy tasks, which should belong to your own server. And people complain the serverless. The comment section is full of "devs" who say serverless is bad or host your own server is bad. Joke about the web dev, without understanding of those subtle details. The current generation has huge skill issues imo.
@@doc8527 Yeah, I get to witness some real nutball spaghetti lambda design. If you need to mange over 50+ lambdas for your backend plus have one for every single API, troubleshooting & DevOps becomes a nightmare. Gotta watch every lambda metric, have so many cloudwatch logs etc. Thats where Vercel like companies do serverless a little better, they're taking on more of that burden, but its priiicy! I'd pay for it in a heart beat to save me time though. Docker containers are where its at. Fargate/ECS that thing. Even EC2 management has improved a lot with CDK + SSM scripts.
it's fun to watch people rediscover simple architectures are the best... microservices and serverless are too often just solutions in search of a problem
this is similar to investment, companies need to say something different, since most of them aren't innovative, instead they just go back and forth between things we have done in the past so people will invest in them.
Very nice explanation and product improvement!! I'm curious about how this change would impact in the infrastructure cost and product pricing. Could you explore this topic?
Not surprised. Google charged me 16 dollars a month for a website with almost no traffic in server less. It's not a good idea if you have no traffic. It's only good to handle extremely volatile traffic loads,.
You could still be serverless with a fargate container doing the ingest server work. You can also do a MD5 hash verification on a file wth S3 but that may not be possible in your scenario depending on client's ability to create a hash.
16:00 last sentence in the first paragraph says "meaning you can there is no need for the extra API call." You may want to fix that! Otherwise, great video!
But what did you use to build your ingest server?!?! typescript? .NET? Go? Rust? Something else??? I wanna know the details about your serverFULL architecture!!! There's no details in your blog post either about what you used to build your ingest server in, how it's hosted, etc. I'm extremely interested in what you landed on for those tech choices.
using the AWS SDK to upload files you can pass a callback or use a Promise to handle the completion. Thanks ChatGPT now give me a code example. Now why do I need to pay for a middleman service?
