A big thing here that is missing is the transactional safety of this. I treat all my endpoints/requests as a transactional boundary. If using a second request, the onus is on the client to do more work that still can fail, and your data source wouldn’t be any wiser. It looks like that db has the capability to make sure things are “complete” but not out of the box using s3. It also isn’t free. It’s shifting from one cost to another
We are using signed uploads at work and at one point to frontend would show the upload was complete but the file didn’t make it to S3, as stated this is not free
We had done something similar, but you have to add the extra step of upload confirmation. If there was an error on upload, the entry still exists on the db. This caused name already exists errors. It ends up being two to three steps depending on if all goes well or not.
This is what I have implemented a while to upload files from a mobile device. Lambda creates an url for the file, sign it for write access and then you can just POST the file from the phone. All network load is on S3 now.
Really informative video, but there was one piece I was hoping for but didn't see: what's the process of creating a signed upload URL? Is that something that Xata (and other storage systems) do for you, or is there more code needed for that part?
Nice approach. I was doing something like that but in a different order. I send everything in one call and immediately return the name and a screenshot using ffmpeg. which will return a image and status pending. in the background will process the files without waiting for the server to respond. And with this process you can choose to request the progress in the background or just refresh the page manually to see if data is ready
This does not look as a generally BETTER Way, more like an other way I would say :) I dont see any gain to small projects ... If you are build a YT clone you should have already have the resources required. Also in EU you are probable would have lots of problems regarding GDPR, you may have to sign a GDPR agreement with the third party service handling the storage ... looks like hell already.
From my understanding this seems more performant in that you aren’t passing all of your data through your middleware essentially ? Dunno anything about gdpr tho
why can't we do fs.unlinkSync to the filepath after we finished uploading the file to our blob storage ? this can stop the need to store any data within the server itself.
Feel this is smart and has a couple of use cases that can be applied. But really not a new idea at all. At the end you are switching costs and introduce error creeps. An enterpize app will have storage on their servers. The main issue is that this simplified example of just a video upload with a filename/path is never the case in real production. From video manipulation, validation transactional integrity, queueing, background tasks, this is going to be a nightmare to split and handle unhappy paths. Makes sense if you are renting every bit of a small app and have no infra at all but not really an issue in real prod. This "renting all" mentality of the past few years is actually costing more on small apps where all of these concerns seem like premature opt. If we look at actuall full stack frameworks, these cost/issues are non existent.
never heard of signed urls. Question.. if i rquest to upload a 2kb mp4 video, get the URL, and then continue to upload my 4GB full hd pirate movie to that URL, would that work? seems risky to give the power to the client.
Does this leave you open to security issues? You are telling the client where all your videos live and how to access them directly. If the backend is s3 for example and policies aren't setup perfect could it lead to issue?
But what if I need to post process the file? Let's say from a video I need to create lighter versions of it (360p, 480p etc.). Or if it's an image I then need to save the master for later and then create a thumb and a lighter 1024px version of it? I need to go to my blob repository anyway, fetch the master file and process it, right?
@@gajrajsingh51 That's for sure. But my point is that server will need to download the full file anyway to run the post processes anyway. So you can either upload directly to S3 if you just need to save the file or you can upload to server and then server will produce derivative low quality versions and upload afterwards.
