The idea of Idempotentency for APIs means that clients can retry the request without changing the state of the system. Whether the server processes the request multiple times or once(as in this case) is an internal detail. Informative video nonetheless!
You said when the operation is successful, the server will delete the key from Redis. Wondering how would the server know that operation was successful if a retry comes in with same key? Server should not delete the key but rather should keep it in the key-store till the time the payment url is valid and then delete it based on ttl
If the server crashes right after hitting the Payments DB you will not have updated the Redis. The client will call the API again and hit the Payments DB once again. How do you handle that scenario?
Does the idempotency key have to be generated on the server side? Seems like the client could generate the unique identity instead, collisions should be minimal?
Great video, had some queries 1. Is the idempotence key unique to each transaction? 2. What if the client crashes looses the idempotency key and when it resumes it retries with different idempotency key. That will be treated as a new transaction and same operation is done twice?
@@harendraprasad6209 what you could try is to just put the key somewhere where it is stored even after a page refreshes like as an URL parameter or in the localStorage. There will still be a chance that the client looses its key. If this happens you could implement a protection, which diffs the transaction (amount, user that the payment is directed to) and then notify the user that a similar transaction has already been made and ask for his approval to send this money twice.
Thanks for the video, my question is why should'd we store that idmpotent key on the primary db itself this will help us take leverage of db transactions, because there could be a case where the payment transacition is successful but while deleting the idempotent key from redis an error occurred and key didn't gets deleted.
Hi Arpit, Thank you for such informative video. I really admire you putting your continuous efforts towards "Real Engineering". Can you please tell me if my approach is correct or not: 1. Client requests idempotency-key from server. Server generates one, save it in a key-value store and return it. 2. Client sends idempotency-key as header along with other parameters in its request. 3. Case a: If due to network failure request did not reach the server, then client retries along with same idempotency-key. 4. Server processes the request, deletes the idempotency-key from the store and return response to client. 5. Case b: If request reaches the server and the server starts processing but in between it got failed, in this case as well idempotency-key remains intact, and client can send retry request. 6. Case c: On retry request, say it got successful and idempotency-key got deleted but client did not receive the response. 7. Client will retry with same idempotency-key. Since server could not find idempotency-key, there can be two possibilities which are either this request was already processed, or idempotency-key is invalid.
Thanks for the helpful video. One thing, this will have possibility of race-around condition if one request with idempotent key is under-process and the other request could come-up. How to handle these situations?
Idempotency is a very important concept in software development. We have developed our apis in such a way that the values are always overwritten instead of some other operation like increment operation. We can do this as we are sure that there will always be a singular api call.
How do we handle the following scenario, money is successfully transferred from A to B but we have failed to delete the idempotent key from DB (maybe because of a DB issue). Now if the client retries we will process it again right. How do we handle this kind of scenario?
@@akashshirale1927 Thanks. let's say we failed to process the request because of any xyz reason in that case key should not be deleted that means we have to increase ttl of that key. Now what if we failed to increase the ttl? Then the key would be deleted and we would consider request as processed even though it is not processed
@@koteshwarraomaripudi1080 You might hear arpit saying a lot of this...Simple systems scale...complicated systems don't scale. So here we would just fail the request if the key is not present And just tell the user to retry his operation and give him a new idempotent key.
@@akashshirale1927 Also if the server has failed to process the request, it should not persist the key in cache, it should be a transactional flow, no? And yes, in this case just fail everything and start a new transaction.
How to handle cases when the request is partially processed? In this case, neither i can save the idempotent response, nor I want my system to reprocess the work that has already been done.
I was wondering about it's optimisation further. Will it makes sense to batch GET the keys, in order to avoid two trips to downstream service for every request. Absolutely incredible technique. Thanks for bringing up such informative content consistently.
I really like your way of explaining. The way you put a blog into simple understandable language. Thanks Arpit for bringing such kind of content. Keep inspiring us 🙂
Hi Arpit, Hope you're doing well. I joined Adobe last month. I would like to thank you for providing the best quality content on design [not just for interviews but beyond that] on your channel. It helped me a lot Thanks!!
Hey Arpit, great stuff. Quick question: 8:25 you mentioned client talking to server to generate a random id, but instead of talking twice to the server, it can generate an idempotence key and share it along the payload with the server in a single step. And if server is seeing it for the first time it can store it, right?
@@AsliEngineering Hi Arpit, I had the same question. Can you please explain how it's easily hackable? I read the blog of stripe and they are generating random idempotence key at client-end itself.
@@adityasheth if end user/ frontend is responsible for generating idempotent keys and sending it to backend. What stops then to generate a new key every time and not adhere to the protocol of reusing.
@@AsliEngineering That makes sense. In that case, even a mistake of API call with a new idempotent key from the end-user can make double transactions that way.
@@AsliEngineering You are conflating security with Idempotency. Idempotency covers the case when a client request is left in a situation where it doesn't know if the request was executed or not. The client needs a reliable way to send the SAME request to the server and have it be recognised as a duplicate send. It can generate a uuid. The server can make the decision to ignore or execute. If a "hacker" maliciously double executes requests, it doesn't matter (they transfer money twice), it doesn't hurt anything. Your goal is trying to prevent normal users accidentally double submitting, not malicious users poking around and losing their money twice.
Sorry if I missed out but curious to know what will happen to case 3 with Idempotency key where the server has completed the execution but somehow there was network issue while sending the response back to the client.
Well server knows it has processed the transaction with that idompotent key. client doesn't get the response so it retries with same idompotent key. This time hopefully response doesn't fail again so client gets the success response.
So question still stays the same, if the client does a retry, server will know it has marked that request with the same Idempotency key as completed, so won't it throw error or ignore the request?
@@ayush_walia How long can we keep the key in server? I think we can delete these keys older than a few minutes, right? As old keys anyway don't make sense.
Consider this scenario Server gets a payment request with a idempotency key and beings processing the request, mean while due to some network error the client reties the same request with the same idempotency key. At this point the server is still processing the initial request so the idempotency key is still marked as unused. What happens here? Even if the server marks the idempotency key as "used" immediately when it starts processing the request still we might end up with this problem in some rare scenarios due to race conditions. The only way to the solve this that I can think of is some kind of locking or transactions on the DB layer.
At 12:35 if the server is deleting the key after the request is handled, that means after the response is send, now let's say the response could not reach the client because of network issue, the client will retry, but the server has already delete the key, so it would consider the retry request as a new request so how to handle that
If we don't have the idempotent key in the DB that means, it is processed. If its a new request it should have obtained the key from the key generator and it will be present in the DB
"Keys are eligible to be removed from the system automatically after they're at least 24 hours old": stripe doc. They don't immediate delete it, to handle cases mentioned by you.
Any idea on how to make api calls idempotent in case a new request is submitted within a matter of milliseconds by a user by mistake? Here in this case the new request will have a new idempotent key and hence it will be treated seperately by the server, right?
Great tutorial!! Have some doubts: What if you have a microservice architecture and you have multiple APIs in the backend? Each of those APIs would be storing the idempotent key? and what if request is failed somewhere between whole APIs?
I think in this case, you must need a distributed system which should maintain the idempotent keys and status of it. If there are many APIs which needs this key and need to update the status then you may consider any data structure/a way to store multiple status and have a final status.
Hi Arpit, I think there is an edge case here but I might be mistaken. Consider a scenario where the idempotence key was sent and application tried to do the operation. Once operation is successful, we will commit the same to our key value store. But there is a possibility that our key value store is down or application went down before making that call. In this case, consumer will send the request again and application will still do the reprocessing. Is there a way to handle this gracefully.
You'd typically store this information in your backend as well, and could check for the same transaction there. (If it exists, ignore/return success or error)
@@AsliEngineering If I am correct, you are saying that doing the operation and storing the idempotency key into key-value store should be under single transaction boundary. That means in case, storing fails all changes done till now will rollback.
Feb cohort is full and really sorry I cannot accommodate more. I am just trying to maximize learning outcomes. I hope you understand. Looking forward to seeing you in the April cohort. The admission will open in March first week. Please join the waitlist to get notified.
