Problem

Client sends api requests to server but might not get a response becuase of connection timeout . The client has no way of knowing if the request is successful or failed . To make sure that the request is processed, the client has to retry the request. If the server has allready the processed the request then this would lead to the same logical request executing twice.

In case of message brokers a server may process a message and crash before acknowledging the message. The broker could crash and lose the acknowledgment. Message broker would typically retry untill message is processed successfully (so that atleast once delivery of message is guranteed)

Solution

An idempotent consumer can perform de-duplication and gurantee that message or api is successfully executed only once.

Few techniques are possible

• Technique 1. Idempotent receiver keeps track of completed api reqeuests in PROCESSED_API table. On receiving the api request the consumer processess the api request and then inserts the idempotency key in PROCESSED_API table in a single atomic transaction. Note that in this technique there is reliance on transaction isolation .  While the reqeust is being processed, if an api request with same idempotency key comes then it should wait to acquire locks so that two requests do not execute concurrently. The second duplicate request should execute after the first request has executed and would fail as insert into PROCESSED_API table would fail.
• Technique 2 : In microservices pattern the consumer which is receiving the request may be executing sagas pattern. Hence the request processing would not completely execute on the service. In this case the service can make atomic entries into RECEIVED_API table , different business entities table and sagas tables in a single atomic transaction. (once entry is present in business entities and sagas tables, sagas pattern is guranteed to complete the processing). If duplicate request comes then primary key violation would occur on insert into RECEIVED_API table. If pk violated, the service can check the satus of sagas transactions from business entity tables and return status to caller. Till the client does not get success or failure it should use the same idempotency key. After failure response it must use different idempotency key.
• Technique 3. The idempotency key can be stored in the business entities created or modified by an api request. Eg lets say an api request creates on order. Order can have an attribute idemPotencyKeys which stores idempotency keys of all requests which have created or modified the order. While processin an order modfication api request, a check can be made to verify that idemPotencyKeys does not allready have the idempotency  key of the request.