Clients can interact with a service either synchronously (often using rest) where the client is expecting a timely response or asynchronously using a message broker where the client is not waiting.

Overview of message brokers,messages and channels

• A message broker is the piece of software which enables services and applications to communicate with each other via messaging.
  • rabbitmq and kafka are popular message brokers.
  • when using message broker the sender does not need to know the network location of the consumer. it may not even know how many subscribers are there. loose coupling is a key advantage of using message broker.
  • Message broker enable request and asynch resposne (where client does not wait for response). This coupled with the fact that message broker buffers messages till the consumer is able to process them increases the elasticity of the system especially during peak load. On the client there is no thread waiting for response to come back which saves a lot of memory. This is called Queue-Based Load Leveling pattern.
• message - A message can be command or event. It has 
  • A  header with key val pairs (typically with message_id)
  • body - body can be text/binary
  • optional return address 
  • note that message format should not be language depenedent hence langage specific format (eg java serialization should not be used)
• channels - there are 2 kinds of channels
  • point to point (aka queue)- delivers messsage to exactly one of the consumers that is reading from the channel. command messages are sent over point to point channel. Eg if order service sends message to reporting service the command would be picked up by one of the instance of the reporting service.
  • pub/sub channel - event messages are oftent sent over pub/sub channel where there can be multiple consumers.

the following kind of interactions can happen

• Synchronous one to one interaction 
  • request/ synchronous reponse 
    • the client typically will make REST/gRPC request to service. In imperative programming languages like java the client program thread will making a blocking call and the thread will wait till a synchronous response does not come back from the client. A delay in response would lead to a timeout. In nodejs where the code is executed in non blocking event loop the client app is still expecting a response and if response does not come it would a lead to a timeout which has to be handled by the nodejs application. The key point is that if the response does not come typicall timeout exception will be there. The client assumes that the response will come within a certain time. Synchronous interactions reduce avalaibility as the pariticipants must all be simultaneously available. The avaliability of a system operations is the product of the avalaibility of the services that are invoked synchronously by the operation.
• Asynchronous one to one interaction (there is only one consumer of message )
  • One way notification
    • message is sent but no reply is required. 
    • client sends a message to point to point channel owned by the service
  • Request /async response.
    • if pair of messages exchanged between client and server the request response model can be mimicked. The client does not wait for the response message to come. note that typically the message sent has 
      • message id, message uniquely identifies the message, can be used for ensuring idempotency.
      • A reply channel header , the receiver must publish to reply messaging channel. 
      • correlation id (same value as message identifier),  when the client receives the response, he can use correleation id for request response correlation.
      • note that response can be handled by any of the client service instances. (the message will go to one of the instances of the service)
      • the command is sent over point to point channel owned by service.
      • the reply is sent over point to point channel owned by the client.
• Asynchronous one to many interaction (the same message goes to n consumers )
  • pub/sub (one to many notification) - Message deliverred to all attached consumers. The service that publishes domain events owns a pub/sub channel eg oder service will own order channel,delivery service will own delivery channel . A service which wants events emmitted by delivery service will subscribe to delivery channel .
  • request/async responses
    • a client publishes a request message and then has to wait for certain amount of time for response for interested services.

Pitfalls of asynchronous interaction using message broker

• message ordering  typically for handling load one logical service will have several (typically stateless) instances. As the the buffered messsages increase the number of service instances should increase to handle the increased load. The issue is that these instances of the same service which are processing messages on the same messaging channels become competing consumers. but this creates the issue of processing the messages in the correct order. for instance a request to create an order and reqeust to cancel the same order could route to different instances of the same service. depending on load , gc and various other factors the two instances may be handling requests at varying speed and the order cancellation could get processed before order creation. SHARDED /PARTIONED CHANNELS provided by some message brokers can ensure that each event of the same entity route to the same shard.Lets see how SHARDING OF CHANNELS works in kafka.
  • Kafka has the concept of consumer group.  A consumer group is considered as a single logical consumer.multiple instances of a service can be within the same consumer group. The consumer group will represent the service. A message will be delievered to only one of the instances within the consumer group. Message broker will assign partions the message channel into shards and assigns one shard to each instance within the message group.
  • The sender specifies a shard key in the message header. the shard key will determine the shard (and hence the service instance) to which the message will go. so for example entity id can be shard key, this will ensure that all events of the same entity go to same shard. Note that even though all the messages are going to the same instance of the service, as the service is multithreaded in order processing is still not guranteed (although the probablity is much reduced) . For instance a order cancellation event may start processing before order creation event . The application logic has to handle such out of order events for an entity. Note that this issue can come even when IPC is through  synchronous  request reponse model. Here is one pattern which can be used for maitaining order when messages are coming from input channel/queue and need to be processed concurrently yet output queue must have the same order as input queue. check the answer by anthony williams. https://stackoverflow.com/questions/3227042/maintaining-order-in-a-multi-threaded-pipeline.
• idemptoency - most message brokers gurantee atleast one delivery. a service can crash after receiving the message, processing it including updating the db but before acknowledging the message. the message broker can deliver the message again. Idempotent consumer can handle duplicate messages. https://clarifyforme.com/posts/5658330156498944/Idempotent-consumer. 
• preserving order when redelivering message.
  • if the message was processed by idempotent receiver and only the ack failed then when the message is redelivered the order preservation does not matter how ever if there is an error while processing order , if the service wants to retry the message it has to decide whether it wants presevation of order. read this. https://stackoverflow.com/questions/60318890/redelivery-from-queue-is-unordered-in-activemq-artemis-2-11-0 and https://dzone.com/articles/processing-messages-of-users-in-order-with-complet

Maximing avaliability during interactions

• Asynchronous interactions where the client does not wait for the response increases avaliablity. The services interact with each other using messages and no participant is blocked while waiting for response. Such architecture would be very reslient because of persistent messsage queues provided by message brokers hence peak load situation will be handled much more gracefully how ever the problem is that applicationa ofent have external api that uses synchnous protocol like rest . With use of websockets the architecture can be completely asynchnous.
• Data replication by services by subscribing to domain events of other services is another technique which can improve avaialibility during synchnous communication.
• A service can respond to client without fully processing the request. For client to know the final outcome of processing the client must poll or receive notification. this kind of model can be achieved using websocket.
• read more here https://accenture.github.io/reactive-interaction-gateway/docs/architecture.html