Event sourcing

Applications typically maintain the current state of data/entity by updating It. Typically locks are used for isolation to avoid problems like dirty read and lost updates. Additionally the history of transasctions is either lost if not explicitly maitained or will cause futher performance over heads as , in addtion to maintaining current state of entity, history of changes is also maitained. Event sourcing is an architectural pattern in which the state of the appliction is determined by a sequence of events .

In event soucring

1. The events which lead to state changes in an entity are appeneded in an immutable, authoratative data store called event store. (hence there is no contention during processing of transactions). 
2. The current state of entity is not maintained, instead the entity is persisted as sequence of events in an event store. (In RDBMS each entity is persisted as rows in a table and entity fields become columns)
3. Current state of an entity can be determined by replaying the events for the entity. (The code must iterate through each event for the entity in the correct chronological order).
4. When a service receives a command it converts the command into list of events representing the state changes. It is these events which are persisted in event store and emitted by event store.
5. When ever entity / domain object state changes event store must publish events usually for 
   1. Transaction/operation completion
   2. CQRS
   3. Materialized views.
6. As event store act as storage and publish events , they solve the problem of updating local db and emitting event.
   1. From the microservice perspective event store is a hubrid of datastore + message broker as event only has to be pushed to event store.
      1. under the hood event store may be using rdbms + message broker like kafka. (a message relay would pick non published events from eventstore and publish to kafka, note that in case of eventstore transaction outbox will not be there) 
   2. Event store does need transaction outbox. It can use polling approach to determine the unpublished events (based on published column in the events table which tracks if an event is published or not.) and publish them. Note that in case event ids are monotonically increasing, determining unpublished events based on last published event id can run into edge conditions.
7. As the history of changes are maitained and hence temporal queries are possible.
8. Consider using version of event schema if the format of events can change. event schema change can often be backward compatible as adding a filed is not likely to impact consumers of event. the consumer will simply ignore unknown fields. Some times the older events may need to to be migrated to the new schema
9. If the number of events for an entity are large (eg accounts ), snapshots of the aggregate/entity state can be created at specific intervals, so that current state can be created by replaying the events created after the snapshot. ie firs the aggregate instance is created with the help of snapshot and then iterate through events created after snapshot creation. The snapshot can be JSON.
10. Note that events emitted by cosumers have to be handled by idempotent consumers. If acid datbase is used then event/message id should be put in processed_messages table as part of atomic transaction that inserts into the event table.In case nosql db is used and acid gurantees are not there the message consumer stores the message id in the events that are generated while processing it. Duplicate detection can be done by ensuring none of aggregates events contain the message id. Read more https://clarifyforme.com/posts/5658330156498944/Idempotent-consumer   note that processing of message must necessarily generate event (a dummy if necessary ) so that message id can be stored in the event/dummy event.
11. Deleting data can be a challenge in event store hence some times encrypting data which is deletable and deleting the encryption key is one way of deleting.Pseudoanomization is another solution.
12. Some points to remember about events.
    1. Note that the names of the events are formulated in the past tense. The names of the event should be expressive and part of the ubiquitous language of the project (Domain driven design principle) A good example will be orderDeliveredEvent. createOrderEvent would be a bad example as createOrderEvent would be command style syntax.
    2. In domain driven design Aggregate is a logical construct which is collection of entities that is bounded to specific business context. There is a root entity which is responsible for lifecycle of the whole object graph under it. Eg loan application can be root entity and within it income source and many other entities or value objects can be present. Events should be based on aggregates , they should not be too fine grained.
    3. Events are always immutable
    4. Events are not deleted from event store, delete event is another event.
       1. some times based on regulatory complilance may need to be deleted . One option is to anonamize the event.
       2. dump pipes(light weight message brokers) and smart end points is good for event sourcing.
    5. Event are commonly stored as json. Event can also be a row in an RDBMS.

When to use Event sourcing

• To avoid / minimize contention when changing state of entity.
• Data history is matained without application taking any special effort. 

Disadvantages of event soucing

• Event sourcing can reduce read performance .
• Evnet scouring in conjuction with CQRS can give fast , contention free writes and fast reads as well as read data store in an optimized model/ platform for reads but the read store is eventually consistent and this approach should be avoided if very strong consistency requirements are there.
• Since the current state is not maintained hence queries like get all accounts where balance is more than 100 can be very difficult. this issues can be resolved by CQRS. Hence event sourcing and CQRS go hand in hand!

Event store

As allready mentioned event store is essentially database + event realy + message broker. Event store typically would have 3 tables

• events for storing events , events for all entities will be stored here, event must have entity id as one of the cols.
• entities , there will be on row for an entity, in case of optimistic locking version number will be one of the columns.
  • each time an entity is update the version number will be updated.
  • this can be used for implement optimistic concurrency control when modifying entity. see more in this video https://youtu.be/A0goyZ9F4bg?t=4035.
• snapshots  , the seralized representation of the entity snapshot will be stored here.

CQRS(command query responsibility seggregation)

Typically read and write of data is done through the same layer in n layer architecture. In microservices (with each service obeying the hexagonal architecture also does this) The read and write data model is the same.

• CQRS separates the data model into a read data model and write data model(create/update/delete)
  • CQRS is a generalization of an architecture where an enterprise uses RDBMS as transactional system of record and text search database as Algolia/Elastisearch. 
• Commands are used to update data and queries are used to read data.
• The write datastore can be rdbms, nosql or even an event store(in case of event soucring) where as the read data model typically is optimized for reading and can be a rdbms (with schema more optimized for reading , eg denormalization) or olap system or a system specializing in geospatial queries.
• The write data model is syched up with read data model asynchronously using event driven architecture. 
• The read and write stores can be scaled indepedently depending on requriements.
• CQRS can be implemented within a service or a seprate query service can also be created. The query service will only have api's for querying .
• Note that the query side could subscribe to implements of multiple services.
• CQRS can be used to implement queries that retrieve data from multiple services (alternative to API composition).
  • some times this can be beneficial as developer's do not have to execute complex joins which can be network and peformance bottleneck. The CQRS view would be prejoined!

Event Sourcing + CQRS challenges

• consistency 
  • read store is eventually consistent.
    • note that one way to have high conssistency would be the read store and write store in in rdbms . the read tables are more suitable for reads (eg denormalized). The updates of read tables and write tables can happen in the same acid transaction. This system would not scale that well .
• validations
  • eg email address must be unique.
    • the read store may not have be consistent yet.hence when u make check if email exists it will say no, although insert has happened on the write side. One option would be validate from event store but this will be slow.
• concurrent transactions.
  • optimistic concurrency control is commonly used in microservices architecture to handle vairous anamolies as sagas transactions lack isolation.
  • pessimistic locking is a misfit in event sourcing. pessimistic locking is discouraged even in traditional architectures.

When to avoid CQRS

• Note that the read store is eventually consistent. When the business domain has very strong consistency requirements  , then this model is not suitable. Note that you should only be as consistent as your domain requires.One way of avoiding this inconsistency is that the command and query side api , provide client with version information to determine if the query side is not in synch with the command side. The way it would work is the command side would return the id of the published event to the client. The client makes query operation with the id of published event. If the query side has not received the event and updated the view then the query side will return an error.