Thread safety

• Thread safety can be achieved in an event loop model like javascript where the code is executing in a single thread.
• Thread safety can be achived via mutex.

Thread pool

Server Programs such as web servers repeatedly execute requests from multiple clients and these are oriented around processing a large number of short tasks.Creating a new request every time a request is received would be very expensive in some programming laguages like java. A thread pool is a collection of pre created threads. Since the thread already exists in threadpool when the request arrives, the delay introduced by thread creation is eliminated, making the application more responsive. Advantages of thread pool are

•  Application is more responsive as the over head of creating the thread when the request is received is eliminated.
• Thread pool can act as throttle  ie help in setting a max limit on concurrent requests handled. If no thread is avalaible in thred pool then the request can be rejected or queued up. Typically even the queue will have a max limit . Throttling is important for various reasons.
  • Protecting internal resouces(memory /cpu) : Threads depending on language can occupy significant memory hence if there is no max limit to number of threads created then system will run out of memory
  • Protecting external resource : When protecting a resource like db, where more threads than max conn is not advisable
  • You want to restrict the system resources avaliable to a low priority task.
  • CPU intensive tasks and IO intensive tasks : CPU intensive tasks need fewer threads primarily for utilizing the cpu cores. Infact if number of threads is more the performance may actually degrade. IO intesive tasks in imperative programming languages like java need more threads.

Difference between concurrency and parallelism

Concurrency is when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same instant. For example, multitasking on a single-core machine.Concurrency as a property of a program or system

Parallelism is when tasks literally run at the same time, e.g., on a multicore processor. Parallelism (in the sense of multithreading) is not possible with single core processors.

What are synchronous, asynchronous , blocking and non blocking calls

Synchronous - Thread will complete the action, either by success or failure. ie the work is done in the thread that calls the function and the method does not return until it is finished. 

Blocking - Thread will wait on action untill success or failure . Failure is commonly a timeout.Examples would be 

• When a java thread is making jdbc request to rdbms. The thread is simply waiting for response on socket.
• Wait and notify mechanism. Thread blocks/waits for notification by another thread.
• Fanout pattern where thread blocks , waiting for a group of threads to finish a job.
• File read operations can be blocking calls.

Non-blocking - Thread will not wait to complete the action. Non blocking io is a very good example of non blocking calls. Event loop in nodejs is non blocking in nature..(non blocking sockets using epoll). Note that there is no call back function like asynchronous calls hence polling is required.

Asynchronous - Another thread/threadpool/process will complete the action . The action may simply be added to queue which is being processed .In a nutshell there is delegation. ie asynchronous methods return immediately because another thread does the work and raises a flag or fires an event  or executes a call back function when the work is done. Asynchnous calls are very common , for example

• Used for lower criticality tasks Eg while executing a transaction , notifications may be sent asynchnously using queue and threadpool or just threadpool. This will reduce the response time to user.

TPC , TPR ,Event loop (not fully reactive) ,Fully reactive non blocking event loop

• Thread per connection(TPC) : The webserver allocates a thread when the client starts a TCP connection, and it uses that thread for all of the requests arriving over that connection. The downside is that you end up with a mostly idle thread which is set aside for the connection but there are no active requests over the connection, If there are a lot of concurrent clients, you can exhaust the number of threads available to the system. In java especially per thread memory is high hence this model will take a lot of memory as a lot of threads are sitting idle doing nothing.

• Thread per connection(TPR)In the thread per request model a thread from a thread pool is allocated to handle an active request. Thread is not tied to connection.The whole point of “thread per request” architectures is to be able to support relatively large numbers of mostly idle connections with a smaller number of mostly busy request threads. This used commonly in imperative programming for instance with java. Note that the thread is bound to the request from start to finish so if that thread is waiting for a response from a DB call, a cache, or, heck, anything!, that thread cannot be reused in the meantime. TPR may not scale as the request traffic increases exponentially and there a lot of active requests over connections.

• Single threaded Event loop(not fully non blocking / reactive) : The event loop is a programming construct or design pattern that waits for and dispatches events or messages in a program.When the event loop forms the central control flow construct of a program, as it often does, it may be termed the main loop or main event loop. Typically event loop is single Threaded and hence threadsafe .This is very different from imperative prorgramming where u need a thread per request .You have only one task execution at a time. Single threaded event loop architecture in nodejs and java frameworks like netty , vertx,jboss untertow are based on non blocking socket calls. Note that in nodejs the entire event loop is non blocking. All components are non blocking . Full benefits of event loop come into effect only if the entire stack is non blocking/reactive and if all participating components (application code, runtime container, integrations) respect deferred execution, non-blocking APIs. Hence it’s possible to bring blocking components to an event loop but the net result will be that the scalability and stability effects, the actual expected benefits, decrease. In the worst case, there’s little or no difference in runtime behavior. For instance java applications talk to RDMBS via JDBC and JDBC is blocking in nature. There are efforts to create reactive relational database connectivity which is non blocking. Read more here https://spring.io/blog/2018/12/07/reactive-programming-and-relational-databases. Since jdbc calls are blocking , single threaded event loop + (taks queue + thread pool model) is used where main networking is exeucted by single threaded evenloop but blocking file/outbound http /jdbc calls are sent to (task queue + thread pool) for concurrent processing. 
• Event loop (fully non blocking) . Nodejs is fully non blocking, including calls to rdmbs. Note that you can use reactive programming in javascript using RxJS. With javascript you can do functional programming and functinal reactive programming.

Note on reactive programming: Reactive programming promotes event driven,non blocking approach to data processing . Reactive programming involves modeling data and events as observable data streams and implementing data processing routines to react to the changes in those streams. Reactive programming style can be used for event loops. Note that reactive programming only promotes non blocking behaviour, it does not guratee it. As allready mentioned reactive is a programming style(there can be a library in any programming language) . Note that one confusing thing is that when articles use the term reactive component  it is often meant non blocking component, not necessarily coded in reactive style. Hence reactive and non blocking are often used synonymously eg reactive database.

Event loop in java : Java frameworks like netty, vertx, jboss undertow , Spring Webflux are based on signle threaded event loop. Spring Webflux by default has Netty as an embedded server. Netty and Undertow are non-servlet runtimes and Tomcat and Jetty are well-known servlet containers. Before the introduction of Spring Webflux, Spring MVC used to have Tomcat as its default embedded server, which uses the Thread Per Request Model.  Now, with Webflux, Netty has been preferred as a default, which uses the Event Loop Model.  Although Webflux is also supported on Tomcat, it's only with versions that implement Servlet 3.1 or later APIs as serverlet 3.1 introduced NIO . Servlet 3.0 provided Async Servlets which allowed asynchronous request processing, but only the (traditional /blocking) I/O was permitted. This can restrict that scalability of your applications . Bascially what happens is that servlet thread can dissociate itself from request , delegate task to a throttled thread pool, and then return to servlet thread pool for handling http requests to webserver. Servelt 3.0 Asynch servelet was useful for handling a situation where long running tasks could exhaust the servlet thread pool. 

 Servlet 3.1 permitted non blocking io .Read more  https://dzone.com/articles/spring-webflux-eventloop-vs-thread-per-request-mod AND https://dzone.com/articles/servlet-31spring-mvc-non-blocking-io.

http multiplexing

Question> What is http multiplexing and it decrease server load?

Multiplexing helps to reduce server load by reducing number of connections but in TPR model  (more connections != more server threads)

Double checked locking pattern

Double checked design pattern is a concurrency design pattern used to reduce locking over head while implementing lazy initialization / singleton pattern . The code should first check if the lock has to be acquired before acquring the lock.

Here is an implementation of singleton pattern without double checking pattern.

class Singleton {
    private Singleton singleton;
   /* this introduces an locking over head every time getInstance() is called  */
    public synchronized Singleton getInstance() {
        if (singleton == null) {
            singleton = new Singleton();
        }
        return singleton;
    }
}

As acquring a lock can introduce a significant over head , this implementation is suboptimal.

Here is the implementation of singleton pattern with double checking pattern.

// Note that volatile key word does not work under Java 1.4
class Singleton {
    private volatile static Singleton instance;
    public static Singleton getInstance()
    {
        if (instance == null) {
            /*The lock will be acquired only once when instance is not created, on all subsequent calls to getInstance()
               lock is not acquired.
              */
            synchronized (Singleton.class)
            {
                if (instance == null) {
                    instance = new Singleton();
                }
            }
        }
        return instance;
    }
}

Note the use of volatile keyword. Volatile Keyword is applicable to variables. volatile keyword in Java guarantees that value of the volatile variable will always be read from main memory and not from Thread's local cache. 

Fan-out/fan-in 

Fan-out/fan-in refers to the pattern of executing multiple functions concurrently and then performing some aggregation on the results. The fanout pattern could be used for better utlization of multiple cores of cpu or in case of io intensive application performing blocking calls.

Read more

https://rsocket.io/