Concepts
The Need
REST APIs are synchronous and follow a client-server model (point-to-point).
Async API: put message in a third party (queue) and move on (“fire and forget”)
- deferred processing: place message in queue and process it later
- fault-tolerance: even if a service goes down we still have previously sent messages in MQ
- decoupling: services aren’t time bound w.r.t each other
- load balancing: delegate load to multiple consumers
- data-streaming: large volumes of data exchange between services
Not only data messages, but tasks messages can be put in queues too.
Event-driven Architecture: put a task in the queue, send it to services and trigger processing in them
Streaming Data Pipelines: huge volume of data (~100K requests/sec) can be sent between services
Prod-Con Model (MQ)
Simple MQ - single FIFO pipeline. Ex - IBM WebSphere MQ, Rabbit MQ, Apache Active MQ, etc.
Messages in MQs can be ordered (FIFO order) or unordered (high priority ones are processed first).
- one-to-one; one producer, one consumer
- message is deleted from MQ by MQ system after consumption, they can also configured to be deleted on consumer-ack (RabbitMQ has this config)
The message deletion can be turned off in most MQ platforms but the general idea of MQ is remove-on-consume.
Disadvantages:
- low latency but slower than Kafka
- throughput is not as high as Kafka
Pub-Sub Model (Kafka)
Publisher puts messages in a central system, subscriber(s) consume them. Broadcasting. Acts as a Distributed Commit Log.
- one-to-many; one producer, many consumers
- message is not deleted from Topic after consumption by consumers (see Kafka notes)
- unordered mostly
- scalability: add multiple consumers to consume messages faster
Disadvantages:
- not for mission critical synchronous systems where ack is required
- not for non-idempotent tasks like financial transactions; because a single message can be processed multiple times
References