Written by Dakshitha Ratnayake, Associate Director of Technology at WSO2
Where is my cab? What time will my train arrive? Was there a price drop in the stock that I invested in? Today, users increasingly demand ever more interactive experiences, and they expect to be automatically informed when something has changed without having to hit the refresh button.
Receiving notifications about someone liking your picture or reacting to your story on Instagram, or your Facebook news stream showing the latest updates about what your friends have been up to, are just a few examples of how frontend applications react to events in the backend. This is all made possible through event-driven APIs, which are supported through various modes of asynchronous event-driven communication between the client application and the backend.
So, the question is how do you build apps that require (near) real-time updates based on remote events? How should developers and architects who create reactive web applications or expose APIs for the consumption of such applications, handle asynchronous or event-driven communication in the world of APIs where synchronous communication is predominant, and most firewalls block non-HTTP traffic?
The answer? This is all about having an event-driven backend, choosing an asynchronous web API technology based on exact async requirements, and a client that can work with remote events.
Synchronous APIs versus asynchronous web API technology
Many APIs that make up the web today are synchronous APIs, where the client initiates all communication between the client and the backend by sending a request to the backend and waiting for a response. However, these request/reply API calls happen one at a time, in a pre-arranged sequence, and each interaction blocks the progress of the process until its completion.
If the client application wishes to know about an update, it must continuously invoke the API for updates at a regular interval. This is known as polling and has been a common approach for client apps that need to become aware of new data. But this method is largely inefficient and less reactive than if the backend immediately notified the client when an event occurs. The term “polling madness” was coined because most of these polling calls are wasted because the data has not changed.
When should you use event-driven APIs?
Event-enabling APIs is a relatively recent phenomenon compared to event-driven architecture (EDA), and it uses EDA to support scalable, real-time or near-real-time, push-based communication in APIs published to third parties. Being event-driven is about taking advantage of events as they happen, and consumers can subscribe and register their interest and react to events. Making APIs event-driven or asynchronous (along with an underlying server-side event-driven architecture) can eliminate the need for inefficient polling requests and send updates to the client or event subscriber as soon as they occur. This provides a much better experience for users.
So, in a nutshell, event driven APIs should be used if:
- The application needs to push changes from the server to the client rather than waiting for a client request.
- The application must provide users with a highly active two-way communication channel.
- The application requires many continuous interactions between the client and the backend that create scaling issues for the backend if synchronous APIs are used.
- The application needs to be able to monitor system-wide events.
Choosing asynchronous API technology
Event-enabling APIs do come with multiple complexities, starting from what frameworks and networking protocols to choose, to building the reliability of delivery, and ensuring the scalability of the solution. The asynchronous protocols commonly used today solve the problem of real-time communication, but they solve different aspects of this problem in different ways which means that some protocols serve different purposes better than others:
- For multiplexed, bidirectional streaming and for applications that need a considerable number of messages from both ends of the connection, WebSockets is ideal.
- Server-Sent Events are especially useful in cases where a browser client is just interested in subscribing to a stream of data, without needing to communicate back with the server in the same connection.
- Webhooks can be used for a simple implementation of pushing notifications to one or a small number of servers.
- The newer async variants of GraphQL (Subscriptions and Live Queries) come with the benefits associated with GraphQL, however, the implementation is relatively complex and entails a considerable learning curve.
- gRPC and Kafka are generally used for communication between internal microservices.
Understanding what the backend architecture should look like
While the client architecture must subscribe to state changed events that originate from the backend, the typical backend architecture can be extended to create remote event connections with the clients via:
- Event-driven APIs (ideally exposed through an API management platform that supports eventing semantics).
- A message broker (if one is not already present).
- Microservices that publish and process state changed events.
Although event-driven APIs do not explicitly require a broker, using an intermediary between event producers and consumers helps to implement the required patterns to deliver more manageable and scalable solutions. The broker receives events from IoT devices, change data capture (CDC) tools, other backend systems and services, and from client applications if two-way communication is enabled. It can then alert the services that subscribe to those events. A scalable microservices architecture (MSA) is the optimal architecture for complex event-driven backend services. These event-driven microservices can act as event subscribers or publishers to process events, handle errors, and persist event-driven states.
The remote event connection between the backend and web clients can be established through event-driven APIs powered by various asynchronous API technologies — such as Webhooks, WebSockets, Server-Sent Events, GraphQL subscriptions, etc. The backend uses these APIs to send events pertinent to the clients and receive events that originate from clients.
Managing event-driven APIs
An event-driven backbone will manage the overall real-time data flow securely and at scale, while asynchronous APIs can be managed for external and internal consumption with a traditional API management solution that comes with inherent or extended capabilities to support event-driven semantics. Most organisations have basic event processing infrastructure, but many do not have the capabilities to design, develop, test, and manage event-centric APIs. Essential API management capabilities — particularly governance, access control, monitoring, analytics, and monetisation — provided by API management solutions can be used to manage asynchronous and REST APIs. Some API management solutions already support Webhooks or/and WebSockets.
Expanding business reach and adoption
Synchronous APIs are ideal for many scenarios, particularly if you need an instant response. In other cases, when clients need to be informed of events or the processing required for the response happens at a different time, ordinary synchronous messaging becomes tricky and event-driven APIs can aptly address such needs.
EDA is flexible enough that you can start with simple notifications or combine two-way event communication with a synchronous approach to deliver an optimal architecture. So, using an event-driven design alongside a traditional request/response design for your APIs gives you the best of both worlds to build highly scalable and reactive web applications. Furthermore, combining traditional API management capabilities with an event-driven architecture provides tremendous value additions to expand business reach and adoption.
