Here’s an example of a distributed system using REST for inter-service communication:

  • A user has come to an online travel booking platform and purchased some airline tickets
  • the UI communicates user actions to a Booking Gateway backend service
  • The UI puts a loading spinner on the screen until Booking Gateway responds
  • meanwhile, the Booking Gateway service sends a http POST request to the Ticket Service to confirm the tickets are available
  • Booking Gateway waits for a response from Ticket Service
  • Once a response is received, Booking Gateway then makes a POST request to the Billing Service to charge the card
  • Booking Gateway waits for a response from Billing Service
  • Once a response is received, Booking Gateway finally updates the UI which removes the spinner and prints a confirmation.
  • Booking Gateway also sends a HTTP request to the Email Service to send user a confirmation

rest diagram

please forgive the amateur diagram - I’m an engineer not an architect

–

Here are the key points that make this RESTful:

  • the services are using synchronous HTTP request/response communication
  • services own their own “resources”, aka business objects, and they take requests to modify those object. (i.e. create a new credit card charge etc.)

The obvious issue to me here is that one link can break the chain. Say for example the tickets are available, but the Billing Service is already overwhelmed with requests and a 503 Unavailable response is received. What happens to the user experience?

There are many way to handle this. One is to implement an alternative protocol for service communication using 1-way, Asynchronous messaging similar to the one seen in my Mesos example a few weeks back.

In this new approach, services communicate thru a message broker, such as Kafka, instead of direct HTTP calls. Here’s how it might look:

  • A user has come to an online travel booking platform and purchased some airline tickets
  • the UI communicates user actions to a Booking Gateway service
  • Booking gateway responds immediately to the UI request and creates transaction (with a UUID) and puts it into a “pending” state
  • User is informed their transaction is pending - no loading spinner needed
  • Booking gateway drops a message into the Tickets topic about this order
  • Tickets topic is consumed by Ticket Service
  • Ticket Service processes the request and drops a response in the Order Updates topic
  • Booking Gateway consumes the Order Updates topic
  • Once Booking Gateway gets an update on the transaction, it changes the internal state for the order and sends a new message to the Billing topic
  • Meanwhile, the UI can poll the Booking Gateway and find out that its order has changed state
  • Billing Service consumes from Billing topic, processes the message and drops an update in the Order Updates topic when finished
  • Booking Gateway service consumes the message, updates the order state and sends a message to the Email topic

broker diagram

holy shit that is a crappy diagram - don’t quit your day job

–

In this new pattern, we have Actors sending Messages to topics and then moving on to their next order of business. There’s no waiting around for a response, or breaking when another service is down (unless the messaging broker system is down…that’s a whole another situation)

The nice thing about this model is that messages are not lost in the HTTP abyss if a service call fails…messages live in topics and consumers can come back online and continue reading them, never missing a message. This means the state of the system will “eventually” be correct. Additionally the state of the system is easy to reason about by checking expected state against actual state using consumer offsets for example.

–

Diagrams made on https://www.youidraw.com/