---

I’ve been working on decoupling a large monolithic API and separating its core concerns from processes that can be done asynchronously offline to make it more performant. Things like billing, aggregation, counting, global caching and deep analysis don’t make sense in a workflow to quickly return value-added data based on some inputs. They can be done elsewhere.

A technique I’m using a lot to extract these workflows is creating broadcasters. There's nothing new about this design pattern, but it's effective. A publish-subscribe style allows us to send out messages with details for microservice actions without the core workflow caring who receives them or what is done with them. We simply broadcast a message with some details and subscribers receive them and do some work.

So we can separate our billing, aggregation, reporting and deep analysis workflows by creating broadcasters that send out JSON messages with the details needed for each of these pieces. For an API that operates in different regions around the world, messages can fan out through cross-region subscriptions and make it to where they need to go.

As an example, let’s say we are using AWS and trying to remove a core billing process that is embedded in an API call to return data that has nothing to do with billing. The billing process saves to a data store that replicates to a few data centers around the world, but the core API product calculates data that customers pay for and this is our primary concern…the billing is secondary.

Instead of using a thread in each API call to save data to SQL (for example), we can broadcast a JSON message with the billing data to an SNS topic that is subscribed to by workers that will handle the saving. The API can fire the message and forget about it and do less work in the process, leaving more resources available to our core concerns.

While we’re at it, maybe we want to get rid of SQL (and its licensing requirements and replication lags), and use Lambda functions in different regions to subscribe to the broadcast topic and save message payloads to new data stores like DynamoDB or Postgres or RedShift hosted in different global regions.

Using Lambda is compelling because we end up with something like cross-region replication in the traditional SQL environment, but done via messaging in a disconnected and asynchronous manner. Using feature flags smartly in the API, we can even run both systems in parallel, broadcasting using our disconnected workflow while still saving to SQL and replicating, and then comparing the results in both data stores over time.

The broadcaster pattern lets the API focus on the core competencies of adding immediate value to requests while still handling sensitive workflows with integrity, but via a disconnected workflow that leverages microservices and the separation of concerns. By eliminating unnecessary work and doing the remaining work more efficiently, our API can perform and scale better and the code driving it can be less monolithic and more manageable.

And we end up with the same results we need to function as an API and a company, while working more smartly and efficiently with more resources ready to perform and scale to meet future needs.