Publish – Subscribe Messaging with Anypoint MQ

Well, after three years of working almost exclusively on the Mulesoft Anypoint Platform, solving critical business problems for both on-premise as well as cloud based customers I’ve finally decided to start my own blog.

As part of an ongoing client engagement, I got the opportunity to work with Mulesoft’s cloud based messaging solution, AnypointMQ – during the project we implemented point to point messaging for the most part but the publish-subscribe messaging is a powerful paradigm to connect distributed systems so I thought it would be a good idea to explore it as part of this blog.
Anypoint MQ is a multi-tenant, cloud based messaging service which allows integrations deployed on Mulesoft’s Cloud Hub environment to talk to each other using either point to point or publish-subscribe messaging. Other alternatives for cloud based messaging include the Azure Service Bus and Amazon SQS messaging service both of which you can integrate seamlessly with your Mulesoft applications.

While the two other options I mentioned above(the Azure Service Bus and Amazon SQS messaging) are available via trial accounts or low cost subscriptions to Microsoft Azure and Amazon AWS respectively, Mulesoft’s AnypointMQ offering is exclusively subscription  based and is not available for trial usage. To use this service, you need to have a Anypoint Platform enterprise license. Even if you do not have an enterprise license, this short blog should offer some insights and interesting read for those of you curious to know how well the cloud based messaging offering works – we’ll also be leveraging the work here in a subsequent blog which will explore usage of Anypoint MQ to implement reliability patterns in Mulesoft.

So let’s get started – to implement a publish-subscribe flow you first need to setup an exchange – and once the exchange is setup you can go ahead and attach one or more queues to it so they receive messages published to the exchange. You’ll also need to include a global Anypoint MQ connector as part of your project – now the defaults may be fine for most cases but we’ll be tweaking them a little here to meet our specific requirements.

By specifying ASYNC above we’re first and foremost leveraging the improved HTTP client to interact with Anypoint MQ – interaction with the messaging bus is through HTTP over REST and you can indeed publish and receive message’s but doing just that – more on this in another blog post – but getting back to the subject – the default client mode is SYNC but we’ve decided to use ASYNC here to get some better performance out of our implementation – we need to validate that ASYNC is in fact faster than SYNC – but that’s for another blog post. Plus ASYNC free’s us from the need to define threading profiles and takes advantage of Mulesoft defaults to process messages at scale. With SYNC you’d obviously have to tune the different thread pools to achieve the desired performance.

The next setting of interest is the max re delivery currently set to 3 – by default this value is set to -1 which will redeliver the message an unlimited amount of times – probably not a good idea if we’d like to control the number of re delivery attempts – also to get a better handle on this we’ll be using a roll back exception strategy.

Next we set the acknowledgement mode – we’ve chosen to set this to AUTO which enables the connector to automatically send a ACK or NACK at the end of the flow depending on whether it detected  a catch exception strategy or a rollback exception strategy. The other options are MANUAL and NONE both of which don’t really fit in with the scenario we’re trying to build here.

Moving onto the ‘Prefetch’ tab we’ve decided to go with a fetch size of 10 which will retrieve a batch of 10 messages once they appear on the queue and poll for messages every 5000 ms.

Now that we have all the setup, it’s time to write our flow. We’ll setup an auction service which gets bids on an inbound HTTP endpoint and publishes them to a topic – we have two separate queues listening on the topic – one of our queues happily consumes the bid and moves it to the next step in the flow, our other queue however accepts the bid but encounters an issue during processing which triggers an exception – this causes our thread to enter a rollback exception strategy and retry a number of times before giving up and submitting the message to a DLQ( in our case a simple VM queue).

The code above has three flows (a) a receiving flow with a HTTP inbound endpoint – this receives a set of bids and publishes them one at a time to a topic (b) mydemoqueue-1 which is listening on the topic and consuming the message (c) mydemoqueue-2 which also listens on the same topic but throws an error to simulate the rollback strategy.

In our rollback strategy we retry a fixed number of times and then move the message to a VM queue to simulate a DLQ.