Microservices with Docker, Spring Boot and Axon CQRS/ES

The pace of change in software architecture has rapidly advanced in the last few years. New approaches like DevOps, Microservices and Containerisation have become hot topics with adoption growing rapidly. In this post, I want to introduce you to a microservice project that I’ve been working on which combines two of the stand out architectural advances of the last few years: command and query responsibility separation (CQRS) and containerisation.

In this first installment, I’m going to show you just how easy it it to distribute and run a  multi-server microservice application using containers.

In order to do this I’ve used Docker to create a suite of containers containing all the microservices required to run the demo. At the time of writing there are seven microservices  in this suite; they are:-

 

The source code for this demo is available on Github and demonstrates how to implement and integrate several of the features required for ‘cloud native’ Java including:-

• Microservices with Java and Spring Boot;
• Build, Ship and Run anywhere using Docker containers;
• Command and Query Responsibility Separation (CQRS) and Event Sourcing (ES) using the Axon Framework v2, MongoDB and RabbitMQ;
• Centralised configuration, service registration and API Gateway using Spring Cloud;

How it works

The microservice sample project introduced here revolves around a fictitious `Product` master data application similar to that which you’d find in most retail or manufacturing companies. Products can be added, stored, searched and retrieved from this master  data using a simple RESTful service API. As changes happen, notifications are sent to interested parties using messaging.

The Product Data application is built using the CQRS architectural style. In CQRS commands like `ADD` are physically separated from queries like `VIEW (where id=1)`. Indeed in this particular example the Product domain’s codebase has been quite literally split into two separate components – a command-side microservice and a query-side microservice.

Like most 12 factor apps, each microservices has a single responsibility; features its own datastore; and can be deployed and scaled independently of the other. This is CQRS and microservices in their most literal interpretation. Neither CQRS or microservices have to be implemented in this way, but for the purpose of this demonstration I’ve chosen to create a very clear separation of the read and write concerns.

The logical architecture looks like this:-

Both the command-side and the query-side microservices have been developed using the Spring Boot framework. All communication between the command and query microservices is purely `event-driven`. The events are passed between the microservice components using RabbitMQ messaging. Messaging provides a scalable means of passing events between processes, microservices, legacy systems and other parties in a loosely coupled fashion.

Notice how neither of the services shares it’s database with the other. This is important because of the high degree of autonomy it affords each service, which in turn helps the individual services to scale independently of the others in the system. For more on CQRS architecture, check out my Slideshare on CQRS Microservices which the slide above is taken from.

The high level of autonomy and isolation present in the CQRS architectural patterns presents us with an interesting problem – how should we distribute and run components that are so loosely coupled? In my view, containerisation provides the best mechanism and with Docker being so widely used, it’s format has become the defacto standard for container images, with most popular cloud platforms offering direct support. It’s also very easy to use, which definitely helps.

The Command-side Microservice

Commands are “actions which change state“. The command-side microservice contains all the domain logic and business rules. Commands are used to add new Products, or to change their state. The execution of these commands on a particular Product results in `Events` being generated which are persisted by the Axon framework into MongoDB and propagated out to other processes (as many processes as you like) via RabbitMQ messaging.

In event-sourcing, events are the sole record of state for the system. They are used by the system to describe and re-build the current state of any entity on demand (by replaying it’s past events one at a time until all previous events have been re-applied). This sounds slow, but actually because events are simple, it’s really fast and can be tuned further using rollups called ‘snapshots’.

In Domain Driven Design (DDD) the entity is often referred to as an `Aggregate` or an `AggregateRoot.`

The Query-side Microservice

The query-side microservice acts as an event-listener and a view. It listens for the `Events` being emitted by the command-side and processes them into whatever shape makes the most sense (for example a tabular view).

In this particular example, the query-side simply builds and maintains a ‘materialised view’ or ‘projection’ which holds the latest state of the individual Products (in terms of their id and their description and whether they are saleable or not). The query-side can be replicated many times for scalability and the messages held by the RabbitMQ queues can be made to be durable, so they can even temporarily store messages on behalf of the query-side if it goes down.

The command-side and the query-side both have REST API’s which can be used to access their capabilities.

For more information, see the Axon documentation which describes how Axon brings CQRS and Event Sourcing to your Java apps as well as lots of detail on how it’s configured and used.

---

Running the Demo

Running the demo code is easy, but you’ll need to have the following software installed on your machine first. For reference I’m using Ubuntu 16.04 as my OS, but I have also tested the app on the new Docker for Windows Beta successfully.

• Docker (I’m using v1.8.2)
• Docker-compose (I’m using v1.7.1)

If you have both of these, you can run the demo by following the process outlined below.

If you have either MongoDB or RabbitMQ already, please shut down those services before continuing in order to avoid port clashes.

Step 1: Get the Docker-compose configuration file

In a new empty folder, at the terminal execute the following command to download the latest docker-compose configuration file for this demo.

$ wget https://raw.githubusercontent.com/benwilcock/microservice-sampler/master/docker-compose.yml

Try not to change the file’s name – Docker defaults to looking for a file called ‘docker-compose.yml’. If you do change the name, use the -f switch in the following step.

Step 2: Start the Microservices

Because we’re using docker-compose, starting the microservices is now simply a case of executing the following command.

$ docker-compose up

You’ll see lots of downloading and logging output in the terminal window as the docker images are downloaded and run.

There are seven docker images in total, they are mongodb, rabbitmq, config-service, discovery-service, gateway-service, product-cmd-side, & product-qry-side.

If you want to see which docker instances are running (and also get their local IP address), open a separate terminal window and execute the following command:-

$ docker ps

Once the instances are up and running (this can take some time at first) you can have a look around immediately using your browser. You should be able to access:-

1. The Rabbit Management Console on port `15672`
2. The Eureka Discovery Server Console on port `8761`
3. The Configuration Server mappings on port `8888`
4. The API Gateway Routes on port ‘8080’

Step 3: Working with Products

So far so good. Now we want to test the addition of products.

In this manual system test we’ll issue an `add` command to the command-side REST API.

When the command-side has processed the command a ‘ProductAddedEvent‘ is raised, stored in MongoDB, and forwarded to the query-side via RabbitMQ. The query-side then processes this event and adds a record for the product to it’s materialised-view (actually a H2 in-memory database for this simple demo). Once the event has been processed we can use the query-side microservice to lookup information regarding the new product that’s been added. As you perform these tasks, you should observe some logging output in the docker-compose terminal window.

Step 3.1: Add A New Product

To perform test this we first need to open a second terminal window from where we can issue some CURL commands without stopping the docker composed instances we have running in the first window.

For the purposes of this test, we’ll add an MP3 product to our product catalogue with the name ‘Everything is Awesome’. To do this we can use the command-side REST API and issue it with a POST request as follows…

$ curl -X POST -v --header "Content-Type: application/json" --header "Accept: */*" "http://localhost:8080/commands/products/add/01?name=Everything%20Is%20Awesome"

If you don’t have ‘CURL’ available to you, you can use your favourite REST API testing tool (e.g. Postman, SoapUI, RESTeasy, etc).

If you’re using the public beta of Docker for Mac or Windows (highly recommended), you will need to swap ‘localhost’ for the IP address shown when you ran docker ps at the terminal window.

You should see something similar to the following response.

* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080(#0)
> POST /commands/products/add/01?name=Everything%20Is%20Awesome HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.47.0
> Content-Type: application/json
> Accept: */*$ http://localhost:8080/commands/products/01
< HTTP/1.1 201 Created
< Date: Thu, 02 Jun 2016 13:37:07 GMTThis
< X-Application-Context: product-command-side:9000
< Content-Length: 0
< Server: Jetty(9.2.16.v20160414)

The response code should be `HTTP/1.1 201 Created.` This means that the MP3 product “Everything is Awesome” has been added to the command-side event-sourced repository successfully.

Step 3.2: Query for the new Product

Now lets check that we can view the product that we just added. To do this we issue a simple ‘GET’ request.

$ curl http://localhost:8080/queries/products/1

You should see the following output. This shows that the query-side microservice has a record for our newly added MP3 product. The product is listed as non-saleable (saleable = false).

{
  name: "Everything Is Awesome",
  saleable: false,
  _links: {
    self: {
    href: "http://localhost:8080/queries/products/1"
    },
  product: {
    href: "http://localhost:8080/queries/products/1"
    }
  }
}

That’s it! Go ahead and repeat the test to add some more products if you like, just be careful not to try to reuse the same product ID when you POST or you’ll see an error.

If you’re familiar with MongoDB you can inspect the database to see all the events that you’ve created. Similarly if you know your way around the RabbitMQ Management Console you can see the messages as they flow between the command-side and query-side microservices.

About the Author

Ben Wilcock is a freelance Software Architect and Tech Lead with a passion for microservices, cloud and mobile applications. Ben has helped several FTSE 100 companies become more responsive, innovate, and agile. Ben is also a respected technology blogger who’s articles have featured in Java Code Geeks, InfoQ, Android Weekly and more. You can contact him on LinkedIn, Twitter and Github.

Book review: SOA Made Simple – Packt Publishing

Packt Publishing’s latest SOA book: ‘SOA Made Simple‘, claims to lay bare the fundamental strategies, goals, principals, benefits and impacts of service oriented architecture in a way that is easily accessible. In this review we’ll see if these claims are justified, and if they are, what it might mean for the SOA community as a whole.

As a certified SOA Architect, I’m often surprised by how difficult it is for companies to create a shared consensus and understanding of what it means to become service-oriented and how important it is for the future survival of commercial and non-commercial organisations alike. Many of these difficulties are related to two basic issues: lack of knowledge and lack of experience. This book intends to help alleviate both.

The authors (Lonneke Dikmans & Ronald van Luttikhuizen) certainly have the knowledge and the experience required to create successful SOA implementations. Fortunately for us, they also have have a fluid and easy to read writing style and the advice that they dispense in this book is accurate, valuable, practical, consistent and of a very high standard throughout.

Many of my SOA gotcha’s are dealt with in the text. Registries aren’t for everyone, tick. ESB’s can be good for some use cases, but using them for EAI is a backwards step, tick. Canonical models are essential for broad-brush interoperability, tick. Data standards can be useful internally sometimes, but often more so at enterprise boundaries, tick.

Setting the scene.

In the preface, the book identifies the types of people that it is intended to help: Architects, Designers, Developers, and Team-leads involved in delivering SOA.  However, I would go much further. I think this book would also be of use to many of the UK’s CIO’s, CTO’s, IT Directors, Enterprise Architects, Department Managers, Project Managers and Programme Managers. Basically anyone who is new to SOA but has some responsibility to deliver it within their enterprise.

The book begins with a chapter that discusses the problems faced by modern businesses, most of which stem from a lack of alignment between business and IT leading to an increased exposure to risk. Duplication of functionality and data in application (and process) ‘silos’ is also identified as a common issue. These two themes are revisited throughout the book using various example case studies.

The following chapter covers Service-Orientation as a solution to these problems. It starts by discussing the SOA architectural style but quickly moves on to services, discussing what a service is and how services can be described using the concepts of contracts, interfaces and implementations (these three being separate facets of the same service).

Service Inventory, Service Design & Service Implementation.

Chapter three discusses the logical starting point for any service inventory: service identification and service design.  It uses simple business process models to illustrate the activities and decision making points in the process (as do I when working with my corporate clients). Top-down, bottom-up and meet-in-the-middle design approaches are covered in detail, with their respective advantages and disadvantages made clear. A set of service design principals are offered, but one of the few criticisms I have is that these are not simply taken from Thomas Erl’s more definitive SOA Principals of Service Design (albeit re-written to make them more accessible to the layperson).

Chapter four discusses the process of ‘classifying’ (grouping & typing) services, and offers a simple classification system that introduces three basic service types: Elementary services, composite services and process services. It’s not a classification system that I’ve used personally (I usually use Erl’s) but I will be considering it in future for use with clients thanks to the additional simplicity it affords.

SOA Platforms are discussed in chapter five with a look at the common building blocks of service oriented enterprise architectures and the technologies that support them. Services, events, compositions, rules, UI’s, security, registry & repository and design & development tooling are all examined and their associated technologies identified and explained (for example: application servers for hosting services, ESB’s for exposing endpoints, etc.). Chapter six then goes on to explore the platforms and components offered by the big three SOA vendors (Oracle, IBM and Microsoft).

Management and Politics.

The latter third of the book is devoted to what I call the ‘management and politics of SOA implementation’. First up is a section on how to create a viable SOA business case and migration roadmap, including an examination of SOA investment choices using basic business scenarios such as cost-cutting or reduced time to market. The reader is also warned about the naturally wavering enthusiasm for SOA as complicated change programmes progress through various emotionally charged stages experiencing both euphoria and despair.

Chapter eight covers the important topic of SOA life-cycle management, whilst chapter nine continues this thread with a discussion on SOA Governance.

In life-cycle management, the authors elaborate on techniques for versioning services and for the management of various service related artefacts (contracts, implementations, etc.). Registries and repositories are explained and their differences noted, alongside some refreshingly practical advice from an enterprise level book of this kind – namely that registries are often not required and repositories can be provided using the simplest of tools (if you’ve seen my InfoQ article on Simple Service Repositories and you’ll already know I’m a strong advocate of this approach).

When discussing SOA Governance, the authors suggest a strategic approach when ‘picking your battles’ – choosing which principles and practices to protect and which can be sacrificed for the greater good. Governance, they say, is about protecting the SOA strategy by guiding and influencing the architectural design choices and remaining outcome-oriented. There are many ways to tie a shoe-lace they say, the important thing is that the shoe stays on! Deviations are sometimes necessary, but they should always be recorded, monitored and corrected at the most suitable juncture available. I couldn’t agree more. It’s a sensible approach to an important SOA practice that’s often misaligned, misinterpreted, misunderstood or simply missed out.

The final section is devoted to methodologies and SOA. Europe’s most commonly used methodologies for demand management (business change), project management, IT management and development are all examined and SOA’s impacts on their processes and practices are explained. Modern SOA (what the management consultants now refer to as ‘digital business’) supersedes most of these methodologies, so it’s good that the authors have taken the time to explain where conflicts in approach may arise and what you can do about them.

My Summary.

As you may have already noticed, I really liked this book. Personally, I think it should be the minimum required reading for any Architect, Developer or Project Manager who adds ‘SOA’ to their profile. Let me explain why…

People who genuinely ‘get SOA’ understand that becoming service-oriented means realising a new strategic design direction for both business and architecture alike. It’s not simply about web service technology. That’s very much the underlying theme of this particular book. It provides the reader with clear information regarding the business motivation behind SOA and then backs up this new found understanding with some insights regarding the tools and technology used to support this new architectural design paradigm.

However, over the years I’ve met a small number of ‘SOA Charlatans’ – people who are ‘a bit vague’ on the motivation and business benefits behind SOA, but figured that they would go and get a SOA job anyway because they’ve “done loads of integration” or because “no-one understands SOA, so why not?”. In the past, this ‘SOA bluff’ strategy probably worked in many cases, but from now on beware – if the interviewer has read and understood just a fraction of this book, they’ll tear these imposters to pieces!

That’s my two-cents, what’s yours? Leave a comment or share…

About the Reviewer:

Ben Wilcock is a freelance SOA Certified Architect with a reputation for delivering exceptional Service Oriented Architectures. You can read his blog at https://benwilcock.wordpress.com/ or contact him via twitter (@benbravo73) or via his company website at http://www.soagrowers.com/.

SOA Service & Microservice Design Cheat Sheet

This simple cheat sheet contains all the key goals, principals and design patterns that you should be aware of when designing services or microservices for SOA. It also and contains helpful links to places where you can find more in-depth information on each topic.

Implementing Entity Services using NoSQL – Part 5: Improving autonomy using the Cloud

In the previous posts I discussed how I went about building my SOA ‘Entity’ service for Products by using a combination of Java Web Services, Java EE and the CouchDB NoSQL database. In this final post in the series I’m going to leverage some of the technical assets that I’ve created and implement some new user stories using some popular SOA patterns.

My current Product Entity Service implementation is very business process agnostic, and therefore highly re-usable in any scenario where consumers want to discover or store Product information. However, as it stands the Product Entity Service is designed to be used within a trusted environment. This means that there are no restrictions on access to operations like Create, Update or Delete.  This is fine within a strictly controlled corporate sandbox but what if I want to share some of my service operations or Product information with non trusted users?

Lets imagine that in addition to our in-house use of the Product Entity Service we also wanted to cater for the following agile ‘user story’…

Implementing Entity Services using NoSQL – Part 4: Java EE

Now that I have prepared a skeleton contract-first web-service and created a data access layer using Ektorp and CouchDB, it’s time to wire them together into a fully working entity service. To do this I’m going to use Java EE and Glassfish 3.1.

It’s worth noting at this point that it’s not strictly necessary that I use Java EE at all for his kind of R&D work. I don’t need the security or the transaction features that are provided by a JEE server like Glassfish and I could probably use something a little lighter like Tomcat or Jetty. However, I do like the convenience and the features of JEE, and many applications that begin life on an standard Java application server like Tomcat do end up either grafting JEE features into Tomcat (like JAX-WS) or migrating to a full JEE server like Glassfish.

Implementing Entity Services using NoSQL – Part 3: CouchDB

Following on from part two of this series where I created and deployed the Product Entity Service using the SOA ‘contract-first’ technique, I’m now going to work on the NoSQL database aspects of the service implementation.

Implementing Entity Services using NoSQL – Part 2: Contract-first

It’s time to begin the coding of my SOA entity service with NoSQL project, and as promised I’m starting with the web service’s contract.

This technique of starting with a web service contract definition is at the heart of the ‘contract-first’ approach to service-oriented architecture implementation and has numerous technical benefits including…