The Lotus Expeditor micro broker component is a small message broker that provides a messaging fabric for integrating various parts of a solution such as applications and small devices located at the edge of the network. A message broker ensures that messages arrive at the correct destination and are transformed to the format required by each destination. The micro broker runs on a wide range of standard devices including PDAs, laptops, and desktops. In addition, micro broker runs on more specific devices, such as programmable logic controllers (PLCs), smart home information hubs (for example, TV set-top boxes), and automobile-mounted devices. The micro broker is implemented in Java and runs on the desktop and device run-times provided by Lotus Expeditor as well as a J2SE Java run-time. The micro broker is suitable for embedding in applications and solutions that have a need for messaging, notification, and event services.
The micro broker allows use of both publish and subscribe and point-to-point messaging models. It provides a messaging infrastructure that lightweight messaging clients use to communicate with each other, within a single device, across a network, and when integrating with enterprise brokers. This enables an end-to-end integration fabric, reaching from sensor and actuator devices, to client applications and up to back-end applications. The micro broker provides an integration capability near the edge of the network, as well as providing a gateway to the enterprise services bus (ESB).
Micro broker is made up of three key components, that are described in the following sections:
The key difference with respect to IBM enterprise messaging products (WebSphere MQ, WebSphere Message Broker) is that micro broker is capable of running in environments that have a small footprint. Of course delivering a product that can run in these constrained environments, means a number of features provided by the enterprise messaging systems are not built into the micro broker. One big difference is that enterprise brokers allow message based applications to run in the context of the broker very much like application servers allow applications to run in the context of the server. All micro broker based applications interact with the broker through a client that runs outside of the context of the broker.
Other features built into enterprise messaging systems that are not supported by micro broker include: message transformation, message routing and message aggregation. It is possible to run limited message transformation and message routing but not to the same extent as in the enterprise offerings (further details in the bridge section below).
The broker is the component to which client applications connect to perform publish and subscribe or point to point messaging. It handles putting and getting messages from queues, and with publish and subscribe messaging, the matching of publications with subscriptions and the distribution of publications to subscribing applications. The broker also handles the persistence of messages to ensure message delivery at the quality of service required. The broker acts as a hub for routing messages between clients, and with the aid of the bridge, other messaging servers. The broker can store messages on behalf of a client that is not connected and make them available to the client when it reconnects. In addition, the broker can store messages on behalf of the bridge and make them available when the messaging servers that the bridge connects to are available. The broker supports different types of persistence for storing messages and subscriptions. The type of persistence is selected when a broker is created and is dependent on how the broker is to be used.
Types of persistence supported include:
- Memory only - Where data (messages and subscriptions) only ever reside in memory. In the event of a failure or the broker being shutdown, the data will be lost.
- Shutdown - Data resides in memory until the broker is shutdown, at which point the data is persisted. In the event of a failure, the data will be lost.
- Full persistence - Data is always written to a persistent store using transactions and will survive both a failure and shutdown. The persistent store is a purpose-designed, high-performance message store that resides in flash memory, on disk, or where appropriate for the particular device running the micro broker.
Application programs do not directly interact with the broker; instead they interact with clients which in turn interact with the broker. This effectively decouples the application program from the broker, allowing the application program to run in a different process or even a different system than the broker. The diagram below shows that a client can:
- Run in the same process space as the broker. As the micro broker is 100 percent Java, this means that the application, client and broker can run inside of the same Java Virtual Machine (JVM)
- A client can run in a different process on the same box as the broker
- A client can run in a different box to the broker.
In fact the broker can concurrently handle multiple clients running in any of the models described above.
Clients use the MQTT protocol to communicate with the broker. They simply provide a means for an application to interact with other clients (applications and services) using a broker. Before a client can interact with a broker, it must first connect to the broker. Once connected, messages can be sent and received. Clients can persist messages (for example, in memory or on disk) by implementing the persistence API interface. A client does not need to be continuously connected to a broker to receive messages. The broker can store messages on behalf of a client, allowing the client to receive stored messages when it reconnects. This provides an additional level of decoupling between sending and receiving clients. Put another way, a client can send messages to other clients that may not be running when the message is sent.
For Java, J2EE defines the Java Messaging Service (JMS) standard, that specifies a standard set of Java interfaces and behavioral semantics to which vendors can develop Java client libraries. This standard enables applications to be written to the standard JMS interfaces and to expect certain behavior from the client without depending upon any given software vendor’s product. If the JMS provider supports the use of a Java Naming and Directory Interface (JNDI) provider, it is possible to decouple the application even further from its administered run-time environment by having named messaging resources stored as administered objects in the JNDI repository instead of the application having to hard-code references to queues and topics in the code. In this way, if (for example) the administrator of the J2EE application changes the name of a queue, as long as the administered object is stored using the same name in the JNDI repository, the application does not need to be changed. Many enterprise messaging vendors support this capability, including IBM WebSphere MQ, IBM WebSphere Process Server, and IBM WebSphere Enterprise Server Bus as well as the IBM Lotus Expeditor micro broker.
Three clients are provided for use with the IBM Lotus Expeditor micro broker:
- A JMS client that supports the JMS API specification. The JMS client is the recommended client library to use when developing applications for the micro broker. The micro broker JMS client does not support XA transactions.The micro broker JMS client features a buffering functionality. When buffering is enabled, the client is able to create sessions and send messages regardless of the true state of the connection with the micro broker. This allows the client to operate in environments where the connection to the micro broker may not always be available. The client maintains a persisted buffer of messages for forwarding to the micro broker when a connection is available. It is important to note that some JMS features (transacted sessions, temporary destinations, message browsing) are not supported when using the buffered client. For further details, please visit IBM Lotus Expeditor InfoCenter.
- A Java MQTT client (version-3), whose API closely maps the MQTT protocol. This client is used for more specialized situations such as devices with constrained environments or where there is a specific requirement to use the MQTT protocol (for instance, low bandwidth, high latency networks).
- An enhanced Java MQTT client (version-5), whose API closely maps the MQTT protocol. This client is introduced in Lotus Expeditor 6.2 and offers improved and additional
capabilities while reducing the client footprint :
- Support for both publish/subscribe and point-to-point messaging paradigms
- Support for generic header properties. These allow meta-data to be added to messages, and improve interoperability and bridging with other messaging systems.
- Support for durable and non-durable subscriptions from the same connection.
- Support for message expiry. Messages can be marked with a time after which they can be discarded.
- Support for message priority. Marking a message with a higher priority signals to the server that this message should be delivered before low priority messages.
- Retained publications can now be cleared by publishing a new message without the "retain" flag set.
- Delivery of asynchronous messages can be started and stopped by the client.
- The ability for a client to reconnect after a network failure, and continue where it left off.
- Standard UTF-8 is now used for string encoding, meaning that any Unicode characters can be included in strings.
It can be used in the same situations that MQTT client version-3 to communicate with client applications with the micro broker or in micro broker to micro broker connections.
The micro broker works in conjunction with MQTT clients, which is insufficient in the enterprise messaging space with a plurality of different messaging protocols, clients and servers. This is where the built-in bridge comes in, with the capability to exchange messages with a range of messaging products. We describe this capability below and to what messaging products the bridge can provide integration into the micro broker's space.
A significant new capability for the bridge in Lotus Expeditor 6.2 that deserves a mention in this introduction is the upgrade of MQTT protocol to version 5 and security for MQTT when bridging to WebSphere MQ.
- MQ to the space (and hence access to many IBM enterprise messaging products)
The bridge has bridging support for WebSphere MQ. The WebShere MQ messaging protocol is a popular solution that drives the messaging backbone of an enterprise messaging systems. Through this MQ connectivity, the bridge enables the micro broker to message with a large number of enterprise messaging systems.
A scenario that the micro broker often supports is a system that includes WebSphere Message Broker. The Message Broker has capabilities and requirements that are more suited for the core of an advanced messaging system whereas the micro broker's, on the other hand, is more suited for the edge. This will be the subject of a case study in this document.
- Non-MQTT based JMS to the space (and hence access to many JMS messaging products)
The bridge has support for third-party JMS products. To clear any confusion, while the micro broker includes a JMS client and supports JMS applications, there are scenarios where enterprises would like to connect to a third-party JMS provider that is not micro broker. The bridge enables this, offering the JNDI-based JMS connectivity that can bridge messaging between the micro broker and the third party JMS provider. On the surface, the JMS API does not change, however, the underlying messaging protocol is provider-specific, with the micro broker JMS client, a MQTT protocol is used to ensure low overheads and latency between JMS application and the micro broker. The bridge allows a different JMS client, with a different protocol, typically with higher overheads and latency, be connected to the micro broker.
The micro broker can bridge to other messaging servers and clients that use the supported MQTT protocols. Other micro brokers and non-micro broker messaging products that support MQTT protocols can be bridged to. WebSphere Message Broker and Really Small Message Broker (http://www.alphaworks.ibm.com/tech/rsmb) are two examples. This allows the pervasive space to be expanded and also for the edge to be widened or extended as requirements demand.
- Bridge to itself
The micro broker bridging to itself is also a very useful scenario. This allows the use of transformation and routing technologies between clients (or, indeed, to/from one client - for example, a mobile phone that requires some intelligent persisting and messaging) of the micro broker, rather than endpoints of the bridge.
The figure below provides a view of the micro broker's connectivity options enabled by the bridge component:
Transformation and Routing
To aid the bridge connecting to different messaging protocols, or to meet different messaging requirements, the bridge contains built-in transformation and routing technologies.
An intrinsic part of the bridge is the conversion of destination names and between topic spaces and queue spaces. This topic-queue conversion is often used in the scenario where the enterprise messaging endpoint that the bridge connects to supports only queueing while the edge clients of the micro broker use the topic space. The destination conversion is very useful as it allows destinations from one side to be changed to what is expected on the other side.
The destination is an important factor for the bridge, but transforming the Quality of Service (QoS) is another important consideration. This can be changed on-the-fly, in order to support different Quality of Service requirements on the edge and the core sides of the micro broker, often due to different resources and connectivity availability.
These are the essential routing and transformation capabilities. The routing can be more sophisticated, based on other header properties (such as the correlated identifier) and the payload. Shortly we will explain four commonly used scenarios with these capabilities.
It should be noted that there may be cases where the needed transformation and routing capabilities can not be configured on the bridge and would have to be provided as a bundle. To help with this, there is a predefined message type that the bridge holds for any incoming and outgoing message. This means writing the transformation/routing code that works for any remote endpoint's type of message, is a straightforward task. A set of transformation and routing behaviors can be defined and dynamically deployed as separate bundles. Multiple transformations can be applied to any message.
The bridge pays special attention to a set of properties for all incoming and outgoing messages where there are MQTT, MQ or JMS header properties and performs appropriate mapping between the three protocols, meaning the user need not be concerned with converting standard message properties.
The micro broker is often deployed on the edge where the connectivity becomes increasingly less always-on, reliable, or more costly and this is where transmission control comes in. Transmission control has a set of predefined policies that can be employed for a variety of requirements.
We are going to cover four uses that the bridge is typically deployed to perform - context-based routing, static/deterministic routing, message enhancement/enrichment, and logging/monitoring. There are of course many other possibilities that are not covered below.
Content based Routing
The messages are be routed to different destinations (targets) based on the message content. Consider an order processing system where special orders identified with a property "OrderType=Special" in the message. These messages need to be routed to different destinations. The routing algorithm needs to parse this field for every message and take the decision of which destination the message needs to be sent.
The bridge provides the facility of parsing the message properties at the transformation level in the bridge. Based on the message property the destination of the message can be changed. The bridge will route the message to the new target.
In most cases the messages may need to be routed from many sources to one target. The target and the sources will have been predetermined. This form of routing is called static routing and very typically used in a scenario where data collection needs to happen from various sources.
The bridge has the concept of flows, that allow for messages to be received from multiple sources (topics and queues) and sent to a single target.
Messages coming from some sources can have too little detail. In a scenario where the enterprise messaging system expects particular fields to be present in the message, the messages need to be enhanced or enriched. This might need to be done at various points in a deployment. The messages coming to a bridge would usually be in MQTT format when received from devices. The MQTT version 3 protocol does not allow message properties to be set, hence the sender of the message on the device would have embedded everything into the message payload. This message payload can be parsed at the transformation level and the message properties appropriately set. If the connection definition on the bridge is set as JMS, these message properties would be mapped to the JMS message properties and sent across by the bridge.
Logging or Monitoring
In this context logging or monitoring is related to the messages flowing though a messaging system. The messages might need to be logged to suite a non-functional requirement of a business at various points in a deployment. Also a monitor showing the message flowing through a messaging system might be needed.
The bridge at the transformation level, can achieve this by logging some message properties to a file or update a monitor about the message information.
Security is an integral part of micro broker in Lotus Expeditor 6.2. Security is present at two levels in the protocol stack:
- Encryption on the wire
- At the messaging (or MQTT) level:
- Authentication, that is, is a user/client who they claim to be?
- Authorization, that is, is this user/client allowed to perform ‘this’ operation on ‘that’ data or resource.
The MQTT protocol’s data stream on the network is encrypted using the industry standard SSL/TLS protocol which guarantees confidentiality, data integrity and authentication of the entities involved in the communication. In a common scenario, micro brokers authenticate themselves to clients with a X.509 based certificate that is exchanged during the SSL/TLS handshake. The micro broker can be configured to have a secure listener, that will accept encrypted network connections via SSL/TLS. This SSL listener may be run in parallel with non-encrypted TCP listeners, but will need to use a different port. The default secure TCP port is 8883, which is registered with IANA (http://www.iana.org/
Clients authenticate themselves to brokers using a flexible scheme that includes login/password at the MQTT level. The micro broker uses the Java Authentication and Authorization Service (JAAS) to authenticate clients. If running on a platform supporting security features, then any client application which specifies a user name and password will be authenticated. If authentication succeeds, then the user name provided by the client will be used as the subject name for authorization. If a client does not specify a user name and password, then the client is considered to be authenticated, and the user name "anonymous" is used for authorization (which may of course be set up to disallow access to anonymous users; so this is not a security hole).
In the micro broker, JAAS is used for authentication, but not for authorization.
Once authenticated, clients are checked to see if they are authorized to access the broker. The types of authorized access are: connection, per topic / per queue, or administration. The authorization mechanism is part of the broker, and calls out to a Java interface at pre-defined policy enforcement points. At these points (for example, when a client is publishing a message), the access controller is asked if the client is authorized to perform the specified operation. Specifically, the micro broker’s access controller uses a policy based authorization scheme. Policies are specified in access control language (ACL), that is specified in XML. A default ACL file is created when a micro broker is created. Any changes to the file will take effect when the broker is started or restarted. To use an existing ACL file with a new broker, first create the broker then overwrite the generated default ACL file with the pre-written one, and finally start the new broker.
The Security added to the micro broker secures:
- MQTT client-broker connection.
- MQTT bridge connection, for example, broker to broker or broker to WebSphere Message Broker .
- MQ bridge connection, that is, broker to WebSphere MQ.
- JNDI bridge connection can be used to communicate the micro broker with JMS providers. In this case, the SSL settings of the JMS client needs to be pre-configured in JNDI, as the micro broker will see the client as a generic JMS client.