This section describes some of the key elements of the solution architecture provided for the customer. It contains the important elements that are necessary to understand what the solution design consists of and why several architecture decisions have been made.
Lotus Expeditor integrator compliments an Enterprise Service Bus. The focus of Lotus Expeditor integrator is to provide reliable and secure data exchange between clients at remote locations and the head office in a lightweight and near zero maintenance fashion.
The JKHLE overall environment including the new parts at retail stores looks like following.
Customer's new architecture - big picture
Please recognize that the retail store environment on the right hand side is reduced to a few building blocks. Key elements of JKHLE's new architecture are marked with a thick red line, the integration EndPoints ("EP").
To design a new architecture it should be clear what is in scope for the new system, and what are the actors and interfaces needed by the system. The following picture shows the system as a component in the middle and the integration End Points at JKHLE's retail stores.
The System in the middle is the future retail store dispatcher to be implemented for bi-directional data transfer from the head office to JKHLE's stores and vice versa. At this stage it doesn't matter what it looks like in detail but let's have a deeper look into the different interfaces of this system.
- This is the connectivity to JKHLE's Backbone Service Bus for incoming and outgoing data based on messaging. A messaging endpoint is already being used at stores today but is doesn't provide all the additional capabilities JKHLE requires.
- The POS Server needs to be fed with data (for example, price and article updates) as well as needs to send data (for example, collected transaction data - TLOG) to the System. JKHLE already asked to replace the FTP connectivity by a more reliable and near real-time interface at her POS Servers. This topic will be reconsidered later on again.
- In opposition to the current infrastructure JKEHL decided to deploy the Back Office Application for their Self Checkout Boxes at the head office and provide on-line access for their store staff. This follows the overall strategy of non-complex IT operations in store. Self Checkout Boxes should be connected to the System through messaging for data exchange with the centralized Back Office Application.
- Data for Scales and Label Printers need to be stored in a local file system for further processing of Scale and Label Printer control applications. Today JKHLE is using a custom developed File Handler to read data from a message queue that is already in places and stores them as a file into the appropriate directory. This Custom File Handler caused several problems in past and JKHLE is looking for an alternative.
- This is an interface to the Transaction Monitoring of JKHLE. It should provide transaction events, whether a transaction has been executed successfully or not. With this end-to-end Transaction Monitoring JKHLE is expecting to improve their control about critical business processes.
- The same as for Transaction Monitoring should be reached with this interface to head office Technical Monitoring: End-to-end view of overall system health. As JKHLE has the TEC (Tivoli Event Console) established for their other systems, an appropriate data format should be delivered to be consumed by TEC.
Customer requirements (functional / non-functional)
Only an excerpt of overall use cases is described here. They will be used to explain the configuration of Lotus Expeditor integrator and the use in different scenarios in detail.
Functional requirements (use cases)
The use case description has been shortened to show only the essential information to give a brief overview about the cases. In real life a solution architect must exactly describe all details to avoid further misunderstanding. Use case discovery and description is one of the most important parts of a solution architecture.
Use case UC001 - Article data transfer to store and print article labels
A message containing article data is being sent from Backbone Service Bus to the System. It must be recognized in terms of purpose, life time and destination where data should be delivered to. The message payload must be extracted and detached to a directory at local file system of the Back Office Server. After delivery of the file a batch file must be executed. All parameters are provided in message header of the incoming message. At the beginning and end of the processing in the System must be generated events containing information about the transaction status.
Use case UC002 - TLOG data capture from POS and transfer to head office
The 4690 POS Server is collecting transaction (TLOG) data and is storing this data as files at a FTP accessible directory. The System must watch this directory through a remote FTP connection. If an appropriate file occurs the System must capture the file and wrap file content into a message with a parameter for the message purpose. The message must be sent to the head office Backbone Service Bus. At the beginning and end of the processing in the System must be generated events containing information about the transaction status.
For further realization the 4690 POS will use a software called DIF (Data Interchange Facility) that allows sending TLOG data as JMS messages. In this case the System must watch a pre-defined queue and transfer the incoming message to the head office Backbone Service Bus in same manner.
Use case UC003 - Self checkout boxes (Pecuron) status data transfer to a central Pecuron management software
A Self Checkout Box will be sending status data as JMS message to the System. The System must watch the appropriate message queue and transfer the incoming message to the head office Backbone Service Bus. At the beginning and end of the processing in the System must be generated events containing information about the transaction status.
Use case UC004 - configuration update of Lotus Expeditor integrator
A message containing System configuration update data is being sent from Backbone Service Bus to the System. It must be recognized in terms of purpose and life time. The message payload must be extracted and detached to a configuration directory of the System. After delivery the System must take this new configuration, validate it and update itself. All parameters are provided in the message header of the incoming message. At the beginning and end of the processing in the System must be generated events containing information about the transaction status.
- The System must be able to work offline without lost of data.
- Overall runtime memory consumption must not extend 256 MByte.
- The System must be self-contained.
- Local maintenance effort of the System must be zero.
Finally JKHLE decided to implement Lotus Expeditor integrator at their retail stores as integration hub between the head office Backbone Service Bus and the applications / devices in stores. The architecture overview shows Lotus Expeditor integrator and the components being used. For an overview see also chapter 4.0 - Design Patterns
Description of the Lotus Expeditor integrator components
PubSub Messaging (micro broker with embedded bridge)
The publish / subscribe messaging service (micro broker) handles the reliable and de-coupled (off-line capable) communication over different messaging providers. JKHLE is using WebSphere Message Broker / WebSphere MQ at Head Office, the connectivity between Lotus Expeditor integrator and WMQ is being handled by the Messaging Client Plug-in (micro broker bridge with embedded WMQ Client). As the micro broker is the main topic of this document please refer to other chapters for a more detailed description.
The Dispatcher improves performance and availability of Lotus Expeditor integrator and allows for parallel processing of external messages received at the standard inbound queue from the Backbone Service Bus. The Dispatcher is not working in transactional context in order to gain the performance advantage. Incoming messages will be routed depending on their header parameters (type of resource and transport and message purpose) to another internal micro broker queue for further processing. This allows parallel processing of the different message types assuring that a lock in the processing of one type can not block the processing of messages of another type.
The Resource Adapters allow integrating the different remote resources and devices. The following Resource Adapters are currently available
- JMS (for micro broker that acts as JMS provider, or for other JMS provider through micro broker Bridge - Messaging Client Plugin)
- Flat Files
- HTTP (inbound over REST)
The Resource Monitor observes all types of resources to trigger events for process execution. Resource Monitor works in synchronous mode (for example, for Flat Files) or in asynchronous mode (for example, for message queues). It needs to be configured for each resource that should be monitored with an unique ID, called a trigger topic.
OSGi EventAdmin Service
All processes within Lotus Expeditor integrator are event driven by messages or other connected resources for near real-time process execution. The leveraged standard component is the OSGi EventAdmin Service that acts as an in-memory PubSub mechanism with pre-defined topics. A Resource Monitor acts as a publisher to such a trigger topic, based on a Resource Adapter configuration. The Process Execution (Application Control Service) acts as a subscriber to this topic, based on Flow definition. Once an event is fired to this topic the corresponding Flow is being executed. This event processing is described in more detail in the use case sections (UC001, UC002 and UC003).
The Process Execution (ACS - Application Control Service) is the central component of Lotus Expeditor integrator. Basic and custom Activities can be flexibly combined with Flow descriptions within XML files. An Activity is a component for a single task that implements an activity interface. A set of basic Activities is delivered with the product, e.g. for read and write different Resources through Resource Adapters. Based on the Activity interface one could create custom Activities to extend the capabilities of Lotus Expeditor integrator. A Flow is defined by a XML file containing a sequence of Activitiy definitions with matching parameters. This principle allows a flexible use case execution based on configuration only. For further details please check also section 6.3 Setup and configuration of a template for further rollout
. The principle of writing custom Activities is described in section 6.6 Extensibility with custom development
Custom Event Service
Transactional events are published by the ACS (Application Control Service) to the OSGi EventAdmin Service. The Custom Event Service acts as a subscriber to the OSGi EventAdmin Service for these transactional events and sends them through the JMS Resource Adapter / PubSub Messaging to a central WMQ message queue for further analysis. A more detailed description of integration into an end-to-end transaction monitoring may be found in section 6.5 Transaction Monitoring capabilities
OSGI Log Service / Custom Log Service
All components are using the OSGi Log Service as their logging mechanism. This acts as a publisher to the OSGi EventAdmin Service for all log events, depending on pre-configured log level. The Custom Log Service acts as a subscriber to the OSGi EventAdmin Service for these transactional events and sends them through a Monitoring Adapter to the Head Office for further analysis.
Technical and Business Monitoring Adapters
The Monitoring Adapters could also be used for event format transformation from CBE format to the needed monitoring data structure at the head office. Out-of-the-box available are CBEs through messaging. There is also a TEC Monitoring Adapter that communicates with Tivoli Event Console or Netcool OMNIbus. JKHLE is using TEC and so far they're leveraging this TEC Monitoring Adapter.
Custom Config Service
The Custom Config Service wraps the OSGI Config Service to extend the standard OSGi configurations mechanisms. This service implements for example an initial boot load that sends a "Hello" message containing his unique ID to a central management system to get the final production configuration. This will be described in section 6.4 Rollout and update mechanisms
JKHLE decided to use the existing Back Office Server for the deployment of Lotus Expeditor integrator. It consumes only 256 MByte runtime memory at maximum and so far there's no further hardware necessary.
Rollout and deployment will be handled by an existing software distribution system in combination with the Backbone Service Bus that is sending binaries and configuration information to Lotus Expeditor integrator instances in same manner as transactional data. Since Lotus Expeditor integrator is self-contained it is even able to handle these incoming configuration update messages by itself. Details of rollout and deployment are in section 6.4 Rollout and update mechanisms