Debugging real-time distributed embedded apps

Real-time system designers and embedded software developers are very familiar with the tools and techniques for designing, developing and debugging standalone or loosely coupled embedded systems. The Unified Modeling Language (UML) may be used at the design stage, an IDE during development, and debuggers and logic analyzers (among other tools) at the integration and debug phases.

However, as connectivity between embedded systems becomes the norm, what used to be a few nodes connected together with clear functional separation between the applications on each node, is now often tens or hundreds of nodes with logical applications spread across them. Such distributed systems are becoming increasingly heterogeneous in terms of both operating systems and executing processors with tight connectivity between real-time and enterprise systems becoming the norm.

This article identifies the issues of real-time distributed system development and discusses how development platforms and tools have to evolve to address this challenging new environment.

The idea of a 'platform' for development has long pervaded the real-time embedded design space as a means to define the application development environment separately from the underlying (and often complex) real-time hardware, protocol stacks and device drivers.

As the OS evolved to provide the fundamental building blocks of standalone system-development platforms, real-time middleware has also evolved to address the distributed-systems development challenges of real-time network performance, scalability, and heterogeneous processor and OS support.

As what happened in the evolution of the standard RTOS, new tools are becoming available to support development, debug and maintenance of the target environment—in this case, real-time applications in large distributed systems.

New platform
From the application developer's perspective, there are three basic capabilities an application development platform must provide when a logical application spans multiple networked computers:

1. Communication between threads of execution
2. Synchronization of events
3. Controlled latency and efficient use of network resources

Communication and synchronization are fairly obvious distributed platform service requirements and are analogous to the services provided by an OS. For distributed applications, however, they have to run transparently across a network infrastructure of heterogeneous operating systems and processors with all that implies in terms of byte ordering and data representation formats.

Ideally, it should use a mechanism that does not require the developer to explicitly understand about the location of the intended receiver of a message or synchronizing thread so that the network can be treated as a single target system from an application development perspective.

Typically, a user employs a commercial or homegrown middleware to provide these key capabilities. There are several middleware solutions supporting this approach such as Java Message Service (JMS) and Data Distributions Service (DDS) from the Object Management Group (OMG).

Solutions such as DDS (Figure 1) explicitly address the controlled latency and efficient use of target network resources, which is a critical issue in real-time applications. DDS provides messaging and synchronization similar to JMS and also incorporates QoS.

QoS brings to the application level the means to explicitly define the level of service (priority, performance and reliability) required between an originator of a message or synchronization request, and the recipient.

DDS treats the target network like a state machine, recognizing that real-time systems are data driven and that the arrival, movement, transition and consumption of data fundamentally defines the operation of a real-time system.

Some data is critical and must be obtained and processed within controlled/fixed latencies, most especially across the network. Some data need to be persisted for defined periods of time for use in computation; other data may need to be reliably delivered but are less time critical. QoS facilitates these requirements and more.

The greatest advantage of using middleware isn't often appreciated until late in the application development process: defining interfaces in a rich middleware format makes it easier to integrate, debug and maintain a system. A good middleware allows you to completely specify the data interaction through QoS, which forms a "contract" for the application.

Figure 1: DDS provides a framework for providing controlled latency and efficient use of target network resources.

DDS, for example, allows a data source to specify the data type and if the data is sent with a "send once" or "retry until" semantic. DDS also determines the size of a history to store late arriving receivers, the priority of a source compared with others, the minimum rate at which data will be sent and more possibilities.

By setting these explicitly, many soft issues that appear during integration can be addressed quickly by matching promised behavior to that requested. DDS middleware even provides warnings at runtime when contracts aren't met.