Communication¶

Three kinds of communication are implemented in the B-Human framework: inter-thread communication, debug communication, and team communication.

Inter-thread Communication¶

The representations sent back and forth between the threads (so-called shared representations) are automatically calculated by the ModuleGraphCreator based on the representations required by modules loaded in the respective thread but not provided by modules in the same thread. The directions in which they are sent are also automatically determined by the ModuleGraphCreator.

All inter-thread communication is triple-buffered. Thus, threads never block each other, because they never access the same memory blocks at the same time. In addition, a receiving thread always gets the most current version of a packet sent by another thread.

Message Queues¶

The debug communication and logging are both based on the same technology: message queues. The class MessageQueue allows storing and transmitting a sequence of messages. Each message has a type (defined in Src/Libs/Streaming/MessageIDs.h) and a content. Each queue has a maximum capacity, which is defined in advance. On the robot, the amount of memory required is pre-allocated to avoid allocations during runtime. On the PC, the memory is allocated on demand, because several sets of robot threads can be instantiated at the same time, and the maximum capacity of the queues is rarely needed.

Since almost all data types are streamable, it is easy to store them in message queues. The class MessageQueue offers three methods to create a stream to write a message in different formats: messages that are stored in the stream returned by bin(MessageID) are written in binary format. The stream returned by text(MessageID) formats data as text and the one returned by textRaw(MessageID) as raw text. A message type must be passed to all of these three methods:

MessageQueue queue;
queue.reserve(1000); // can be omitted on PC
queue.text(idText) << "Hello world!";

Note that only a single message can be written at the same time, i.e. before calling one of these three methods again, the previous message must have been written completely.

To declare a new message type, an id for the message must be added to the enumeration type MessageID in Src/Libs/Streaming/MessageIDs.h. The enumeration type has two sections: the first for representations that should be recorded in log files, and the second for infrastructure messages.

Messages are read from a queue through an iterator. Its operator * returns an object of the type MessageQueue::Message. It allows to determine the message's id() (i.e. its type) and its size() and also provides two methods that return a stream the contents of the message can be read from: bin() to interpret the contents in binary format and text() to interpret them as text. The iterator makes message queues compatible with C++'s for-each loop:

for(MessageQueue::Message message : queue)
  if(message.id() == idText)
  {
    std::string text;
    message.text() >> text;
  }

Debug Communication¶

For debugging purposes, there is a communication infrastructure between the threads and the PC. This is accomplished by message queues. Each thread has two of them: theDebugSender and theDebugReceiver. The macro OUTPUT(<id>, <format>, <sequence>) defined in Src/Libs/Streaming/Output.h simplifies writing data to the outgoing debug message queue. id is a valid message id, format is text, bin, or textRaw, and sequence is a streamable expression, i.e. an expression that contains streamable objects, which – if more than one – are separated by the streaming operator <<.

OUTPUT(idText, text, "Could not load file " << filename << " from " << path);
OUTPUT(idCameraImage, bin, CameraImage());

For receiving debugging information from the PC, each thread also has a message handler, i.e. a method handleMessage(MessageQueue::Message) that is called for each message the thread receives. The method returns whether it processed the message or not.

The thread Debug manages the communication of the robot control program with the tools on the PC. For each of the other threads, it has a sender and a receiver for their debug message queues. Messages that arrive via Wi-Fi or Ethernet from the PC are stored in debugReceiver. The method Debug::handleMessage(MessageQueue::Message) distributes all messages in debugReceiver to the other threads. The messages received from the other threads are stored in debugSender. When a Wi-Fi or Ethernet connection is established, they are sent to the PC via TCP/IP.

The debug communication and the thread Debug are not available on the actual NAO when the software deployed was compiled in the configuration Release.

Team Communication¶

The purpose of the team communication is to send messages to the other robots in the team. These messages are always broadcasted via UDP, so all teammates can receive them. Sending and receiving team messages is done in the thread Cognition using the representations ReceivedTeamMessages for handling received messages and BHumanMessageOutputGenerator for generating messages to send. These representations are both generated and filled with their functionality by the module TeamMessageHandler.

The team messages use a custom format (defined in Src/Tools/Communication/BHumanMessage.(cpp|.h)) that is much shorter than the currently allowed maximum of 128 bytes. It contains a header with the player number and data for clock synchronization (see below), followed by a variable length container of compressed representations.

Representations that should be communicated must inherit from BHumanMessageParticle, either directly or via the template BHumanCompressedMessageParticle<Representation>. The former requires the programmer to implement operator>>(BHumanMessage&) and operator<<(const BHumanMessage&), while the latter automatically implements them to stream the representation into/from a team message using the name the<Representation>, which is the right thing in most cases.

Clock Synchronization¶

Throughout the system, timestamps are used to describe when something happened or will happen (only in some cases, the difference to the current time is additionally provided for convenience). When these timestamps are communicated to other players in a team message, there must be some estimate of the offset of the robots' clocks so that the receiver can interpret the timestamp relative to its own clock.

We employ a Reference Broadcast Synchronization¹ based scheme in which GameController packets serve as reference broadcasts. The GameController application sends a packet roughly twice a second, which includes an incrementing 8 bit counter. Therefore, within approximately a two-minute interval, the packet number uniquely identifies a packet and, assuming a single wireless access point and ignoring differences in propagation time to each robot, defines an event that all robots can measure at the same time. The only significant delay that can be different for each robot is the time it takes until the timestamp of the local clock for the receive event is taken. We minimize this by fetching the timestamp from the kernel using the ioctl(SIOCGSTAMPNS) call.

To actually calculate the clock offsets, each sent team message contains the packet number of the most recent GameController packet and the local timestamp when this was received. Upon reception of a team message, the receiver looks up the packet number in a buffer of recent GameController packets to find its own local timestamp that corresponds to that packet. The difference between these is the estimate of the clock offset. If the GameController packet number is not known to the receiver, e.g., because it was lost, the previous offset is kept. In the worst case, no offset was known before and the message has to be dropped because the timestamps are invalid. A known offset is also invalidated if the timestamp of the sender is outside estimated bounds, as this indicates that the robot was restarted in the meantime. However, a new offset can be estimated immediately from the received message if the referenced GameController packet number is known. We have measured the resulting system to estimate clock offsets of pairs of robots with millisecond precision, which is what we generally use to represent timestamps.

Message Format¶

The encoding of the representations is defined in the file Config/Scenarios/Default/teamMessage.def. The record type TeamMessage in this file describes the overall structure (which must correspond to the representations listed in the macro FOREACH_TEAM_MESSAGE_REPRESENTATION in Src/Modules/Communication/TeamMessageHandler.cpp), while the other records correspond to representations or other streamables included in representations. Only the members that are included in this file are actually sent, the others keep their default values in a ReceivedTeamMessage. Specifically,

Booleans are encoded as one bit.
Enums take as many bits as needed to encode the number of alternatives (known by the TypeRegistry).
Integers can be limited to a value range, which automatically determines the number of bits.
Timestamps are automatically translated via the offsets established by RBS, and they can be encoded relative to the current time and quantized.
Floats can be quantized by specifying their value range and a number of bits (which determines the resolution).
Arrays can have either fixed length (so that it does not need to be communicated) or have their length bounds specified.
Matrices can be specified to be symmetric (e.g. covariance matrices) so that the off-diagonal entries are encoded only once.

However, this is still not an optimal encoding as each individual value must occupy an integer number of bits. For example, two enums with 5 alternatives take 3 bits each, instead of the theoretical 4.64 bits needed to encode 25 different values.

The structural information from the TypeRegistry combined with teamMessage.def not only allows to automatically encode/decode team messages on the robots, but also to generate Java code for B-Human's Team Communication Monitor plugin. This is done using the debug request module:TeamMessageHandler:generateTCMPluginClass, which creates the file Config/BHumanMessage.java. This file has to be copied to the directory resources/plugins/05/B-Human/src/bhuman/message in the Team Communication Monitor before compiling it.

Message Budget Allocation¶

According to the rules, the team has a limited budget of messages per game (1200 in 2023). The BHumanMessageOutputGenerator contains a function provided by the TeamMessageHandler that decides whether a team message can and shall be sent out in the current frame or not. The current approach is described in the B-Human Team Report and Code Release 2022 Section 3.2.1.

Jeremy Elson, Lewis Girod, and Deborah Estrin. Fine-grained network time synchronization using reference broadcasts. ACM SIGOPS Operating Systems Review 36(SI):147-163, 2002. ↩

Last update: October 12, 2023