tools

Tools for handling logged network data.

The tools module provides methods and objects designed to simplify loading and handling logged network data. The following methods are available:

Functions

dump_to_array(source, keys, min_time=None, max_time=None)[source]

Load log file data into a numpy array.

The dump_to_array() function parses network data logs into a numpy.array. To parse data into a numpy.array, the following conditions must be met:

  • All messages loaded must be the same MCL Message type.
  • All logged messages must contain the specified keys.
  • The contents of the message keys must be convertible to a float.
Parameters:
  • source (str) – Path to network data log(s) to convert into a list. source can point to a single file or a directory containing multiple log files. If the log files are split, provide the prefix to the log files.
  • keys (list) – List of message attributes to load into numpy array. The items in this list specify what is copied into the numpy columns.
  • min_time (float) – Minimum time to extract from dataset.
  • max_time (float) – Maximum time to extract from dataset.
Returns:

A numpy.array containing the requested keys (columns) from each message (rows) in the network log.
Return type:

numpy.array
Raises:
  • IOError – If the input source does not exist.
  • TypeError – If the input keys is not a string or list of strings. A TypeError will also be raised if all loaded message packets are not of the same type.
  • KeyError – If the input keys do not exist in the loaded objects.
dump_to_csv(source, csv_file, keys, min_time=None, max_time=None)[source]

Write log file data to a CSV file.

Parameters:
  • source (str) – Path to network data log(s) to convert into a list. source can point to a single file or a directory containing multiple log files. If the log files are split, provide the prefix to the log files.
  • csv_file (str) – Path to write CSV file.
  • keys (list) – List of message attributes to load into columns of the CSV file.
  • min_time (float) – Minimum time to extract from dataset.
  • max_time (float) – Maximum time to extract from dataset.
Raises:
  • IOError – If the input source does not exist.
  • TypeError – If the input keys is not a string or list of strings. A TypeError will also be raised if all loaded message packets are not of the same type.
  • KeyError – If the input keys do not exist in the loaded objects.
dump_to_list(source, min_time=None, max_time=None, message=False, metadata=True)[source]

Load log file data into a list.

The dump_to_list() function parses a log file or directory of log files into a list. Each element in the list is returned as it is recorded in the log file(s) (see metadata):

{'elapsed_time': float(),
 'topic': str(),
 'message': dict or Message()}
Parameters:
  • source (str) – Path to network data log(s) to convert into a list. source can point to a single file or a directory containing multiple log files. If the log files are split, provide the prefix to the log files.
  • min_time (float) – Minimum time to extract from dataset.
  • max_time (float) – Maximum time to extract from dataset.
  • message (bool) – If set to True messages will automatically be decoded into the MCL Message type stored in the log file(s). If set to False (default), message data is returned as a dictionary. Note: to read data as MCL messages, the messages must be loaded into the namespace and recorded in the log file header.
  • metadata (bool) – If set to True (default), each element in the list will store a dictionary containing the elapsed time, topic and payload. If set to False only the payload will be stored in each element of the list.
Returns:

A list of chronologically ordered network messages.
Return type:

list