Tani Historian File Description

Directory structure

The historian has a configurable data directory. All historian variables that use file storage write the data in this directory. It can be changed in <install directory>/Historian/bin/Historian.Settings, key "HistorianFilePath".

Below this directory, each historian variable has its own sub-directory (name is the variable name). Each directory contains these files:

• Var.ini: is a copy of the variable configuration. This allows reconstructing the variable settings and file formats, even if the variable itself has been deleted from the historian.
• Structs.ini: exists only for variables that have a structure type. It contains all required structure definitions.
• data_*.bin: are the data files.

Var.ini contents

The Var.ini file is an ini-File with one section. The section name is "[Var.<variable name>]". In this section, the following keys can exist:

Active

Boolean; yes enables the variable in the runtime system.

AggregateAvg

Boolean; yes enables calculation of Average

AggregateCalcOffset

Number; for DataType=struct, the offset of the element that is used for aggregate calculation

AggregateCalcDataType

Enum; for DataType=struct, the data type of the element that is used for aggregate calculation; see Data Types

AggregateCount

Boolean; yes enables calculation of Count

AggregateMax

Boolean; yes enables calculation of Maximum

AggregateMin

Boolean; yes enables calculation of Minimum

AggregateStddev

Boolean; yes enables calculation of Standard Derivation

ArrayLength

Number; number of array elements to store

DataType

Enum; data type of value to store; see Data Types; for structures, a : and the structure name is appended

ElementLength

Number; for DataType=string, the maximum string length; for DataType=struct, the data length of the structure

FileResolution

Enum; The resolution for creating new data files; see Resolution

FileSave

Boolean; yes to enable file save

MaxFileCount

Number; the maximum number of live data value files to create

MaxMem

Number; number of bytes to use for live data storage of memory variable

MinimumFreeDiskSpace

Number; mimimal required free disk space; 0 uses globally defined value; values < 100 are percent; value >= 100 are bytes

OPCActive

Boolean; yes to enable automatic data acquisation of OPC data

OPCConnName

String; OPC Connection name to use

OPCGroupName

String; OPC Connection group name to use

OPCVarName

String; OPC Variable name to use

ReadResolution

Enum; The resolution of OPC data acquisation; see Resolution

Structs.ini contents

The Structs.ini is an ini-file with multiple sections. Each section describes one data structure used in the data files. The section name is equal to the structure name, some characters with special meaning are replaced by <xx> where xx is the hex-code of the character. The sequence of section entries describes the sequence of structure elements. Valid keys are:

Header

<Structure Name>,<Flags>
<Structure Name> - the structure name (repeated, for historical reasons)
<Flags> - hexadecimal encoded flag bits; internal use only; not required for decoding

Element

<Element Name>,<Data Type>,<Data Type Flags>,<Element Flags>,<Array Size>,<Original Type>
<Element Name> - the element name; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.
<Data Type> - data type of the structure element; see Data Types; for structures and enumerations, a : and the structure/enumeration name is appended; for strings, a : and the maximum string length is appended
<Data Type Flags> - hexadecimal encoded flag bits; internal use only; not required for decoding
<Element Flags> - hexadecimal encoded flag bits; valid values from the following list are OR'ed tokether:   0x0010 - PACKED - if set, no bit padding exists before this element
  0x0020 - PADDING - if set, this element is a padding element and can be ignored or hidden by user application
  0x0040 - VARIABLE_ARRAY_LENGTH - if set, the array length is given by another structure element instead of the ArrayLength field
  0x0080 - VARIABLE_STRING_LENGTH - if set, the string length is dynamic, instead of a fixed string length with padding
  0x0100 - OPTIONAL - if set, the element is optional; its presence is determined by another structure element
  all other bits are reserved for internal use only
<Array Size> - the array length in elememnts <Original Type> - a protocol specific "original type" for imported structs; may be safely ignored

ElementDynamic

optional; <Length Element>,<Switch Element>,<Switch Value>; this line is present only if at least one of these settings are required
<Length Element> - used if VARIABLE_ARRAY_LENGTH is set; the name of a structure element that gives the array length of this element; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.
<Switch Element> - used if OPTIONAL is set; the name of a structure element that determines if this element is present or not; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.
<Switch Value> - used if OPTIONAL is set; if the <Switch Element> has this value, then this element is present, otherwise it is absent

ElementComment

optional; the element comment; some characters with special meaning are replaced by <xx> where xx is the hex-code of the character.

Data file contents

The data files contain the stored values in a binary format.

File name rules: data_<resolution>_<timestamp>.bin. <resolution> is a number (0 to 7) that specifies the resolution of the data contained in this file; see Resolution. <timestamp> is the start time of this file. The timestamp format is "yyyymmddhhmm" (4 digits for year, 2 digits each for month, day, hour and minute).

All values are stored in the little-endian format (least significant byte first). Each file contains a number of records. The length of each record is determined by the settings in Var.ini.

Each entry has a 16-byte header with the following fields:

tv_sec

unsigned int64; Timestamp, given as seconds since 1970-01-01 00:00 UTC

tv_nsec

unsigned int32; Timestamp, nanoseconds in the second given by tv_sec

quality

unsigned int32; Quality for the stored value

After the header follow the user data. After the user data (without any padding) follows the next header.

Live data format (resolution 0)

These are the unmodified values as given by OPC or written by an application. Data length is given by the DataType entry in Var.ini, multiplied by the ArrayLength entry (1 if not present).

DataType=string: Each string has a 4-byte header (giving the really used number of bytes) followed by the data bytes (always ElementLength bytes).

DataType=struct: Each structure has a length as given in ElementLength

Structure encoding rules

• "bit" elements occupy 1 bit.
• all numeric elements occupy as many bytes as needed (documented below).
• "string" elements have a 4-byte length element followed by the character data (usually UTF-8 encoded). If a maximum string length is given, the character data are padded to this length with "\x00" bytes to ensure a fixed structure layout. Othweise no implicit padding exists.
• "time" elements occupy 12 bytes: 8 byte UNIX-Timestamp (seconds since 1970-01-01 00:00:00 UTC) and 4 byte Nanoseconds
• "struct" elements occupy as many bytes as required for their encoding
• all elements except "bit" start at a byte boundary (align 1). Padding bytes (if required) must be specified explicitly via PADDING elements
• "bit" elements start at a byte boundary if:
  • they have an Array Size other than 1 or VARIABLE_ARRAY_LENGTH is set
  • and PACKED is not set
• "bit" elements are packed in one byte if:
  • the Array Size is 1 and VARIABLE_ARRAY_LENGTH is not set
  • or PACKED is set
• If required, implicit padding bits are inserted after "bit" elements.

Aggregate data format (resolutions 1 - 7)

These are aggregated values calculated during data recording. Data length is determined by the selected aggregate functions. The following fields may be present (in the given sequence):

To calculate the average value in the interval: Avg = Sum / (tv_sec + tv_nsec / 1000000000)

To calculate the standard derivation in the interval: StdDev = sqrt(SumSq / (tv_sec + tv_nsec / 1000000000) - Avg * Avg)

Data types

Resolution values

The following resolutions are available for data acquisation:

Fastest

Save values as fast as the source generates them

Second

Save at most one value per second

Minute

Save at most one value per minute

Hour

Save at most one value per hour

Day

Save at most one value per day

The following resolutions are available for file rotation:

Hour

Start a new file at the beginning of each hour (UTC time): 2000-01-01 00:00, 2000-01-01 01:00, 2000-01-01 02:00, ...

Day

Start a new file at midnight (UTC time) of each day: 2000-01-01 00:00, 2000-01-02 00:00, 2000-01-03 00:00

Week

Start a new file at midnight (UTC time) of each monday: 2000-01-03 00:00, 2000-01-10 00:00, 2000-01-17 00:00, 00

Month

Start a new file at midnight (UTC time) of the first day of each month: 2000-01-01 00:00, 2000-02-01 00:00, 2000-03-01 00:00, ...

Year

Start a new file at midnight (UTC time) of january 1st: 2000-01-01 00:00, 2001-01-01 00:00, 2002-01-01 00:00, ...