Introduction

History reconsolidating means the recalculation and recollection to upper levels, but often the terms reconsolidating, recalculation and recollection are used as synonyms. The RTDB-Scheduler service conducts the history reconsolidation processing by sending commands to the transformator and calculation processes. The reconsolidating of history is needed in systems where history values are being corrected, or the history values arrive late from the source system. Especially the energy management systems most often require the possibility to perform history reconsolidating.

Think what it means if a couple of yesterday’s one-hour average periods for some variables are updated. First, the calculation program that uses these variables as source values must be re-run. After that, the upper-level history values must be recollected. For example, the one-day average period must be recollected from the hour period values. After that, the calculation program that uses the one-day value as a source data must be re-run. After that, the upper-level history periods must be recollected (for example the on-going month period). The RTDB-Scheduler program, by using its configuration settings, does the activation of all these programs.

Activation of History Reconsolidation

Local RTDB applications can activate the history reconsolidation by calling the method M_RecalculateHistory of class C_DataPointConstructor. Applications can also activate the history recalculation with the ODBC/SQL stored procedure RecalculateHistory, for example:

{ call RecalculateHistory('OneMinAve', { ts '2000-08-01 00:10:00' }, { ts '2000-08-01 00:30:00' }, '', 1) }

Application programs call history recalculation after they have updated some values in the history table. They give the name of the updated history table and the time range of the updated data in the recollection request. The starting time of the range is inclusive. For event-based history tables (for example CurrentHistory), the ending time is inclusive, but for period-based tables (for example the average history types) the ending time is exclusive. So, for example, if you modified two periods of the ten-minute average starting with the period that has the beginning time 00:10, the begin time is given as 00:10 and the ending time as 00:30. While the processing of the history recalculation is executed, the given time range is passed to the transformator service and to the calculation programs. Note that the time range is always the original time range of the source table. When the calculation program or the transformator receives this time range, and it is calculating or retransforming upper-level histories, it must adjust the time range to correspond to the given time level (the begin time must be rounded to a smaller time and the end time must be rounded to a greater time to match the period times).

The history reconsolidation configuration uses the DoTaskList mechanism of the RTDB service framework).

What actually happens in the C_DataPointConstructor::M_RecalculateHistory function or in the RecalculateHistory stored procedure is that the application (or the ODBC server) sends a DoTaskList command to the RTDB Scheduler service via CommandQueue.

For example:

DoTaskList=R-OneMinAve;Def=HsUpdStart_UTC=2002-05-20 10:00:00;Def=HsUpdEnd_UTC=2002-05-20 11:00:00

The format of the DoTaskList command is:

DoTaskList=TaskListName{;Def=macrovariable=macrovalue}

*TaskListName* is the name of the task list to execute. The task list names are just entries in RTDB SimpleConfig, with section name RTDB-Scheduler. The task list names for the history recollection tasks are R- + the name of the historical database table. The DoTaskList command may contain macro value definitions by using the keyword Def. The history recollection request messages always define values for the macros HsUpdStart_UTC and HsUpdEnd_UTC, which define the updated time range.

Configuration of History Reconsolidation

Before RTDB-Scheduler is able to correctly run all tasks that relate to the history recalculation, the tasks must be configured to the RTDB database tables SimpleConfig and TimedTasks. First of all, for each history level that can be updated, an entry must be placed to SimpleConfig.

For example, if we have history tables CurrentHistory, OneMinAve (one minute averages), and OneHourAve (one hour averages), the configuration settings in SimpleConfig could be the following (for clarity, the INSERT INTO part is not repeated):

INSERT INTO SimpleConfig(SectionName, KeyName, StrValue) 
VALUES('RTDB-Scheduler', 'R-CurrentHistory',   'H=641,642,643,644,645,646');
VALUES('RTDB-Scheduler', 'R-OneMinAve',                'H=643,644,645,646');
VALUES('RTDB-Scheduler', 'R-OneHourAve',                           'H=646');

The SectionName for the setting is ‘RTDB-Scheduler’ and the key name is R- + the name of the historical database table. The value of the setting has the form ‘H=*comma-separated list of task numbers*’. The task numbers refer to the tasks to be performed, if a time range in the given history table is changed. The tasks are defined in the TimedTasks table, and they typically contain commands that the scheduler must perform. The commands typically contain SendCmdW commands, which tell the Scheduler to send a command to another process, such as the calculation program (usually process number 101), Transformator (process number 2), and CVMC-server (process number 7).

In our example, if the CurrentHistory table is modified, we need to:

• Retransform OneMinAve (i.e. to recollect from CurrentHistory to OneMinAve)
• Recalculate one-minute-level calculations
• Retransform OneHourAve
• Recalculate one-hour-level calculations

In our example, if the OneMinAve table is modified, we need to:

• Recalculate one-minute-level calculations
• Retransform OneHourAve
• Recalculate one-hour-level calculations

If OneHourAve is modified, we only need to recalculate the hour-level calculations.

The following tasks must now be inserted to TimedTasks (for clarity, the INSERT INTO part is not repeated):

--Transform 1 min level
INSERT INTO TimedTasks(TaskNum, TaskSpec) VALUES
(641, 'SendCmdW=7;Timeout=200;Cmd=''ReTransform;Target=OneMinAve;
  TIME1=$(HSUPDSTART_UTC);TIME2=$(HSUPDEND_UTC)''');
(642, 'SendCmdW=2;Timeout=200;Cmd=''ReTransform;Target=OneMinAve;
  TIME1=$(HSUPDSTART_UTC); TIME2=$(HSUPDEND_UTC)''');
--Calculate 1 min level
(643, 'SendCmdW=101;Timeout=20;Cmd=''ReCalc;Id=2;                
  TIME1=$(HSUPDSTART_UTC);TIME2=$(HSUPDEND_UTC)''');
--Transform 1 hour level
(644, 'SendCmdW=7;Timeout=200;Cmd=''ReTransform;Target=OneHourAve;
  TIME1=$(HSUPDSTART_UTC);TIME2=$(HSUPDEND_UTC)''');
(645, 'SendCmdW=2;Timeout=200;Cmd=''ReTransform;Target=OneHourAve;
  TIME1=$(HSUPDSTART_UTC);TIME2=$(HSUPDEND_UTC)''');
--Calculate 1 hour level
(646, 'SendCmdW=101;Timeout=20;Cmd=''ReCalc;Id=5;                
  TIME1=$(HSUPDSTART_UTC);TIME2=$(HSUPDEND_UTC)''');

The format of the SendCmdW command is:

SendCmdW=ReceiverId;Timeout=timeout-in-seconds;Cmd=CommandToSend

SendCmdW asks Scheduler to send a command via CommandQueue to the receiver ReceiverId, and to wait until the receiver has processed the command and sent an acknowledgement back to Scheduler. The sending of the acknowledgement message is done automatically by the RTDB service framework, when the execution returns from M_CalculateModule. If the command is executed by M_DoCalcCommand, the implementation should call M_SendAcknowledgement. If Scheduler does not receive an acknowledgement within the timeout-in-seconds period, Scheduler prints an error message to RTDB MessageLog and continues processing of the recalculation tasks.

(Version info: the documentation was defective before 2008-02-11: it said that acknowledgement was automatically sent after return from M_DoCalcCommand)

Note that the CommandToSend string must be enclosed in quotation marks (either single or double), because the string itself contains semicolon characters. In the above example, we used single quotation characters that must be written doubled in SQL string literals.

In addition to SendCmdW commands, the TaskSpec may contain any commands recognized by RTDB-Scheduler (such as QuickRun, or DoTaskList, or Comment).

The command string contains macro replacement parts. Scheduler replaces $(HSUPDSTART_UTC) and $(HSUPDEND_UTC) with the starting time and ending time of the modified time range, respectively. The time is in UTC time, and in ISO format (YYYY-MM-DD HH:MM:SS.fffffff).

The command verb to be sent to the RTDB Transformator service is . The Transformator service understands this command, and it has the following format:

This command asks the Transformator service to recollect the given range of periods of the target-table from its source table.

First, the identical ReTransform message needs to be sent to RTDB current value service (RTDB-CVMCServer, process number 7), so that it knows that the current period of the history table may need to be recollected.

The command verb to be sent to the RTDB calculation program is . The RTDB calculation program understands this command, and it has the format:

The module-number defines the time level that the calculation program needs to recalculate. Note that the calculation programs are project-specific applications, and the module numbers are also project-specific.

The Scheduler flushes (using a certain criteria) values in CurrentValues to CurrentHistory before starting recollection tasks that are listed (comma-separated) in the SimpleConfig parameter HR_CVTOHV_TASKS. The default value in RTDB 3.3 is "R-CurrentHistory", and an empty string in later RTDB versions (this behavior was implemented to version 2007-01-10).

Merging of Reconsolidation Requests

When Scheduler receives a reconsolidation request command, it puts the request to its internal queue of reconsolidation requests. If the request is for the same history table (or tables with equal recollection task lists), and has an overlapping or adjoining time range, and has equal additional parameters, the Scheduler combines the requests so that only a single reconsolidation run is needed for them. The Scheduler does not start execution of the received request before a small time interval (default 10 seconds), which can be adjusted with the SimpleConfig setting RTDB-Scheduler/HRStartDelay. The setting takes effect for the next reconsolidation request received.

RTDB-Scheduler executes the reconsolidation tasks in a separate thread, so it is able to concurrently continue its other tasks.

For diagnostics purposes, the Scheduler can maintain an RTDB variable for the length of the internal reconsolidation request queue. If the variable is needed, create the variable and enter its name to SimpleConfig with the following INSERT command, and restart the RTDB-Scheduler service:

INSERT INTO SimpleConfig(SectionName, KeyName, StrValue) VALUES('RTDB-Scheduler', 'HRQueueLengthVar', 'RTDB_HRQUEUELENGTH')

(Let’s agree that the name of the RTDB variable is always RTDB_HRQUEUELENGTH).

The following table describes the configuration settings that affect how the merging of recalculation requests is done. The history recalculation requests may also be created from calculation activation definitions, and the calculation activations contain also other information in addition to the history table and time range (such as the updated value and variable). By default, the Scheduler does not merge requests that contain different variables or values (except if the further requests for the same variable completely cover the older requests).

Diagnostics

RTDB-Scheduler prints diagnostic messages about the history reconsolidation to RTDB MessageLog. Use the RTDB_MsgType command to analyze and monitor the messages.