Introduction

CpmPlus Connector, from henceforth referred to as Connector in this document, is an ABB Ability™ Edge module designed with the aim of providing Ability Cloud connectivity to ABB Ability™ History and the potential pre-existing equipment behind it.
Connector is designed to run only in conjunction with ABB Ability™ History , and is built directly on top of NetSync, another component created by the team.

In the classic structure of Ability Platform, connected devices are defined by registering their model as a type definition on the Ability Platform Information Model, and instantiated by an Edge Module by sending device.created messages to the cloud, which will result in an object.model getting created respective to the connected device. Afterwards, the module will be allowed to collect variable(sensor) data from these connected devices and send it to the cloud to be stored on appropriate database component. In this scenario, the information model is the one and only source of truth and a database component on it is the one and only storage; However, certain legacy systems existed long before the platform was born, and those were not designed for this architecture. To explain further, there are some setups in which the connected devices are attached to the ABB Ability™ History database, with ABB Ability™ History acting as both source of truth and storage for them. In order to make such setups compatible with Ability Platform, Connector has been created, which sits between the cpmPlus, and an Ability Platform deployment.

To summarize, Connector is constantly monitoring the states of devices, and attempting to propagate all the events which occur, to the Ability Platform in a compatible format. As an example, if a device gets added to the structure behind ABB Ability™ History, Connector is notified and translates this event to a device.created message, sending it on the correct MQTT topic to be consumed by the appropriate component. The same goes for model updates, which will be translated as device.updated message, and produced telemetry(variable) data which will also be translated to be stored on the cloud. This allows the compatibility of the legacy system with the Ability Platform, while maintaining a synchronization between Information Model and ABB Ability™ History database as sources of truth, and storing the variable data on the cloud in addition to the ABB Ability™ History database if required. It should be noted that in this scenario, the actual source of truth is still the ABB Ability™ History database, and the Information Model is simply synchronized on a best effort basis. This means that the connector is not awaiting confirmation for the results of the platform messages it sends, and will not act upon issues and/or failures.

Last but not least, Connector also allows the writeback of values to the ABB Ability™ History database. This implies that it is possible to manually modify values for the connected devices behind the ABB Ability™ History database. The use case for this is the capability to use components such as a Calc-Engine, developed as an Edge Module, to apply data processing on the variable data produced by these connected devices, and storing them back on the database to change actual device behavior.

Internal Structure

As mentioned before, Connector is built on top of NetSync. Netsync is a component built on top of ABB Ability™ History, which is basically responsible for listening to all kind of events happening on the connected devices behind the ABB Ability™ History Plus database, and exposing a subscription mechanism for other components (such as the Connector) to act on these events. NetSync component is configured with Equipment publication, which defines how the Equipment model is exposed and data updated to other systems.

Messaging events

At this moment, the Connector listens for 3 types of events on NetSync (and ABB Ability™ History by proxy):

1. ABB Ability™ History PropertyUpdate: This is equivalent of incoming telemetry from ABB Ability™ History Database side. These are converted into platform supported variable data(telemetry) and sent out on the messages_out topic. Based on your edge setup, you may be able to listen to this topic and act on this variable data. In a standard edge deployment, such data will be apprehended by Ability Edge Proxy, and pushed to the Azure Cloud, which ultimately will be stored in a timeseries database. In a custom Edge setup, you may use the data to store it on a custom database, or apply processing on it and write it back to the CpmPlus database through the Connector.
2. ABB Ability™ History ObjectUpdate: This is equivalent of a device.created event of the platform. It creates connected devices based on the declared type. Connector will construct the device.created message and send it to the messagesout topic. In a standard edge deployment, proxy will redirect this to the Information Model in the cloud, and returns an acknowledgement indicationg the success status, and an**_Object.Model** in case of an actual success; However, Connector is not actually concerned with the reply as mentioned previously.
3. ABB Ability™ History EquipmentEvent: This is the equivalent of an event message on the platform. The Connector will construct an event message and send it out on messages_out topic. In a standard edge deployment, this will be redirected by the proxy to the Azure cloud and stored in Azure Time-series insights.

In addition, the Connector also listens to warm topic from the platform side for the writeback functionality. This indicates that you are able to write to this topic, and have the variable data entered into the ABB Ability™ History database.

Setup, Deploying and Usage

Deploying

The method which you deploy with is out of the scope of this documentation. However, we try to provide guidance on what should be done in a couple of scenarios.

Standard Edge

Deploying Connector is performed just like a normal typed module on the Ability Edge. A configuration type and a device type must be registered in the type definition registry, and a reference to those must also be added to the edge's configuration type. It must be noted that cpmPlus History should be available for Connector module to connect to. You can see a sample of types for Connector module below. Remember that you must add a reference to these in your edge types.

Configuration type:

{
  "model": "abb.ability.configuration",
  "typeId": "abb.ability.configuration.edge.modules.cpm-connector",
  "version": "1.0.0",
  "unique": ["name"],
  "properties": {
    "name": {
      "value": "cpm-connector",
      "description": "Name of this module (in the context of the edge device)",
      "dataType": "string",
      "isMandatory": true
    },
    "docker": {
      "env": {
        "dataType": "map",
        "values": "string"
      },
      "replicas": {
        "description": "How many replicas of this module should be created",
        "dataType": "integer",
        "minimum": 1
      },
      "image": {
        "description": "Docker image of this module",
        "dataType": "string",
        "value": "cpm-connector:latest",
        "isMandatory": true
      },
      "password": {
        "description": "Will force the MQTT password for debugging purpose. Don't use on production",
        "dataType": "string",
        "value": "mqttPasswordForCpmConnector",
        "isMandatory": false
      }
    },
    "RuntimeConfiguration": {
      "NetSyncConfiguration": {
        "ConnectionString": {
          "dataType": "string",
          "value": "wss://cpmplushistory:9443/view"
        },
        "Username": {
          "dataType": "string",
          "value": "root"
        },
        "Password": {
          "dataType": "string",
          "value": "root"
        },
        "EarliestTime": {
          "dataType": "string",
          "value": "2020-01-01T12:00:00.000Z"
        },
        "EquipmentModelName": {
          "dataType": "string",
          "value": "abb.ability.yourarbitrarymodel"
        }
      },
      "BufferConfiguration": {
        "MaxBufferSize": {
          "dataType": "integer",
          "value": 50000
        },
        "BufferFlushTimerInterval": {
          "dataType": "integer",
          "value": 10000
        }
      }
    }
  },
  "attributes": {
    "minimum": {
      "dataType": "integer",
      "appliesTo": ["integer"]
    }
  }
}

Device Type:

{
  "model": "abb.ability.device",
  "typeId": "abb.ability.device.edge.modules.cpm-connector",
  "version": "1.0.0",
  "unique": ["name"],
  "properties": {
    "name": {
      "description": "Name of this module (in the context of the edge device)",
      "dataType": "string",
      "isMandatory": true
    }
  },
  "variables": {
    "docker": {
      "image": {
        "description": "Docker image of this module. Reported by the edge runtime",
        "dataType": "string"
      },
      "state": {
        "description": "Runtime state of this module. Reported by the edge runtime",
        "dataType": "string",
        "enum": ["running", "stopped"]
      }
    }
  },
  "relatedModels": {
    "abb.ability.configuration": {
      "type": "abb.ability.configuration.edge.modules.cpm-connector@1",
      "uniqueMapping": {
        "name": "name"
      }
    }
  }
}

Custom Edge

The module is expecting environment values to be set as follows. It is your decision how these are set:

The module is also expecting runtime configuration through an MQTT message on topics_model_in/abb.ability.configuration/object_id Please note that topics_model_in and object_id are dependant on what you previously set in the environment and also that the retain flag for this message has to be set as true. Next the format of the actual configuration received must look like the json sample below:

{
  "objectId": "3f17bfbe-357f-430a-ae84-4b0566b4a49d",
  "model": "abb.ability.configuration",
  "type": "abb.ability.configuration.edge.modules.cpm-connector@1",
  "properties": {
    "name": { "value": "b1f7b7f5-8e77-4326-b4b2-71efdff76f52/cpm-connector" },
    "docker": {
      "image": { "value": "cpm-connector:latest" },
      "password": { "value": "edgeHelsinki" }
    },
    "RuntimeConfiguration": {
      "NetSyncConfiguration": {
        "ConnectionString": { "value": "wss://cpmplushistory:9443/history" },
        "Username": { "value": "root" },
        "Password": { "value": "root" },
        "EarliestTime": { "value": "2020-01-01T12:00:00Z" },
        "EquipmentModelName": { "value": "abb.ability.yourarbitrarymodel" }
      },
      "BufferConfiguration": {
        "MaxBufferSize": { "value": 3000 },
        "BufferFlushTimerInterval": { "value": 6000 }
      }
    },
    "version": 1,
    "lastModified": "2020-02-27T12:02:03.998Z"
  }
}

Please note that from this configuration, only the properties.RuntimeConfiguration and what's inside is required, and you can also adjust what you put in there based on what was explained in the configuration section. i.e. you can omit Username/Password if you choose to do so since they're not mandatory.

Configuration

Providing the configuration to the module is the minimum requirement for Connector to launch and operate. The way of providing the configuration will be explained in the next section, but this section focuses on what each configurable value means.

NetSync Configuration

BufferConfiguration

Connector uses an internal telemetry batching system to batch multiple time-series data-points into one and send all in one MQTT message. This is particularly useful due to the limits imposed by the Azure IoT Hub component on the number of messages you can transmit per day.

Logger

Log level can also be adjusted based on your needs. The default level is Information.

Usage

Registration of Types

Currently, storing telemetry for any device, or equipment on the Ability Azure Cloud requires a respective instance of that device to exist on the information model. The process to achieve this, is through the method called creating a device. This process, requires you to first register a type for on the Type definition Registry, and instantiate that type.

As of this moment, the CpmPlus Connector does not support registration of types. Therefore, this has to be done manually through the Type Definition Registry of your edge deployement. Please note that these endpoints are unique per resource group(edge deployement) and may not be shared accross multiple edge deployements depending on the plaftom version you are using; The insantiating act however(sending device.created), is automatically performed by Connector and you do not need to worry about it as long as its respective type exist. It should be noted that due to these limitations, it is currently not possible by Connector to confirm the existance of a type before sending a device.created. This implicates that in case Connector is launched without the proper types registered, the respective device.created messages will fail and may cause dataloss, and in case the types are registered after the first launch, a connector restart will be required.

There are three possible scenarios in which you may want to register devices. The first scenario is when your equipment(devices) hierarchy already exist and you are running Connector for the first time. For this, we have provided some scripts in the next section to simplify your work. These scripts simply extract the types you're required to register through Connector, and provide you with a curl script to perform this on a machine with access to the TypeDefRegistry endpoint. It is not mandatory to use this scripts, and you can perform this task manually by following the steps of the second scenario, and register the device types one by one.

The second scenario is when you want to add a new equipment at runtime(after using the first-time registration scripts). This scenario requires you to manually create a new type using the Connector.

The third scenario is adding new properties to an existing equipment at runtime. This scenario is currently unsupported.

First-Time Registration of Devices

In this section, we provide you with some scripts to simplify your effort in the type registration process. For this to work, you will need POST URL for Adding new type definition to database for the Type Definition Registry of your deployement, console access on your edge's host, and a running Connector. The reason that you need a running Connector is that Connector currently generates and writes the correct format of the type you will need. These scripts will generate a Curl script which you can execute anywhere with access to the Type Definition Registry endpoint(could be your local machine).

First, you need to get the correct URI, and you need to construct it. The format looks like this:

{platformtyperegistryuri}/api/v1.0/modelDefinitions/{EquipmentModelName}/types

Equipment Model name is what you defined in the configuration of the module, and platformtyperegistryuri is specific to your Edge deployement. It looks something like this:https://abitydefregaspag21eundev.azurewebsites.net

Next create these two scripts with proper execution permissions on your host machine:

findtypesfromfile.sh:

#!/bin/bash
if [ "$#" -ne 2 ]; then
    echo "Needs docker container id and output file as argument."
	exit 1
fi
docker cp $1:/tmp/connector_seen_types.txt .
cat connector_seen_types.txt 2>&1 | sort | uniq | grep { > $2

createcurlscript.sh :

#!/bin/bash
if [ "$#" -ne 2 ]; then
    echo "Needs input file and registry url as arguments."
	exit 1
fi
dos2unix $1
rm curl.sh || true
echo "#!/bin/bash" > curl.sh
cat $1 | xargs -I % -d "\n" echo "curl -X POST -H 'Content-Type: application/json' -d '%' $2" >> curl.sh
chmod a+x curl.sh
echo "Created curl.sh file. Validate and execute that to register types."

After creation of the scripts, first find out the Connector's containerId(docker ps) and enter the following into the console(typefile can be anything you like):

./findtypesfromfile.sh <container id> <typefile>

Then, use the typefile from the previous step, and the TypeDefRegistry URL in the next command and execute:

./createcurlscript.sh <typefile> <platformtyperegistry>

The output will be a file named curl.sh which will contain the proper curl script for registrering your types. You can run this on any machine with internet access to the endpoint(your host machine may not have internet access). Please review the contents before executing as it can be time consuming to purge pre-existing types.

Adding New Equipment to the System and Registering its type

This scenario is for when you have already used the above scripts, registered your devices, and are now looking to add new equipment to the system OR for when you have decided to register your devices manually. For this, you once again need access to the Connector container, and access to the TypeDefRegistry endpoint(preferably swagger).

After adding the equipment, look for the /tmp/connector_seen_types.txt file on the Connectors container, and find the json respective to the equipment you have added. Please make sure that the JSON payload includes the name of the equipment you have recently added.

For the next step, go to the Typedefregistry swagger, select the field of "Adds new type definition to database". For the model, put the EquipmentModelName which you have entered in the Connector configuration, and for the payload, paste the JSON you extracted earlier. Make sure the type is registered correctly.

Finally, restart the connector so the device.created messages are sent again.

Please note that this step can also be done manually for each type at the first run, instead of using the scripts.

Modifying Existing Equipment

In most cases, modification of the equpiment model requires updates to the type definitions. The version number can be referred in the Connector configuration.

Listening for what happens on the CpmPlus side

This depends on your setup and equipment structure. Not much can be done here as they're all performed automatically and abstracted under several layers. As long as the configuration is set properly, you can expect:

1. device.created messages on topics_messages_out/type=deviceCreated&ack=all
2. device.updated messages on topics_messages_out/type=deviceUpdated&ack=all
3. variable messages on topics_messages_out/type=timeSeries
4. event messages on topics_messages_out/type=event

In a normal edge, these are all sent and handled by the Edge Proxy. In a custom edge, its your responsibility to apprehend these topics and read the messages.

Writing variable data to the CpmPlus

Currently only variables are supported to be written. You must send an MQTT message to the topic warm and the payload must follow the structure below:

{
  "objectId": "0c055b6c-004d-493d-8caa-96c952c0e207",
  "model": "abb.ability.device",
  "timestamp": "2020-01-01T00:00:00Z",
  "variable": "sampleVariable",
  "value": 15
}

The value will get written to the Equipment instance that is found with the objectId as its ID and the variable name as its property name.

In a normal edge, you probably will not have permissions to write to this topic in the first place. In a custom edge, this is possible through proper setup of Broker Access Control List.

Configuring source database

Source ABB Ability™ History database needs to be configured. The equipment publication is configured the same way as when using Vtrin-Netsync. Only difference is that the target database node has to be of type AbilityCloud. Tutorial for configuring both is in this turotial.