DE EN EN (Google)

Custom Data Type Updater

If a Custom Data Type plugin receives data from a repository on an external server, the server has the ability to check if the custom data in the repository has changed, and objects that contain the custom data are updated.

The server collects Custom Datatype values from fields in objects. A set of distinct Custom Datatype values is then sent to the update script, that should return (only!) the values which changed in the meantime. The server will then map the returned data using an unique identifier, and will update affected objects.

The Updater is set up to run once a day at a full hour.

Alternatively, it can be started immediately using the API endpoint /api/v1/settings/updatecustomdata. This endpoint must be enabled in the server configuration: server.api.settings.updatecustomdata must be set to true.

Plugin Configuration

The plugin needs to specify the settings for the update in the plugin configuration (YAML file).

Under custom_types.<custom_type>.update the following variables are defined:

custom_types:
  <custom_type>:
    update:
      script: build/scripts/update.js
      timeout: 10
      batch_size: 1000
      interval_base_config: update_interval_<custom_type>.days

Base Configuration

The custom datatype plugin can define an own interval in days to wait between updates. If this value is not defined, the default value for all updates is used

The value must be an int and must correspond with the value of custom_types.<custom_type>.update.interval_base_config: "update_interval_<custom_type>.days" defines a parameter days for the base config entry update_interval_<custom_type>.

base_config:
  - name: update_interval_<custom_type>
    group: update_custom_data_type
    parameters:
      days:
        type: int
        min: 0
        default: 1
        position: 0

The value should have a minimum value of 0. If it is set to 0, the server interpretes this as a disabling of all updates for this custom datatype.

Example Configuration

A working example of a plugin that supports the update functionality is the Custom Datatype Plugin Gazetteer. The plugin defines the following configuration:

custom_types:
  gazetteer:
    update:
      script: build/scripts/gazetteer-update.js
      timeout: 10
      batch_size: 1000
      interval_base_config: update_interval_gazetteer.days
    mapping:
      ...

base_config:
  - name: update_interval_gazetteer
    group: update_custom_data_type
    parameters:
      days:
        type: int
        min: 0
        default: 1
        position: 0
  - name: gazetteer_plugin_settings
    ...

Update Payload

The update script has to receive and return data in a specific JSON structure. The requests always contain the following keys:

The returned JSON data has to have the following structure:

Received and returned data for action "start_update"

Data received from the Custom Datatype Updater:

{
    "action": "start_update",
    "plugin_config": {}
}
Mandatory fields

Data returned by the Update Script:

{
    "status_code": 200,
    "body": {
        "state": {}
    }
}
Mandatory fields
Optional fields

Received and returned data for action "update"

Data received from the Custom Datatype Updater:

{
    "action": "update",
    "plugin_config": {},
    "state": {},
    "batch_info": {
        "offset": 100,
        "total": 800
    },
    "objects" [
        {
            "identifier": "987654321abc",
            "data": {}
        }
    ]
}
Mandatory fields
Optional fields

Data returned by the Update Script:

{
    "status_code": 200,
    "body": {
        "state": {},
        "payload": [
            {
                "identifier": "987654321abc",
                "data": {}
            }
        ]
    }
}
Mandatory fields
Optional fields

If there were no updates, the script should return a status code of 200 and an empty JSON array as "payload".

Update Script

The update script is specific for each Custom Datatype plugin. It has to be able to receive and return the data in the JSON strucutures described above. It has to implement functions that can handle the actions "start_update" and "update". It also should handle any error and return correctly formatted error messages. Errors will be logged as events, and should contain enough information to debug and reproduce errors.

The script also has to able to use the API of the repository server. This API communication can be implemented in any way and is completely independent of the easydb server.

A working example can be found in the GazetteerUpdate.coffee file of the Gazetterr Plugin. This file must be compiled to a JavaScript file and must be provided at the path which is defined in the config variable custom_types.gazetteer.update.script.