Back to top
Vastuu Group

Data Product 101

Last updated: Oct. 1, 2020
|
Reading time: 8 min

Platform of Trust harmonizes incompatible data coming from various sources, enabling it to be merged and utilized. Users can request and use this data by requesting data products.

Defines both data content and the business elements of real-world entities in a machine-readable format. Data product defines how to query harmonized data from the relevant data source via the means of Platform of Trust Broker API.

– 

Data Product

3rd party developers always use Broker API to access data product data. For more information, visit developer portal API profile to learn about Broker API. Or read more about /broker/v1/fetch-data-product in API documentation.

Note: We expect the raw data provider to have a beta API environment which can be used in the Platform of Trust environment. The beta API is used for testing the data product before launch.

What is a connector and why do I need it?

An integration component that enables the Platform to pull and harmonize data from external sources.

– 

Connector

Main connector functions:

  1. Fetch and standardize data
  2. Validate all communications between broker and data source

Get familiar with Platform of Trust ontology

Local data most likely use a different model to construct data. If you haven't done data integration before, use a good amount of time in this step. It will pay back later and reduce the amount of time on refactoring and resources.

You need to be familiar with the ontology to harmonize your source data fields to standard formats used in Platform of Trust . With a harmonized ontology and semantics we can be sure that everyone understands the content in a similar way. You can also learn about our harmonized data model from our Ontology Viewer.

1. What Data is available and why it is published

You might be tempted to jump directly to the technical integration, but think about the business reasons first. Use a business model canvas or similar to iterate business reasons and other aspects if needed.

First you need to find out what data is available in the source system. On top of that you need to have a reason why it is published. You can checkout currently supported Data Source Integrations in Platform of Trust here.

If you act on behalf of another company and your task is to setup technical facilities for the data product, discuss with your client.

Business-related questions which normally needs to be answered before proceeding:

  • What data is available?
  • Why should that data be published via Platform of Trust (business reasons)?
  • Who are the expected consumers?
  • What are the use cases for the data?

Technical questions which need to be answered before proceeding

You're about to start building the connector component or functionality between your data source system and Platform of Trust.

  • How is the data accessed? API? Something else?
  • Where's the API (if there is)?
  • Data source system/integration documentation?
  • How to get credentials which are needed?

The 3rd party developers will access the data via the Platform of Trust Data Broker API, which forwards the request to the connector component that continues processing the request and responses accordingly. The beta API is also expected to stay available after the data product has been created since 3rd party developers will continue to use it via Data Broker API while developing applications instead of calling your production API.

2. Create a Platform of Trust user and get Bearer token

To get started, you need to create a user on Platform of Trust. You can do it by simply signing up on Login Portal. In order to get bearer token to able to perform authorized API requests to Platform of Trust, you need to get Bearer Token first.

Register to sandbox

3. Create and host a connector

A connector transforms the local data model and attributes to match the Platform of Trust ontology

To access the data behind a connector, the 3rd party developers request data through the Data Broker API, which forwards the request to the connector.

The Platform of Trust Data Broker API does not store or cache the data anywhere. Each request is always relayed as is, with extra security added in signed payload headers. The request data or response data is never mangled in any way in the Data Broker API.

Read more about how to setup your own connector

You can work with our Multi-connector which is now actively being developed with new features added frequently. This connector provides a huge starting point for those who want to develop connectors. Basically, building a connector, in most cases, only involves writing a few configuration files.

You can also fork any of the public connectors repositories available in our GitHub organization. A few of our connectors are:

  • Entose - a connector to retrieve electricity prices from the Entsoe API
  • Accuweather - to provide weather products (weather today and forecast) from Accuweather

You need to deploy the code yourself, e.g. use your own server or a cloud provider. So this would require more work from you.

The third popular option is the use of Connector Engine service in Platform of Trust. We have a list of dedicated partners who are willing to implement customized connectors tailored to your needs. Contact us for more information about this with the message "Require a Connector".

4. Register data product in the sandbox and test the connector

You can register the data product with our Product API. The documentation always points to the sandbox APIs, which means that you should first test against that. To learn more about Product API, visit the API profile in the Developer Portal.

You can directly test Product API live from the Developer Portal. On running the following snippet, you can get a list of Data Products available in the Sandbox environment

Product API 1.0 - List all products

const request = require('request'); request('https://api-sandbox.oftrust.net/products/v1', { json: true }, (err, res, body) => { if (err) { return console.log(err); } console.log('response', body); });

Here's an example of how you would register a data product with the Product API in Platform of Trust. Besides Bearer token, you will also need groupId parameter. Read more about how to create your own group

Request: POST /products/v1

curl -X POST https://api-sandbox.oftrust.net/products/v1 \
-H "Content-Type: application/json" \
-H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzY29w...DVs5aaf" \
-d '{
  "dataContext": "https://platformoftrust.github.io/standards/contexts/product-data.jsonld",
  "parameterContext": "https://platformoftrust.github.io/standards/contexts/product-parameters.jsonld",
  "productCode": "example-product-1",
  "name": "Example product",
  "groupId": "25d5c931-7479-45a6-b7fb-f66bc795345d",
  "translatorUrl": "https://example.com/translator/url",
  "organizationPublicKeys": [
    {
      "url": "https://example.com/example.pub",
      "type": "RsaSignature2018"
    }
  ]
}'

Response: Ok
Status: 201 Created
Body:

{
  "@context": "https://standards.lifeengine.io/v1/Context/Identity/Thing/HumanWorld/Product/DataProduct/",
  "@type": "DataProduct",
  "@id": "https://api-sandbox.oftrust.net/products/v1/example-product-1",
  "productCode": "example-product-1",
  "dataContext": "https://platformoftrust.github.io/standards/contexts/product-data.jsonld",
  "parameterContext": "https://platformoftrust.github.io/standards/contexts/product-parameters.jsonld",
  "translatorUrl": "https://example.com/translator/url",
  "name": "Example product",
  "organizationPublicKeys": [
    {
      "type": "RsaSignature2018",
      "url": "https://example.com/example.pub"
    }
  ],
  "description": null,
  "imageUrl": null,
  "identityId": "5d58025b-3764-4f3d-8d09-5e17b064f0f9"
}

Read more about /products/v1 request in Product API documentation.

5. Testing the data product in sandbox

In order to test your new data product, you need to make a request to the Data Broker API. Read more about how to construct a valid request.

Request: POST /broker/v0.1/fetch-data-product

curl --request POST \
  --url https://api-sandbox.oftrust.net/broker/v1/fetch-data-product \
  --header 'content-type: application/json' \
  --header 'x-app-token: eyJ…FY' \
  --header 'x-pot-signature: MYZlyZR+V5+auLqMMXCUnvaVHfD2wvSujFreOrMrASw=' \
  --data '{
    "timestamp": "2019-10-25T00:00:00.000Z",
    "productCode": "example-product-1",
    "parameters": {
        "example1": "example2"
    }
}'

Response: Ok
Status: 200 OK
Body:

{
  "@context": "https://standards.lifeengine.io/v1/Context/Identity/Thing/HumanWorld/Product/DataProduct/",
  "data": {
    "@context": "https://standards.lifeengine.io/v1/Context/Identity/Thing/HumanWorld/Product/DataProduct/",
    "@type": "DataProduct",
    "items": [
      {
        "id": "example1",
        "data": [
          {          
            "example1": "example2",
            "example2": 1234
          },
        ]
      }
    ]
  },
  "signature": {
    "type": "RsaSignature2018",
    "created": "2019-10-25T00:00:05.000Z",
    "creator": "https://example.com/example.pub",
    "signatureValue": "OaSx5PCjG/yYFdXvguUx...vtjEOqctQDqs0uPFpEKL0E0L4="
  }
}

Read more about /broker/v1/fetch-data-product request in Data Broker API

6. Register data product in the production

The final step is to register the data product in the production environment.

Configure your data product and connector to work with real data. And, make the same API calls but point all requests to Platform of Trust production servers https://api.oftrust.net instead of https://api-sanbox.oftrust.net

Create an account and get started!

If you are an application developer, it might be a good idea to read the Application Development Guide first.

Identities are fundamental features of the platform, take a look at the Identities and Links

Open sandbox is your friend! Isolated environment for testing applications and data product integrations, read more from Open Platform of Trust Sandbox

Ready to explore more?

Try Platform Sandbox

Improvement Suggestions? or a New Guide?

Tell us in GitHub