Federation

A core concept within the model data explorer is the federation between multiple Model Data Explorer instances. This is essential as multiple research centers might offer different services for one single dataset. One research center might make the data available via THREDDS, the other one might make it available as WMS.

Challenges

When synchronizing two MDE instances, we want to have the full metadata available in both instances. An essential part of this metadata is the relation to data groups and the resulting permissions of the users on the dataset.

For reasons of data protection we do not want to synchronize the user profiles between the two instances as exchanging these personal information between two independent research centers is difficult from a legal point of view.

Core concepts

Our federation framework is built upon three central concepts:

1. Each data groups has a unique hosting MDE

2. OAuth for synchronizing user permissions between two MDE instances

3. A websocket messaging for synchronizing MDE instances

Unique hosting

The unique hosting MDE for a data group means that only the MDE hosted at Hereon (for instance) can manage the members of the Hereon datagroup. The MDE hosted at another research center imports the metadata of the Hereon data group from the MDE hosted at Hereon, but it does not import the members of the data group (for data protection reasons).

OAuth-authentication

The OAuth-authentication can connect the two instances on a per-user basis. The user who has already an account at the Hereon MDE decides by himself whether he wants to create an account at the other research center. He then authenticates via OAuth against the MDE at Hereon and allows the MDE at the other research center to query the datagroups that the users is a member of. From this, the MDE at the other research center knows what data groups the user should be part of.

Websocket messaging

The final synchronization between the two MDE instances goes via websocket communication through a message broker (the dasf-broker-django). Each MDE instance has a background worker that opens a websocket connections to the other MDE (using channels workers framework). When there is a change to a dataset or a service on one instance, it sends a message via the websocket connection to this background worker of the other instance. In that sense, the originating MDE instance becomes a producer of a message and the background worker becomes a consumer.

The connection via websocket uses an API token. This token corresponds to a bot user at the other MDE. As such, whether a dataset is included in the federation or not depends on the permission of the user account that is used for the federation.

Todo

It might be better to use an OAuth-Workflow here instead of sending around API tokens

Embedded

Image

Workflow for creating a federation

Models

Description

The optional mde-federation plugin implements the functionality. It defines two models:

FederatedDataGroup

A subclass of the DataGroup of the mde-core plugin that holds a link to a Federation. If there is a FederatedDataGroup present, the owners, etc. of a DataGroup are managed by the mde-federation plugin.

Federation

A federation to another model data explorer. This federation has a domain (e.g. hereon.de), a websocket endpoint and an API token to connect via websocket.

Graph

Graph

Django source

from django.db import models
from django.contrib.auth.models import User

class DataGroup(models.Model):
    """A DataGroup (implemented by mde-core)"""

    name = models.CharField(max_length=100)


class Federation(models.Model):
    """A federation with another model data explorer instance."""

    domain = models.CharField(
        "Domain name", max_length=100, unique=True
    )

    websocket_endpoint = models.CharField(
        "Websocket endpoint for the consumer",
        max_length=300,
    )

    api_token = models.CharField(
        "API token for the foreign MDE", max_length=100
    )


class FederatedDataGroup(DataGroup):
    """A datagroup that is owned by a federation."""

    federation = models.ForeignKey(
        Federation,
        on_delete=models.CASCADE,
        help_text=(
            "The foreign MDE instance that owns this datagroup."
        )
    )