WIP-0009: Plugin Database Management

Introduction

This document proposes an implementation of plugins' database interactions and database schema management.

As a highly modular system, Whitebox uses plugins to provide and augment its functionalities, as well as allowing plugins to augment each other. The complexity increases as the augmention increases, as the database as a persistence layer requires careful management to ensure data consistency and integrity.

Goals

• Ensure data consistency and integrity across plugins' data
• Allow plugins to interact with each other's data
• Allow plugins to reference each other's data
• Allow plugins to manage their own database schema
• Allow plugins to be replaced without affecting other plugins

Problems and Challenges

• How to safely reference data between plugins

  As plugins can be replaced or disabled, we need a way to ensure that the data they reference is still available and does not cause inconsistencies or errors when the plugin is no longer available. Foreign key references are not sufficient for this purpose, as they can lead to dangling references and errors when the referenced plugin is no longer available.

• How to manage database schema changes for plugins

  The Django ORM is very opinionated about database schema changes, and does not provide a way to manage schema changes that are not tied to a specific model at the boot time. This makes it difficult to introduce concepts like referencing other plugins.

  While Django has a concept of a "swappable" model since its early age, but the feature is barely documented and not nearly as flexible as we would need it for our use case:

  • Any swappable model needs to be created in the initial migration, meaning any additionally-added models would require migration history rewriting
    to ensure that the new model is available in plugins' 0001 migration (Django docs)

  • Migrating to a swappable model on an already running project is not supported. There are no official guides for when you need to perform such an operation, but rather a collection of helpful comments on their bug tracker, with steps such as deleting migration files, rewriting history, and performing backups to ensure no inadvertent data loss.

• How to manage data migrations during plugin replacements

  When one plugin is installed, its database schema in place, and it already contains data, plugin replacement gets tricky. We need a solution that allows plugins to manage their own data migrations without affecting other plugins, and without requiring manual intervention or data loss.

Proposed Implementation

Cross-plugin foreign key references

The whitebox.db.get_model cross-plugin-model-loading mechanism allows plugins to reference each other's models, even if they are not installed at the time of the reference. This is achieved by using a lazy proxy that is resolved at runtime, allowing plugins to reference each other's models without causing errors or inconsistencies.

As an example:

from django.db import models

from whitebox.db import get_model

FlightSession = get_model("flight.FlightSession")

class FlightAnnotation(models.Model):

    flight_session = models.ForeignKey(
        FlightSession,
        on_delete=models.CASCADE,
        related_name="annotations",
    )

In the above case, to ensure that the plugin defining the FlightAnnotation model does not crash Django, it has to have flight capability defined in its requires_capabilities definition in its pyproject.toml.

Implementation detail

Django's relational fields don't accept our LazyModelProxy instances directly, and instead require either a concrete model instance, or a string representation of the model (app_label.ModelName). As mocking the full behavior of a Model into LazyModelProxy is not feasible, a monkey-patch for relational fields' __init__ method is applied, which allows the proxy to provide a string to be used as the target for the field reference.

The monkey-patch is minimal, and you can find it here: utils.monkey_patching.model_proxy_relation_field

Augmenting plugins' models by inheritance

Django provides a mechanism for one model to inherit another model with multi-table inheritance. This allows plugins to extend other plugins' models while still keeping the parent model intact, functional, and up-to-date.

For example, let's say that you want to extend the bare-bones flight session with a replacement model that provides new functionality, like aircraft type. You could do that by creating a new model that inherits from the original flight session model, like so:

from django.db import models

from whitebox.db import get_model

FlightSession = get_model("flight.FlightSession")

class CustomFlightSession(FlightSession):
    aircraft_type = models.CharField(max_length=100)

    # Parent model's methods/properties can be overridden here

For the inherited model to be fully recognized by the plugin system, it needs to be defined in the plugin's pyproject.toml file under the same name as the parent model, in this case, the flight.FlightSession, as well as have the original plugin defined as a dependency:

[tool.poetry.dependencies]
"whitebox-plugin-flight-management" = "*"

[tool.whitebox-plugin.plugin-models]
"flight.FlightSession" = "whitebox_plugin_<plugin_name>.CustomFlightSession"

After the plugin is installed, when the plugin system loads the models, it will automatically recognize the CustomFlightSession model as a replacement of the flight.FlightSession model, and any plugins importing it will receive the CustomFlightSession model instead of the original FlightSession model.

Implementation detail

As the Django's ORM, and with it the plugin system, will not be fully initiated as the time of plugin's model definition, we will be using an interim version of the model we want to inherit, which the proxy's instance provides via its __mro_entries__ override. This override provides a potentially un-ready version of the model, provided by django.apps.apps.get_model(..., require_ready=False), and it should exclusively be used for model inheritance.

---

This approach ensures that even if plugins extend and effectively and functionally replace each other's models, the data is separated and will not cause inconsistencies once the plugin providing the extension is replaced or disabled. Other plugins, relying on the extended model, will simply be able to define that specific plugin as a dependency of theirs, while all other plugins will, by power of inheritance, be able to seamlessly use the extended model without needing to know about the plugin that provided it.

To ensure stability and prevent cases causing data loss or inconsistencies, plugins are able to inherit only from the models that initially defined the model by their pyproject.toml file.

Data Structure and Handling of Inheritance Migrations

When a plugin is being replaced by inheritance, its database schema and data are not removed from the database or altered. Instead, the inheriting plugin's migrations will create a new database table, linking to the parent model's table, ensuring that there is a clean separation of data between the original and the extended models.

For this reason, the inheriting models must ensure that the original model's data remains valid and consistent with the parent's behavior, and it is up to the plugin's developers to ensure that their plugins will play nice after eventual plugin uninstallation.

More details in INSERT TICKET ID