WIP-0006: Plugin-Defined Docker Services¶

Introduction¶

This document defines a mechanism for plugins to define and manage their own Docker services (containers) that run alongside the core Whitebox stack. Previously, plugin-specific services had to be hardcoded into the main compose.yml and compose.dev.yml files, which violated the plugin architecture's modularity principle.

Problem¶

The former plugin architecture allowed plugins to extend backend and frontend functionality, but they could not declare their own containerized services.

Former State: Hardcoded Services¶

Services like Stratux that should logically belong to plugins were previously defined in the core repository:

compose.yml:

services:
  stratux:
    build:
      context: .
      dockerfile: packaging/docker/stratux.Dockerfile
    privileged: true
    container_name: stratux
    # ...

Issues¶

Tight Coupling: Plugin services were baked into core compose files even when the plugin wasn't installed
Manual Integration: Adding a new plugin with service requirements needed manual modification of core repository
Not Scalable: As more plugins needed services, the core compose files and packaging/ directory became cluttered
Distribution Problem: Plugins could not be truly self-contained - their infrastructure lived in a different repository
Cannot Install/Uninstall Cleanly: Installing a plugin didn't automatically spin up its required services, and uninstalling didn't clean them up

The Core Challenge¶

Plugins run inside the backend container, but Docker commands must run on the host. This created a fundamental problem:

Plugins are installed via poetry add whitebox-plugin-stratux inside the backend container
Plugin code can execute in Python within the container
But Docker services need to be managed from the host machine where the Docker daemon runs
The backend container cannot start new sibling containers

Even if we packaged Dockerfiles and compose files with the plugin, we have no way to use them from within the container to start services on the host.

Solution¶

Whitebox Host Manager #393 - a privileged service running on the host that exposes an API for container management. Plugins can now define their service requirements, and the backend communicates with the Host Manager to spin up those services.

Architecture Overview¶

Host Machine
|-Whitebox Host Manager
|   |-Runs on host
|   |-Exposes REST API
|   |-Has Docker socket access
|   |-Loads plugin packages directly from source
|   |-Can manage plugin defined services
|   |-Device management, etc.
|
|-Backend Container
|   |-Plugin installed in site-packages
|   |-Notifies Host Manager of plugin metadata
|   |-Calls Host Manager API to register/start/stop services
|

Key Components¶

1. Whitebox Host Manager

A privileged service running on the host:

Exposes REST API for management
Has Docker socket access to run docker/compose commands
Downloads and extracts plugin packages directly from source (PyPI/Git)
Provides keyword resolver for Jinja templates
Segregates plugin services by namespace to prevent naming conflicts
Processes Jinja templates (.yml.j2) with resolved keywords to generate final compose files
Maintains registry of which plugins have registered which services
Handles service lifecycle: build, start, stop, remove

2. Plugin Manager (Backend)

Discovers installed plugins and their metadata
Notifies Host Manager with plugin details: name, version, source (PyPI/Git)
Provides cleanup on plugin unload by unregistering services from Host Manager

3. Host Manager Client (Backend)

An API via which plugins can interact with the Host Manager
Methods to register/unregister services
Methods to query service status
Methods to start/stop/restart services

Implementation Flow¶

Phase 1: Enable Plugins to Package Services¶

Update plugin packaging to include service definitions as Jinja templates:

pyproject.toml:

[tool.poetry]
name = "whitebox-plugin-stratux"
include = [
    { path = "packaging", format = "wheel", destination = "whitebox_plugin_stratux" },
    { path = "compose.yml.j2", format = "wheel", destination = "whitebox_plugin_stratux" },
    { path = "compose.dev.yml.j2", format = "wheel", destination = "whitebox_plugin_stratux" },
]

[tool.whitebox-plugin]
privileged = true  # Set to true if the plugin requires privileged container access

Plugin structure:

whitebox-plugin-stratux/
|   |-whitebox_plugin_stratux/
|   |     |-whitebox_plugin_stratux.py
|   |     |-compose.yml.j2
|   |     |-compose.dev.yml.j2
|   |     |-packaging/
|   |         |-docker/
|   |             |-stratux.Dockerfile
|   |...
|-pyproject.toml

Example compose.yml.j2 with keywords:

services:
  stratux:
    build:
      context: .
      dockerfile: packaging/docker/stratux.Dockerfile
    privileged: true
    container_name: {{ whitebox.plugin_container_name }}
    networks:
      - {{ whitebox.network }}

networks:
  {{ whitebox.network }}:
    external: true

Keyword Resolver System:

The Host Manager also acts as a keyword resolver, providing a context of available keywords that plugins can use in their templates:

Example Keywords:

{{ whitebox.network }} - Resolves to whitebox-net or whitebox-dev-net based on environment
{{ whitebox.plugin_container_name }} - Resolves to plugin namespace (e.g., whitebox_plugin_stratux)
Additional keywords can be added as needed

Benefits:

Plugins remain agnostic of deployment specifics
Host Manager controls all resolution logic based on configuration
Easy to extend with new keywords without changing plugin code
Prevents hardcoding of environment-specific values in plugins

Phase 2: Notify Host Manager During Plugin Load¶

Detect installed plugin packages
Extract plugin metadata: name, version, source (PyPI/Git repository URL)

Phase 3: Register Service with Host Manager¶

Backend makes the API call to Host Manager to register and start the service:

host_manager_client.docker_services.register_service(
    plugin_name="whitebox-plugin-stratux",
    version="1.0.0",
    source="pypi",  # or git repository URL
)
host_manager_client.docker_services.start_service(service_name="stratux")

Host Manager Processing:

Downloads the plugin package from the specified source (PyPI/Git)
Extracts the plugin package to a temporary working directory
Reads and validates pyproject.toml to check [tool.whitebox-plugin] permissions
Locates compose.yml.j2, compose.dev.yml.j2, and Dockerfiles within the package

Builds keyword resolution context based on plugin metadata and host configuration:

keyword_context = {
    'whitebox': {
        'network': 'whitebox-net',
        'container_name': f"whitebox_plugin_{plugin_name}",
        # Any additional keywords can be added here
    }
}

Processes Jinja templates with keyword resolver
Security validation: Verifies that rendered compose file doesn't request privileges not declared in pyproject.toml
If compose file contains privileged: true but [tool.whitebox-plugin] privileged = false or unset, reject the service
Writes rendered compose file to working directory
Runs: docker compose -f <rendered_compose> up -d
Records service in registry with plugin metadata and declared permissions

Benefits:

Plugin services run in isolated namespaces via resolved {{ whitebox.plugin_container_name }}
No naming conflicts between plugins
Network assignment handled via keyword resolution (external network marked in template)
Plugins remain deployment-agnostic, all specifics resolved by Host Manager
Easy to extend with new keywords for future requirements

Phase 4: Unregister Service¶

When plugin is removed, unregister service and clean up:

host_manager_client.docker_services.stop_service(service_name="stratux")
host_manager_client.docker_services.unregister_service(plugin_name="whitebox-plugin-stratux")

Host Manager Processing:

Runs: docker compose -f <rendered_compose> down
Cleans up rendered compose files and temporary extracted plugin files
Removes plugin from registry

Plugin Permissions¶

The pyproject.toml file serves as the canonical source of truth for plugin permissions.

1. Permission Declaration in pyproject.toml:

[tool.whitebox-plugin]
privileged = true  # Requires privileged container access

2. Multi-Layer Permission Enforcement:

Plugin Manager (Backend): Reads [tool.whitebox-plugin] section from package metadata to display warnings to users before installation
Host Manager: Validates that rendered compose files don't request privileges beyond what's declared in pyproject.toml

3. Security Validation Flow:

Plugin Installation Request
    ↓
Plugin Manager reads pyproject.toml
    ↓
Display warning to user if privileged = true
    ↓
User confirms installation
    ↓
Backend notifies Host Manager
    ↓
Host Manager downloads & validates pyproject.toml
    ↓
Host Manager renders compose template
    ↓
Host Manager validates rendered compose against declared permissions
    ↓
If mismatch detected → Reject service creation
    ↓
If valid → Create service

4. User Trust & Transparency:

Users ultimately decide what to install on their Whitebox:

Plugin Manager UI displays clear warnings for plugins requiring elevated permissions
Installed plugin list in UI shows which plugins run with elevated privileges
No friction added for plugins, but full transparency for informed decisions

5. Future Extensions:

The [tool.whitebox-plugin] section can be extended with additional permission fields:

[tool.whitebox-plugin]
privileged = true
devices = ["/dev/stratux"]
capabilities = ["NET_ADMIN"]

Each permission would follow the same validation pattern: declared in pyproject.toml, verified by Host Manager.