Whitebox Plugin - Transfer Manager¶
Manages file transfers from external devices (cameras, etc.) with a two-phase workflow: download from device, then process (transcode/stitch/convert). Tracks persistent state, emits real-time progress to the frontend, and handles retries and crash recovery automatically.
Installation¶
Capabilities¶
| Provides | Requires |
|---|---|
transfer-manager |
device |
Exposed to Other Plugins¶
Plugin Classes (get_plugin_classes_map)¶
Accessible via import_whitebox_plugin_class():
| Key | Type | Description |
|---|---|---|
transfer.TransferManager |
TransferManager instance |
Singleton service for queueing, dispatching, and tracking transfers |
transfer.TransferAdapter |
Protocol class |
Interface that device plugins implement for download/processing |
transfer.RemoteFile |
dataclass class |
Represents a file on a remote device |
Models (model_registry)¶
| Key | Model | Description |
|---|---|---|
transfer.Transfer |
Transfer |
Persistent record for each file transfer |
Frontend Components (exposed_component_map)¶
| Category | Key | Component | Description |
|---|---|---|---|
service-component |
transfer-service |
TransferServiceComponent |
Background service that listens for WebSocket transfer events, updates the Zustand store, and triggers flight session re-fetches on completion |
transfer-manager |
transfer-panel |
TransferPanel |
UI panel showing active/completed transfers with progress bars, cancel, and retry actions |
Frontend Slots (slot_component_map)¶
| Slot | Component |
|---|---|
transfer.panel |
TransferPanel |
State Stores (state_store_map)¶
| Key | Module | Description |
|---|---|---|
transfer |
stores/transfer |
Zustand store tracking transfer state (transfers, activeCount, upsertTransfer, removeTransfer, clearCompleted, dismissAll) |
Daemon¶
Command: daemon_transfer_manager
Runs as a BaseDaemonCommand that orchestrates all transfers.
Subscribed Events¶
| Event | Action |
|---|---|
observation.flight.end |
Dispatch any pending transfers |
observation.device.connection_status.update |
On reconnect, dispatch pending transfers for that device |
command.transfer.cancel |
Cancel a transfer (sets Redis flag for in-progress workers) |
command.transfer.retry |
Retry a failed/cancelled transfer |
command.transfer.queue_batch |
Create transfer records from a batch of files and dispatch |
Periodic Checks (every 5s)¶
- Stale transfer recovery: Resets transfers stuck in active states with no DB update for 2+ minutes (worker likely dead)
- Dispatch pending: Picks up PENDING transfers respecting concurrency limits
Startup Recovery¶
- Resets any in-progress transfers (DOWNLOAD_STARTED, DOWNLOAD_COMPLETED, PROCESSING_STARTED) back to PENDING
- Clears retry_after on PENDING transfers that were in backoff when daemon stopped
Emitted Events¶
All events are sent to the management WebSocket group.
| Event | When |
|---|---|
observation.transfer.queued |
Transfer created or reset to pending |
observation.transfer.download.started |
Download begins |
observation.transfer.download.progress |
Download progress (throttled: 1s or 5% delta) |
observation.transfer.download.completed |
Download finished |
observation.transfer.processing.started |
Processing begins |
observation.transfer.processing.progress |
Processing progress (throttled: 1s or 5% delta) |
observation.transfer.processing.completed |
Processing finished, FlightSessionRecording created |
observation.transfer.download.failed |
Download failed permanently |
observation.transfer.processing.failed |
Processing failed permanently |
observation.transfer.retry_scheduled |
Transfer scheduled for retry with backoff |
observation.transfer.cancelled |
User cancelled transfer |
observation.transfer.paused |
User paused transfer |
Registering an Adapter¶
Device plugins expose adapters via their plugin class:
# In device plugin's whitebox plugin class
def get_transfer_adapters(self):
from .transfer_adapter import MyDeviceTransferAdapter
return [("my_device_codename", MyDeviceTransferAdapter())]
The daemon discovers these on startup; workers discover lazily on first use.
Process file recording command¶
Command: process_file_recording [device_codename] [source_path] [destination_path]
This command is used to process a file recording for a transfer manually.
Example:
docker exec backend-dev poetry run python whitebox/manage.py process_file_recording \
insta360_x4 \
/opt/whitebox/media/transfers/downloads/VID_001.insv \
/opt/whitebox/media/transfers/processed/VID_001.m3u8
Running this command will process the specified source path with the transfer
adapter defined by the device plugin for the given device_codename, and store
it into the destination path, just like the transfer manager would do it.
It can be used for debugging or testing purposes.
TransferManager API¶
Queueing¶
from plugin.registry import import_whitebox_plugin_class
transfer_manager = import_whitebox_plugin_class("transfer.TransferManager")
RemoteFile = import_whitebox_plugin_class("transfer.RemoteFile")
# Queue a batch of files (sync, for use in RQ tasks or daemon)
files = [RemoteFile(path="/DCIM/VID_001.insv", size=1_000_000)]
transfers = transfer_manager.queue_batch_sync(
device_connection=conn,
files=files,
download_dir=Path("/media/transfers/downloads"),
processed_dir=Path("/media/transfers/processed"),
associate_with=flight_session,
)
Concurrency¶
| Setting | Default | Description |
|---|---|---|
| Max concurrent global | 4 | Total active transfers across all devices |
| Max concurrent per device | 2 | Active transfers per device connection |
Retry Logic¶
- Transient errors (network, device) retry with exponential backoff: 2s, 4s, 8s, ..., capped at 60s
- Retry window: 7 days from initial queue time
- Safety cap: 500 retries max
- Non-transient errors (disk full, corrupt file) do not retry