Development Guide

This guide covers the development process and tools used in the Whitebox project and how you can develop new features or plugins for Whitebox and contribute to the project.

Prerequisites

• Docker (for running the Whitebox server)
• git-lfs (for downloading asset files)
• git (for cloning the repository)

Understanding Docker Configuration

To facilitate easy deployment and testing, Whitebox includes Docker configurations. Dockerfiles define the necessary environment for running or developing the Whitebox, including all required dependencies and configuration. Compose files orchestrate the setup of the server, database, and other services.

The Dockerfile is usually configured in two stages: one for building the application and another for running it. This separation optimizes the build process and reduces the final image size.

Setting up Development Environment

The recommended way developers working on Whitebox is to use the dev containers. It is a pre-configured Docker container that includes all the necessary dependencies and tools for development. Once launched, the container will simply start an environment.

To access the environment, you will need to enter the shell of the container (think "SSH-ing" into the container). From there you can access the environment, install project dependencies, run the required services, etc, same as you would with an SSH session.

This allows you to develop code on your local machine and run it in the container without having to rebuild the container each time you make a change.

Starting Development Containers

First you need to create .env file in the root of the project. You can use the provided .env.example file as a template:

cp .env.example .env

Edit the .env file to set env vars as needed. After saving the .env file, start the dev containers with:

docker compose -f compose.dev.yml up -d

You may encounter a warning about orphan containers being found for this project if you already have whitebox production container running as they share the same project and expose same ports. It's recommended to stop the production container before starting the development container. The warning can look like this:

WARN[0073] Found orphan containers ([frontend backend redis postgres]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.

To enter the backend container shell, run the following command:

docker exec -it backend-dev bash

To enter the frontend container shell, run the following command:

docker exec -it frontend-dev bash

Both containers automatically install all the necessary dev dependencies during build. So, as soon as containers are up, you are ready to start developing!

To install new dependencies, make sure to use dev container shell. It is also recommended to rebuild the dev container after adding new dependencies to ensure they are cached at the build stage making future builds faster. To rebuild the containers:

docker compose -f compose.dev.yml build --no-cache

Then start the containers:

docker compose -f compose.dev.yml up -d

Running the Django backend development server

1. Ensure development environment is set up.

2. Run the Whitebox server:

   docker exec -it backend-dev make run
   

This will start the Django development server on http://localhost:8000. Any changes you make to the backend will be automatically reloaded.

Plugins' JSX files are transpiled only on backend startup. If you make changes to the plugins' JSX code, you will need to restart the backend server for them to get re-transpiled, and then refresh the frontend app.

Running the React frontend development server

1. Ensure development environment is set up.

2. Run the frontend development server:

   docker exec -it frontend-dev make run
   

This will start the development server on http://localhost:3000. The development server has automatic reloading enabled, so you can see your changes in real-time.

This applies only the frontend project's code. In case you make changes to the plugins' JSX code, you will need to refresh the page for changes to take effect.

Running the RQ workers

1. Ensure development environment is set up.

2. Run the RQ worker:

   docker exec -it backend-dev make run-worker
   

This will start an RQ worker process that is going to process the tasks as they are added into the queue. If you need to run multiple workers for parallelized processing, you can run the run-worker command multiple times from multiple terminals.

Commands

Whitebox includes Makefiles with commands you can run inside the development containers for common development tasks.

Backend Commands

Setup Commands

These commands are for Whitebox setup, and need to be run inside the whitebox directory.

• setup-network: Configure network interface naming for Orange Pi.

  Usage:

  sudo make setup-network
  

  • Only needed for Orange Pi devices
  • Disables predictable network interface naming
  • Required for Wi-Fi access point setup

• setup-udev: Set up udev rules for SDR devices.

  Usage:

  sudo make setup-udev
  

  • Downloads and installs Stratux/SDR device udev rules
  • Reloads udev rules automatically

Development Commands

First, enter the container's shell:

docker exec -it backend-dev bash

and then you can use the following Makefile commands (format: make <COMMAND>):

• migrate: Apply database migrations
• run: Start the backend server for production
• run-dev: Start the backend server for development
• run-for-test: Start the backend server in test mode

• test: Run backend tests

• build_federation_modules: Builds federation modules for all the plugins that are available to Whitebox. Useful for quicker iterations during development.

• build_federation_modules_watch: Builds all plugin modules just as above, but also includes tests and watches for their file changes & rebuilds them

• download_external_assets: For whitebox to function offline, it needs to download external assets like videos, external JS libraries, etc. for both plugins and whitebox itself. This command downloads all the external assets

• clean: All the downloaded external assets will be removed. Whitebox will not function without these assets.

• makemigrations: Create migrations for all the plugins

• fmt: Format code using ruff

• run-mkdocs: Run the MkDocs server for documentation

Frontend Commands

First, enter the container's shell:

docker exec -it frontend-dev bash

and then you can use the following Makefile commands (format: make <COMMAND>):

• run: Start the frontend server for development
• run-for-test: Start the frontend server in test mode
• test: Run all frontend tests
  • unit_test: Run kernel's unit tests
  • integration_test: Run kernel's integration tests
  • federation_unit_test: Run plugins' unit tests (requires backend server running in test mode)
  • federation_integration_test: Run plugins' integration tests (requires backend server running in test mode)
  • test_e2e: Run end-to-end tests (requires frontend server running in test mode)
• lint: Run lint checkers on the code

Testing

Backend testing

Whitebox implements a custom test runner that allows for running tests for whitebox and plugins within a single test suite by discovering all plugin tests and loadin them dynamically.

To run tests, along with plugin tests, you need to make sure your plugin adheres to guidelines outlined in the Plugin Guide.

1. Ensure the development environment is set up.

2. Run the backend tests:

   docker exec -it backend-dev make test
   

Frontend testing

Whitebox uses Vitest (with Playwright in Browser Mode) for frontend unit testing, and Playwright for integration testing. To run frontend tests:

1. Ensure the development environment is set up.

2. Run the frontend tests:

   docker exec -it frontend-dev make test
   

This will run first the unit testing suite, and then the integration testing suite.

Federation testing

Federation modules require both frontend and backend to be running in order to test them - backend to build and serve the modules, and frontend to actually load and render the modules.

To run the federation tests, first run both servers in the testing mode (the two commands need to be run in separate terminals):

docker exec -it backend-dev make run-for-test
docker exec -it frontend-dev make run-for-test

Then, in a separate terminal, you can run the unit tests for the federation modules:

docker exec -it frontend-dev make federation_test

or, run the integration tests for the federation modules:

docker exec -it frontend-dev make federation_integration_test

End-to-end testing

Whitebox uses Playwright for end-to-end testing. The setup steps for end-to-end testing are slightly different from the other tests, as it requires running both the frontend and backend servers in conjunction.

As the frontend and backend are running in separate containers, and the tests will be run from the frontend container, the target hosts will be different—frontend will effectively live on localhost, while the backend will be on backend host (if using production containers), or backend-dev host (if using development containers).

For the tests to run successfully, you will need to ensure that the frontend application is configured to use the correct backend host from within the container.

To run end-to-end tests:

1. Ensure development environment is set up.
2. Run the backend environment that the frontend will use:

docker exec -it backend-dev make run

1. Run the frontend environment, ensuring that the backend host is set correctly for the frontend build:

docker exec -it frontend-dev make run-for-test

1. Run the test suite on the frontend container:

docker exec -it frontend-dev make e2e_test

Running the tests on LambdaTest

Whitebox uses LambdaTest for running frontend integration, and end-to-end tests on multiple browsers and devices.

When ran during CI, the end-to-end tests will be run against the sandbox environment, which is setup as one of the steps before the LambdaTest testing step. You can also run the tests on LambdaTest from localhost, if you have an account.

To run the tests on LambdaTest, you will need the following:

• LambdaTest username, later set as environment variable named LT_USERNAME
• LambdaTest access key, later set as environment variable named LT_ACCESS_KEY

Similarly to the standard end-to-end test run, you'll need to setup frontend and backend environments, and then run the tests against them:

1. Ensure development environment is set up.

2. Run the backend environment that the frontend will use:

   docker exec -it backend-dev make run
   

3. Run the frontend environment, ensuring that the backend host is set correctly for the frontend build:

   docker exec -it frontend-dev make run-for-test
   

4. Run the LambdaTest test suites on the frontend container:

   # Enter the frontend container shell
   docker exec -it frontend-dev bash
   
   # Set the LambdaTest credentials in the environment
   export LT_USERNAME=<your_lambdatest_username>
   export LT_ACCESS_KEY=<your_lambdatest_access_key>
   
   # Run the integration tests on LambdaTest
   yarn run test_integration:lambdatest-tunnel
   
   # Run the end-to-end tests on LambdaTest
   yarn run test_e2e:lambdatest-tunnel
   

Using *-tunnel commands, the tests will run using the LambdaTest tunnel which is preconfigured in the dev container. This allows for testing on local environments. This is useful when you want to run tests against the dev container running on your local machine.

You may also set environment variables like LT_USERNAME and LT_ACCESS_KEY in the .env file in the root of the project. Refer to the .env.example file for the required variables. This will allow you to run the tests from the host machine without having to set them each time like so:

docker exec -it frontend-dev env $(./get-env) npm run test_integration:lambdatest-tunnel

Ensure you do poetry install at the root of the project to install all the required dependencies for the get-env script to work.

If you want to run the tests without the tunnel, against a sandbox, you can run the integration tests with:

FRONTEND_URL="TARGET_URL_OF_THE_SANDBOX" yarn run test_integration:lambdatest

or end-to-end tests with:

FRONTEND_URL="TARGET_URL_OF_THE_SANDBOX" yarn run test_e2e:lambdatest

Debugging

For comprehensive debugging procedures including Insta360 connection troubleshooting, check the Troubleshooting section.

VSCode

1. Install VSCode and open the whitebox project in VSCode.
2. Add Python language support by installing the extension.
3. JS Language support and chrome debugger support is available by default.
4. If using firefox browser, you would need to install this extension.

Backend Debugging

1. Enter the backend dev container's shell: docker exec -it backend-dev bash
2. Run: make debug (server won't start, move to next steps)
3. Click the Run & Debug Logo located on the left sidebar.
4. From the drop-down next to the play button, select Debug Backend.
5. Click the play button to debug.

Frontend Debugging

1. Enter the frontend dev container's shell: docker exec -it frontend-dev bash
2. Run: make debug
3. Click the Run & Debug Logo located on the left sidebar.
4. From the drop-down next to the play button, select Debug Frontend (Chrome) OR Debug Frontend (Firefox).
5. Click the play button to debug.

PyCharm

Remote debugging on PyCharm works only on the Professional Edition and can take some work to fully set up. Community edition does not support remote debugging in any capacity.

You can find the detailed explanation and setup steps in PyCharm debugging.

Contributing

1. Fork the repository
2. Create a new branch for your feature
3. Make your changes
4. Run tests and ensure they pass
5. Submit a pull request

Google Docstring Conventions should be followed for all code documentation.

Documentation

Documentation for Whitebox is generated using MkDocs. To run documentation locally, make sure you have Whitebox repository cloned and set up as per the instructions above.

To run the documentation server:

1. Ensure development environment is set up.

2. Enter the backend container shell:

   docker exec -it backend-dev bash
   

   3. Run the documentation server:

   make run-mkdocs
   

Versioning

Whitebox uses Semantic Versioning for versioning. In the CI configuration, the update_version stage is responsible for updating the version of the project. Backend and frontend versions are kept in sync, with the backend's version used as a reference point.

When a merge request is merged to main, first, patch version will be bumped in the pyproject.toml file for backend, and that same version number will then be applied to package.json for frontend. This is done by the script located in packaging/scripts/maintenance/whitebox_update_version.py.

Afterward, a commit will be made with the new version, which will then be tagged, and the CI will push these changes to the repository.

Adding temporary dependencies to CI

Sometimes you need to add a temporary dependency to the CI environment that you do not want to be included in the project's dependencies upon merge. You may want to do this when you are working on Whitebox core and a plugin in parallel. In these cases, the plugins' changes would only be available on its own branch, which are not published in PyPI, so you have to install them directly from Git.

To do this, you can add the dependency to the temporary-dependencies Poetry group. Tests and sandbox will be run with these temporary dependencies installed (they take precedence over the "default" ones from the pyproject.toml file), but as it's an optional group, they won't be included in the final project dependencies. This change will be safe to merge, as the CI will perform the cleanup (for more info, take a look how the maintenance CI step works).

You can reference a Git branch directly in the poetry add command, by doing:

poetry add --group temporary-dependencies git+GIT_BRANCH#BRANCH_NAME

For example, to add a temporary dependency with Git URL https://gitlab.com/whitebox-aero/whitebox-plugin-gps-display.git, with a branch feature/my-new-feature, you can run:

poetry add --group temporary-dependencies git+https://gitlab.com/whitebox-aero/whitebox-plugin-gps-display.git#feature/my-new-feature

During development, dev containers automatically include the temporary dependencies group, so you can use them right away. However, if you want temporary dependencies in production, you need to set the TEMPORARY_DEPENDENCIES environment variable to 1 in .env file before building the production containers.

Next Steps

• Learn more about the Whitebox Architecture
• Check out the Development Guide