Pioreactor Calibration Dev

Pioreactor Calibration Dev

Goal

Help Codex change Pioreactor calibration protocols and session flows without breaking the storage, CLI, UI, or web-task contracts.

High-level terminology

0. A calibration relates two variables together. One variable is what we can vary and the other is the target.
1. The calibration relates them via a calibration curve, which is a mapping between the variables.
2. The Pioreactor has many devices that can be calibrated. A device can have multiple calibrations, but only one can be active at a time.
3. A protocol creates a calibration for a device. Devices can have multiple protocols.
4. A calibration can apply to multiple devices. Example: the same calibration can be used for waste and media pumps.

In practice, calibrations are stored as YAML files in ~/.pioreactor/storage/calibrations/<device>/.

Tools

• Use scripts/generate_calibration_flow_graphs.py to generate protocol directed graphs.

Registries (authoritative maps)

• calibration_protocols in core/pioreactor/calibrations/registry.py: maps device -> protocol_name -> CalibrationProtocol class. Populated on subclassing.
• StepRegistry in core/pioreactor/calibrations/session_flow.py: maps step_id -> SessionStep class per protocol.
• calibration_actions in core/pioreactor/web/tasks.py: maps action_name -> handler that returns a Huey task, label, and normalizer.

Workflow

1. Confirm target device and protocol name.
2. Implement calibration routine to return the correct structs.*Calibration type.
3. Define session steps as SessionStep subclasses with step_id, render(ctx) -> CalibrationStep, and advance(ctx) -> SessionStep | None.
4. Provide a StepRegistry mapping {step_id: StepClass} for the protocol.
5. Expose:
   • start_<protocol>_session(...) -> CalibrationSession
6. Define a CalibrationProtocol subclass with:
   • target_device, protocol_name
   • title, description, requirements for the UI protocol list
   • step_registry and start_session(...)
   • optional on_session_abort(session, executor=None) for cleanup when UI abort is pressed
7. Register the protocol in core/pioreactor/calibrations/__init__.py and CLI entrypoint if required.
8. The UI protocol list is served by /api/workers/<unit>/calibration_protocols (task response). There is no static frontend registry anymore. Update the frontend only if new step types are required.

Session Flow (UI + API)

• Session engine: core/pioreactor/calibrations/session_flow.py.
• Steps are classes, not functions. Keep each render and advance single-purpose.
• Use SessionContext for ctx.data and ctx.complete(...) results.
• Store calibrations via ctx.store_calibration(calibration, device) so UI can link.
• Store estimators via ctx.store_estimator(estimator, device) so UI can link.
• ctx.read_voltage() uses the read_aux_voltage action.
• For UI sessions, hardware actions must run in Huey tasks via SessionContext.executor.
• _execute_calibration_action uses the calibration_actions registry with a fixed 300s timeout.

Persistence Contract (Critical)

• Treat persistence from SessionStep.advance() as mode-sensitive:
  • UI mode runs in Flask (www-data context).
  • Huey tasks run as pioreactor and are the correct place for privileged writes.
• In UI sessions, always persist through SessionContext helpers:
  • calibrations: ctx.store_calibration(calibration, device)
  • estimators: ctx.store_estimator(estimator, device)
• Do not call save_to_disk_for_device(...) directly from UI SessionStep.advance().
• Do not call set_as_active_calibration_for_device(...) directly from UI SessionStep.advance().
• Why: direct writes from UI steps can create ownership drift (www-data-owned YAMLs) and bypass the session executor/action pipeline.
• Call chain (UI):
  • ctx.store_calibration / ctx.store_estimator
  • unit_calibration_sessions_api._execute_calibration_action(...)
  • core/pioreactor/web/tasks.py registered action (save_calibration / save_estimator)
  • *.save_to_disk_for_device(...) in Huey task context
• CLI sessions have no executor; direct save/set-active is acceptable there.
• Recommended branch pattern:

if ctx.mode == "ui":
    save_result = ctx.store_estimator(estimator, device)
    saved_path = save_result["path"]
else:
    saved_path = estimator.save_to_disk_for_device(device)
    estimator.set_as_active_calibration_for_device(device)

• If you introduce a new persisted artifact type, register a calibration action in core/pioreactor/web/tasks.py and route persistence through the executor in UI mode.

Abort Cleanup Hook (on_session_abort)

• CalibrationProtocol supports optional:

  • @classmethod
  • def on_session_abort(cls, session: CalibrationSession, executor=None) -> None

• This is invoked by the unit calibration session abort route before the session is marked aborted.

• Prefer graceful, cooperative shutdown over process killing:

  • publish b"disconnected" to pioreactor/<unit>/<experiment>/<job_name>/$state/set
  • avoid PID-based kill for jobs launched within Huey task workers.

• If your protocol starts jobs via /unit_api/jobs/run/job_name/..., store the exact experiment used in session.data. Also specify the JOB_SOURCE in the payload:

    experiment = get_testing_experiment_name()
    job_source = f"fir_bias_trim_{session_id}"
  
    payload = {
        "args": [],
        "options": {
            "automation_name": "thermostat",
            "target_temperature": target_temperature,
        },
        "env": {
            "EXPERIMENT": experiment,
            "JOB_SOURCE": job_source,
        },
        "config_overrides": [],
    }
    response = post_into(
        address,
        "/unit_api/jobs/run/job_name/temperature_automation",
        json=payload,
        timeout=8,
    )

• Aggregate cleanup failures into one ValueError message when multiple resources fail to stop, so the abort route can surface useful diagnostics.

Field builders for forms in SessionStep

Protocol steps build forms using fields.* helpers from session_flow.py. These correspond to SessionInputs.* accessors used in advance() methods.

• fields.str(name, label=None, default=None) -> ctx.inputs.str(name, default=..., required=...)
  • field_type: string
  • Use for names, labels, and free-form text.
• fields.choice(name, options, label=None, default=None) -> ctx.inputs.choice(...)
  • field_type: choice
  • Validates input against the provided options.
• fields.float_list(name, label=None, default=None) -> ctx.inputs.float_list(...)
  • field_type: float_list
  • Accepts a list of numbers or a comma-separated string.
• fields.bool(name, label=None, default=None) -> ctx.inputs.bool(...)
  • field_type: bool
  • Accepts true/false and common yes/no string inputs.
• fields.float(name, label=None, minimum=None, maximum=None, default=None) -> ctx.inputs.float(...)
  • field_type: float
  • Enforces numeric bounds when provided.
  • Supports custom bound errors: min_error_msg and max_error_msg (stored on CalibrationStepField).
• fields.int(name, label=None, minimum=None, maximum=None, default=None) -> ctx.inputs.int(...)
  • field_type: int
  • Enforces integer bounds when provided.
  • Supports custom bound errors: min_error_msg and max_error_msg (stored on CalibrationStepField).