Dpdisp Submit

dpdisp-submit

This Skill guides the Agent to use the DPDispatcher tool to convert Shell commands into computational jobs and submit them to local machines or High-Performance Computing (HPC) clusters (supporting environments such as Shell, Slurm, PBS, LSF, Bohrium, etc.).

Prerequisites: uv and internet access.

Syntax & Protocol

This section defines the field mappings, variable syntax, and special flags for the configuration file.

Protocol Acquisition (Initialize)

As an Agent, before gathering information and building the configuration, you MUST FIRST execute the following command to read and learn the latest Schema protocol specifications and requirements:

uvx --with dpdispatcher dargs doc dpdispatcher.entrypoints.submit.submission_args

Field Mapping

You must accurately translate the gathered user requirements into the following core JSON hierarchy:

• machine: Defines the execution location and method (connection protocol, scheduler type).
• resources: Defines the computational resource requirements (nodes, CPUs, GPUs, queues, etc.).
• task_list: Defines the specific Shell commands to be executed and the file flow paths.

Environment Variable Syntax & Injection Rules

If user-specified values (e.g., username, Token, remote path) need to be read from local environment variables, you must strictly use the ${ENV_VAR_NAME} format in the template.

• Example: "remote_root": "${USER_HPC_WORKSPACE}"

Reference & Reuse

The protocol allows the use of the {"$ref": "other.json"} syntax to nest and reference reusable configuration snippets from other JSON files (the referenced file is loaded first, and then the current file's fields override or extend it). The relative path for $ref is resolved relative to the execution directory where submission.json is located. You must ensure that the execution path strictly matches the path pointed to by $ref.

Path Resolution Rules

• Base Directory (work_base): Defines the base working directory level for all tasks, typically set to . (i.e., the current execution directory).
• Task and File Path Resolution: task_work_path is resolved relative to work_base, whereas the file paths specified in forward_files are strictly resolved relative to task_work_path.

Dry-Run Testing (--dry-run)

Parses the configuration, generates local directories, and validates the Schema, but DOES NOT actually submit the job to the machine or cluster. You can use this flag for a final safety check before real execution.

Execution Workflow

As an Agent, you MUST strictly execute tasks in the sequence of the following stages, without skipping any steps:

Information Gathering

When feeling vague or uncertain about the specific parameters and configuration information for running the job, you MUST proactively ask the user in natural language to supplement the necessary information.

Secure Build

You MUST generate the configuration file based on the acquired Schema protocol and the gathered information.

• Pure Static Configuration: If no environment variable injection is needed, directly generate the final submission.json.
• Environment Variable Injection Required:
  • You must generate a submission.template.json file, using the ${VAR_NAME} syntax ONLY for the variables that need to be replaced.
  • You must use the envsubst command and explicitly list the variables to be replaced to prevent unrelated $… symbols in the JSON (such as "$ref") from being accidentally expanded.
  • Example:

    envsubst '${USER_HPC_WORKSPACE} ${USER_OTHER_VAR}' < submission.template.json > submission.json

Validate & Submit

You MUST choose the command chain that matches the configuration and execute it in sequence. Keep the same choice for every later resume or synchronization run.

For a configuration without $ref, keep external references disabled:

# Logic and Schema Validation
uvx --with dpdispatcher dargs check -f dpdispatcher.entrypoints.submit.submission_args submission.json
# Submit Job
uvx --from dpdispatcher dpdisp submit submission.json

For a configuration that uses $ref, pass --allow-ref to both validation and submission; omitting it makes a valid referenced configuration fail:

# Logic and Schema Validation
uvx --with dpdispatcher dargs check --allow-ref -f dpdispatcher.entrypoints.submit.submission_args submission.json
# Submit Job
uvx --from dpdispatcher dpdisp submit --allow-ref submission.json

Reporting Standard

After execution finishes, you MUST output a structured report to the user with the following fixed elements:

• Task Summary: Briefly describe the user's request (execution location, executed command, allocated resources).
• Current State: Explicitly point out the status of the job (started / running / finished / failed).
• Artifact Path: Explicitly point out the location of the output files (for example, when task_work_path is ., point out the specific paths of log and err).
• Exception Guidance: If the job encounters an interruption or partial failure, provide the user with detailed issue information and execute according to the user's further instructions.

Long-Running Jobs

High-performance computing tasks usually take an extremely long time (from hours to weeks), and there is a long time gap between the submission command and the final result. This is not a one-off, instant Q&A process, and you must choose the appropriate disconnect-prevention execution mode based on the specific scenario:

Blocking Mode

• Wrap in tmux: Run the standard dpdisp submit submission.json. The program will continuously hang and wait until the job is truly finished calculating on the cluster and the files are downloaded back before exiting. You must run it inside a tmux session to prevent any possible disconnection from killing the process.