Test Runner Architecture

Tutorial and Example Test Architecture

Purpose

ISET repositories run tutorials and examples as smoke tests with one shared architecture. The architecture must:

• use the same return value and checkpoint format in every repository;
• survive MATLAB termination with a readable checkpoint;
• isolate scripts so figures, variables, and ISET session data do not accumulate;
• support explicit file skips through % SkipFile;
• produce one report format through ieTestReport; and
• require only thin repository-specific entry points.

ISETCam owns the shared implementation because ISETBio, ISET3D, and related repositories already depend on ISETCam.

Entry Points

The four entry points are:

• ieTutorialTest
• ieExampleTest
• isetbioTutorialTest
• isetbioExampleTest

All four entry points are thin configuration wrappers over the shared ieRunTutorialExampleTests engine. They return the same canonical run record, and checkpoint.mat contains that identical record.

ieTestReport accepts MATLAB unit-test results, a canonical run record, a checkpoint file, or a run directory. Temporary adapters remain for checkpoints and result arrays created by the pre-engine runners.

Public Interface

Shared execution engine

ISETCam provides one public execution function:

run = ieRunTutorialExampleTests(config)

config is a scalar struct with these required fields:

Optional fields are:

The engine supplies common defaults for discovery, % SkipFile, logging, cleanup, error capture, and reporting. Repository wrappers must not copy the engine's helper functions.

Thin repository wrappers

Each repository keeps two small public entry points in its top-level validate/ directory:

function run = isetbioExampleTest(varargin)
[selector,start] = localParseSelection(varargin{:});

config = struct();
config.repositoryName = 'ISETBio';
config.repositoryRoot = isetbioRootPath;
config.suiteKind = 'examples';
config.runnerName = mfilename;
config.selector = selector;
config.start = start;
config.skipPathPatterns = {'library'};

run = ieRunTutorialExampleTests(config);
end

ISETCam uses ieTutorialTest and ieExampleTest. Other repositories use <repository>TutorialTest and <repository>ExampleTest.

The public wrappers accept one optional name-value pair:

repositoryTutorialTest('selection','t_oneTutorial')
repositoryTutorialTest('start','t_firstTutorialToRun')

With no arguments, a wrapper runs the complete suite. selection runs only the named file, while start runs the named file and all subsequent files in the deterministically sorted plan.

Canonical Run Record

The function return value and the contents of checkpoint.mat are the same versioned run struct, so completed and interrupted runs report identically.

Required top-level fields:

schemaVersion
repositoryName
repositoryRoot
suiteKind
runnerName
targetDir
selector
state                 % Running or Completed
startedAt
lastEventAt
finishedAt
plannedFiles
currentIndex
currentFile
results
runDir
checkpointFile
progressFile

results is a struct array with this contract:

file
status                % Passed, Failed, or Skipped
error                 % compact error text for Failed, otherwise empty
startedAt
finishedAt
durationSeconds

If MATLAB terminates, state remains Running. ieTestReport interprets the difference between plannedFiles and results as unfinished work and reports currentFile as the last active file.

Increment schemaVersion only for incompatible checkpoint changes. ieTestReport should give a clear error for unsupported future versions.

Discovery and Selection

• Recursively discover t_*.m and s_*.m below the target directory.
• Sort by normalized relative path for deterministic execution.
• Do not execute .mlx files. Plain-text Live Code .m is the canonical source, and executing .mlx introduces name-shadowing and renderer crashes.
• A selector may be a bare stem, file name, relative path, or full path.
• start accepts the same forms and trims the selected plan so execution begins with that file.
• Duplicate stems are configuration errors and should be reported before the run begins.

Skipping

The common source marker is:

% SkipFile

The engine may continue recognizing % UTTBSkip during migration, but new files must use % SkipFile.

Skip decisions occur in this order:

1. common path exclusions;
2. repository skipPathPatterns;
3. % SkipFile source marker; and
4. conditionalSkipFcn for environment-dependent cases.

A skipped result should eventually include a machine-readable reason. Until that field is added, place the reason in nearby source comments and the progress log.

Script Isolation and Cleanup

Each script runs from its own directory inside a helper function workspace. The engine performs the same lifecycle before and after every script:

1. set wait bar off;
2. set init clear true;
3. call ieInit to close figures and reset vcSESSION;
4. execute the script and capture errors;
5. call ieInit again; and
6. flush pending graphics events.

Tutorial/example smoke runs should start in a dedicated MATLAB session because ieInit intentionally closes figures. Preferences changed by the engine are restored when the complete run exits normally.

No script may depend on variables or ISET objects created by an earlier script.