Caliber Testing

Caliber Testing

Critical

• All test files MUST be placed in __tests__/ directories parallel to source files: src/[module]/__tests__/[module].test.ts
• Register any new __tests__/ directories in vitest.config.ts's include glob (already configured: src/**/*.test.ts)
• NEVER mock src/test/setup.ts — it is the global LLM provider mock already applied to all tests
• Environment variable tests MUST save process.env in beforeEach, restore in afterEach, and explicitly delete env vars to test absence
• Temporary file/directory cleanup MUST happen in afterEach, not in individual test cleanup. Use fs.rmSync(dir, { recursive: true, force: true })
• When a test requires unmocking modules mocked in global setup, call vi.unmock('../module.js') BEFORE the import statement
• Run pnpm test locally and pnpm test:coverage before committing to verify coverage thresholds (lines: 50, functions: 50, branches: 50, statements: 50)

Instructions

Step 1: Create the test file in the correct directory

Create src/[module]/__tests__/[module].test.ts. The parent source file is src/[module]/[module].ts.

Verify: The __tests__ directory exists at the same level as the source file being tested.

Step 2: Import test framework

At the top of every test file, import from vitest:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';

Add additional imports based on what you're testing:

• File system: import fs from 'fs'; import path from 'path'; import os from 'os';
• Temporary files: Use fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-prefix-'))
• Exec: import { execSync } from 'child_process';

Verify: All required test utilities are imported before test definitions.

Step 3: Set up module mocking (if needed)

If testing a module that imports other modules you want to mock:

vi.mock('../config.js', () => ({
  loadConfig: vi.fn(),
  writeConfigFile: vi.fn(),
}));

For complex mocks with test-time factory functions (hoisted):

const { mockLoadConfig } = vi.hoisted(() => ({
  mockLoadConfig: vi.fn(),
}));

vi.mock('../config.js', () => ({
  loadConfig: () => mockLoadConfig(),
}));

For unmocking global setup mocks (e.g., to test llm/index.js itself):

vi.unmock('../index.js');

Place all vi.mock() and vi.unmock() calls BEFORE importing the module under test.

Verify: Mock declarations appear before the import of the module being tested.

Step 4: Organize tests in describe blocks

Group related tests with describe():

describe('functionName', () => {
  it('returns X when Y', () => {
    // test body
  });
});

Verify: Each it() test has a clear, complete assertion.

Step 5: Manage environment and process state

For tests that modify process.env or process.argv:

describe('config tests', () => {
  const originalEnv = process.env;
  const originalArgv = process.argv;

  beforeEach(() => {
    process.env = { ...originalEnv }; // Copy, not reference
    process.argv = [...originalArgv];
    delete process.env.SPECIFIC_VAR; // Explicitly remove vars to test absence
  });

  afterEach(() => {
    process.env = originalEnv;
    process.argv = originalArgv;
  });

  it('tests env var behavior', () => {
    process.env.MY_VAR = 'test';
    // test code
  });
});

Verify: beforeEach creates a copy of env/argv; afterEach restores originals; unused env vars are explicitly deleted with delete process.env.VAR.

Step 6: Manage temporary files and directories

For file system tests, create temporary directories and clean them up:

describe('file tree', () => {
  const dirs: string[] = [];
  
  afterEach(() => {
    for (const d of dirs) {
      try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
    }
    dirs.length = 0;
  });

  it('processes files', () => {
    const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-test-'));
    dirs.push(tmp);
    // use tmp directory
  });
});

Verify: All temporary directories are pushed to a cleanup array and removed in afterEach.

Step 7: Handle mock state cleanup

Before each test, clear mock call history to avoid pollution between tests:

beforeEach(() => {
  vi.clearAllMocks(); // Reset all mock call counts and return values
});

afterEach(() => {
  vi.restoreAllMocks(); // Restore original implementations
  vi.resetModules(); // Reset cached module imports (if you reload modules)
});

Verify: beforeEach calls vi.clearAllMocks() for providers and afterEach calls vi.restoreAllMocks().

Step 8: Test assertions

Write assertions using expect(). Match the patterns from existing tests:

// Simple checks
expect(value).toBe(expected);
expect(array).toContain(item);
expect(fn).toThrow('error message');

// Instance checks
expect(obj).toBeInstanceOf(ClassName);

// Mock checks
expect(mockFn).toHaveBeenCalledTimes(1);
expect(mockFn).toHaveBeenCalledWith(arg);

// File system checks
expect(fs.existsSync(path)).toBe(true);

Verify: Each test has at least one assertion and uses appropriate expect() matchers.

Step 9: Run tests

Run tests locally before committing:

pnpm test                # Run all tests in watch mode
pnpm test:coverage       # Check coverage thresholds
pnpm test -- src/my/path/__tests__/my.test.ts  # Run single test file

Verify: All tests pass and coverage thresholds are met (lines: 50%, functions: 50%, branches: 50%).

Examples

Example 1: Testing a module with environment variables (config.test.ts pattern)

User asks: "Write tests for my config loader that reads from env vars and a config file."

Actions taken:

1. Create src/lib/__tests__/config.test.ts
2. Import test framework and fs module
3. Mock fs and os
4. Save/restore process.env in beforeEach/afterEach
5. Delete specific env vars to test absence
6. Test each branch: env vars set, file exists, both, neither

Result:

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import fs from 'fs';

vi.mock('fs');
vi.mock('os', () => ({ default: { homedir: () => '/home/user' } }));

import { loadConfig } from '../config.js';

describe('config', () => {
  const originalEnv = process.env;

  beforeEach(() => {
    vi.clearAllMocks();
    process.env = { ...originalEnv };
    delete process.env.ANTHROPIC_API_KEY;
    delete process.env.OPENAI_API_KEY;
  });

  afterEach(() => {
    process.env = originalEnv;
  });

  it('returns env config when ANTHROPIC_API_KEY is set', () => {
    process.env.ANTHROPIC_API_KEY = 'sk-ant-test';
    const config = loadConfig();
    expect(config?.provider).toBe('anthropic');
  });

  it('returns null when no env vars set', () => {
    expect(loadConfig()).toBeNull();
  });
});

Example 2: Testing file system operations with temp cleanup (file-tree.test.ts pattern)

User asks: "Write tests for file tree analysis."

Actions taken:

1. Create src/fingerprint/__tests__/file-tree.test.ts
2. Set up a cleanup array for temp directories
3. Create temp dirs in each test and push to cleanup array
4. Clean up all temp dirs in afterEach
5. Test with various file mtimes and directory structures

Result:

import { describe, it, expect, afterEach } from 'vitest';
import fs from 'fs';
import path from 'path';
import os from 'os';
import { getFileTree } from '../file-tree.js';

const dirs: string[] = [];
afterEach(() => {
  for (const d of dirs) {
    try { fs.rmSync(d, { recursive: true, force: true }); } catch {}
  }
  dirs.length = 0;
});

describe('getFileTree', () => {
  it('returns files sorted by mtime descending', () => {
    const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'caliber-ft-'));
    dirs.push(tmp);

    fs.writeFileSync(path.join(tmp, 'a.ts'), 'a');
    fs.writeFileSync(path.join(tmp, 'b.ts'), 'b');
    const tree = getFileTree(tmp);
    expect(tree).toHaveLength(2);
  });
});