Interval Guided Regression Test Update

Interval-Guided Regression Test Update

Automatically update regression tests to maintain interval coverage when program code changes.

Core Concept

Intervals represent ranges of values that variables can take during execution. When code changes, intervals may be added, removed, or modified. This skill ensures regression tests continue to cover all important intervals.

Example:

# Old code
def process(x):
    if x < 10:
        return x * 2
    else:
        return x + 10

# Intervals: [0, 10), [10, ∞)

# New code (added negative handling)
def process(x):
    if x < 0:
        return -x
    elif x < 10:
        return x * 2
    else:
        return x + 10

# Intervals: (-∞, 0), [0, 10), [10, ∞)
# Need to add test for new interval: x < 0

Workflow

1. Analyze Existing Tests

Parse the current regression test suite:

• Identify test inputs
• Determine which intervals each test covers
• Calculate current interval coverage

# Example test analysis
test_process_small: input=5, covers [0, 10)
test_process_large: input=15, covers [10, ∞)
# Coverage: 2/2 intervals = 100%

2. Extract Interval Information

Obtain interval information from the updated program:

From static analysis:

• Parse conditionals (if, switch, etc.)
• Extract comparison operators
• Derive interval constraints

From profiling:

• Run existing tests
• Track observed value ranges
• Identify intervals exercised

From symbolic execution:

• Execute with symbolic values
• Collect path constraints
• Derive interval constraints

See references/interval-analysis.md for detailed extraction methods.

3. Identify Coverage Gaps

Compare current coverage with new intervals:

old_intervals = {[0, 10), [10, ∞)}
new_intervals = {(-∞, 0), [0, 10), [10, ∞)}

added_intervals = {(-∞, 0)}  # Need new test
removed_intervals = {}
modified_intervals = {}

4. Update Tests

Apply appropriate update strategies:

Add tests for new intervals:

# New test for (-∞, 0)
def test_process_negative():
    assert process(-5) == 5

Adjust inputs for modified intervals:

# If boundary changed from x < 10 to x < 20
# Old: test_process_large with input=15
# New: test_process_large with input=25

Update assertions for changed behavior:

# If logic changed
# Old: assert process(5) == 10
# New: assert process(5) == 15

Remove redundant tests:

# If intervals merged, remove duplicate coverage

See references/test-update-strategies.md for detailed strategies.

5. Validate Updated Tests

Run the updated test suite:

• Execute all tests
• Verify all pass
• Check interval coverage
• Identify any remaining gaps

6. Generate Coverage Report

Provide comprehensive summary:

• Before/after interval coverage
• Tests added, modified, removed
• Validation results
• Remaining gaps (if any)

Quick Start Examples

Example 1: New Interval Added

Scenario: Function adds handling for negative numbers

Original code:

def abs_double(x):
    if x < 10:
        return x * 2
    else:
        return x + 10

Original tests:

def test_small():
    assert abs_double(5) == 10

def test_large():
    assert abs_double(15) == 25

Updated code:

def abs_double(x):
    if x < 0:
        return -x * 2
    elif x < 10:
        return x * 2
    else:
        return x + 10

Updated tests:

def test_negative():  # NEW
    assert abs_double(-5) == 10

def test_small():
    assert abs_double(5) == 10

def test_large():
    assert abs_double(15) == 25

Summary:

• Added interval: (-∞, 0)
• Added test: test_negative
• Coverage: 2/2 → 3/3 (100%)

Example 2: Interval Boundary Changed

Scenario: Threshold value modified

Original code:

def categorize(score):
    if score < 60:
        return "fail"
    else:
        return "pass"

Original tests:

def test_fail():
    assert categorize(50) == "fail"

def test_pass():
    assert categorize(70) == "pass"

Updated code:

def categorize(score):
    if score < 50:  # Changed from 60
        return "fail"
    else:
        return "pass"

Updated tests:

def test_fail():
    assert categorize(40) == "fail"  # Changed input from 50

def test_pass():
    assert categorize(60) == "pass"  # Changed input from 70

Summary:

• Modified interval: [0, 60) → [0, 50)
• Modified interval: [60, ∞) → [50, ∞)
• Updated 2 test inputs
• Coverage: 2/2 → 2/2 (100%)

Example 3: Intervals Merged

Scenario: Simplified logic combines cases

Original code:

def classify(x):
    if x < 0:
        return "negative"
    elif x == 0:
        return "zero"
    else:
        return "positive"

Original tests:

def test_negative():
    assert classify(-5) == "negative"

def test_zero():
    assert classify(0) == "zero"

def test_positive():
    assert classify(5) == "positive"

Updated code:

def classify(x):
    if x <= 0:
        return "non-positive"
    else:
        return "positive"

Updated tests:

def test_non_positive():  # Merged
    assert classify(-5) == "non-positive"
    assert classify(0) == "non-positive"

def test_positive():
    assert classify(5) == "positive"

Summary:

• Merged intervals: (-∞, 0) and {0} → (-∞, 0]
• Removed test: test_zero (redundant)
• Updated test: test_negative → test_non_positive
• Coverage: 3/3 → 2/2 (100%)

Update Strategies

Strategy 1: Input Adjustment

Modify test inputs to cover new or changed intervals.

When to use:

• New interval added
• Interval boundary changed
• Need to cover previously uncovered interval

How:

1. Identify target interval
2. Select representative value from interval
3. Update test input
4. Verify test still valid

Strategy 2: Assertion Update

Modify expected values when behavior changes.

When to use:

• Logic changed within interval
• Return value modified
• Side effects changed

How:

1. Execute test with new code
2. Observe actual output
3. Verify output is correct
4. Update assertion

Strategy 3: Test Removal

Remove redundant or obsolete tests.

When to use:

• Intervals merged
• Multiple tests cover same interval
• Interval no longer exists

How:

1. Identify redundant tests
2. Keep most representative test
3. Remove others
4. Verify coverage maintained

Strategy 4: Test Generation

Create new tests for uncovered intervals.

When to use:

• New interval added
• Coverage gap identified
• No existing test covers interval

How:

1. Identify uncovered interval
2. Generate representative input
3. Determine expected output
4. Create new test

Coverage Analysis

Computing Coverage

Interval Coverage = (Covered Intervals) / (Total Intervals) × 100%

Example:

Total intervals: 4
Covered by tests: 3
Coverage: 75%