Perforatedai Analyze

PerforatedAI Results Analysis Skill

Overview

This skill analyzes completed PerforatedAI training runs and provides optimization recommendations based on the training outputs.

When to use this skill:

• After your dendritic training has completed
• When you want to understand how dendrites impacted performance
• To get recommendations for improving future training runs

For other PerforatedAI tasks:

• Setup/Integration: Say "Perforate my model" (uses perforatedai skill)
• Debugging: Say "Debug my perforated model" (uses perforatedai skill)

Entry Point: "Analyze my perforated results"

When the user says "Analyze my perforated results", perform a comprehensive analysis of their PAI training outputs.

Step 1: Locate Result Files

Find the save_name from their training script by searching for the UPA.perforate_model() call. The save_name parameter shows where results are stored.

Only ask "What was your save_name?" if:

• The script has a variable or argument for save_name that could change
• You cannot find the perforate_model call in their script

The results are stored in: {save_name}/{save_name}_*.csv

Look for these files:

• {save_name}/{save_name}_scores.csv - Validation scores over epochs
• {save_name}/{save_name}_best_arch_scores.csv - Best architecture performance at each dendrite count
• {save_name}/{save_name}_switch_epochs.csv - Epochs when dendrites were added
• {save_name}/{save_name}_learning_rate.csv - Learning rate schedule
• {save_name}/{save_name}Best_PBScores.csv - Perforated Backpropagation scores (if enabled)
• {save_name}/*noImprove_lr* files - Check if these exist (indicates no dendrites were added)
• {save_name}/{save_name}_train_scores.csv - Training scores if tracked (for overfitting detection)

Step 2: Read and Analyze Score Files

Read all available CSV files and analyze:

From {save_name}_scores.csv:

• Validation score progression over epochs
• Calculate overall improvement from baseline to final
• Identify training phases (neuron mode vs dendrite mode)
• Check for plateaus or instabilities

From {save_name}_switch_epochs.csv:

• Identify exactly when dendrites were added (epoch numbers)
• Correlate dendrite additions with score changes from _scores.csv
• Determine if dendrite additions coincided with improvements
• Check if dendrites were added before actual plateau: Look at score history before each switch - if scores were still improving significantly when dendrites were added, they may have been added prematurely

Check for noImprove_lr files:

• If noImprove_lr files exist in the save folder: This means dendrites were NEVER added during training
• This is a critical issue - training completed without ever attempting to add dendrites
• See troubleshooting section for loop settings recommendations

From {save_name}_train_scores.csv (if exists):

• Compare training scores vs validation/test scores
• If training scores significantly better than validation/test: Indicates overfitting
• Calculate the gap between train and validation performance
• Recommend regularization if gap is large (>5-10% difference)

From {save_name}_best_arch_scores.csv:

• Compare performance across different dendrite counts (0, 1, 2, 3, etc.)
• If test tracking was enabled: This file will contain test scores for each architecture, not just your final one
• Identify if there are diminishing returns after a certain number of dendrites
• Determine optimal dendrite count: If score gains plateau or decrease after N dendrites, recommend setting max_dendrites to that value
• Example: If dendrites 0→1→2→3 show good gains but 4→5 show minimal improvement, recommend max_dendrites=3

From {save_name}_learning_rate.csv:

• Show learning rate schedule
• Correlate LR changes with score changes
• Identify if LR scheduling worked well with dendrite additions

From {save_name}Best_PBScores.csv (if exists):

• Analyze correlation scores for each module
• Identify modules with low correlation scores (< 0.3 or bottom 20%)
• Recommendation: If certain modules show very low correlation scores, they may not benefit from dendrites
  • Suggest adding those module IDs to append_module_ids_to_track() to skip them
  • Focus dendrite resources on high-correlation modules instead

Step 3: Generate Insights

Provide a comprehensive summary including:

1. Training Summary:

   • Initial score vs final score (% improvement)
   • Total epochs trained
   • Number of dendrites added
   • Training phases observed

2. Performance Analysis:

   • Best validation score achieved
   • Score stability (variance in later epochs)
   • Whether training converged properly

3. Dendrite Impact:

   • CRITICAL: Check if dendrites were added at all (absence of switch_epochs data or presence of noImprove_lr files)
   • Score improvement after each dendrite addition (from switch_epochs.csv + scores.csv)
   • Which dendrite additions had the most impact
   • Whether dendrites helped or hurt performance
   • Optimal dendrite count based on diminishing returns in best_arch_scores
   • Timing of additions: Were dendrites added prematurely (before plateau) or at appropriate times?

4. Generalization Analysis (if training scores available):

   • Train vs validation/test score gap
   • Whether the model is overfitting (train >> val/test)
   • If overfitting is present, recommend regularization strategies

5. Comparison to Baseline:

   • Baseline score is the first score in best_arch_scores
   • Calculate parameter count increase (if available)
   • Assess efficiency: did dendrites provide good accuracy/parameter ratio?

6. Module-Level Analysis (if PB scores available):

   • Which modules benefited most from dendrites (high correlation)
   • Which modules should be excluded (low correlation)
   • Parameter distribution: Which layers received the most dendrites?

Step 4: Show Visualizations

Tell them: "Training visualizations have been automatically generated at {save_name}/{save_name}.png. This shows:

• Score progression over epochs
• Learning rate schedule
• Dendrite addition timeline
• Architecture performance comparison"

Step 5: Next Steps

Based on the analysis results:

If training went well (dendrites improved performance):

Say: "Your dendritic training was successful! Here's what I found worked well and recommendations for optimization."

Then direct them to the Optimization Recommendations section below.

If training had issues (dendrites didn't help or training was unstable):

Say: "I see some issues in your training results. Let's troubleshoot:"

• If NO dendrites were added (noImprove_lr files exist or switch_epochs.csv is empty):

  • This means training completed without ever attempting dendrite addition
  • Most likely cause: Your training loop exited before PAI could detect a plateau and attempt adding dendrites
  • Recommend: Verify your training loop structure:
    • CRITICAL: Training should use an infinite loop (while True:) with PAI's training_complete flag controlling when to exit
    • The loop should only break when training_complete returns True after add_validation_score()
    • Check set_n_epochs_to_switch() allows enough time (e.g., 20-30 epochs per phase)
    • Check set_n_epochs_for_switch_history() - needs sufficient history to detect plateau (e.g., 10 epochs)
  • Example problem: If you used for epoch in range(50): instead of while True:, training may have ended before PAI finished
  • Correct pattern:

    epoch = -1
    while True:
        epoch += 1
        # training code
        model, restructured, training_complete = GPA.pai_tracker.add_validation_score(val_score, model)
        if training_complete:
            break

  • The PAI system needs: warmup epochs + history epochs + time to detect plateau before attempting dendrite addition
  • To control/minimize training time with infinite loop:
    • Use GPA.pc.set_max_dendrites(N) to limit how many dendrites are added (e.g., set_max_dendrites(3) stops after 3 dendrites)
    • Training will complete faster since fewer dendrite phases are needed
    • Optionally use FIXED_SWITCH_MODE for more consistent/predictable training time:

      GPA.pc.set_when_to_switch_mode("FIXED_SWITCH_MODE")
      GPA.pc.set_n_epochs_to_switch(20)  # Adds dendrite every 20 epochs

    • With FIXED mode, you know exactly when dendrites are added, making total training time predictable

• If dendrites didn't improve performance:

  • Check if you're converting the right layers
  • Is improvement_threshold too strict? Try [0]
  • Try different input_dimensions or module configurations
  • Consider using the perforatedai skill to debug: say "Debug my perforated model"

• If training was unstable:

  • Consider reducing learning rate
  • Adjust candidate_weight_initialization_multiplier lower (0.01 instead of 0.1)
  • Try a different scheduler
  • Check for dimension mismatches
  • Consider using the perforatedai skill to debug: say "Debug my perforated model"

---

Optimization Recommendations

When dendritic training has successfully improved performance, provide targeted recommendations based on the analysis:

1. Optimize Dendrite Count

Based on best_arch_scores.csv analysis:

If diminishing returns detected:

• Example: Dendrites 0→1→2→3 showed gains of +5%, +3%, +2%, but 4→5 showed only +0.1%
• Recommend: GPA.pc.set_max_dendrites(3) to focus resources on high-impact dendrites
• Explain: "Your results show the biggest improvements came from the first 3 dendrites. Setting max_dendrites=3 will make training more efficient without sacrificing performance."

If all dendrites contributed equally:

• Keep current max_dendrites or increase slightly
• Suggest running longer to see if more dendrites could help

2. Adjust Improvement Threshold and Switch Timing

Based on score progression from scores.csv and switch_epochs.csv:

If dendrites were added too frequently:

• Current threshold may be too lenient
• Recommend: Tighten threshold, e.g., [0.02, 0.01, 0.001, 0] instead of [0.01, 0.001, 0.0001, 0]
• This makes dendrite additions more selective

If dendrites were added BEFORE scores actually plateaued:

• Look at score history in the epochs before each dendrite addition
• If scores were still improving significantly (e.g., +2% in the last few epochs), dendrites were added prematurely
• Premature dendrite addition wastes capacity - the base network could have improved more first
• Recommend for FIXED switch mode: Increase set_n_epochs_to_switch() to give more time before adding dendrites
• Recommend for HISTORY switch mode:
  • Raise the improvement threshold (e.g., from [0.001] to [0.005] or [0.01])
  • Increase set_n_epochs_for_switch_history() to require longer plateau (e.g., from 10 to 15 epochs)
• Goal: Only add dendrites when base network has truly plateaued

If dendrites were rarely added but helpful:

• Threshold may be too strict
• Recommend: Relax threshold or set to [0] to always try adding dendrites when performance plateaus