Resilient Context Extraction

Resilient Context Extraction

Objective

Prevent data hallucination and inefficiency by mandating that agents inspect, validate, and fully extract data from provided reference files before attempting web searches or generating synthetic data. When extraction is incomplete, use fallback strategies before making assumptions.

Critical Rule

If a reference file is provided in the task context, it is the source of truth. Do not fabricate data or search the web for information that may exist within the provided attachments. If extraction appears incomplete, attempt alternative methods before proceeding.

Workflow Steps

1. Scan Context for Attachments

At the start of every task, explicitly list all files provided in the context window or attachment panel.

• Check for spreadsheets (.xlsx, .csv), documents (.pdf, .docx, .pptx), or data dumps (.json, .txt, .xml, .yaml).
• Note the filename, file size (if available), and inferred content type.
• Record this list for later verification.

2. Evaluate Relevance

Determine if any provided file contains the data required to complete the task.

• Match Keywords: Do filenames or expected column headers match task requirements?
• Check Scope: Does the data cover the necessary timeframe or region?
• Prioritize: Rank files by likelihood of containing required data.

3. Extract Data with Validation

If relevant files are found:

• Read the file content using appropriate tools (e.g., read_file, pandas, pdf_reader).
• Validate Extraction Completeness (NEW CRITICAL STEP):
  • Check if output appears truncated (e.g., sudden cutoff mid-sentence, character limits hit).
  • Compare expected data points vs. extracted data points (e.g., "Task mentions pricing tiers; did extraction include pricing numbers?").
  • Look for structural indicators of incompleteness (e.g., unclosed tables, missing document endings).
• If extraction is incomplete or suspicious, proceed to Step 4 (Fallback Strategies) before using the data.

4. Apply Fallback Extraction Strategies

When initial extraction is incomplete or critical data is missing, attempt these strategies in order before making assumptions:

4a. Alternative Library/Tool Approach

• For .docx: Try python-docx directly via execute_code_sandbox if read_file was truncated.
• For .xlsx/.csv: Try pandas with explicit sheet selection or openpyxl directly.
• For .pdf: Try pdfplumber, PyPDF2, or shell-based pdftotext if available.
• For any format: Try reading as raw bytes/text via shell commands (cat, xxd, strings).

4b. Shell-Based Extraction

• Use shell_agent or run_shell for format-specific tools:
  • unzip -p file.docx word/document.xml | xmllint --format - (extract raw docx XML)
  • in2csv file.xlsx (convert Excel to CSV via shell)
  • pdftotext file.pdf - (extract PDF text via command line)
• This bypasses potential sandbox or library limitations in read_file.

4c. Targeted Re-Reading

• If specific sections are missing (e.g., "pricing table on page 3"), attempt to re-read with focus on that region.
• Use tools that support page/section ranges if available.

4d. Document What's Missing

• Before any assumptions or external searches, explicitly document:
  • What data was expected but not found.
  • What extraction methods were attempted.
  • Why each method failed or what portions remain unavailable.
• Example: "Attempted read_file on Pricing_email.docx (truncated at 980 chars). Attempted python-docx extraction via sandbox—pricing table in section 2 not present in file. Pricing numbers unavailable from provided context."

5. Cite Source Explicitly

When presenting data in the final output:

• Explicitly state which file the data came from.
• Example: "According to Massabama_active_listings.xlsx..."
• If fallback methods were used, note this: "Extracted via shell-based pdftotext from report.pdf..."
• This verifies to the user that real data was used, not hallucinated.

6. Fallback to Search (Last Resort Only)

If all extraction strategies fail to provide the specific data needed:

• State clearly what was missing from the files.
• Document all extraction methods attempted and why they failed.
• Then proceed with web search or estimation.
• Mark any non-file data clearly as "External Search" or "Estimated (file data unavailable)".
• Never present estimated data as if it came from the file.

Checklist

• Did I list all attached files with their types?
• Did I open and read the relevant files?
• Did I validate extraction completeness (check for truncation, missing sections)?
• If extraction was incomplete, did I try at least one fallback strategy?
• Did I explicitly document what data is unavailable before making assumptions?
• Did I cite the file name (and extraction method if non-standard) in my output?
• Did I avoid fabricating numbers that should have come from the file?
• Is any external/estimated data clearly marked as such?