Community Intelligence Research

Community Intelligence Methodology

Extract behavioral specifications from user-generated content: tutorials, blog posts, reviews, forum threads, issue reports, video transcripts, and changelogs. These are external observations of behavior by real users — zero structural contamination, pure behavioral surface.

When to Use This Mode

Use Community Intelligence mode when:

• The target product has active users who write about it
• Official documentation has gaps (community fills in what docs miss)
• You need observed behavior to corroborate or contradict official claims
• Edge cases, defaults, and undocumented behavior need coverage
• Version-specific behavioral changes need tracking

This mode runs independently of all other intelligence sources. It requires only web access. All output is public origin and goes to workspace/public/community/.

Why Community Content Matters

Official documentation describes intended behavior. Community content describes observed behavior. The gap between the two is where edge cases, undocumented features, surprising defaults, and breaking changes live.

A tutorial author who writes "when I ran tool --flag, it output X" has performed a runtime observation. A GitHub issue that says "expected Y but got Z" documents a behavioral contract violation. A blog post walkthrough that shows step-by-step output is equivalent to a test vector recorded by a human.

1. Search Channels

Search proceeds across six channels. Execute multiple patterns per channel before moving on.

digraph community_search {
    rankdir=TB;

    "Start community research" [shape=doublecircle];
    "Channel 1: Tutorials & blog posts" [shape=box];
    "Channel 2: Forums & Q&A" [shape=box];
    "Channel 3: Issue trackers" [shape=box];
    "Channel 4: Reviews & marketplace" [shape=box];
    "Channel 5: Version-specific content" [shape=box];
    "Channel 6: Video content" [shape=box];
    "Diminishing returns?" [shape=diamond];
    "Source budget exhausted?" [shape=diamond];
    "Build consensus analysis" [shape=box];
    "Write claims, consensus, gaps" [shape=box];
    "Research complete" [shape=doublecircle];

    "Start community research" -> "Channel 1: Tutorials & blog posts";
    "Channel 1: Tutorials & blog posts" -> "Channel 2: Forums & Q&A";
    "Channel 2: Forums & Q&A" -> "Channel 3: Issue trackers";
    "Channel 3: Issue trackers" -> "Channel 4: Reviews & marketplace";
    "Channel 4: Reviews & marketplace" -> "Channel 5: Version-specific content";
    "Channel 5: Version-specific content" -> "Channel 6: Video content";
    "Channel 6: Video content" -> "Diminishing returns?";
    "Diminishing returns?" -> "Build consensus analysis" [label="yes"];
    "Diminishing returns?" -> "Source budget exhausted?" [label="no"];
    "Source budget exhausted?" -> "Build consensus analysis" [label="yes"];
    "Source budget exhausted?" -> "Channel 1: Tutorials & blog posts" [label="no, continue"];
    "Build consensus analysis" -> "Write claims, consensus, gaps";
    "Write claims, consensus, gaps" -> "Research complete";
}

Channel 1: Tutorials & Blog Posts

Channel 2: Forums & Q&A

Channel 3: Issue Trackers

Issue tracker content is extremely high-value because bug reports document the gap between expected and actual behavior — both are behavioral specifications.

Channel 4: Reviews & Marketplace

Channel 5: Version-Specific Content

Version-specific content is critical for understanding behavioral evolution and resolving contradictions between community sources (different versions may explain different observed behaviors).

Channel 6: Video Content

For video results, fetch the page to extract title, description, and any transcript/caption data available. Video descriptions often contain command sequences and expected outputs.

Source Prioritization

1. Tutorials with concrete examples (commands + output) over opinion pieces
2. Recent content over old (behavioral changes over time)
3. Multiple independent authors confirming same behavior over single sources
4. Bug reports with "steps to reproduce" over vague complaints
5. Content matching target version over other versions

Termination Criteria

Stop when ALL of the following are true:

1. At least 3 search patterns executed per channel
2. At least 10 distinct tutorial/blog sources reviewed
3. At least 10 Stack Overflow threads reviewed
4. At least 5 GitHub issues reviewed (if issue tracker exists)
5. No new behavioral claims from the last 5 sources fetched (diminishing returns)

Source budget: 75 sources per agent run. Community sources are individually smaller than documentation pages but more numerous. Record unfetched sources in gaps.md.

2. Fetching Rules

Rate Limiting

• Insert at least 2 seconds between consecutive fetches to the same domain.
• On HTTP 429, respect Retry-After header. If absent, wait 30 seconds.
• Maximum 3 retries per URL. After 3 failures, record in gaps.md.

Content Handling

• Convert HTML to markdown for storage.
• For forum threads, ensure the full thread is captured (not just the question).
• For multi-page tutorials, follow "next" / "part 2" links.
• Track visited URLs to avoid cycles.

Sources to Skip

• Paywalled or authenticated content: record in gaps.md.
• Non-English content: record URL and language in gaps.md.
• Content clearly about a different product with the same name.
• Pure marketing with no behavioral content.

3. Extraction Methodology

High-Value Observations (EXTRACT)

Low-Value Content (SKIP)