Promql Validator

How This Skill Works

This skill performs multi-level validation and provides interactive query planning:

1. Syntax Validation: Checks for syntactically correct PromQL expressions
2. Semantic Validation: Ensures queries make logical sense (e.g., rate() on counters, not gauges)
3. Anti-Pattern Detection: Identifies common mistakes and inefficient patterns
4. Optimization Suggestions: Recommends performance improvements
5. Query Explanation: Translates PromQL to plain English
6. Interactive Planning: Helps users clarify intent and refine queries

Workflow

When a user provides a PromQL query, follow this workflow:

Working Directory Requirement

Run validation commands from the repository root so relative paths resolve correctly:

cd "$(git rev-parse --show-toplevel)"

If running from another location, use absolute paths to scripts/ files.

Step 1: Validate Syntax

Run the syntax validation script to check for basic correctness:

python3 devops-skills-plugin/skills/promql-validator/scripts/validate_syntax.py "<query>"

Output parsing notes:

• Exit 0: syntax valid
• Exit non-zero: syntax failure; include stderr and pinpoint token/position
• Prefer quoting the smallest failing fragment, then provide corrected query

The script will check for:

• Valid metric names and label matchers
• Correct operator usage
• Proper function syntax
• Valid time durations and ranges
• Balanced brackets and quotes
• Correct use of modifiers (offset, @)

Step 2: Check Best Practices

Run the best practices checker to detect anti-patterns and optimization opportunities:

python3 devops-skills-plugin/skills/promql-validator/scripts/check_best_practices.py "<query>"

Output parsing notes:

• Treat script sections as independent findings (cardinality, metric-type misuse, regex misuse, etc.)
• If script output is empty but query is complex, add a manual sanity pass and mark it as manual-review
• Preserve script wording for finding labels, then add remediation in plain English

The script will identify:

• High cardinality queries without label filters
• Inefficient regex matchers that could be exact matches
• Missing rate()/increase() on counter metrics
• rate() used on gauge metrics
• Averaging pre-calculated quantiles
• Subqueries with excessive time ranges
• irate() over long time ranges
• Opportunities to add more specific label filters
• Complex queries that should use recording rules

Step 3: Explain the Query

Parse and explain what the query does in plain English:

• What metrics are being queried
• What type of metrics they are (counter, gauge, histogram, summary)
• What functions are applied and why
• What the query calculates
• What labels will be in the output
• What the expected result structure looks like

Required Output Details (always include these explicitly):

**Output Labels**: [list labels that will be in the result, or "None (fully aggregated to scalar)"]
**Expected Result Structure**: [instant vector / range vector / scalar] with [N series / single value]

Example:

**Output Labels**: job, instance
**Expected Result Structure**: Instant vector with one series per job/instance combination

Line-Number Citation Method (Required)

When citing examples/docs in recommendations, include file path + 1-based line numbers:

examples/good_queries.promql:42
docs/best_practices.md:88

Rules:

• Cite the most relevant single line (or start line if multi-line snippet)
• Keep citations tight; do not cite full files
• If line numbers are unavailable, state line number unavailable and provide file path

Step 4: Interactive Query Planning (Phase 1 - STOP AND WAIT)

Ask the user clarifying questions to verify the query matches their intent:

1. Understand the Goal: "What are you trying to monitor or measure?"

   • Request rate, error rate, latency, resource usage, etc.

2. Verify Metric Type: "Is this a counter (always increasing), gauge (can go up/down), histogram, or summary?"

   • This affects which functions to use

3. Clarify Time Range: "What time window do you need?"

   • Instant value, rate over time, historical analysis

4. Confirm Aggregation: "Do you need to aggregate data across labels? If so, which labels?"

   • by (job), by (instance), without (pod), etc.

5. Check Output Intent: "Are you using this for alerting, dashboarding, or ad-hoc analysis?"

   • Affects optimization priorities

IMPORTANT: Two-Phase Dialogue

After presenting Steps 1-4 results (Syntax, Best Practices, Query Explanation, and Intent Questions):

⏸️ STOP HERE AND WAIT FOR USER RESPONSE

Do NOT proceed to Steps 5-7 until the user answers the clarifying questions. This ensures the subsequent recommendations are tailored to the user's actual intent.

Step 5: Compare Intent vs Implementation (Phase 2 - After User Response)

Only proceed to this step after the user has answered the clarifying questions from Step 4.

After understanding the user's intent:

• Explain what the current query actually does
• Highlight any mismatches between intent and implementation
• Suggest corrections if the query doesn't match the goal
• Offer alternative approaches if applicable

When relevant, mention known limitations:

• Note when metric type detection is heuristic-based (e.g., "The script inferred this is a gauge based on the _bytes suffix. Please confirm if this is correct.")
• Acknowledge when high-cardinality warnings might be false positives (e.g., "This warning may not apply if you're using a recording rule or know your cardinality is low.")

Step 6: Offer Optimizations

Based on validation results:

• Suggest more efficient query patterns
• Recommend recording rules for complex/repeated queries
• Propose better label matchers to reduce cardinality
• Advise on appropriate time ranges

Reference Examples: When suggesting corrections, cite relevant examples using this format:

As shown in `examples/bad_queries.promql` (lines 91-97):
❌ BAD: `avg(http_request_duration_seconds{quantile="0.95"})`
✅ GOOD: Use histogram_quantile() with histogram buckets

Citation sources:

• examples/good_queries.promql - for well-formed patterns
• examples/optimization_examples.promql - for before/after comparisons
• examples/bad_queries.promql - for showing what to avoid
• docs/best_practices.md - for detailed explanations
• docs/anti_patterns.md - for anti-pattern deep dives

Citation Format: file_path (lines X-Y) with the relevant code snippet quoted

Step 7: Let User Plan/Refine

Give the user control:

• Ask if they want to modify the query
• Offer to help rewrite it for better performance
• Provide multiple alternatives if applicable
• Explain trade-offs between different approaches

Key Validation Rules

Syntax Rules

1. Metric Names: Must match [a-zA-Z_:][a-zA-Z0-9_:]* or use UTF-8 quoting syntax (Prometheus 3.0+):
   • Quoted form: {"my.metric.with.dots"}
   • Using name label: {__name__="my.metric.with.dots"}
2. Label Matchers: = (equal), != (not equal), =~ (regex match), !~ (regex not match)
3. Time Durations: [0-9]+(ms|s|m|h|d|w|y) - e.g., 5m, 1h, 7d
4. Range Vectors: metric_name[duration] - e.g., http_requests_total[5m]
5. Offset Modifier: offset <duration> - e.g., metric_name offset 5m
6. @ Modifier: @ <timestamp> or @ start() / @ end()

Semantic Rules

1. rate() and irate(): Should only be used with counter metrics (metrics ending in _total, _count, _sum, or _bucket)
2. Counters: Should typically use rate() or increase(), not raw values
3. Gauges: Should not use rate() or increase()
4. Histograms: Use histogram_quantile() with le label and rate() on _bucket metrics
5. Summaries: Don't average quantiles; calculate from _sum and _count
6. Aggregations: Use by() or without() to control output labels

Performance Rules

1. Cardinality: Always use specific label matchers to reduce series count
2. Regex: Use = instead of =~ when possible for exact matches
3. Rate Range: Should be at least 4x the scrape interval (typically [2m] minimum)
4. irate(): Best for short ranges (<5m); use rate() for longer periods
5. Subqueries: Avoid excessive time ranges that process millions of samples
6. Recording Rules: Use for complex queries accessed frequently

Anti-Patterns to Detect

High Cardinality Issues

❌ Bad: http_requests_total{}

• Matches all time series without filtering

✅ Good: http_requests_total{job="api", instance="prod-1"}

• Specific label filters reduce cardinality