Dt Obs Hosts

Infrastructure Hosts Skill

Monitor and manage host and process infrastructure including CPU, memory, disk, network, and technology inventory.

When to Use This Skill

Use this skill when the user needs to:

• Inventory: "Show me all Linux hosts in AWS us-east-1"
• Agent Inventory: "Which hosts run FULL_STACK, INFRASTRUCTURE, or DISCOVERY mode?" / "Show OneAgent version distribution"
• Monitor: "What hosts have high CPU usage?"
• Troubleshoot: "Which processes are consuming the most memory?"
• Discover: "What databases are running in production?"
• Plan: "Track Kubernetes version distribution for upgrade planning"
• Cost: "Calculate infrastructure costs by cost center"
• Security: "Find all processes listening on port 22"
• Compliance: "Identify hosts running EOL Java versions"
• Quality: "Check data completeness for AWS hosts"
• Optimize: "Find rightsizing candidates based on utilization"

Cross-source join required: If the query must combine host data with logs or other telemetry sources (e.g. "show logs from Linux hosts with their IP addresses") → also read dt-dql-essentials/references/smartscape-topology-navigation.md before writing the query.

Core Concepts

Entities

• HOST - Physical or virtual machines (cloud or on-premise)
• PROCESS - Running processes and process groups
• CONTAINER - Kubernetes containers
• NETWORK_INTERFACE - Host network interfaces
• DISK - Host disk volumes

Metrics Categories

1. Host Metrics - dt.host.cpu.*, dt.host.memory.*, dt.host.disk.*, dt.host.net.*
2. Process Metrics - dt.process.cpu.*, dt.process.memory.*, dt.process.io.*, dt.process.network.*
3. Inventory - OS type, cloud provider, technology stack, versions
4. Cost - dt.cost.costcenter, dt.cost.product
5. Quality - Metadata completeness, version compliance

Alert Thresholds

• CPU/Memory/Disk: 80% warning, 90% critical
• Network: >70% high, >85% saturated
• Disk Latency: >20ms bottleneck
• Network Errors: Drop rate >1%, error rate >0.1%
• Swap: >30% warning, >50% critical

Key Workflows

1. Host Discovery and Classification

Discover hosts, classify by OS/cloud, inventory resources.

smartscapeNodes "HOST"
| fieldsAdd os.type, cloud.provider, host.logical.cpu.cores, host.physical.memory
| summarize host_count = count(), by: {os.type, cloud.provider}
| sort host_count desc

OS Types: LINUX, WINDOWS, AIX, SOLARIS, ZOS

→ For cloud-specific attributes, see references/inventory-discovery.md

2. Resource Utilization Monitoring

Monitor CPU, memory, disk, network across hosts.

timeseries {
  cpu = avg(dt.host.cpu.usage),
  memory = avg(dt.host.memory.usage),
  disk = avg(dt.host.disk.used.percent)
}, by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| filter arrayAvg(cpu) > 80 or arrayAvg(memory) > 80
| sort arrayAvg(cpu) desc

High utilization threshold: 80% warning, 90% critical

Key CPU Metrics:

• dt.host.cpu.usage — Total CPU utilization (0-100%)
• dt.host.cpu.idle — CPU idle time (inverse of usage; useful for anomaly detection)
• dt.host.cpu.user — CPU time in user mode
• dt.host.cpu.system — CPU time in kernel mode
• dt.host.cpu.iowait — CPU waiting for I/O (Linux only)

→ For detailed CPU analysis, see references/host-metrics.md
→ For memory breakdown, see references/host-metrics.md

Disk Free Space — Find Hosts with Most/Least Free Disk

timeseries disk_used_pct = avg(dt.host.disk.used.percent), by: {dt.smartscape.host}
| fieldsAdd host_name = getNodeName(dt.smartscape.host)
| fieldsAdd avg_disk_used = arrayAvg(disk_used_pct),
    free_pct = 100 - arrayAvg(disk_used_pct)
| sort free_pct desc
| limit 10

3. Process Resource Analysis

Identify top resource consumers at process level.

timeseries {
  cpu = avg(dt.process.cpu.usage),
  memory = avg(dt.process.memory.usage)
}, by: {dt.smartscape.process}
| fieldsAdd process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(cpu) > 50
| sort arrayAvg(cpu) desc
| limit 20

→ For process I/O analysis, see references/process-monitoring.md
→ For process network metrics, see references/process-monitoring.md

4. Technology Stack Inventory

Discover and track software technologies and versions.

smartscapeNodes "PROCESS"
| fieldsAdd process.software_technologies
| expand tech = process.software_technologies
| fieldsAdd tech_type = tech[type], tech_version = tech[version]
| summarize process_count = count(), by: {tech_type, tech_version}
| sort process_count desc

Common Technologies: Java, Node.js, Python, .NET, databases, web servers, messaging systems

→ For version compliance checks, see references/inventory-discovery.md

5. Service Discovery via Ports

Map listening ports to services for security and inventory.

smartscapeNodes "PROCESS"
| fieldsAdd process.listen_ports, dt.process_group.detected_name
| filter isNotNull(process.listen_ports) and arraySize(process.listen_ports) > 0
| expand listen_port = process.listen_ports
| summarize process_count = count(), by: {listen_port, dt.process_group.detected_name}
| sort toLong(listen_port) asc
| limit 50

Well-known ports: 80 (HTTP), 443 (HTTPS), 22 (SSH), 3306 (MySQL), 5432 (PostgreSQL)

→ For comprehensive port mapping, see references/inventory-discovery.md

6. Container and Kubernetes Monitoring

Track container distribution and K8s workload types.

smartscapeNodes "CONTAINER"
| fieldsAdd k8s.cluster.name, k8s.namespace.name, k8s.workload.kind
| summarize container_count = count(), by: {k8s.cluster.name, k8s.workload.kind}
| sort k8s.cluster.name, container_count desc

Workload Types: deployment, daemonset, statefulset, job, cronjob

Note: Container image names/versions NOT available in smartscape.

→ For K8s version tracking, see references/container-monitoring.md
→ For container lifecycle, see references/container-monitoring.md

7. Cost Attribution and Chargeback

Calculate infrastructure costs by cost center.

smartscapeNodes "HOST"
| fieldsAdd dt.cost.costcenter, host.logical.cpu.cores, host.physical.memory
| filter isNotNull(dt.cost.costcenter)
| fieldsAdd memory_gb = toDouble(host.physical.memory) / 1024 / 1024 / 1024
| summarize 
    host_count = count(),
    total_cores = sum(toLong(host.logical.cpu.cores)),
    total_memory_gb = sum(memory_gb),
    by: {dt.cost.costcenter}
| sort total_cores desc

→ For product-level cost tracking, see references/inventory-discovery.md

8. Infrastructure Health Correlation

Correlate host and process metrics for cross-layer analysis.

timeseries {
  host_cpu = avg(dt.host.cpu.usage),
  host_memory = avg(dt.host.memory.usage),
  process_cpu = avg(dt.process.cpu.usage)
}, by: {dt.smartscape.host, dt.smartscape.process}
| fieldsAdd
    host_name = getNodeName(dt.smartscape.host),
    process_name = getNodeName(dt.smartscape.process)
| filter arrayAvg(host_cpu) > 70
| sort arrayAvg(host_cpu) desc

Health scoring: Critical if any resource >90%, warning if >80%

→ For multi-resource saturation detection, see references/host-metrics.md

9. OneAgent Inventory

Count and list hosts by OneAgent monitoring mode, version, or cloud region.

ONEAGENT entity: OneAgent is a separate smartscape entity type (smartscapeNodes "ONEAGENT"). Access it by traversing backward from HOST via the monitors edge (the edge runs ONEAGENT → HOST, so HOST→ONEAGENT is direction: backward).

Key ONEAGENT fields:

• dt.agent.monitoring_mode — monitoring coverage level: FULL_STACK / INFRASTRUCTURE / DISCOVERY
• dt.agent.module.version — installed version string, e.g. 1.347.0.20260809-172428

Count by monitoring mode:

smartscapeNodes "HOST"
| traverse edgeTypes: {monitors}, targetTypes: {ONEAGENT}, direction: backward
| fieldsAdd oa_mode = `dt.agent.monitoring_mode`
| summarize host_count = count(), by: {oa_mode}
| sort host_count desc

Count by agent version:

smartscapeNodes "HOST"
| traverse edgeTypes: {monitors}, targetTypes: {ONEAGENT}, direction: backward
| fieldsAdd oa_version = `dt.agent.module.version`
| summarize host_count = count(), by: {oa_version}
| sort host_count desc

Combined: mode + version (for upgrade planning):

smartscapeNodes "HOST"
| traverse edgeTypes: {monitors}, targetTypes: {ONEAGENT}, direction: backward
| fieldsAdd oa_mode = `dt.agent.monitoring_mode`, oa_version = `dt.agent.module.version`
| summarize host_count = count(), by: {oa_mode, oa_version}
| sort host_count desc

Monitoring modes: FULL_STACK (full code-level monitoring + infrastructure), INFRASTRUCTURE (infrastructure metrics only, no code-level monitoring), DISCOVERY (topology discovery and basic host monitoring)

→ For listing hosts by mode/version, see references/inventory-discovery.md

Response Construction

When the user asks for data retrieval or a DQL query (e.g., "show me top hosts by CPU"), include the DQL query in the response alongside the results. Users want to see and reuse the query — it is the deliverable, not just a means to get results.

When the user asks for analysis (anomaly detection, forecasting, seasonality), the analysis results are the deliverable. Focus on presenting findings clearly:

• Prioritize metric-level findings over data collection artifacts. If an analysis tool reports data gaps alongside actual anomalies, lead with the metric behavior the user asked about and mention gaps only as supplementary context.
• Include host names (not just IDs) using getNodeName(dt.smartscape.host) or the get-entity-name tool.
• State the timeframe analyzed and the tools/parameters used.

Analytical Workflows

Host metric queries often serve as inputs to analytical tools (anomaly detection, forecasting, seasonality analysis). This skill helps construct the right DQL query; the actual analysis is performed by dedicated tools.

Anomaly Detection and Pattern Analysis

When users ask about "unusual behavior", "anomalies", "spikes", or "sudden changes" in host metrics, the workflow is:

1. Construct the timeseries query using this skill's patterns
2. Pass it to the appropriate analysis tool (anomaly detector, novelty detection)

Choosing between detectors:

• adaptive-anomaly-detector — use when the user asks about magnitude: "spikes", "abrupt changes", "values that went above normal", "sudden jumps". It answers "did this metric cross an unexpected threshold?" and reports alert durations and peak values.
• timeseries-novelty-detection — use when the user asks about behavioral change: "unusual patterns", "something changed", "trends", "new behavior". It answers "did the shape of the signal change?" without implying a specific threshold was crossed.

Response format for anomaly results: Include both the host name (resolved via getNodeName(dt.smartscape.host) or get-entity-name) and the host entity ID alongside timestamps and values. Entity IDs alone are opaque to users; names alone prevent follow-up queries.

Novelty type selection rule: When using novelty detection, set analysisNoveltyType to only [SPIKE, CHANGE_IN_VALUES, TREND_IN_VALUES] by default. EXCLUDE GAP_WITH_MISSING_VALUES and CHANGE_IN_MISSING_VALUES unless the user explicitly asks about data gaps or monitoring coverage. Data gaps are infrastructure issues, not metric behavior anomalies — reporting them when the user asks about CPU or memory patterns is incorrect.

Queries for analysis tools should use simple timeseries format with a single aggregated metric and appropriate time range:

timeseries avg(dt.host.cpu.idle), by: {dt.smartscape.host}

timeseries avg(dt.host.memory.usage), by: {dt.smartscape.host}

Avoid adding filters or field transformations that reduce the data — the analysis tools work best with complete timeseries data.

Forecasting

When users ask to "predict", "forecast", or "estimate future" host metrics:

1. Construct the timeseries query with sufficient historical data (e.g., 7d for short-term, 30d for longer predictions)
2. Pass to the forecasting tool with the desired forecast horizon

The forecast horizon (how far ahead to predict) and the historical window (how much past data the model trains on) are independent. A request like "forecast the next 2 hours" sets the horizon to 2h — it says nothing about the lookback. Always use at least 7 days of historical data regardless of how short the forecast horizon is. Too few training data points cause the forecast model to fail and fall back to raw historical values.

timeseries avg(dt.host.cpu.usage), by: {dt.smartscape.host}