Implementing Security Monitoring With Datadog

# /etc/datadog-agent/datadog.yaml

api_key: <YOUR_DATADOG_API_KEY>
site: datadoghq.com   # or datadoghq.eu, us3.datadoghq.com, etc.

# Enable log collection for Cloud SIEM
logs_enabled: true

# Enable security features
runtime_security_config:
  enabled: true          # Workload Protection (CSM Threats)
  activity_dump:
    enabled: true        # Record process activity for investigation

compliance_config:
  enabled: true          # CIS benchmark checks (CSM Misconfigurations)
  host_benchmarks:
    enabled: true

# /etc/datadog-agent/conf.d/auth.d/conf.yaml
logs:
  - type: file
    path: /var/log/auth.log
    source: auth
    service: linux-auth
    tags:
      - env:production
      - security:authentication

  - type: file
    path: /var/log/syslog
    source: syslog
    service: linux-syslog

# /etc/datadog-agent/conf.d/win32_event_log.d/conf.yaml
logs:
  - type: windows_event
    channel_path: Security
    source: windows.events
    service: windows-security
    filters:
      - id: [4624, 4625, 4648, 4672, 4688, 4720, 4726, 4740, 4767]

# /etc/datadog-agent/system-probe.yaml
runtime_security_config:
  enabled: true
  fim_enabled: true        # File Integrity Monitoring
  network_enabled: true    # Network activity monitoring

sudo systemctl restart datadog-agent
sudo datadog-agent status | grep -A5 "Security Agent"

Datadog App > Security > Cloud SIEM > Configuration > Content Packs

AWS Content Pack:
  1. Enable the AWS integration in Datadog (Integrations > Amazon Web Services)
  2. Configure CloudTrail log forwarding via the Datadog Forwarder Lambda
  3. Enable VPC Flow Logs forwarding to Datadog
  4. Enable GuardDuty findings forwarding

Required IAM permissions for the Datadog role:
  - cloudtrail:LookupEvents
  - logs:FilterLogEvents
  - guardduty:ListDetectors, guardduty:GetFindings
  - s3:GetObject (for CloudTrail S3 bucket)

Azure Content Pack:
  1. Configure Azure Activity Logs via Event Hub to Datadog
  2. Forward Azure AD Sign-in Logs and Audit Logs
  3. Enable Microsoft Defender for Cloud alerts forwarding

GCP Content Pack:
  1. Configure GCP Audit Logs export via Pub/Sub to Datadog
  2. Forward Cloud Audit Logs (Admin Activity, Data Access)

Datadog App > Logs > Search
  Filter: source:(cloudtrail OR aws.guardduty OR azure.activitylogs)
  Verify: Logs appearing with correct source tags and parsed attributes

Datadog App > Security > Detection Rules

Out-of-the-box rule categories:
  - AWS: IAM policy changes, root account usage, S3 public access
  - Azure: Suspicious sign-ins, resource group deletions
  - GCP: IAM policy modifications, firewall rule changes
  - Authentication: Brute force, impossible travel, credential stuffing
  - Network: Port scanning, DNS tunneling, C2 beaconing
  - Application: SQL injection attempts, XSS, SSRF patterns

Datadog App > Security > Detection Rules > New Rule

Rule Name: "Brute Force Login Detection - Custom"
Rule Type: Log Detection (Real-time)

Define Search Query:
  source:auth status:error @evt.name:authentication @evt.outcome:failure
  Group By: @usr.id

Set Rule Cases:
  Case 1: When count > 10 in 5 minutes
    Name: "High volume failed logins"
    Severity: HIGH
    Notification: @slack-security-alerts @pagerduty-soc

  Case 2: When count > 50 in 5 minutes
    Name: "Extreme brute force attempt"
    Severity: CRITICAL
    Notification: @slack-security-alerts @pagerduty-soc-critical

Signal Settings:
  Keep signal alive for: 10 minutes
  Maximum signal duration: 24 hours
  Evaluation window: 5 minutes

Rule Name: "AWS Root Account Console Login"
Rule Type: Log Detection

Query:
  source:cloudtrail @evt.name:ConsoleLogin @userIdentity.type:Root

Severity: CRITICAL
Notification Message:
  "AWS Root account console login detected from IP {{@network.client.ip}}.
   Account: {{@usr.account_id}}
   Region: {{@cloud.region}}
   MFA Used: {{@additionalEventData.MFAUsed}}"

Tags: attack:initial-access, mitre:T1078

Datadog App > Security > Cloud Security Management > Setup

Enable Workload Protection:
  1. Verify Agent has runtime_security_config.enabled: true
  2. Review default Agent rules (file integrity, process execution)
  3. Customize rules for your environment

Default detection categories:
  - Process Execution: Detect reverse shells, crypto miners, exploitation tools
  - File Integrity: Monitor changes to /etc/passwd, /etc/shadow, SSH keys
  - Network Activity: Detect unexpected outbound connections, DNS tunneling
  - Container Escape: Detect privileged container breakout attempts
  - Kernel Module: Detect rootkit or unauthorized kernel module loading

Datadog App > Security > CSM > Agent Rules > New Agent Rule

Rule Expression:
  open.file.path == "/root/.ssh/authorized_keys" &&
  open.flags & (O_WRONLY | O_RDWR | O_CREAT) > 0 &&
  process.file.name != "sshd"

Rule Name: ssh_key_modification
Description: Detect non-sshd processes modifying root authorized_keys
Tags: attack:persistence, mitre:T1098.004

Datadog App > Dashboards > New Dashboard > "Security Operations Overview"

Widgets:
  1. Signal Count Over Time (timeseries)
     Query: count:security_signal by {signal.rule.name}
     Display: Line chart, last 24 hours

  2. Top Triggered Rules (top list)
     Query: count:security_signal by {signal.rule.name}.as_count()
     Display: Top 10

  3. Critical Signals (query value)
     Query: count:security_signal{severity:critical}
     Conditional format: Red if > 0

  4. Signals by Source (pie chart)
     Query: count:security_signal by {source}

  5. Geographic Threat Map (geomap)
     Query: count:security_signal by {network.client.geoip.country.name}

  6. Top Targeted Users (top list)
     Query: count:security_signal by {usr.id}

  7. Mean Time to Triage (query value)
     Query: avg:security_signal.triage_time

  8. Open Signals by Severity (table)
     Query: count:security_signal{status:open} by {severity}

Datadog App > Security > Notification Rules

Rule 1: Critical Signal Escalation
  Condition: severity:critical
  Recipients: @pagerduty-soc-critical @slack-security-incidents
  Message: "CRITICAL security signal: {{signal.rule.name}}
            Source: {{signal.attributes.network.client.ip}}
            Target: {{signal.attributes.usr.id}}
            Details: {{signal.message}}"

Rule 2: High Signal SOC Alert
  Condition: severity:high
  Recipients: @slack-security-alerts
  Suppress: After first notification, suppress for 15 minutes

Rule 3: Compliance Violation
  Condition: rule_type:compliance
  Recipients: @slack-compliance-team @jira-compliance-board

Workflow Automation (Datadog Workflows):
  Trigger: Security signal with severity:critical
  Steps:
    1. Enrich signal with threat intelligence lookup
    2. Create Jira incident ticket
    3. Send Slack notification with investigation context
    4. If source is AWS: Trigger Lambda to isolate resource

# Generate a test security event (failed SSH login)
ssh -o StrictHostKeyChecking=no invalid_user@localhost 2>/dev/null

# Verify the event appears in Datadog Logs
# Datadog App > Logs > source:auth status:error

# Check that a security signal was generated
# Datadog App > Security > Signals > Filter by rule name

# Tune noisy rules by adding suppression queries:
# Datadog App > Security > Detection Rules > [Rule] > Edit
# Add suppression: Suppress signal when @usr.id:service-account-*

from datadog_api_client import Configuration, ApiClient
from datadog_api_client.v2.api.security_monitoring_api import SecurityMonitoringApi

configuration = Configuration()
# Reads DD_API_KEY and DD_APP_KEY from environment

with ApiClient(configuration) as api_client:
    api = SecurityMonitoringApi(api_client)
    signals = api.search_security_monitoring_signals(
        body={
            "filter": {
                "query": "status:open severity:critical",
                "from": "now-24h",
                "to": "now",
            },
            "sort": {"field": "timestamp", "order": "desc"},
            "page": {"limit": 25},
        }
    )
    for signal in signals.data:
        attrs = signal.attributes
        print(f"[{attrs.severity}] {attrs.title}")
        print(f"  Rule: {attrs.custom.get('rule', {}).get('name', 'N/A')}")
        print(f"  Time: {attrs.timestamp}")