Data Distributed Storage

# hdfs-site.xml
dfs.replication: 3
dfs.block.size: 268435456  # 256MB
dfs.namenode.handler.count: 100
dfs.datanode.handler.count: 50
dfs.namenode.gc.time.threshold: 60s
dfs.namenode.checkpoint.period: 3600  # 1 hour
dfs.permissions.enabled: true

{
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::data-lake/*",
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    }
  ]
}

Cloud adoption strategy?
├── Single cloud → Native object store (S3/ADLS/GCS)
├── Multi-cloud → MinIO (abstraction layer)
├── On-premise
│   ├── Hadoop ecosystem → HDFS
│   └── Kubernetes-native → MinIO (S3-compatible)
└── Edge / IoT → MinIO (lightweight, S3 API)

Data access pattern?
├── Accessed frequently → Hot tier (delete when no longer needed)
├── Accessed occasionally → Intelligent tiering or manual tiering
├── Compliance retention → Write once, tier to cold → delete after retention
└── Temporary data → Hot tier → delete in 7-30 days

# MinIO Kubernetes Operator
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: data-lake-storage
spec:
  image: ${MINIO_IMAGE_REF:?Set MINIO_IMAGE_REF to quay.io/minio/minio@sha256:<reviewed-digest>}
  credsSecret:
    name: minio-creds
  pools:
    - servers: 4
      volumesPerServer: 4
      volumeClaimTemplate:
        spec:
          storageClassName: ssd
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 4Ti
  mountPath: /export
  requestAutoCert: true
  buckets:
    - name: raw
    - name: staging
    - name: analytics

# Erasure coding parity by deployment size
# 4 nodes × 4 drives = 16 drives
#   EC:8 → tolerate 8 drive failures, 2x overhead
#   EC:4 → tolerate 4 drive failures, 1.33x overhead
#   EC:2 → tolerate 2 drive failures, 1.14x overhead

# Set via environment variable
MINIO_STORAGE_CLASS_STANDARD: "EC:4"
MINIO_STORAGE_CLASS_RRS: "EC:2"     # Reduced redundancy for temp data

# Drive-level
# Use XFS filesystem with noatime,nodiratime mount options
# Separate write-intensive (metadata, WAL) from read-intensive drives

# Network-level
# MinIO uses S3-compatible HTTP/REST
# Enable HTTP/2 for multiplexing
# Use load balancer (nginx, HAProxy) for multi-node distribution

# Kernel tuning
# net.core.rmem_max: 134217728    # 128MB receive buffer
# net.core.wmem_max: 134217728    # 128MB send buffer
# net.ipv4.tcp_congestion_control: bbr  # BBR for better throughput
# vm.dirty_ratio: 30              # Page cache dirty ratio
# vm.dirty_background_ratio: 10

# AWS S3 Transfer Acceleration
# Uses CloudFront edge locations, TCP optimization
# Enable: s3.put_bucket_accelerate_configuration
# Best for: cross-region uploads, large objects > 1GB
# Cost: premium per GB transferred
# Test: s3-accelerate-speedtest.s3-accelerate.amazonaws.com

# Azure Data Box / Import-Export
# Physical transfer: Data Box (80TB), Data Box Disk (8TB)
# For: > 10TB initial loads, slow networks, air-gapped
# Timeline: order → receive → load → return → ingest (2-4 weeks)

# GCS Storage Transfer Service
# Online: from S3, HTTP, or GS URLs
# Schedule: one-time or recurring
# Best for: ongoing sync from S3 to GCS

migration_phases:
  phase_1_assessment:
    - inventory current storage (total TB, object count, avg size)
    - identify access patterns (hot vs cold, frequency)
    - document retention and compliance requirements
    - estimate egress costs and transfer time

  phase_2_pilot:
    - migrate 1-2 TB of non-critical data first
    - validate performance, cost, and access patterns
    - document issues and adjust strategy

  phase_3_bulk:
    - parallel multi-threaded copy (aws s3 sync, rclone, distcp)
    - validate data integrity checksums post-transfer
    - monitor for failures (rate limits, timeouts)

  phase_4_cutover:
    - final sync of delta changes
    - update application configs to point to new storage
    - redirect DNS / endpoint to new location
    - decommission old storage after 30-day validation window

# S3 Object Lock (compliance mode)
# Prevents object deletion or overwrite for retention period
# Governance mode: users with special permissions can override
# Compliance mode: no one (including root) can override
# Legal hold: prevents deletion regardless of retention

object_lock_examples:
  - name: "Financial records (SOX)"
    mode: COMPLIANCE
    retention_days: 2555  # 7 years
    bucket: "data-lake/finance/"

  - name: "Logs (internal policy)"
    mode: GOVERNANCE
    retention_days: 365  # 1 year
    bucket: "data-lake/logs/"

# ADLS Gen2: immutability policy
# Set at container level
# Time-based retention or legal hold

# Data residency requirements
# GDPR: data must stay in EU region
# Financial services: data within country borders
# Healthcare: HIPAA requires US-only storage (for US patients)

# Implementation:
# - Per-region buckets with IAM restrictions
# - S3 bucket policies denying cross-region replication for sensitive data
# - MinIO multi-tenant deployment per region
# - Regular audit to verify no data leaves allowed regions

cost_components:
  storage:
    - per-GB-month by tier (hot, warm, cold, archive)
    - minimum storage duration charges (Glazer: 90 days min)
    - early deletion fees (cold/archive tiers)

  operations:
    - PUT/COPY/POST/LIST requests (per 1000)
    - GET/SELECT requests (per 1000)
    - lifecycle transition requests (per 1000)

  data_transfer:
    - upload (usually free)
    - download / egress (per GB, tiered pricing)
    - cross-region replication (per GB)
    - transfer acceleration (premium per GB)

  additional:
    - encryption (KMS: per key, per API call)
    - monitoring (CloudWatch, metrics per custom metric)
    - backup / replication (CRR storage in destination region)

# Monthly storage cost estimate
cost_estimate:
  hot_data:
    volume: 50 TB
    unit_cost: $23/TB/mo
    subtotal: $1,150/mo

  warm_data:
    volume: 150 TB
    unit_cost: $12.50/TB/mo
    subtotal: $1,875/mo

  cold_data:
    volume: 300 TB
    unit_cost: $4/TB/mo
    subtotal: $1,200/mo

  archive_data:
    volume: 500 TB
    unit_cost: $1/TB/mo
    subtotal: $500/mo

  operations:
    requests: $200/mo  # Based on 10M PUT + 100M GET

  data_transfer:
    egress: $300/mo    # Based on 10TB egress

  monitoring_backup:
    cross_region_replication: $500/mo  # 50TB replicated
    monitoring: $100/mo

  total_monthly: $5,825/mo
  total_annual: $69,900/yr

# Object store monitoring (CloudWatch, Azure Monitor, GCS Ops)
metrics:
  storage:
    - BucketSizeBytes (by storage tier)
    - NumberOfObjects
    - AverageObjectSize

  requests:
    - AllRequests (count)
    - GetRequests / PutRequests / ListRequests
    - 4xxErrors / 5xxErrors
    - FirstByteLatency / TotalRequestLatency

  throughput:
    - BytesDownloaded / BytesUploaded
    - GetBandwidth / PutBandwidth (MB/s)

  cost:
    - StorageCost (by bucket/tag)
    - RequestCost
    - DataTransferCost
    - TotalCost

alerts:
  - name: "Storage growth anomaly"
    metric: BucketSizeBytes
    threshold: "> 20% week-over-week increase"
    severity: warning

  - name: "Error rate spike"
    metric: 5xxErrors
    threshold: "> 1% of total requests"
    severity: critical

  - name: "Latency degradation"
    metric: FirstByteLatency
    threshold: "p99 > 500ms"
    severity: warning

  - name: "Budget threshold"
    metric: TotalCost
    threshold: "> 80% of monthly budget"
    severity: warning

  - name: "Lifecycle failure"
    metric: LifecycleTransitions
    threshold: "transitions failed > 10 per hour"
    severity: warning

Workload type?
├── Hot data, frequent queries → ZSTD (level 3) — fast decompression
├── Cold data, archive → GZIP (level 6) — best compression ratio
├── Streaming, low-latency → Snappy or LZ4 — minimal CPU overhead
├── Columnar formats (Parquet/ORC)
│   ├── Default → ZSTD (best balance)
│   └── Legacy compatibility → Snappy
└── JSON/CSV files
    ├── Compressed → GZIP (most tools support it)
    └── Splittable → BZIP2 or LZ4

Compliance requirement?
├── Disaster recovery (region outage)
│   ├── RTO < 1 hour → Synchronous CRR
│   └── RTO < 4 hours → Asynchronous CRR
├── Data sovereignty (must stay in region)
│   └── Same-region replication (SRR) + backup to another AZ
├── GDPR right to erasure
│   ├── Replicate selectively (no unnecessary copies)
│   └── Document replication topology for audit
└── No compliance requirement
    └── No replication — rely on cloud provider durability