Implementing Mlops

MLOps Patterns

Operationalize machine learning models from experimentation to production deployment and monitoring.

Purpose

Provide strategic guidance for ML engineers and platform teams to build production-grade ML infrastructure. Cover the complete lifecycle: experiment tracking, model registry, feature stores, deployment patterns, pipeline orchestration, and monitoring.

When to Use This Skill

Use this skill when:

• Designing MLOps infrastructure for production ML systems
• Selecting experiment tracking platforms (MLflow, Weights & Biases, Neptune)
• Implementing feature stores for online/offline feature serving
• Choosing model serving solutions (Seldon Core, KServe, BentoML, TorchServe)
• Building ML pipelines for training, evaluation, and deployment
• Setting up model monitoring and drift detection
• Establishing model governance and compliance frameworks
• Optimizing ML inference costs and performance
• Migrating from notebooks to production ML systems
• Implementing continuous training and automated retraining

Core Concepts

1. Experiment Tracking

Track experiments systematically to ensure reproducibility and collaboration.

Key Components:

• Parameters: Hyperparameters logged for each training run
• Metrics: Performance measures tracked over time (accuracy, loss, F1)
• Artifacts: Model weights, plots, datasets, configuration files
• Metadata: Tags, descriptions, Git commit SHA, environment details

Platform Comparison:

MLflow (Open-source standard):

• Framework-agnostic (PyTorch, TensorFlow, scikit-learn, XGBoost)
• Self-hosted or cloud-agnostic deployment
• Integrated model registry
• Basic UI, adequate for most use cases
• Free, requires infrastructure management

Weights & Biases (SaaS, collaboration-focused):

• Advanced visualization and dashboards
• Integrated hyperparameter optimization (Sweeps)
• Excellent team collaboration features
• SaaS pricing scales with usage
• Best-in-class UI

Neptune.ai (Enterprise-grade):

• Enterprise features (RBAC, audit logs, compliance)
• Integrated production monitoring
• Higher cost than W&B
• Good for regulated industries

Selection Criteria:

• Open-source requirement → MLflow
• Team collaboration critical → Weights & Biases
• Enterprise compliance (RBAC, audits) → Neptune.ai
• Hyperparameter optimization primary → Weights & Biases (Sweeps)

For detailed comparison and decision framework, see references/experiment-tracking.md.

2. Model Registry and Versioning

Centralize model artifacts with version control and stage management.

Model Registry Components:

• Model artifacts (weights, serialized models)
• Training metrics (accuracy, F1, AUC)
• Hyperparameters used during training
• Training dataset version
• Feature schema (input/output signatures)
• Model cards (documentation, use cases, limitations)

Stage Management:

• None: Newly registered model
• Staging: Testing in pre-production environment
• Production: Serving live traffic
• Archived: Deprecated, retained for compliance

Versioning Strategies:

Semantic Versioning for Models:

• Major version (v2.0.0): Breaking change in input/output schema
• Minor version (v1.1.0): New feature, backward-compatible
• Patch version (v1.0.1): Bug fix, model retrained on new data

Git-Based Versioning:

• Model code in Git (training scripts, configuration)
• Model weights in DVC (Data Version Control) or Git-LFS
• Reproducibility via commit SHA + data version hash

For model lineage tracking and registry patterns, see references/model-registry.md.

3. Feature Stores

Centralize feature engineering to ensure consistency between training and inference.

Problem Addressed: Training/serving skew

• Training: Features computed with future knowledge (data leakage)
• Inference: Features computed with only past data
• Result: Model performs well in training but fails in production

Feature Store Solution:

Online Feature Store:

• Purpose: Low-latency feature retrieval for real-time inference
• Storage: Redis, DynamoDB, Cassandra (key-value stores)
• Latency: Sub-10ms for feature lookup
• Use Case: Real-time predictions (fraud detection, recommendations)

Offline Feature Store:

• Purpose: Historical feature data for training and batch inference
• Storage: Parquet files (S3/GCS), data warehouses (Snowflake, BigQuery)
• Latency: Seconds to minutes (batch retrieval)
• Use Case: Model training, backtesting, batch predictions

Point-in-Time Correctness:

• Ensures no future data leakage during training
• Feature values at time T only use data available before time T
• Critical for avoiding overly optimistic training metrics

Platform Comparison:

Feast (Open-source, cloud-agnostic):

• Most popular open-source feature store
• Supports Redis, DynamoDB, Datastore (online) and Parquet, BigQuery, Snowflake (offline)
• Cloud-agnostic, no vendor lock-in
• Active community, growing adoption

Tecton (Managed, production-grade):

• Feast-compatible API
• Fully managed service
• Integrated monitoring and governance
• Higher cost, enterprise-focused

SageMaker Feature Store (AWS):

• Integrated with AWS ecosystem
• Managed online/offline stores
• AWS lock-in

Databricks Feature Store (Databricks):

• Unity Catalog integration
• Delta Lake for offline storage
• Databricks ecosystem lock-in

Selection Criteria:

• Open-source, cloud-agnostic → Feast
• Managed solution, production-grade → Tecton
• AWS ecosystem → SageMaker Feature Store
• Databricks users → Databricks Feature Store

For feature engineering patterns and implementation, see references/feature-stores.md.

4. Model Serving Patterns

Deploy models for synchronous, asynchronous, batch, or streaming inference.

Serving Patterns:

REST API Deployment:

• Pattern: HTTP endpoint for synchronous predictions
• Latency: <100ms acceptable
• Use Case: Request-response applications
• Tools: Flask, FastAPI, BentoML, Seldon Core

gRPC Deployment:

• Pattern: High-performance RPC for low-latency inference
• Latency: <10ms target
• Use Case: Microservices, latency-critical applications
• Tools: TensorFlow Serving, TorchServe, Seldon Core

Batch Inference:

• Pattern: Process large datasets offline
• Latency: Minutes to hours acceptable
• Use Case: Daily/hourly predictions for millions of records
• Tools: Spark, Dask, Ray

Streaming Inference:

• Pattern: Real-time predictions on streaming data
• Latency: Milliseconds
• Use Case: Fraud detection, anomaly detection, real-time recommendations
• Tools: Kafka + Flink/Spark Streaming

Platform Comparison:

Seldon Core (Kubernetes-native, advanced):

• Advanced deployment strategies (canary, A/B testing, multi-armed bandits)
• Multi-framework support
• Integrated explainability (Alibi)
• High complexity, steep learning curve

KServe (CNCF standard):

• Standardized InferenceService API
• Serverless scaling (scale-to-zero with Knative)
• Kubernetes-native
• Growing adoption, CNCF backing

BentoML (Python-first, simplicity):

• Easiest to get started
• Excellent developer experience
• Local testing → cloud deployment
• Lower complexity than Seldon/KServe

TorchServe (PyTorch official):

• PyTorch-specific serving
• Production-grade, optimized for PyTorch models
• Less flexible for multi-framework use

TensorFlow Serving (TensorFlow official):

• TensorFlow-specific serving
• Production-grade, optimized for TensorFlow models
• Less flexible for multi-framework use

Selection Criteria:

• Kubernetes, advanced deployments → Seldon Core or KServe
• Python-first, simplicity → BentoML
• PyTorch-specific → TorchServe
• TensorFlow-specific → TensorFlow Serving
• Managed solution → SageMaker/Vertex AI/Azure ML

For model optimization and serving infrastructure, see references/model-serving.md.

5. Deployment Strategies

Deploy models safely with rollback capabilities.

Blue-Green Deployment:

• Two identical environments (Blue: current, Green: new)
• Deploy to Green, test, switch 100% traffic instantly
• Instant rollback (switch back to Blue)
• Trade-off: Requires 2x infrastructure, all-or-nothing switch

Canary Deployment:

• Gradual rollout to subset of traffic
• Route 5% → 10% → 25% → 50% → 100% over time
• Monitor metrics at each stage, rollback if degradation
• Trade-off: Complex routing logic, longer deployment time

Shadow Deployment:

• New model receives traffic but predictions not used
• Compare new model vs old model offline
• Zero risk to production
• Trade-off: Requires 2x compute, delayed feedback

A/B Testing:

• Split traffic between model versions
• Measure business metrics (conversion rate, revenue)
• Statistical significance testing
• Use Case: Optimize for business outcomes, not just ML metrics

Multi-Armed Bandit (MAB):

• Epsilon-greedy: Explore (try new models) vs Exploit (use best model)
• Thompson Sampling: Bayesian approach to exploration
• Use Case: Continuous optimization, faster convergence than A/B

Selection Criteria:

• Low-risk model → Blue-green (instant cutover)
• Medium-risk model → Canary (gradual rollout)
• High-risk model → Shadow (test in production, no impact)
• Business optimization → A/B testing or MAB

For deployment architecture and examples, see references/deployment-strategies.md.

6. ML Pipeline Orchestration

Automate training, evaluation, and deployment workflows.

Training Pipeline Stages:

1. Data Validation (Great Expectations, schema checks)
2. Feature Engineering (transform raw data)
3. Data Splitting (train/validation/test)
4. Model Training (hyperparameter tuning)
5. Model Evaluation (accuracy, fairness, explainability)
6. Model Registration (push to registry if metrics pass thresholds)
7. Deployment (promote to staging/production)

Continuous Training Pattern:

• Monitor production data for drift
• Detect data distribution changes (KS test, PSI)
• Trigger automated retraining when drift detected
• Validate new model before deployment
• Deploy via canary or shadow strategy

Platform Comparison:

Kubeflow Pipelines (ML-native, Kubernetes):

• ML-specific pipeline orchestration
• Kubernetes-native (scales with K8s)
• Component-based (reusable pipeline steps)
• Integrated with Katib (hyperparameter tuning)

Apache Airflow (Mature, general-purpose):

• Most mature orchestration platform
• Large ecosystem, extensive integrations
• Python-based DAGs
• Not ML-specific but widely used for ML workflows

Metaflow (Netflix, data science-friendly):

• Human-centric design, easy for data scientists
• Excellent local development experience
• Versioning built-in
• Simpler than Kubeflow/Airflow

Prefect (Modern, Python-native):

• Dynamic workflows, not static DAGs
• Better error handling than Airflow
• Modern UI and developer experience
• Growing community

Dagster (Asset-based, testing-focused):

• Asset-based thinking (not just task dependencies)
• Strong testing and data quality features
• Modern approach, good for data teams
• Smaller community than Airflow

Selection Criteria:

• ML-specific, Kubernetes → Kubeflow Pipelines
• Mature, battle-tested → Apache Airflow
• Data scientists, ease of use → Metaflow
• Software engineers, testing → Dagster
• Modern, simpler than Airflow → Prefect

For pipeline architecture and examples, see references/ml-pipelines.md.

7. Model Monitoring and Observability

Monitor production models for drift, performance, and quality.

Data Drift Detection:

• Definition: Input feature distributions change over time
• Impact: Model trained on old distribution, predictions degrade
• Detection Methods:
  • Kolmogorov-Smirnov (KS) Test: Compare distributions
  • Population Stability Index (PSI): Measure distribution shift
  • Chi-Square Test: For categorical features
• Action: Trigger automated retraining when drift detected

Model Drift Detection:

• Definition: Model prediction quality degrades over time
• Impact: Accuracy, precision, recall decrease
• Detection Methods:
  • Ground truth accuracy (delayed labels)
  • Prediction distribution changes
  • Calibration drift (predicted probabilities vs actual outcomes)
• Action: Alert team, trigger retraining

Performance Monitoring:

• Metrics:
  • Latency: P50, P95, P99 inference time
  • Throughput: Predictions per second
  • Error Rate: Failed predictions / total predictions
  • Resource Utilization: CPU, memory, GPU usage
• Alerting Thresholds:
  • P95 latency > 100ms → Alert
  • Error rate > 1% → Alert
  • Accuracy drop > 5% → Trigger retraining

Business Metrics Monitoring:

• Downstream impact: Conversion rate, revenue, user satisfaction
• Model predictions → business outcomes correlation
• Use Case: Optimize models for business value, not just ML metrics

Tools:

• Evidently AI: Data drift, model drift, data quality reports
• Prometheus + Grafana: Performance metrics, custom dashboards
• Arize AI: ML observability platform
• Fiddler: Model monitoring and explainability

For monitoring architecture and implementation, see references/model-monitoring.md.

8. Model Optimization Techniques

Reduce model size and inference latency.

Quantization:

• Convert model weights from float32 to int8
• Model size reduction: 4x smaller
• Inference speed: 2-3x faster
• Accuracy impact: Minimal (<1% degradation typically)
• Tools: PyTorch quantization, TensorFlow Lite, ONNX Runtime

Model Distillation:

• Train small student model to mimic large teacher model
• Transfer knowledge from teacher (BERT-large) to student (DistilBERT)
• Size reduction: 2-10x smaller
• Speed improvement: 2-10x faster
• Use Case: Deploy small model on edge devices, reduce inference cost

ONNX Conversion:

• Convert models to Open Neural Network Exchange (ONNX) format
• Cross-framework compatibility (PyTorch → ONNX → TensorFlow)
• Optimized inference with ONNX Runtime
• Speed improvement: 1.5-3x faster than native framework

Model Pruning:

• Remove less important weights from neural networks
• Sparsity: 30-90% of weights set to zero
• Size reduction: 2-10x smaller
• Accuracy impact: Minimal with structured pruning

For optimization techniques and examples, see references/model-serving.md.

9. LLMOps Patterns

Operationalize Large Language Models with specialized patterns.

LLM Fine-Tuning Pipelines:

• LoRA (Low-Rank Adaptation): Parameter-efficient fine-tuning
• QLoRA: Quantized LoRA (4-bit quantization)
• Pipeline: Base model → Fine-tuning dataset → LoRA adapters → Merged model
• Tools: Hugging Face PEFT, Axolotl

Prompt Versioning:

• Version control for prompts (Git, prompt management platforms)
• A/B testing prompts for quality and cost optimization
• Monitoring prompt effectiveness over time

RAG System Monitoring:

• Retrieval quality: Relevance of retrieved documents
• Generation quality: Answer accuracy, hallucination detection
• End-to-end latency: Retrieval + generation time
• Tools: LangSmith, Arize Phoenix

LLM Inference Optimization:

• vLLM: High-throughput LLM serving
• TensorRT-LLM: NVIDIA-optimized LLM inference
• Text Generation Inference (TGI): Hugging Face serving
• Batching: Dynamic batching for throughput

Embedding Model Management:

• Version embeddings alongside models
• Monitor embedding drift (distribution changes)
• Update embeddings when underlying model changes

For LLMOps patterns and implementation, see references/llmops-patterns.md.

10. Model Governance and Compliance

Establish governance for model risk management and regulatory compliance.

Model Cards:

• Documentation: Model purpose, training data, performance metrics
• Limitations: Known biases, failure modes, out-of-scope use cases
• Ethical considerations: Fairness, privacy, societal impact
• Template: Model Card Toolkit (Google)

Bias and Fairness Detection:

• Measure disparate impact across demographic groups
• Tools: Fairlearn, AI Fairness 360 (IBM)
• Metrics: Demographic parity, equalized odds, calibration
• Mitigation: Reweighting, adversarial debiasing, threshold optimization

Regulatory Compliance:

• EU AI Act: High-risk AI systems require documentation, monitoring
• Model Risk Management (SR 11-7): Banking industry requirements
• GDPR: Right to explanation for automated decisions
• HIPAA: Healthcare data privacy

Audit Trails:

• Log all model versions, training runs, deployments
• Track who approved model transitions (staging → production)
• Retain historical predictions for compliance audits
• Tools: MLflow, Neptune.ai (audit logs)

For governance frameworks and compliance, see references/governance.md.

Decision Frameworks

Framework 1: Experiment Tracking Platform Selection

Decision Tree:

Start with primary requirement:

• Open-source, self-hosted requirement → MLflow
• Team collaboration, advanced visualization (budget available) → Weights & Biases
• Team collaboration, advanced visualization (no budget) → MLflow
• Enterprise compliance (audit logs, RBAC) → Neptune.ai
• Hyperparameter optimization primary use case → Weights & Biases (Sweeps)

Detailed Criteria:

Recommendation by Organization:

• Startup (<50 people): MLflow (free, adequate) or W&B (if budget)
• Growth (50-500 people): Weights & Biases (team collaboration)
• Enterprise (>500 people): Neptune.ai (compliance) or MLflow (cost)

For detailed decision framework, see references/decision-frameworks.md.

Framework 2: Feature Store Selection

Decision Matrix:

Primary requirement:

• Open-source, cloud-agnostic → Feast
• Managed solution, production-grade, multi-cloud → Tecton
• AWS ecosystem → SageMaker Feature Store
• GCP ecosystem → Vertex AI Feature Store
• Azure ecosystem → Azure ML Feature Store
• Databricks users → Databricks Feature Store
• Self-hosted with UI → Hopsworks

Criteria Comparison:

Recommendation:

• Open-source, self-managed → Feast
• Managed, production-grade → Tecton
• AWS ecosystem → SageMaker Feature Store
• Databricks users → Databricks Feature Store

For detailed decision framework, see references/decision-frameworks.md.

Framework 3: Model Serving Platform Selection

Decision Tree:

Infrastructure:

• Kubernetes-based → Advanced deployment patterns needed?
  • Yes → Seldon Core (most features) or KServe (CNCF standard)
  • No → BentoML (simpler, Python-first)
• Cloud-native (managed) → Cloud provider?
  • AWS → SageMaker Endpoints
  • GCP → Vertex AI Endpoints
  • Azure → Azure ML Endpoints
• Framework-specific → Framework?
  • PyTorch → TorchServe
  • TensorFlow → TensorFlow Serving
• Serverless / minimal infrastructure → BentoML or Cloud Functions

Detailed Criteria:

Recommendation:

• Kubernetes, advanced deployments → Seldon Core or KServe
• Python-first, simplicity → BentoML
• PyTorch-specific → TorchServe
• TensorFlow-specific → TensorFlow Serving
• Managed solution → SageMaker/Vertex AI/Azure ML

For detailed decision framework, see references/decision-frameworks.md.