Nginx To Higress Migration

Nginx to Higress Migration

Automate migration from ingress-nginx to Higress in Kubernetes environments.

⚠️ Critical Limitation: Snippet Annotations NOT Supported

Before you begin: Higress does NOT support the following nginx annotations:

• nginx.ingress.kubernetes.io/server-snippet
• nginx.ingress.kubernetes.io/configuration-snippet
• nginx.ingress.kubernetes.io/http-snippet

These annotations will be silently ignored, causing functionality loss!

Pre-migration check (REQUIRED):

kubectl get ingress -A -o yaml | grep -E "snippet" | wc -l

If count > 0, you MUST plan WASM plugin replacements before migration. See Phase 6 for alternatives.

Prerequisites

• kubectl configured with cluster access
• helm 3.x installed
• Go 1.24+ (for WASM plugin compilation)
• Docker (for plugin image push)

Pre-Migration Checklist

Before Starting

• Backup all Ingress resources

  kubectl get ingress -A -o yaml > ingress-backup.yaml

• Identify snippet usage (see warning above)
• List all nginx annotations in use

  kubectl get ingress -A -o yaml | grep "nginx.ingress.kubernetes.io" | sort | uniq -c

• Verify Higress compatibility for each annotation (see annotation-mapping.md)
• Plan WASM plugins for unsupported features
• Prepare test environment (Kind/Minikube for testing recommended)

During Migration

• Install Higress in parallel with nginx
• Verify all pods running in higress-system namespace
• Run test script against Higress gateway
• Compare responses between nginx and Higress
• Deploy any required WASM plugins
• Configure monitoring/alerting

After Migration

• All routes verified working
• Custom functionality (snippet replacements) tested
• Monitoring dashboards configured
• Team trained on Higress operations
• Documentation updated
• Rollback procedure tested

Migration Workflow

Phase 1: Discovery

# Check for ingress-nginx installation
kubectl get pods -A | grep ingress-nginx
kubectl get ingressclass

# List all Ingress resources using nginx class
kubectl get ingress -A -o json | jq '.items[] | select(.spec.ingressClassName=="nginx" or .metadata.annotations["kubernetes.io/ingress.class"]=="nginx")'

# Get nginx ConfigMap
kubectl get configmap -n ingress-nginx ingress-nginx-controller -o yaml

Phase 2: Compatibility Analysis

Run the analysis script to identify unsupported features:

./scripts/analyze-ingress.sh [namespace]

Key point: No Ingress modification needed!

Higress natively supports nginx.ingress.kubernetes.io/* annotations - your existing Ingress resources work as-is.

See references/annotation-mapping.md for the complete list of supported annotations.

Unsupported annotations (require built-in plugin or custom WASM plugin):

• nginx.ingress.kubernetes.io/server-snippet
• nginx.ingress.kubernetes.io/configuration-snippet
• nginx.ingress.kubernetes.io/lua-resty-waf*
• Complex Lua logic in snippets

For these, check references/builtin-plugins.md first - Higress may already have a plugin!

Phase 3: Higress Installation (Parallel with nginx)

Higress natively supports nginx.ingress.kubernetes.io/* annotations. Install Higress alongside nginx for safe parallel testing.

# 1. Get current nginx ingressClass name
INGRESS_CLASS=$(kubectl get ingressclass -o jsonpath='{.items[?(@.spec.controller=="k8s.io/ingress-nginx")].metadata.name}')
echo "Current nginx ingressClass: $INGRESS_CLASS"

# 2. Detect timezone and select nearest registry
# China/Asia: higress-registry.cn-hangzhou.cr.aliyuncs.com (default)
# North America: higress-registry.us-west-1.cr.aliyuncs.com
# Southeast Asia: higress-registry.ap-southeast-7.cr.aliyuncs.com
TZ_OFFSET=$(date +%z)
case "$TZ_OFFSET" in
  -1*|-0*) REGISTRY="higress-registry.us-west-1.cr.aliyuncs.com" ;;      # Americas
  +07*|+08*|+09*) REGISTRY="higress-registry.cn-hangzhou.cr.aliyuncs.com" ;; # Asia
  +05*|+06*) REGISTRY="higress-registry.ap-southeast-7.cr.aliyuncs.com" ;;   # Southeast Asia
  *) REGISTRY="higress-registry.cn-hangzhou.cr.aliyuncs.com" ;;          # Default
esac
echo "Using registry: $REGISTRY"

# 3. Add Higress repo
helm repo add higress https://higress.io/helm-charts
helm repo update

# 4. Install Higress with parallel-safe settings
# Note: Override ALL component hubs to use the selected registry
helm install higress higress/higress \
  -n higress-system --create-namespace \
  --set global.ingressClass=${INGRESS_CLASS:-nginx} \
  --set global.hub=${REGISTRY}/higress \
  --set global.enableStatus=false \
  --set higress-core.controller.hub=${REGISTRY}/higress \
  --set higress-core.gateway.hub=${REGISTRY}/higress \
  --set higress-core.pilot.hub=${REGISTRY}/higress \
  --set higress-core.pluginServer.hub=${REGISTRY}/higress \
  --set higress-core.gateway.replicas=2

Key helm values:

• global.ingressClass: Use the same class as ingress-nginx
• global.hub: Image registry (auto-selected by timezone)
• global.enableStatus=false: Disable Ingress status updates to avoid conflicts with nginx (reduces API server pressure)
• Override all component hubs to ensure consistent registry usage
• Both nginx and Higress will watch the same Ingress resources
• Higress automatically recognizes nginx.ingress.kubernetes.io/* annotations
• Traffic still flows through nginx until you switch the entry point

⚠️ Note: After nginx is uninstalled, you can enable status updates:

helm upgrade higress higress/higress -n higress-system \
  --reuse-values \
  --set global.enableStatus=true

Kind/Local Environment Setup

In Kind or local Kubernetes clusters, the LoadBalancer service will stay in PENDING state. Use one of these methods:

Option 1: Port Forward (Recommended for testing)

# Forward Higress gateway to local port
kubectl port-forward -n higress-system svc/higress-gateway 8080:80 8443:443 &

# Test with Host header
curl -H "Host: example.com" http://localhost:8080/

Option 2: NodePort

# Patch service to NodePort
kubectl patch svc -n higress-system higress-gateway \
  -p '{"spec":{"type":"NodePort"}}'

# Get assigned port
NODE_PORT=$(kubectl get svc -n higress-system higress-gateway \
  -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}')

# Test (use docker container IP for Kind)
curl -H "Host: example.com" http://localhost:${NODE_PORT}/

Option 3: Kind with Port Mapping (Requires cluster recreation)

# kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 80
  - containerPort: 30443
    hostPort: 443

Phase 4: Generate and Run Test Script

After Higress is running, generate a test script covering all Ingress routes:

# Generate test script
./scripts/generate-migration-test.sh > migration-test.sh
chmod +x migration-test.sh

# Get Higress gateway address
# Option A: If LoadBalancer is supported
HIGRESS_IP=$(kubectl get svc -n higress-system higress-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Option B: If LoadBalancer is NOT supported, use port-forward
kubectl port-forward -n higress-system svc/higress-gateway 8080:80 &
HIGRESS_IP="127.0.0.1:8080"

# Run tests
./migration-test.sh ${HIGRESS_IP}

The test script will:

• Extract all hosts and paths from Ingress resources
• Test each route against Higress gateway
• Verify response codes and basic functionality
• Report any failures for investigation

Phase 5: Traffic Cutover (User Action Required)

⚠️ Only proceed after all tests pass!

Choose your cutover method based on infrastructure:

Option A: DNS Switch

# Update DNS records to point to Higress gateway IP
# Example: example.com A record -> ${HIGRESS_IP}

Option B: Layer 4 Proxy/Load Balancer Switch

# Update upstream in your L4 proxy (e.g., F5, HAProxy, cloud LB)
# From: nginx-ingress-controller service IP
# To: higress-gateway service IP

Option C: Kubernetes Service Switch (if using external traffic via Service)

# Update your external-facing Service selector or endpoints

Phase 6: Use Built-in Plugins or Create Custom WASM Plugin (If Needed)

Before writing custom plugins, check if Higress has a built-in plugin that meets your needs!

Built-in Plugins (Recommended First)

Higress provides many built-in plugins. Check references/builtin-plugins.md for the full list.

Common replacements for nginx features: