Set up Envoy Gateway

View Markdown

Set up Envoy Gateway

This page describes how to set up Envoy Gateway for an Anyscale Kubernetes deployment. If you have an existing deployment using ingress-nginx, this guide also covers migrating traffic to Envoy Gateway.

note

ingress-nginx is deprecated. Anyscale recommends migrating to Gateway API for all Kubernetes deployments.

Prerequisites

Before starting, confirm you have the following:

• Kubernetes 1.24 or later.
• Your Anyscale cloud ID and cloud resource ID. Find both in the Anyscale console under Cloud details.
• The latest version of the Anyscale CLI. Run pip install --upgrade anyscale to update.

Enter your IDs below to auto-populate the commands and configuration on this page.

Cloud Resource ID

Cloud ID

note

If you don't have active Anyscale services running, you don't need to run the migration commands in Step 6. Complete Steps 1 through 5 to set up Gateway infrastructure, then upgrade the operator. Gateway takes over automatically.

Step 1: Install Envoy Gateway

Run the following command to install Envoy Gateway v1.7.0:

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.7.0 \
  --namespace envoy-gateway-system \
  --create-namespace

Run the following command to verify Envoy Gateway is running before proceeding:

kubectl wait --for=condition=available deployment/envoy-gateway \
  -n envoy-gateway-system --timeout=120s

Step 2: Create an EnvoyProxy resource

Create an EnvoyProxy resource to configure the Envoy Gateway load balancer. Anyscale recommends a dedicated EnvoyProxy per Anyscale cloud resource for isolation.

Select your cloud provider and save the following as envoyproxy.yaml:

AWS

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Google Cloud

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
        annotations:
          cloud.google.com/load-balancer-type: "External"

Azure

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "false"

CoreWeave

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/coreweave-load-balancer-type: "public"

Nebius

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: envoy-proxy
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer

Run the following command to apply the EnvoyProxy resource:

kubectl apply -f envoyproxy.yaml

Step 3: Create a GatewayClass

Create a GatewayClass that references the EnvoyProxy from Step 2. Anyscale recommends a dedicated GatewayClass per Anyscale cloud resource for isolation.

Save the following as gatewayclass.yaml:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: envoy-proxy
    namespace: envoy-gateway-system

Run the following command to apply the GatewayClass:

kubectl apply -f gatewayclass.yaml

Step 4: Create the Gateway

The Gateway resource requires TLS certificate secrets that Anyscale provisions automatically. Secret names include the Anyscale cloud resource ID. The following table describes when Anyscale provisions these certificates:

Secret name	Covers	Provisioned
anyscale-<cldrsrc-id>-certificate	Head node access	After operator installation
anyscale-svc-<cldrsrc-id>-certificate	Service access	After your first Anyscale service runs

note

Kubernetes secret names require lowercase characters and hyphens. If you entered your cloud resource ID in the field above, the YAML on this page converts the format automatically. Otherwise, convert underscores to hyphens and lowercase the result manually. For example, cldrsrc_abc123 becomes cldrsrc-abc123. Then the secrets would be named anyscale-cldrsrc-abc123-certificate and anyscale-svc-cldrsrc-abc123-certificate.

Create the Gateway in the Anyscale operator namespace with three listeners. Save the following as gateway.yaml:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: anyscale-operator
spec:
  gatewayClassName: eg
  listeners:
  - name: http
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: Same
  - name: https
    port: 443
    protocol: HTTPS
    hostname: '*.i.anyscaleuserdata.com'
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: anyscale-<cldrsrc-id>-certificate
    allowedRoutes:
      namespaces:
        from: Same
  - name: https-session
    port: 443
    protocol: HTTPS
    hostname: '*.s.anyscaleuserdata.com'
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: anyscale-svc-<cldrsrc-id>-certificate
    allowedRoutes:
      namespaces:
        from: Same

note

The anyscale-svc-<cldrsrc-id>-certificate secret isn't provisioned until your first Anyscale service runs. The Gateway applies successfully regardless. Kubernetes processes the listener configuration and picks up the secret automatically once it's provisioned.

note

The http listener on port 80 is required for the canary migration process in Step 6. During migration, the Ingress controller makes an internal HTTP call to the Envoy service. If you're doing a fresh installation with no active services, it has no effect on external traffic. All client-facing traffic uses the HTTPS listeners.

Run the following command to create the Gateway:

kubectl apply -f gateway.yaml

Run the following command to get the Gateway's load balancer address. Use this value in the next step.

kubectl get gateway gateway -n anyscale-operator \
  -o jsonpath='{.status.addresses[0].value}'

Step 5: Configure the Anyscale operator

Update your values.yaml to enable Gateway mode. This configuration is additive. It doesn't replace your existing ingress settings, and both can coexist during migration.

Run the following command to get your current Anyscale operator values:

helm get values anyscale-operator -n anyscale-operator

Add the following to your values.yaml:

networking:
  gateway:
    enabled: true
    name: "gateway"
    namespace: "anyscale-operator"
    apiVersion: "gateway.networking.k8s.io/v1"
    hostname: "<gateway-lb-address>"  # From Step 4
    ip: ""

Run the following command to upgrade the Anyscale operator:

helm upgrade anyscale-operator anyscale/anyscale-operator \
  -n anyscale-operator \
  -f values.yaml

Run the following command to verify you've successfully created HTTPRoute resources:

kubectl get httproutes -n anyscale-operator

note

To revert Gateway mode at any point after this step, see Roll back the migration.

Step 6: Migrate traffic

important

You must complete this step if you have active Anyscale services.

If you don't have any Anyscale services actively deployed, you can skip this step.

For deployments with active services, use the following migration commands to gradually shift traffic from Ingress to Gateway. The following instructions use Ingress and HTTPRoute to serve traffic in parallel during the transition, ensuring traffic continues to flow without interruption.

note

During migration, traffic shifting applies only to Anyscale services. Jobs and workspaces continue running on Nginx until you set the weight to 100, at which point all workloads switch to Gateway.

Step 6a: Start the migration

Run the following command to start the migration. During this initial phase, both Ingress and HTTPRoute serve traffic, but all traffic flows through the Ingress load balancer.

anyscale cloud start-gateway-migration --cloud-id <cloud-id>

Step 6b: Verify dual-stack operation

Run the following command to confirm that both Ingress and HTTPRoute specs are present in your namespace:

anyscale cloud gateway-migration-status --cloud-id <cloud-id>

Step 6c: Shift traffic to Gateway

Gradually shift traffic from Ingress to Gateway by increasing the canary weight. Monitor your services between each increment before continuing.

Run the following command to shift 10% of traffic to Gateway:

anyscale cloud set-gateway-canary-weight --cloud-id <cloud-id> --weight 10

After monitoring your services, increase the weight:

anyscale cloud set-gateway-canary-weight --cloud-id <cloud-id> --weight 50

After verifying stability, complete the migration:

anyscale cloud set-gateway-canary-weight --cloud-id <cloud-id> --weight 100

note

Wait at least 60 seconds between weight increments before increasing further. To roll back at any point, set the weight back to 0. Traffic returns to Ingress immediately. Contact Anyscale support if requests aren't reaching the Envoy controller or if setting the weight to 0 doesn't restore traffic through Ingress. Check the Envoy controller logs to diagnose routing issues.

At weight=100, migration is complete and Anyscale no longer generates Ingress resources.

Step 6d: Verify the migration

Run the following command to confirm that all traffic flows through Gateway:

anyscale cloud gateway-migration-status --cloud-id <cloud-id>

Roll back the migration

If you encounter issues during the migration, you have the following options depending on how far you've progressed:

• Pause and revert to Ingress: Set the canary weight to 0. All traffic returns to flowing through the Ingress load balancer.

  anyscale cloud set-gateway-canary-weight --cloud-id <cloud-id> --weight 0

• Revert Gateway mode entirely: Set the canary weight to 0 first, then set networking.gateway.enabled to false in values.yaml and upgrade the operator. The operator returns to generating Ingress resources.

  helm upgrade anyscale-operator anyscale/anyscale-operator \
    -n anyscale-operator \
    -f values.yaml

Troubleshoot

The following table lists common troubleshooting commands:

Description	Command
Check Gateway status	kubectl get gateway gateway -n anyscale-operator kubectl describe gateway gateway -n anyscale-operator
Check Envoy Gateway controller logs	kubectl logs -n envoy-gateway-system \ -l control-plane=envoy-gateway --tail=50
Check operator logs	kubectl logs -n anyscale-operator \ -l app.kubernetes.io/name=anyscale-operator --tail=50
Verify HTTPRoutes	kubectl get httproutes -n anyscale-operator
Verify TLS certificates	kubectl get secrets -n anyscale-operator | grep certificate