.. _version_history_1.38.0: 1.38.0 (Pending) ================= Incompatible behavior changes ----------------------------- *Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required* * **boringssl/fips**: The previous flag for specifying FIPS builds (ie ``--define=boringssl=fips``) will no longer work and has been replaced by ``--config=boringssl-fips``. This change will allow us to better support custom SSL libraries, and will allow FIPS-compliant Envoy to be built with the imminent switch to Bazel bzlmod. * **on_demand**: The on_demand filter, when doing On Demand CDS, will no longer do internal redirects after CDS is successful. Filters appearing in the filter chain before the On Demand Filter will no longer be invoked twice. This behavior can be temporarily reverted by setting the runtime guard ``envoy.reloadable_features.on_demand_cluster_no_recreate_stream`` to ``false``. * **tcp_proxy**: The TCP proxy filter now requires :ref:`max_early_data_bytes ` to be explicitly set when using :ref:`upstream_connect_mode ` modes other than ``IMMEDIATE`` (i.e., ``ON_DOWNSTREAM_DATA`` or ``ON_DOWNSTREAM_TLS_HANDSHAKE``). The field can be set to ``0`` to disable early data buffering while still using delayed connection modes. Configurations using these modes without ``max_early_data_bytes`` will now fail validation at startup. * **tls**: Set :ref:`enforce_rsa_key_usage ` to true by default. The handshake will fail if the keyUsage extension is present and incompatible with the TLS usage. In the next version of Envoy, this option will be removed and the enforcing behavior will always be used. Minor behavior changes ---------------------- *Changes that may cause incompatibilities for some users, but should not for most* * **ext_authz**: Changed the behavior of ``timeout: 0s`` in the HTTP ext_authz filter to mean "no timeout" (infinite) instead of immediate timeout. Previously, ``timeout: 0s`` would cause requests to fail immediately. This aligns with other Envoy timeout configurations where ``0`` means disabled or infinite. * **histograms**: Updated libcircllhist to 0.3.2, which changes how bucket bounds are interpreted. This should not impact production monitoring if the number of samples in the histograms is high. Affected tests were adjusted to account for histogram changes. * **mcp**: Changed the default metadata namespace for the MCP filter from ``mcp_proxy`` to ``envoy.filters.http.mcp``. This change can be reverted by setting the runtime guard ``envoy.reloadable_features.mcp_filter_use_new_metadata_namespace`` to ``false``. * **ratelimit**: Changed the behavior of ``timeout: 0s`` in the HTTP rate limit filter to mean "no timeout" (infinite) instead of immediate timeout. Previously, ``timeout: 0s`` would cause requests to fail immediately. This aligns with other Envoy timeout configurations where ``0`` means disabled or infinite. * **stat_sinks**: OpenTelemetry :ref:`SinkConfig ` stopped reporting empty delta counters and histograms. Bug fixes --------- *Changes expected to improve the state of the world and are unlikely to have negative effects* * **drop_overload**: Fixed a bug where ``drop_overload`` failed to use cached EDS resources. * **ext_authz**: Fixed a bug where headers from a denied authorization response (non-200) were not properly propagated to the client. * **ext_authz**: Fixed the HTTP ext_authz client to respect ``status_on_error`` configuration when the authorization server returns a 5xx error or when HTTP call failures occur. Previously, these error scenarios always returned 403 Forbidden regardless of the configured error status. * **ext_proc**: Fixed a bug to support two ext_proc filters configured in the chain. This change can be reverted by setting the runtime guard ``envoy.reloadable_features.ext_proc_inject_data_with_state_update`` to ``false``. * **formatter**: Fixed the log formatter in HTTP router upstream logs by correctly setting the downstream connection's ``ConnectionInfoProvider`` in the ``StreamInfo``. * **http**: Fixed a potential file descriptor leak where HTTP/1.1 connections with zombie streams (waiting for codec completion) would not be properly closed when in draining state. This could occur when a response was sent before the request was fully received, causing connections to remain open indefinitely. This change can be reverted by setting the runtime guard ``envoy.reloadable_features.http1_close_connection_on_zombie_stream_complete`` to ``false``. * **http**: Fixed upstream client to not close connection when idle timeout fires before the connection is established. This change can be reverted by setting the runtime guard ``envoy.reloadable_features.codec_client_enable_idle_timer_only_when_connected`` to ``false``. * **load_report**: Fixed an issue upon load-report shutdown race with ADS stream. Introduced proper cleanup of the gRPC stream. * **mcp_router**: Fixed MCP router to support session-less backends that do not return ``mcp-session-id`` headers. Previously this caused a spurious 500 error. * **oauth2**: Fixed OAuth2 refresh requests so host rewriting no longer overrides the original ``Host`` header value. * **odcds**: Fixed a crash (SIGABRT) when destroying OdCDS handles on worker threads. The handle no longer holds a direct reference to the subscription, preventing thread-safety issues during destruction. The subscription now persists in ClusterManagerImpl and is looked up by a config source key. * **overload_manager**: Fixed a resource leak in global connection limit tracking that caused permanent connection rejections when using load shedding (e.g., ``envoy.load_shed_points.tcp_listener_accept``). When connections were rejected due to load shedding after passing the global connection limit check, the allocated connection limit resource was not released, causing the connection counter to become incorrect and leading to ``failed_updates`` in the resource monitor. This resulted in permanent connection rejections even after load subsided. The fix ensures that connection limit resources are properly released when connections are rejected due to load shedding. Also added defensive resource cleanup for edge cases where address processing fails (e.g., ``localAddress()`` or ``peerAddress()`` errors). * **release**: Published contrib binaries now include the ``-contrib`` suffix in their version string. * **spiffe**: Reduce the number of the file watches needed by :ref:`trust_bundles ` when the validator is used in multiple places. Add support for :ref:`watched_directory ` to support Kubernetes environments that rely on atomic symbolic file updates. Add content hashing during file watching to handle excessive watch notifications. * **tls**: Fix on-demand TLS selector to enforce session resumption settings. New features ------------ * **a2a**: Added the parsing support for A2A(Agent2Agent) protocol. It is used to parse the A2A JSON-RPC messages. * **access_log**: Supported gauge in the :ref:`stats access logger `. * **admin**: Added ``filter`` query parameter support to the ``/clusters`` endpoint. The parameter accepts a RE2 regular expression to filter clusters by name. Compatible with the ``format`` parameter for both text and JSON output (e.g., ``/clusters?filter=service&format=json``). * **config**: Added support for :ref:`set_node_on_first_message_only ` to Delta-xDS. Guarded by runtime flag ``envoy.reloadable_features.xds_legacy_delta_skip_subsequent_node``. * **content_parsers**: Added :ref:`envoy.content_parsers.json ` content parser for extracting values from JSON content using JSON path selectors. Can be used by filters that need to parse structured JSON data and extract specific fields into metadata. * **dynamic modules**: Added drain and shutdown lifecycle hooks for bootstrap dynamic modules. * **dynamic modules**: Added support for dynamic modules authors to register any combination of http, network, listener, udp listener, and bootstrap filters in the rust SDK. * **dynamic modules**: Introduced the extended ABI forward compatibility mechanism for dynamic modules where modules built with a SDK version can be loaded by Envoy binaries of the next Envoy version. For example, A module built with the v1.38 SDK can now be loaded by an Envoy binary of v1.39. * **dynamic_modules**: Added :ref:`TLS certificate validator ` support for dynamic modules, enabling custom TLS certificate validation to be implemented in dynamic modules. * **dynamic_modules**: Added :ref:`load balancing policies ` support for dynamic modules, enabling custom load balancing algorithms to be implemented in dynamic modules. * **dynamic_modules**: Added a process-wide function registry to the dynamic modules ABI. Modules can register functions by name via ``envoy_dynamic_module_callback_register_function`` and other modules can resolve them via ``envoy_dynamic_module_callback_get_function``, enabling zero-copy cross-module interactions analogous to ``dlsym``. * **dynamic_modules**: Added admin handler API to the bootstrap extension dynamic modules ABI, enabling modules to register custom admin HTTP endpoints. * **dynamic_modules**: Added configurable :ref:`metrics_namespace ` field to ``DynamicModuleConfig``. This allows users to customize the prefix used for all metrics created by dynamic modules. Metrics now appear with the standard ``envoy_`` prefix followed by the namespace in prometheus output (e.g. ``envoy_myapp_requests_total``). The legacy behavior (stripping the namespace prefix from prometheus output) can be restored by setting the runtime guard ``envoy.reloadable_features.dynamic_modules_strip_custom_stat_prefix`` to ``true``. * **dynamic_modules**: Added dynamic module input matcher extension which would allow implementing custom matching logic in external languages (Rust, Go, C) via dynamic modules. * **dynamic_modules**: Added filter state read/write support for dynamic module cert validators, allowing modules to set and get string values in the connection's filter state during certificate chain verification. * **dynamic_modules**: Added init manager integration to the dynamic modules bootstrap extension ABI. An init target is automatically registered for every bootstrap extension, blocking traffic until the module signals readiness via ``signal_init_complete``. * **dynamic_modules**: Added metrics definition and update support for bootstrap dynamic modules. * **dynamic_modules**: Added timer API to the bootstrap extension dynamic modules ABI. * **dynamic_modules**: Network filter read and write buffers now persist after ``on_read``/``on_write`` callbacks, allowing modules to access buffered data from ``on_scheduled`` and other callbacks. Added ``envoy_dynamic_module_callback_network_filter_get_cluster_host_count`` to query cluster host counts by name, enabling scale-to-zero and custom load balancing decisions in network filters. * **formatter**: Added ``%UPSTREAM_DETECTED_CLOSE_TYPE%`` and ``%DOWNSTREAM_DETECTED_CLOSE_TYPE%`` to expose the detected close type of downstream and upstream connections. The possible values are ``Normal``, ``LocalReset``, and ``RemoteReset``. * **formatter**: Added ``QUERY_PARAMS`` support for substitution formatter to log all query params. They can either be logged in their original form or decoded. * **formatter**: Added the new access log formatter ``DOWNSTREAM_LOCAL_CLOSE_REASON``. * **formatter**: Extended ``*_WITHOUT_PORT`` address formatters to accept an optional ``MASK_PREFIX_LEN`` parameter that masks IP addresses and returns them in CIDR notation (e.g., ``%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT(16)%`` returns ``10.1.0.0/16`` for client IP ``10.1.10.23``). * **geoip**: Added ``asn_org`` field to :ref:`geo_field_keys ` to populate a header with the autonomous system organization name from the MaxMind ASN database. * **http**: Added :ref:`envoy.filters.http.sse_to_metadata ` filter for extracting values from Server-Sent Events (SSE) streams and writing them to dynamic metadata. Useful for capturing token usage metrics from LLM API responses. Supports pluggable content parsers for different SSE data formats. * **mcp**: Added HTTP DELETE session termination support to the MCP filter. DELETE requests with an ``MCP-Session-Id`` header are now recognized as valid MCP traffic in ``REJECT_NO_MCP`` mode. * **mcp_router**: Added :ref:`statistics ` to the MCP router filter for observability into request routing, fanout operations, and error conditions. * **mcp_router**: Added SSE (Server-Sent Events) streaming support for MCP backend responses. The router now handles SSE responses from backends for ``tools/call`` with direct pass-through streaming, and supports SSE aggregation for fanout operations (``tools/list``, ``initialize``) with incremental event parsing. * **mcp_router**: Added SSE response support for MCP ``resources/list`` fanout aggregation. * **mcp_router**: Added support for MCP client-to-server notification methods ``notifications/cancelled`` and ``notifications/roots/list_changed``. * **mcp_router**: Added support for MCP completion method ``completion/complete`` with routing based on ``ref/prompt`` or ``ref/resource``. * **mcp_router**: Added support for MCP logging method ``logging/setLevel``. * **mcp_router**: Added support for MCP prompt methods ``prompts/list`` and ``prompts/get``. * **mcp_router**: Added support for MCP resource methods ``resources/list``, ``resources/read``, ``resources/subscribe``, and ``resources/unsubscribe``. * **network**: Added support access logging in network filters like http filters, by allowing network filters to register as access logger instances. * **oauth2**: Added ``TLS_CLIENT_AUTH`` for the OAuth2 HTTP filter to support RFC 8705 mutual TLS client authentication. In this mode ``token_secret`` is optional and ignored, and the token endpoint cluster must be configured with mTLS. * **ratelimit**: Added ``RemoteAddressMatch`` action to the rate limit filter. This action will generate a descriptor based on the remote address of the downstream connection by matching it against specified CIDR ranges with support for inversion and formatter substitution. * **ratelimit**: Added per-descriptor ``x-ratelimit-*`` headers support. See the :ref:`x_ratelimit_option ` field documentation for more details. * **ratelimit**: Added support for shadow mode in the local rate limit filter. * **resource_monitors**: Added cgroup v2 support to the CPU utilization resource monitor. The monitor now automatically detects and selects between cgroup v1 and v2 at runtime by checking available cgroup files on the system. This enables the resource monitor to work correctly in both cgroup v1 and v2 environments without configuration changes. * **sse_parser**: Extended the SSE parser utility to support all standard SSE fields: ``id``, ``event`` (as ``event_type``), and ``retry``, in addition to the existing ``data`` field. The ``retry`` field is parsed as a ``uint32_t`` and only accepts values consisting of ASCII digits per the SSE specification. * **stat_sinks**: Added support for exporting OpenTelemetry metrics via HTTP. The new ``http_service`` field in :ref:`SinkConfig ` enables direct OTLP metrics export to backends that only accept HTTP (Dynatrace, Datadog, Elastic), without requiring an intermediate collector sidecar. * **stats**: Added support to limit the number of stats stored in each stats scope in the stats library. * **stats**: The admin prometheus stats endpoint now supports the protobuf exposition format, and will automatically use it if the request contains the correct Accept header, or if query parameter ``prom_protobuf=1`` is set. In a prometheus scrape configuration, add ``PrometheusProto`` to ``scrape_protocols`` to use the protobuf format. Additionally, when using the protobuf exposition format, the admin prometheus stats endpoint now supports `native histograms `_ when using the prometheus protobuf exposition format, using query ``/stats/prometheus?histogram_buckets=prometheusnative``. * **tcp_proxy**: Added :ref:`proxy_protocol_tlv_merge_policy ` to control how TLVs in ``proxy_protocol_tlvs`` are merged with existing PROXY protocol state. Supports ``ADD_IF_ABSENT`` (default), ``OVERWRITE_BY_TYPE_IF_EXISTS_OR_ADD``, and ``APPEND_IF_EXISTS_OR_ADD``. * **tcp_proxy**: Added an option to emit a log entry when the connection is accepted. * **tls**: Added support for fetching certificates on-demand via SDS in the upstream TLS transport socket using the extension :ref:`on-demand certificate selector `. * **tracers**: Add log events to spans created by OpenTelemetry tracer.