On October 10, 2023, the National Institute of Standards and Technology (NIST) published CVE-2023-44487 that describes a potential issue with many HTTP/2 implementations:
The HTTP/2 protocol allows a denial of service (server resource consumption) because request cancellation can reset many streams quickly, as exploited in the wild in August through October 2023.
Referred to as HTTP/2 Rapid Reset, this zero-day vulnerability can affect different implementations and configurations of the HTTP/2 protocol differently.
When virtual servers are configured with an HTTP profile that enables HTTP/2, this vulnerability may increase a load on NetScaler (CPU usage). Note that HTTP/2 is not enabled by default.
We recommend that you upgrade any HTTP/2 backend servers that you may be using in conjunction with NetScaler load balancing and content switching to versions that patch CVE-2023-44487. Our initial assessment and internal tests show that NetScaler can withstand an HTTP/2 Rapid Reset attack, but will forward requests for new streams to backend servers.
If you are using virtual servers with HTTP/2 enabled, you may want to consider the following options to mitigate the vulnerability and reduce the stress on backend servers. Please note that each of the options may have an impact that you will want to consider.
Additionally, you can lower the limit of maximum concurrent HTTP/2 streams from its default of 100 as follows:
set httpprofile <http-profile-name> -http2maxConcurrentStreams 50Lowering the limit of maximum concurrent HTTP/2 streams has been found to be effective against some exploit variants, and should be done in conjunction with these three options:
Option 1: Use a responder policy on load balancing and content switching virtual servers that use HTTP/2
Description: Configure the -threshold and -timeSlice parameters in the below policy to rate limit the number of requests per TCP connection to be allowed.
The configuration in the example below will drop the connection if 1,000 HTTP/2 requests are received in a single TCP connection over 180 seconds:
add streamstream selector myapp_sel CLIENT.IP.SRC CLIENT.TCP.SRCPORT
add ns limitIdentifier myapp_limitreq -threshold 1000 -timeSlice 180000 -selectorName myapp_sel
add responder policy myapp_resp_pol "sys.check_limit(\"myapp_limitreq\")" DROP
bind lb vserver <http2-virtual-server-name> -policyName myapp_resp_pol -priority 1
-gotoPriorityExpression END -type REQUESTYou can adjust the time interval and the threshold as required. For more information on configuring the traffic rate limit in NetScaler, refer to the documentation.
Note: Using a responder policy may cause some legitimate client requests to fail if the threshold or timeSlice is set too low. Choose a suitable value of threshold and timeSlice.
Option 2: Set the maxClients parameter on services bound to HTTP/2 virtual servers
Description: Configure the maxClients parameter to a non-zero value on the HTTP/SSL backend service. This parameter will limit the number of TCP connections that are established between the NetScaler ADC and a backend service. For a detailed explanation on how to do this, see the documentation.
Note: Clients may experience additional latency if the value of the maxClients parameter configured on the service is lower than the capacity of the backend service.
Option 3: Disable HTTP/2 on the HTTP profile for a service
Description: Disable the HTTP/2 parameter on a HTTP profile that is bound to a HTTP/SSL load balancing, content switching, or VPN virtual server until a fix is available.
Note: Applications that explicitly require HTTP/2 will be impacted. Applications that can fall back to lower HTTP protocol versions, such as HTTP/1.1, will suffer negligible impact, only losing out on the performance benefits of HTTP/2.
What’s next
The NetScaler team will continue to monitor the situation with this attack vector, investigate any variation of this attack, and update the advisory for CVE-2023-44487 as additional information becomes available.
FAQs
What is the impact of this vulnerability?
A malicious HTTP/2 client can connect to a server and open multiple HTTP/2 streams by sending HEADERS frames with new stream identifiers and then immediately cancel these streams by sending RST_STREAM frames.
These short-lived streams can cause increased resource consumption by the HTTP/2 service because new streams may be created on the backend that cannot be canceled immediately. This may lead to denial of service.
Have DoS attacks been observed?
Some service providers have observed incidents of DoS attacks as a result of this vulnerability.
Which protocols are impacted by this vulnerability?
Many implementations of the HTTP/2 protocol from multiple vendors are impacted and could potentially be affected by malicious clients.
How do I know if my NetScaler implementation is affected?
Depending on how you are using NetScaler, you may or may not be affected.
If you are using NetScaler as an HTTP/2 reverse proxy, NetScaler will be able to handle an HTTP/2 rapid reset attack but will use additional resources.
An HTTP/2 rapid reset attack could lead to a denial of service (DoS) attack by causing unnecessary streams to be created by NetScaler while it is processing short-lived HTTP/2 connections. And because backend services could have these short-lived connections forwarded on, backend services may be impacted.
Is HTTP/2 enabled by default in NetScaler?
HTTP/2 is not enabled by default. It must be specifically enabled on a HTTP profile.
How can I tell if HTTP/2 is being used on a specific NetScaler ADC?
You can check if HTTP/2 is being used by examining all the HTTP Profiles in the configuration utility under System → Profiles → HTTP Profiles. If the HTTP/2 checkbox is checked in any of the profiles, then HTTP/2 may be in use if it is bound to a virtual server.
From the CLI, you can run the following command:
show run | grep "http2 ENABLED"Similarly, if HTTP/2 is found to be enabled in an HTTP profile, you will need to check if it is bound to any of your virtual servers.
How can I determine if my NetScaler ADC has been attacked?
You can use the “stat lb vserver <http2-vserver-name>” command then compare the “Http2 Requests” and “Http2 Responses” counters. If “Http2 Responses” is very low compared to “Http2 Requests,” then this indicates a high number of cancellations and suggests an attack may have occurred.
For HTTP/1.x backends, you can also compare the counters http2_tot_headers_frames_rcvd and http2_tot_rst_stream_frames_rcvd from “nsconmsg -d stats.” The RST_FRAMES counter would normally be much lower than the total number of frames received. For more information on inspecting counters, see the documentation.