On October 10, 2023, NetScaler published a security bulletin for CVE-2023-4966 — now dubbed by some as “CitrixBleed” — that affects customer-managed NetScaler ADC and NetScaler Gateway. This critical vulnerability was discovered by our internal team. At the time we published the security bulletin, we were unaware that this vulnerability had been exploited in the wild, and we recommended that customers upgrade as soon as possible to an updated version released simultaneously with the security bulletin to resolve this critical issue.
In the NetScaler blog post on CVE-2023-4966 published on October 23, 2023, we shared that the U.S. Cybersecurity and Infrastructure Security Agency (CISA) and Mandiant both reported that this vulnerability had been exploited by threat actors, leading to session hijacking. We also shared remediation guidance for clearing sessions immediately. We continued to encourage customers that had not patched to do so urgently. We also relayed important information concerning exploit reports from CISA and Mandiant.
Until mid-October, we understood from public reporting and through very limited support cases that exploitation of CVE-2023-4966 was targeted and limited in nature. However, we learned of a concerning development when, on October 25, Shadowserver Foundation, a non-profit internet monitoring organization, posted on X (formerly known as Twitter) that there was a sharp increase in attempts to exploit this vulnerability in unpatched NetScaler ADCs.
Now the media is reporting that the LockBit ransomware group is targeting unpatched NetScaler ADCs.
We strongly urge NetScaler customers to review the security bulletin, our October 23 blog post, CISA guidance, and Mandiant’s blogs on remediation and investigation, and patch immediately.
Next step after upgrading
If you are using any of the affected builds listed in the security bulletin, you should upgrade immediately by installing the updated versions. After you upgrade, we recommend that you remove any active or persistent sessions using the following commands:
kill aaa session -all
kill icaconnection -all
kill rdp connection -all
kill pcoipConnection -all
clear lb persistentSessionsNote: Please ensure that the formatting remains intact as you copy and paste these commands.
Investigation recommendations
From our engagements with impacted customers, we’re developing recommendations for investigations of exploits of CVE-2023-4966:
- Look for patterns of suspicious session use in your organizations’ monitoring and visibility tools, particularly relating to virtual desktops if you have these configured.
- If you are forwarding NetScaler’s logs to a syslog server, review these for ‘SSLVPN TCPCONNSTAT’ logs that contain mismatching ‘Client_ip’ and ‘Source’ IP addresses. Note that there are legitimate scenarios where this might occur, such as a roaming user.
- Review the ‘SSLVPN TCPCONNSTAT’ logs for the same ‘Source’ IP address accessing the sessions of multiple users (you can refer to the ‘User’ field in the log).
- Finally, if you are conducting your own forensic investigation on an unpatched instance, see NetScaler product documentation on collecting memory snapshots of the NSPPE process. Note that this will require at least 5GB of space on your NetScaler ADC, and more in some configurations. You should remove these core dumps, located in /var/core, afterwards to avoid filling the partition, which is needed for normal operation. Careful analysis of the memory snapshots of the unpatched instances would help identify if there have been any exploitation attempts.
Improved vulnerability management with ADM
If you use NetScaler Application Delivery Management (ADM), this is an ideal time to explore the security features in ADM. The first two features below can help reduce your mean time to patch, which we believe is critical in the current threat landscape:
- Security Advisory protects your infrastructure by highlighting NetScaler ADCs with CVE exposure, scheduling on-demand vulnerability scans, and suggesting remediations
- Upgrade Advisory helps you with the lifecycle management of NetScaler ADCs
- File Integrity Monitoring ensures the integrity of the files on NetScaler ADCs by determining if changes have been made to your NetScaler build files
Learn more and stay up to date
Please note that we follow Product Security Incident Response Team (PSIRT) standards in disclosing vulnerabilities, and these standards include the opportunity for qualifying customers to sign up for and receive pre-notifications of vulnerabilities. We aim to provide these customers who opt-in for pre-notification with one to two weeks’ advance notice of the vulnerability severity to allow planning for remediation. You can learn more about our pre-notification program after signing into your Citrix account.
With the holidays and year-end change freezes approaching, we strongly urge NetScaler customers to follow our remediation guidance for CVE-2023-4966 and, more generally, our security best practices available via the links below:
- Read the security bulletin
- Sign up for security bulletin notifications
- Consult the best practices deployment guide
- NetScaler security best practices
- NetScaler Security Advisory and File Integrity Monitoring
- NetScaler Upgrade Advisory
Technical assistance
NetScaler and Citrix are both business units of Cloud Software Group, and we share the same ticketing system. If you encounter issues when you are updating your affected builds, please contact Citrix Customer Support, irrespective of whether your product is NetScaler branded or Citrix branded.