The Australian Signals Directorate’s Australian Cyber Security Centre (ACSC) is aware of ongoing attempts to exploit a critical vulnerability in Citrix Application Delivery Controller (ADC) (formerly known as NetScaler ADC), Citrix Gateway (formerly known as NetScaler Gateway) and Citrix SD-WAN WANOP.
The vulnerability, known as CVE-2019-19781, was disclosed on 17 December 2019 and enables an unauthenticated adversary to execute arbitrary code.
Due to active exploitation of this vulnerability, organisations running the affected applications that did not implement Citrix’s mitigations before 11 January 2020 should attempt to identify and remediate successful exploitation of their Citrix servers. Advice is provided in the Detecting Compromise and Remediating Compromise sections below.
What you need to do
On 19 January 2020, Citrix released patches for two versions of the Citrix Application Delivery Controller (ADC) and Citrix Gateway appliances. Citrix expects to have patches available across all supported versions of Citrix ADC, Citrix Gateway and Citrix SD-WAN WANOP before the end of January 2020.
ACSC strongly encourages organisations to immediately apply available patches, available from https://support.citrix.com/article/CTX267027
For versions which do not currently have a patch available, ACSC strongly encourages affected organisations to immediately follow the mitigation steps provided by Citrix, available from https://support.citrix.com/article/CTX267679.
Affected versions include:
- Citrix ADC and Citrix Gateway version 13.0 all supported builds
- Citrix ADC and NetScaler Gateway version 12.1 all supported builds
- Citrix ADC and NetScaler Gateway version 12.0 all supported builds
- Citrix ADC and NetScaler Gateway version 11.1 all supported builds
- Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds.
Check Citrix server “httpaccess.log” and “httperror.log” file for indicators of exploitation. Noting that tradecraft of this vulnerability is evolving, ACSC currently recommends looking for the following:
- POST or GET requests to paths containing “/vpns/” indicating access to potentially vulnerable resources such as “newbm.pl” and “rmbm.pl”
- GET requests which contain code such as
- POST or GET request to XML files which have been recently created or have unusual filenames.
Note that these logs may be compressed due to file size limits or aging policies. Ensure archived versions are also checked.
If packet captures are available, a common HTTP header field used during the exploitation process is “NSC_USER”. The ACSC recommends looking for suspicious field values such as the example below:
If there is any evidence of malicious activity present within the above logs, further analysis should be undertaken using the following artefacts:
- Process Listing - Look for any suspicious child processes of “httpd” owned by user “nobody.”
- File System – Look for any recently created or unusual XML files, specifically in locations which have permission to write and execute files such as:
- bash.log - Look for any suspicious executables such as curl, hostname, uname or whoami, or commands run by user “nobody”. This file contains information on command executions even if the environment variable HISTFILE has been unset.
- Scheduled Tasks – Look for cron jobs that have been created to run as user “nobody”. By default, there should be no scheduled cron jobs run as user “nobody.”
Detecting post-compromise actions
The ACSC’s analysis to date identified instances where actors have installed web shells in additional locations. This is suspected to be used as a secondary access method. The file paths observed by the ACSC include:
Other filenames and directories are possible.
This web shell is a variation of the commonly used China Chopper web shell. More information on this family of web shells is available at https://www.fireeye.com/blog/threat-research/2013/08/breaking-down-the-china-chopper-web-shell-part-i.html.
The ACSC recommends that agencies that have identified successful exploitation of their Citrix NetScaler devices analyse publicly accessible web roots for web shells. Since Citrix NetScaler devices support a variety of programming languages, the ACSC recommends all PHP, Python and Perl scripts are inspected for evidence of web shells.
If you detect compromise, we recommend that you take the following actions to remediate:
- Implement patch if available or follow the mitigations described in the Citrix support article https://support.citrix.com/article/CTX267027
- Copy identified malicious XML files to external device and remove the original malicious files from the following directories:
- Validate all cron jobs created to run as user “nobody”. By default, there should be no scheduled cron jobs run as user “nobody.”
- Clear or reset authenticated session cookies.
- Reboot your Citrix server to disconnect any active connections from malicious actors.
- Reset passwords for all local accounts on the Citrix server.
- Perform analysis of XML files and other forensic artefacts to identify further mitigation actions.
Indicators of Compromise (IoCs) identified by the ACSC
The following list includes locations of tools that were installed post compromise:
The observed instances of the above webshell each use a unique 16-character password. As such, hashes cannot be provided as a reliable indicator to assist organisations in identification efforts.
Read the Citrix Security Bulletin: https://support.citrix.com/article/CTX267027.
Read ACSC’s guidance on how organisations can prepare for and respond to a cyber security incident.
To report a cybercrime, visit ReportCyber.
To learn more about the OAIC Notifiable Data Breaches scheme, visit the OAIC website.