Skip to main content

2020-001-4: Remediation for critical vulnerability in Citrix Application Delivery Controller and Citrix Gateway

On 19 January 2020, Citrix released patches for two versions of the Citrix Application Delivery Controller (ADC) and Citrix Gateway appliances. Citrix expects to have patches available across all supported versions of Citrix ADC, Citrix Gateway and Citrix SD-WAN WANOP before the end of January 2020.

What you need to do

ACSC strongly encourages organisations to immediately apply available patches, available from https://support.citrix.com/article/CTX267027

For versions which do not currently have a patch available, ACSC strongly encourages affected organisations to immediately follow the mitigation steps provided by Citrix, available from https://support.citrix.com/article/CTX267679.

Affected versions include:

  • Citrix ADC and Citrix Gateway version 13.0 all supported builds
  • Citrix ADC and NetScaler Gateway version 12.1 all supported builds
  • Citrix ADC and NetScaler Gateway version 12.0 all supported builds
  • Citrix ADC and NetScaler Gateway version 11.1 all supported builds
  • Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds.

Detecting compromise

Check Citrix server “httpaccess.log” and “httperror.log” file for indicators of exploitation. Noting that tradecraft of this vulnerability is evolving, ACSC currently recommends looking for the following:

  • POST or GET requests to paths containing “/vpns/” indicating access to potentially vulnerable resources such as “newbm.pl” and “rmbm.pl”
  • GET requests which contain code such as GET /vpns/portal/<malcious_code>
  • POST or GET request to XML files which have been recently created or have unusual filenames.

Note that these logs may be compressed due to file size limits or aging policies. Ensure archived versions are also checked.

If packet captures are available, a common HTTP header field used during the exploitation process is “NSC_USER”. The ACSC recommends looking for suspicious field values such as the example below:

NSC_USER: ../../../../netscaler/portal/templates/<malcious_code>

If there is any evidence of malicious activity present within the above logs, further analysis should be undertaken using the following artefacts:

  • Process Listing - Look for any suspicious child processes of “httpd” owned by user “nobody.”
  • File System – Look for any recently created or unusual XML files, specifically in locations which have permission to write and execute files such as:
    • /netscaler/portal/templates
    • /var/tmp/netscaler/portal/templates
  • bash.log - Look for any suspicious executables such as curl, hostname, uname or whoami, or commands run by user “nobody”. This file contains information on command executions even if the environment variable HISTFILE has been unset.
  • Scheduled Tasks – Look for cron jobs that have been created to run as user “nobody”. By default, there should be no scheduled cron jobs run as user “nobody.”

Detecting post-compromise actions

The ACSC’s analysis to date identified instances where actors have installed web shells in additional locations. This is suspected to be used as a secondary access method. The file paths observed by the ACSC include:

  • var/vpn/themes/admin.php
  • var/vpn/themes/default/default.php

Other filenames and directories are possible.

This web shell is a variation of the commonly used China Chopper web shell. More information on this family of web shells is available at https://www.fireeye.com/blog/threat-research/2013/08/breaking-down-the-china-chopper-web-shell-part-i.html.

The ACSC recommends that agencies that have identified successful exploitation of their Citrix NetScaler devices analyse publicly accessible web roots for web shells. Since Citrix NetScaler devices support a variety of programming languages, the ACSC recommends all PHP, Python and Perl scripts are inspected for evidence of web shells.

Remediating compromise

If you detect compromise, we recommend that you take the following actions to remediate:

  • Implement patch if available or follow the mitigations described in the Citrix support article https://support.citrix.com/article/CTX267027
  • Copy identified malicious XML files to external device and remove the original malicious files from the following directories:
    • /netscaler/portal/templates
    • /var/tmp/netscaler/portal/templates
  • Validate all cron jobs created to run as user “nobody”. By default, there should be no scheduled cron jobs run as user “nobody.”
  • Clear or reset authenticated session cookies.
  • Reboot your Citrix server to disconnect any active connections from malicious actors.
  • Reset passwords for all local accounts on the Citrix server.
  • Perform analysis of XML files and other forensic artefacts to identify further mitigation actions.

Indicators of Compromise (IoCs) identified by the ACSC

The following list includes locations of tools that were installed post compromise:

  • var/vpn/themes/admin.php
  • var/vpn/themes/default/default.php

The observed instances of the above webshell each use a unique 16-character password. As such, hashes cannot be provided as a reliable indicator to assist organisations in identification efforts.

Further information

Read the Citrix Security Bulletin: https://support.citrix.com/article/CTX267027.

Read ACSC’s guidance on how organisations can prepare for and respond to a cyber security incident.

To report a cybercrime, visit ReportCyber.

To learn more about the OAIC Notifiable Data Breaches scheme, visit the OAIC website.