Checklist for Citrix ADC CVE-2019-19781

Citrix has released a critical vulnerability warning (CVE-2019-19781) in all Citrix ADC & Gateway systems one week before Christmas. Several working exploits have been released since Jan. 10, 2020 and are available to everyone.

Important ! The fix from Citrix with the Responder Policy does not work on systems with version 12.1.51.16/51.19, 50.31 and older. If this version is in use, please update to the latest 12.1 version.

The exploits allow remote code to be executed anonymously, allowing unauthenticated attackers to take over the various machines with root privileges.

For quite some time now, massive scanning for Citrix ADC machines has been observed by various institutions. This compiled list is now under attack.

There is now an online checker for the exploit.

cve-2019-19781.azurewebsites.net

Firmware Updates

Citrix is currently unable to provide a patch. However, there are already dates set when these will be released:

Citrix ADC and Citrix Gateway
VersionRefresh BuildExpected Release Date
10.510.5.70.x31st January 2020
11.111.1.63.15Patch is here
12.012.0.63.13Patch is here
12.112.1.55.x27th January 2020
13.013.0.47.x27th January 2020
Citrix SD-WAN WANOP  
ReleaseNetScaler ReleaseExpected Release Date
10.2.611.1.63.x27th January 2020
11.0.311.1.63.x27th January 2020

Workaround

Executes the following commands via the GUI command line interface or directly on the appliance (via Putty).

Command line interface

  • Log on to the affected system and go to the navigation points System > Diagnostics
  • In the menu item Diagnostics on the right side go to Command line interface
  • In the CLI you can enter the following commands and execute them one by one by Go

Directly per Putty

  • Starts Putty and enters the management IP of the affected system
  • Connect to the system and enter your username and password
  • Now you can insert and execute the commands listed below

Standalone System

The following commands are used to create the Responder Actions & Policy.

enable ns feature responder

add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""

add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 

bind responder global ctx267027 1 END -type REQ_OVERRIDE

save config

Now we make sure that the changes also apply to the management interface.

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0

shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"

reboot

HA Pair

On the Primary

Enter the following by Putty or CLI.

enable ns feature responder

add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""

add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 

bind responder global ctx267027 1 END -type REQ_OVERRIDE

save config

Afterwards, switch the HA state of the machine to Secondary via GUI or Shell Command.

  • Navigate to System > High Availability

  • In the Nodes tab, select a node and click Force Failover in the Action list.

Or enter the following command via Putty or CLI:

force HA failover

Now we make sure that the changes also apply to the management interface. Again using Putty or CLI, enter the following.

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0

shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"

reboot

ON THE SECONDARY, AFTER THE PRIMARY IS STARTED

Switch the HA state of the machine back to secondary via GUI or shell command and then enters the following via Putty or CLI

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0

shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"

reboot

Cluster

CLIP

Enter the following by Putty or CLI.

enable ns feature responder

add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""

add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 

bind responder global ctx267027 1 END -type REQ_OVERRIDE

save config

Now we make sure that the changes also apply to the management interface and enter the following via Putty or CLI.

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0

shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"

reboot

ON EACH CLUSTER NODE

Enter the following by Putty or CLI.

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0

shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"

reboot

IN THE ADMIN PARTITION

Enter the following by Putty or CLI.

switch ns partition default

enable ns feature responder

add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\""

add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 

bind responder global ctx267027 1 END -type REQ_OVERRIDE

save config

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0

shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler"

reboot

Review of the systems

Even systems with the protection methods in place should be subjected to a thorough review, as it is never possible to say when attacks may have started before they have become broadly based.

Here is a summary of the articles, Twitter entries and own experiences regarding the attack wave.

Bash script to check the system

Daniel Weppeler has created a bash script to check Citrix ADC / Netscaler for corruption.

Download the script in Git under. bash CVE-NetScalerFileSystemCheck.sh

XML files

The exploits write all files into several different directories. Therefore, check the following directories to see if unknown user names appear there. Meanwhile, hackers delete these files after they have been opened. All commands should be executed by Putty or CLI.

shell ls /netscaler/portal/templates/*.xml

This directory normally does not contain XML files and therefore the following display should follow.

If files have been found, the first access to the system has already happened.

shell ls /var/tmp/netscaler/portal/templates

This directory normally does not exist.

But if no error occurs or there are even files in this folder, the system has been successfully attacked.

Here you see an attack with a .xml.ttc2 file.

The content of the .xml.ttc2 file.

shell ls /var/vpn/bookmark/*.xml

This directory should not exist or only contain XML files from users you know.

If there are XML files in this folder that are unknown, the system was successfully attacked.

If files are present, delete them immediately to detect further attacks directly.

Bash logs

There may also be entries in the bash.logs when attacks occur. We are looking for the user nobody here, since these are his rights if he comes through Apache.

shell cat /var/log/bash.log | grep nobody

shell gzcat /var/log/bash.*.gz | grep nobody

Shown are more attacks.

Apache log files

Furthermore, attacks leave entries in the Apache httpaccess log files.

shell cat /var/log/httpaccess.log | grep vpns | grep xml

Attacks…

shell cat /var/log/httpaccess.log | grep "/\.\./"

Shown are more attacks.

shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml

And more attacks..

shell gzcat /var/log/httpaccess.log.*.gz | grep "/\.\./"

Above is an attack with a nice reference to the open vulnerable under CVE-2019-19781 (11 January 2020 07:39:59) and more attacks in the evening (19:18:11 etc.)

Cron Jobs

Attackers could have installed a backdoor via the cron jobs and thus gain persistent access, even if the vulnerability is patched. Check your crontab file for anomalies.

shell cat /etc/crontab

Listed below is an uncorrupted crontab file.

No alt text provided for this image

Now check if new users have been added. Listed below are the normal users in the passwd file for versions before 13.0.

shell cat /etc/passwd

Here a list for version 13.0.

Now check if one of the existing users has assigned a cron job (here e.g. the user nobody).

crontab -u nobody -l

Here you can see the crontab file on a corrupted server.

Content of this file.

Delete the crontab file in the path:

 /var/cron/tabs

The ERROR response just means that these users have no cron jobs assigned, which is normal behavior.

Nobody processes

In the context of the nobody user, new processes were also started during attacks. Here is an example of the permitted processes.

shell ps auxd | grep nobody

And an attacked server. The top line is malware.

Perl & Python Scripts

By adding your own Perl or Python scripts you can also open a backdoor.

shell ps -aux | grep python

shell ps -aux | grep perl

If there are more than the grep queries shown, the other scripts should be checked by a Google search.

These are normal scripts for an ADC instance hosted in Azure.

Crypto Miners

I have seen several ADC instances that were used as Crypto Miners after the attack. With the following command you can see all running processes. Here no other process, besides NSPPE-xx, should be permanently at 100%. Shown is a normal state.

shell top -n 10

Procedure after update by Citrix (Standalone,CLIP, HA Primary)

Executes the following commands in Putty or via the CLI after the patch from Citrix (end of January) is installed

unbind responder global ctx267027

rm responder policy ctx267027

rm responder action respondwith403

save config

Remove the nsapi command from the rc.netscaler file.

shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=1

shell "sed -i '' '/skip_systemaccess_policyeval=0/d' /nsconfig/rc.netscaler"

reboot

A restart of the instances is not necessary to apply the directive. It is a preventative step to ensure that all open sessions received via the vulnerability are deleted.

Countermeasures for affected systems

As DJ Eshelmann writes in his Blog, if the system has been corrupted, you should do the following

  • Take the ADC instance from the network
  • Changes the password of all LDAP or other AD / network accounts stored on the ADC
  • Issues a new SSL certificate and a new key file for all client SSL files on the device (The keys are stored in files on the ADC that could theoretically be read by the attacker)
  • If it is a VPX appliance and snapshots of the appliance (older than January 9, 2020) are available, it may be worthwhile to restore it first, but this is NOT A WARRANTY for security
    • To be on the safe side, export the configuration file of the affected system and copy / restore it to a new fresh VPX appliance
    • Make sure that the same hardware address is used, otherwise the license is invalid and must be requested / imported again
    • Disconnect the network before starting
    • Launch the appliance and check via the console that the VPX is not compromised
    • Change the nsroot password
    • Connect the internal network only
    • Perform the fix listed above
    • Connecting the external network
    • Keep an eye on the protocols and responder hits
  • Replace the SSL certificates on the appliance at the earliest possible time
  • Revokes the compromised SSL certificates and SSL keys

3 thoughts on “Checklist for Citrix ADC CVE-2019-19781”

    1. Yes, I know, therefore my comment that these scripts must then be checked. The first results tell you directly that you don’t have to worry. If you have a screenshot about such a request, I can deposit it gladly. Unfortunately I only had one about the scripts that run by default on a system hosted in Azure.

  1. In remediation steps for a compromised Netscaler/ADC you also have to make sure to actively revoke the old SSL certs for protecting from MITM attacks after having recreated and redeployed the new certs (with new private keys ) and installed theses everywhere they have to be used (i.e other systems than the Netscaler that was compromised if certs are shared between systems in such a way, think about wildcard certs sprawl generally found in so many IT infrastructures)

Leave a Reply

Your email address will not be published. Required fields are marked *

* I consent to having this website store my submitted information so they can respond to my inquiry.