Checklist for NetScaler (Citrix ADC) CVE-2023-3519

Table of Contents

Citrix issued an alert yesterday (07/18/2023) about a critical vulnerability (CVE-2023-3519) in all NetScaler (Citrix ADC) & Gateway systems. To date, no working exploits have been published.

Important ! There are no patches for NetScaler (Citrix ADC) version 12.1 or older. These systems have reached their EOL and will therefore no longer be equipped with the necessary fix. In this case please update to the latest 13.0 or 13.1 version.

The vulnerability allows anonymous remote code execution and thus unauthenticated attackers to take over various machines with root privileges.

As we hear from the Citrix community, more and more attacked systems are being found. The first exploits have also been available for purchase on the dark web for some time.

Thanks to Sander Horsman of Conda Security for the further insights and information.

Assetnote has now reverse engineered a script to detect open systems. In the first step, this only affects machines where SAML is enabled.

The first exploits and scanners are now publicly available via GitHub. Attached is the output of the script from Bryan Smith (@securekomodo).

The output on a patched clean system.

And on a system that is not yet patched.

Firmware Updates

Citrix has made a patch available for all supported versions.

NetScaler (Citrix ADC) and NetScaler (Citrix) Gateway
Version	Refresh Build	Expected Release Date
13.0	13.0-91.13	Patch is here
13.1	13.1.-49.13	Patch is here

Instructions for updating the appliances can be found here.

Citrix ADC 101 – Fundamentals

Review of the systems

Even systems with the protections in place should be subjected to a thorough review, as it is never possible to say with certainty when attacks may have begun before they are widely deployed.

Here is a summary of articles, Twitter entries and our own experiences regarding the wave of attacks.

Find out the time of the last update

At the moment, an attack can only be detected by overwritten timestamps in certain files. These timestamps usually only change when updating the NetScaler and therefore it is important to know this date. A good indicator is to look at the date of the unpacked installation packages.
All commands should be executed via Putty or CLI.

First, of course, you should log in with nsroot or another administrative account. After that you can view your extracted installation data with the following command.

shell ls -ll /var/nsinstall

1	shell ls -ll /var/nsinstall

Here you can see the date of the last NetScaler update package (19.07.2023). We should note this, because we need this for the next commands. It is important, that we do not enter the determined date in the following commands, but the determined date in the form YYYYMMDD +1. So in my case 20230720.

Edited files

Now we can check whether certain files have been adjusted since the last update. This would not be proof yet, but a first indication and should be taken seriously.

shell
find /var/vpn/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;
find /var/netscaler/logon/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;
find /var/python/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

shell

find /var/vpn/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

find /var/netscaler/logon/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

find /var/python/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

A common webshell I see in the wild is identified by the file a.php under /var/netscaler/logon. Please also check if this file is present, this is a sign that an attack has taken place.

The files under ns_gui change the timestamp not only when the files are installed, but at every reboot. Therefore, this should be taken into account when checking the timestamp.

shell
find /netscaler/ns_gui/ -type f -name *.php -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

1 2	shell find /netscaler/ns_gui/ -type f -name *.php -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

Or with the following Python command.

shell
python -c "import os, glob, time; newest_timestamp = max(os.stat(f).st_mtime for f in glob.glob('/var/nsinstall/*')); print('\n'.join(os.path.join(dirpath, f) for dir in ['/var/netscaler/logon/', '/var/python/', '/var/vpn/'] for dirpath, _, files in os.walk(dir) for f in files if os.stat(os.path.join(dirpath, f)).st_mtime > newest_timestamp))"

shell

python -c "import os, glob, time; newest_timestamp = max(os.stat(f).st_mtime for f in glob.glob('/var/nsinstall/*')); print('\n'.join(os.path.join(dirpath, f) for dir in ['/var/netscaler/logon/', '/var/python/', '/var/vpn/'] for dirpath, _, files in os.walk(dir) for f in files if os.stat(os.path.join(dirpath, f)).st_mtime > newest_timestamp))"

If nothing has been edited since the last update this is a good sign.

New files (webshell)

We check various directories for new files as well, this would indicate a webshell.

shell
ls -ll /var/nsproflog/*.php
ls -ll /var/netscaler/gui/*.php
ls -ll /netscaler/portal/*.php

shell

ls -ll /var/nsproflog/*.php

ls -ll /var/netscaler/gui/*.php

ls -ll /netscaler/portal/*.php

HTTP error log files

There may also be entries in the http error logs during attacks. We are looking for anomalies regarding .sh and .php here.

zgrep '\.sh' /var/log/httperror.log*
zgrep '\.php' /var/log/httperror.log*
zgrep '\.pl' /var/log/httperror.log*

zgrep '\.sh' /var/log/httperror.log*

zgrep '\.php' /var/log/httperror.log*

zgrep '\.pl' /var/log/httperror.log*

The picture shows that the search for .sh was successful. You can check this directly by looking into the file with an editor of your choice (e.g. vi).

In this case, I just added .sh in the affected file to check my command.

Shell / Bash log files

In addition, attacks leave entries in the shell or bash log files.

shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/sh.log*
shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/bash.log*

Here in the screenshot you can see only my tests and no attack.

Log files

Check log files for known IOCs.

grep -v '127\.0\.0' /var/log/*.log | grep 'nc -l\|/etc/passwd\|/etc/shadow\|python -c\|curl\|\.php'

1	grep -v '127\.0\.0' /var/log/*.log \| grep 'nc -l\\|/etc/passwd\\|/etc/shadow\\|python -c\\|curl\\|\.php'

Edited files with the setuid bit

find /var -perm -4000 -user root -not -path "/var/nslog/*" -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

1	find /var -perm -4000 -user root -not -path "/var/nslog/*" -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;

Nobody processes

In the context of the Nobody user, new processes were also started during attacks. Here is an example of the allowed processes.

shell ps aux | grep nobody | grep -v '/bin/httpd'

1	shell ps aux \| grep nobody \| grep -v '/bin/httpd'

And an attacked server. The top line is malware.

Check Crontab for new entries

With the following command we can examine the current crontab file. It should look like in the displayed image.

shell grep '' /etc/crontab

1	shell grep '' /etc/crontab

There should be no crontab for nobody. This can be checked with the following command.

shell crontab -l -u nobody

1	shell crontab -l -u nobody

I have received several community comments that the following entries are also normal:

/netscaler/adss-licexp.sh –> For trial licenses
0/5 * * * * root /netscaler/iprep –> When IP Reputation is in use
*/5 * * * * root /var/python/bin/python /netscaler/aslearn_health_monitor.py –> When app firewall is used
*/5 * * * * root /bin/pgrep -f /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py; [ $? != 0 ] && /var/python/bin/python /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py –> When app firewall is used

NetScaler HA Systems

During various attacks, the process responsible for HA (nsfsyncd) was disabled on NetScaler HA systems.

shell ps aux | grep nsfsyncd

1	shell ps aux \| grep nsfsyncd

On standalone machines, the process should not appear either. Shown is the case how it should look like on a working system.

httpaccess-vpn log files

Check log files for successful web accesses to unknown resources.

shell zgrep -E -v 'CitrixReceiver' /var/log/httpaccess-vpn.log* | grep ' 200 '

1	shell zgrep -E -v 'CitrixReceiver' /var/log/httpaccess-vpn.log* \| grep ' 200 '

As well as web accesses from HeadlessChrome.

shell zgrep 'HeadlessChrome' /var/log/httpaccess-vpn.log*

1	shell zgrep 'HeadlessChrome' /var/log/httpaccess-vpn.log*

NPPE Core Dumps

In some attacks, a core dump is created via NPPE (NetScaler Packet Processing Engine) after a critical failure in NetScaler. Normally, the following directory should be empty.

shell ls -ll /var/core/1

1	shell ls -ll /var/core/1

This is what it looks like for an attacked system.

Perl & Python Scripte

Adding custom Perl or Python scripts can also open a backdoor.

shell ps -aux | grep python
shell ps -aux | grep perl

1 2	shell ps -aux \| grep python shell ps -aux \| grep perl

If more than the grep queries shown are seen, the further scripts, should be checked by a Google search.

If something appears, run the command again after a few seconds. Some scheduled tasks on the NetScaler also use python or perl. Only if it still runs on the second call, this should be checked.

These are normal scripts for an NetScaler instance hosted in Azure.

When the StoreFront monitor is used, you will permanently see the following Perl script.

Crypto Miners

I have seen several NetScaler instances used as crypto miners after the attack.

With the following command you can see all running processes.

Here, no other process besides NSPPE-xx should be permanently at 100%. A normal state is shown.

shell top -n 10

1	shell top -n 10

Check network and firewall logs

Checks the network and firewall logs for the following events:

Scans of the subnets sent by NetScaler for the protocols HTTP / HTTPS / SMB (Port 80 / 443 / 445)
Spikes in the queries from NetScaler regarding LDAP / LDAPS / DNS / AD (Port 389 / 636 / 53 / 88 / 135 / 137-138 / 464 / 3268-3269) protocols

Countermeasures for affected systems

My recommendation regarding action when the system has been corrupted.

Remove the NetScaler instance from the network
Changes the password of all LDAP or other AD / network accounts stored on the NetScaler.
Issues a new SSL certificate and key file for all client SSL files on the device (the keys are stored in files on the NetScaler, which could theoretically be read by the attacker)
If it is a VPX appliance and there are snapshots of the appliance (older than 3 months), it may be worth restoring them first, but this is NO GUARANTEE of security
- To be on the safe side, export the configuration file of the affected system and copy / restore it to a new fresh VPX appliance
- Note that the same hardware address is used, otherwise the license is invalid and must be requested / imported again
- Disconnects the network before startup
- Starts the appliance and verifies via the console that the VPX is not compromised
- Changes the nsroot password
- Connect only the internal network
- Connect the external network
- Keeps an eye on the logs and responder hits
Replaces the SSL certificates on the appliance at the earliest possible time
Revokes the compromised SSL certificates and SSL keys

Firmware Updates

Review of the systems

Find out the time of the last update

Edited files

New files (webshell)

HTTP error log files

Shell / Bash log files

Log files

Edited files with the setuid bit

Nobody processes

Check Crontab for new entries

NetScaler HA Systems

httpaccess-vpn log files

NPPE Core Dumps

Perl & Python Scripte

Crypto Miners

Check network and firewall logs

Countermeasures for affected systems

Teilen mit:

Like this:

Related posts: