Global CrowdStrike Outage Impacting Services Update July 19, 2024 UPDATE: We have completed the process of addressing the availability issues affecting windows servers hosted in our infrastructure. We have remediated all servers within our environment at this point. If you continue to experience any issues within your environment, please reach out to us via the support@carbon60.com ticket submission process, please include any specific details about the symptoms you are seeing with your application. —————————————- We are aware of the global outages caused by CrowdStrike and are actively working to remediate the affected Windows Servers. This is not a security incident. The CrowdStrike team has identified a manual workaround which the Carbon60 team is working diligently to implement. To learn more, see below or visit: Statement on Windows Sensor Update – crowdstrike.com Summary CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. Details Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted. Windows hosts which are bought online after 0527 UTC will also not be impacted This issue is not impacting Mac- or Linux-based hosts Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version. Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version. Current Action CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes. If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue: Workaround Steps for individual hosts: Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: Boot Windows into Safe Mode or the Windows Recovery Environment Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Boot the host normally. Note: Bitlocker-encrypted hosts may require a recovery key. Workaround Steps for public cloud or similar environment including virtual: Option 1: Detach the operating system disk volume from the impacted virtual server Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes Attach/mount the volume to to a new virtual server Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory Locate the file matching “C-00000291*.sys”, and delete it. Detach the volume from the new virtual server Reattach the fixed volume to the impacted virtual server Option 2: Roll back to a snapshot before 0409 UTC. Workaround Steps for Azure via serial Login to Azure console –> Go to Virtual Machines –> Select the VM Upper left on console –> Click : “Connect” –> Click –> Connect –> Click “More ways to Connect” –> Click : “Serial Console” Step 3 : Once SAC has loaded, type in ‘cmd’ and press enter. type in ‘cmd’ command type in : ch -si 1 Press any key (space bar). Enter Administrator credentials Type the following: bcdedit /set {current} safeboot minimal bcdedit /set {current} safeboot network Restart VM Optional: How to confirm the boot state? Run command: wmic COMPUTERSYSTEM GET BootupState Latest Updates 2024-07-19 05:30 AM UTC | Tech Alert Published. 2024-07-19 06:30 AM UTC | Updated and added workaround details. 2024-07-19 08:08 AM UTC | Updated 2024-07-19 XXXX AM UTC | Updated Support Find answers and contact Support with our Support Portal
