Introduction

A utility company was having issues with business workflows and production that was impacting their connectivity. After reviewing the current state of their network, Fortis experts determined that unauthorized user(s) were accessing their system via a Citrix server. This could have occurred due to stolen credentials or a brute force login attack. This type of attack not only results in unauthorized access to data, apps, and other resources, but also serves as an entry point for further attacks. The unauthorized access impacted the company’s current backups along with an encrypted SQL server. The attacker then executed malware to disable servers and encrypt file structures.

The utility company required assistance to determine the current state of their network, to stop and remediate the current attack, and to implement additional security measures that would help identify and prevent unauthorized access to their network via Citrix or any other method moving forward. The Fortis Incident Response team provided assistance to disable the unauthorized access to the company’s network and worked to remediate the environment to a state before the attack occurred based on the customer’s backups.

Incident Status

It was determined that attacker(s) gained remote access to the customer’s network via a Citrix server, then used credentials from three different domain admin accounts to access other portions of the environment. The attacker(s) deleted disk-to-disk backups, disabled terminal servers, encrypted SQL servers, and executed malware, all of which significantly impacted business workflow and production. Also during the Fortis network security review, it was discovered that several unauthorized remote logins to the company’s Veeam proxy server had occurred via the server administrator account and were used to access the backups and network.

Resolution / Remediation

The Fortis team assisted with many areas of the incident response, including providing security recommendations, securing the environment, and contributing to restorative activities.

Fortis cyber security engineers started the process by disabling the affected Citrix server and all domain admin accounts, as well as blacklisting all .exe file types in Cisco AMP (Advance Malware Protection) to prevent the situation from becoming worse. The current state of the network was reviewed to determine if there were are additional areas where the attacker(s) would be able to reenter the network. It was discovered that the Veeam proxy server had unauthorized administrator accounts for remote logins that granted access to the backups and to the company network. Fortis engineers disabled all VPN access until new protection methods were in place to combat the unauthorized access to the network.

Once the network was secured and infected servers / workstations cleared of any viruses / malware, our engineers started the remediation of the damaged and compromised systems. We discovered the only tape backups that had not been deleted were a couple of weeks old. The Fortis team rebuilt the Veeam proxy server since it had been compromised during the attack. Once the Veeam proxy server rebuild had been completed, the remaining compromised or damaged servers were restored using the available backup files.

Fortis engineers deployed Cisco AMP for Endpoint on every server to help block / prevent malware at the point of entry. Cisco AMP was also deployed to gain visibility into file and executable-level activity so malware could be removed at this level.

Fortis deployed additional security measures throughout the customer’s environment to significantly improve protection, detection, and recovery capabilities.

Our team started by working with the company’s IT team to harden the password requirements and reset all user passwords to meet these requirements. This provided an additional level of security, so in the event of another brute force attack these more complex passwords would be tougher and take much longer to crack.

We also implemented Cisco Duo for multi-factor authentication. Duo requires users to confirm their identities before granting them access to corporate applications. Controls allow the company to make application access decisions based on the user’s identity and the trustworthiness of their device(s) rather than the networks from where access originates.

Cisco Identity Services Engine (ISE) was also deployed by Fortis engineers to provide identity access to switches, wireless, and VPN connections. The additional layer of security created by ISE enabled the organization to better determine which corporate issued or approved outside devices should have the ability to log in to the company’s private network and which ones should be restricted to the guest-only public Internet. SCSE also implemented additional ASA firewall rules to harden access to and from the Internet.

Engineers implemented Sentinel’s Backup as a Service (BaaS) to provide air gapped backups through Veeam. Sentinel’s BaaS enables organizations to efficiently protect, locate, and recover critical data across all types of environments and platforms so they can return to business quickly and with minimal disruption following a data loss event. The Fortis Security Operations Center (SOC) was also deployed to provide security monitoring and strategic security guidance. Our 24x7x365 SOC keeps a close eye on the company’s critical infrastructure elements to ensure their sensitive data and applications remain protected and satisfy performance metrics.

Conclusion

The Fortis team was able to determine the state of the network during the attack and identify the penetration points used by the attacker(s). Engineers were able to disable the rogue access within the network and begin the remediation. Restoration was completed using the remaining stable backups as necessary, along with any additional updates required to secure the network. We implemented multiple solutions designed to enhance the security within the network and VPN access. The final portion enabled off-site, air gapped backups to add an extra layer of security and allow for faster and easier restoration should the network become compromised again at some point in the future.

If you are interested in learning more about Fortis security offerings and how we can help protect your environment, please contact us for additional information.