On April 22, 2024, at approximately 10 PM CDT, our ticketing system, HaloPSA, inadvertently sent a large number of emails to our current and previous PCMR customers. We want to assure you that this was not a security incident, and all customer data remains safe and secure.
About the Affected Software
HaloPSA is a ticketing software we use at PCMR to efficiently manage incidents and requests. We deployed this software in March 2024 to replace our old system, Kaseya BMS. Our primary email service, Exchange Online, is integrated with HaloPSA for sending and receiving emails. Most emails for HaloPSA are received through our shared mailbox at support@pcmrcomputers.com, which has been active since September 2020.
The Incident
Upon realizing the issue, our staff initiated our incident response plan. We promptly notified affected individuals via email and a Facebook post. We also created tickets for individuals who called into the store and assigned them to the parent problem ticket for the incident. Within the first hour of investigation, we determined that the issue was a configuration error within the software, not a security event. Consequently, we downgraded the problem from an urgent to a high priority issue.
The incident occurred about 10 hours after a configuration change was made to automations within HaloPSA for automatic ticket and report generation. This change was necessary for the proper functionality of these automations. However, the automation to sync all emails in the PCMR Support shared mailbox started syncing over 3,000 emails stored in the shared mailbox. When emails are synced to HaloPSA, a ticket is automatically created, and an acknowledgement email is sent to the end-user. Users not previously entered into HaloPSA also received a welcome email to the public-facing support portal.
Our Response
We notified the HaloPSA support team of the issue on April 23, 2024, and launched an investigation into why this incident occurred. We found that after the automation configuration changes, emails that had not been synced began syncing. This was not intended behavior and has been documented as a bug with the software and reported to the HaloPSA support team.
Going Forward
In the following weeks, our staff discussed next steps to prevent similar incidents in the future. We have instituted several changes:
Change Management Process: We have expanded our change management process to include all changes for internal and client environments. This includes operating system patches, software patching, network changes, hardware configuration changes, and software configuration changes. A copy of the Change Management Process is available to current managed customers upon request.
Communications: We have introduced a communication plan to ensure end-users are aware of changes and incidents as they occur. This includes Facebook posts on the PCMR Facebook page, emails to affected end-users, and announcement banners in the PCMR Support Portal.
Shared Mailbox Management: To prevent large amounts of emails from being synced in the future, all emails will now be synced to Halo for ticket creation. After the sync, the email will be deleted from the shared mailbox.
Employee Training: We have trained our employees on how to create problem parent and child tickets. If a problem (large scale incident affecting multiple clients) has been identified, a problem ticket will be created. All incident tickets from individuals will be marked as a child ticket of the problem.
Conclusion
At PCMR, we strive to be your Trusted Local IT Provider. We take incidents like this very seriously and have made the necessary changes to continue offering the fast, reliable, quality services you have come to know and love.
As a recap, no customer data was lost during this incident. The configuration issue has been resolved, and a mitigation strategy has been implemented.
Thank you for your continued support of our small business. For questions and inquiries regarding this incident, please contact PCMR Support at support@pcmrcomputers.com or call (660) 864-0039.