Brillium platform upgrade performance challenges

Incident Report for Brillium

Postmortem

Brillium Platform Upgrade Incident – Post Mortem

Incident Summary:
On March 10, 2026, Brillium experienced an unplanned service disruption during a scheduled platform upgrade. While Brillium upgrades are typically completed without customer impact and undergo extensive testing, this release resulted in unexpected performance issues that temporarily affected platform availability for our customers.
Our engineering team responded immediately, worked continuously to restore service, and progressively brought customers back online while stabilizing platform performance. The platform is now fully available and operating as expected for all customers.

Customer Impact:
During the incident, customers may have experienced:

  • Temporary inability to access the Brillium platform
  • Slower performance while services were being restored
  • Intermittent access during stabilization efforts

We recognize the importance of platform availability to our customers’ assessment and testing operations and regret the disruption this incident caused.

Root Cause (High Level):
This release introduced new features that required updates to the platform’s underlying data structure, including initializing certain data values as part of the upgrade process. While the upgrade was aggressively tested prior to deployment, the testing did not fully account for the impact of these changes on customers with very large data volumes.
Under those conditions, the upgrade process placed significantly more load on the system than anticipated, which led to degraded performance and ultimately caused platform instability.

Resolution:
To resolve the issue, Brillium’s engineering team implemented targeted adjustments to support the upgrade process at scale, restored customer access in controlled phases, and increased monitoring to ensure platform stability. Once all customers were back online, the platform continued to be closely monitored to confirm normal operation and performance.

Preventative Measures and Improvements:
As a result of this incident, Brillium is implementing the following improvements:

  • Enhancements to pre upgrade testing to better simulate customers with large data volumes
  • Additional safeguards during upgrades that involve data structure changes
  • Improved monitoring and validation steps throughout the upgrade process

These actions are intended to further reduce the risk of future upgrade related service disruptions.

Closing Statement:
We sincerely apologize for the disruption caused by this incident and appreciate the patience shown by our customers throughout the restoration process. Over the past 24 months, Brillium has successfully delivered more than 28 platform upgrades without service interruption through our zero downtime delivery process, which is designed to minimize customer impact.

While we take extensive measures to ensure reliable and seamless upgrades, unplanned issues can still occur in complex technology environments. We take this incident seriously and are applying the lessons learned to further strengthen our upgrade and testing processes. Brillium remains fully committed to providing a stable, secure, and continuously improving platform, and we thank you for your continued trust.

Posted Mar 12, 2026 - 09:45 EDT

Resolved

The Brillium platform is fully available to all customers and is operating as expected. Our engineering team has confirmed system stability following the recent upgrade.

A post‑incident review is currently being completed and will be published on the Brillium Status page once finalized. Thank you for your patience and continued trust.
Posted Mar 11, 2026 - 23:57 EDT

Update

Brillium continues working to ensure all customers are fully up and running following the latest platform upgrade. At this time, only one customer remains offline, and our engineering team is actively working to restore service.

Once complete, we will continue to monitor the platform to ensure stability and expected performance. This upgrade also introduces new AI‑powered essay grading assistance. Additional information about this feature is available on our website. If you are interested in learning more or experience any issues related to the upgrade, please contact the Brillium Support Team.
Posted Mar 11, 2026 - 08:24 EDT

Monitoring

The Brillium platform has been restored for nearly all customers, with the exception of two accounts. Our engineering team is actively working to bring the remaining customers fully online early this morning (Eastern Time).

We will continue to monitor platform performance over the next 24–36 hours to ensure stability and expected operation. If you experience any issues, please contact the Brillium Support Team for assistance.
Posted Mar 10, 2026 - 22:37 EDT

Update

Many customers are now able to access the Brillium platform as service continues to be restored. The platform is still in the process of stabilizing, and some users may experience slower performance while demand remains high. Our team is actively addressing remaining issues and will continue to provide updates as stability improves. Thank you for your continued patience.
Posted Mar 10, 2026 - 14:54 EDT

Update

The upgrade process is nearing completion, with the majority of customers now successfully migrated. We are completing the final steps and will share another update as soon as the process is fully finished. Thank you for your continued patience.
Posted Mar 10, 2026 - 13:45 EDT

Update

Progress continues as our team works diligently to bring the Brillium platform back online. We appreciate your patience while we complete improvements that will enhance overall platform functionality and reliability. This release will introduce our new AI Essay Grading service which required some migration of existing essays. Additional information about this feature is available on our website. Further updates will be shared as progress continues.
Posted Mar 10, 2026 - 11:55 EDT

Update

Our engineering team is continuing work on the upgrade and has made additional adjustments to support customers with larger amounts of data. These steps will help ensure a smoother and more reliable upgrade experience. We appreciate your patience and will continue to share updates as progress continues.
Posted Mar 10, 2026 - 10:54 EDT

Update

Our engineering team continues to make progress on the platform upgrade. We will provide another update as soon as the Brillium Assessment Builder is back online or if there are any material changes to share.
Posted Mar 10, 2026 - 09:25 EDT

Identified

Our engineers have identified the issue and applying a fix now.
Posted Mar 10, 2026 - 08:08 EDT

Investigating

The Brillium Assessment Builder is currently unavailable due to unexpected performance issues encountered during a platform upgrade. While upgrades are typically completed without downtime, this particular update introduced challenges that were not identified during extensive pre-release testing. We sincerely apologize for the disruption and are working as quickly as possible to restore full service. We will provide another update as soon as more information is available.
Posted Mar 10, 2026 - 08:04 EDT
This incident affected: API & Integrations (API, Zapier Integration), Administration (User Administration and Authentication, Partner Central Custom Administration), Assessment Builder (Assessment Authoring, Assessment Delivery), and Talent (Invitation Management, Recruiter & Candidate Management).