Helix users were unable to print any pathology requests after the Release RT-38.
Postmortem report
Instructions
Report
Leadup
RT-38 release was deployed on Wednesday, Feb 17 night.
Fault
Helix users were unable to print any pathology requests.
Impact
Entire Production stack was impacted.
Detection
CS team raised an incident ticket after a site contacted her for Support.
Timeline
8:41 Incident raised. 8:42 Incident posted in the Incident war room teams channel. 8:42 Engineer responds to the post. 8:46 Engineer confirms not infrastructure related. 8:55 Engineer starts investigating. 9:32 Fix is applied on staging. 9:38 Fix is verified in staging. 9:40 Fix is applied on MT-MEGA (Prod). 9:43 Fix is confirmed in Production. 9:45 Fix is applied on MT-NEXT (PROD). 9:46 Incident is resolved
Root Cause
Library version mismatch was identified.
Resolution
The newer library version was applied in production manually.
Posted Feb 19, 2021 - 14:14 AEDT
Resolved
A fix has been applied to Helix, and we can confirm that it has been successful. All functions are now fully restored. We apologise for the inconvenience caused and thank you for your patience.
Posted Feb 18, 2021 - 09:44 AEDT
Identified
We are currently experiencing an incident related to printing Pathology requests in Helix.
The issue has been identified and our engineering and operations teams are working hard to resolve it as soon as possible. Every effort is being made to minimise the impact to you, your staff, and your patients.
We will send an update in the next hour or as soon as more information becomes available. We apologise for any inconvenience.