An unusually high number of paperless prescribing timouts was detected and reported by customers the morning of Jan 15 2021. It was then identified to be affecting all paperless prescriptions and to be caused by a messaging synchronisation issue.
Postmortem report
Instructions
Report
Impact
When issuing a paperless prescription the response would always timeout, resulting in the GP having to then revert to a printed prescription.
Response
SRE team contacted other engineering members on Teams to promptly investigate.
Timeline
All times in 24h AEDT 09:29 - Incident ticket logged after several calls had been received by Customer Care 09:57 - SRE escalated and investigation began 10:10 - Confirmed there was no issue on the MDX server side or processing the payload 11:02 - The app service was restarted 11:34 - The app service needed to be restarted again due to not starting correctly 11:54 - Confirmed app service running correctly, then continued to monitor 12:07 - Confirmed with select customers that the issue was resolved, then continued to monitor 13:50 - Incident status changed to Resolved
Root cause
Suspect a combination of an issue where multiple responses could cause a timeout for creation requests along with load/timing of messages.
Resolution
The issue was resolved after the app service was restarted. Subsequently a fix for the issue of multiple responses causing a timeout for creation requests has been applied.
Posted Feb 01, 2021 - 15:06 AEDT
Resolved
After monitoring the Helix fix, we can confirm that it has been successful. All functions are now fully restored. We apologise for the inconvenience caused and thank you for your patience.
Posted Jan 15, 2021 - 13:50 AEDT
Monitoring
We have implemented our tested solution to Helix. We are closely monitoring all functions. You will be advised of any further developments.
Posted Jan 15, 2021 - 11:59 AEDT
Identified
We are currently experiencing issues with Electronic Prescribing feature.
The issue has been identified and our engineering and operations teams are working hard to resolve it. Every effort is being made to minimise the duration and impact.
We will send an update as more information becomes available. We apologise for any inconvenience this limitation to Helix features has caused.
Posted Jan 15, 2021 - 11:04 AEDT
Update
We are continuing to investigate this issue.
Posted Jan 15, 2021 - 10:58 AEDT
Investigating
We are currently experiencing issues with Electronic Prescribing feature.
Our engineering and operations teams are working hard to resolve it. Every effort is being made to minimise the duration and impact.
We will send an update as more information becomes available. We apologise for any inconvenience this limitation to Helix features has caused.