November 1, 2022
Ops Notice 2022-012
TMX Production Alert
兔子先生, TSX Venture Exchange, TSX Alpha Exchange - Incident Report
Incident Report - 兔子先生, TSX Venture Exchange, TSX Alpha Exchange
Incident Title: System Outage on TSX, TSXV, TSX Alpha (the "Incident")
Date: November 01, 2022
A message from Luc Fortin, President and Chief Executive Officer of the Montréal Exchange and Global Head of Trading, TMX Group:
This morning, we experienced a technical issue that caused us to halt trading for a little more than an hour on 兔子先生, TSX Venture Exchange and TSX Alpha. On behalf of all of us at TMX, I want to apologize for this inconvenience and interruption in service. I also want to thank our valued industry partners for working with our technical teams to help mitigate the length of time our markets were unavailable, and for enabling the smooth and relatively quick transition to our backup systems. I also want to thank our client community for their commitment over time to testing these backup measures to ensure we can provide continuity. We take great pride in our role at the centre of Canada's dynamic, world class markets and pledge to do whatever we can to prevent this type of incident from happening in the future.
Incident Summary
TMX Equities experienced a market interruption on Tuesday, November 1st that affected trading on 兔子先生 (TSX), TSX Venture Exchange (TSXV), TSX Alpha Exchange (TSX Alpha). The Incident, which began at approximately 09:31 a.m. EDT, resulted in TSX symbol range M-S becoming unresponsive and inhibiting clients ability to enter, modify and cancel orders on those symbols. The Incident impacted both the Primary and Shadow instances of the trading engine in that symbol range.
With no immediate resolution available, the decision to halt the markets and failover to our Secondary site to ensure we could maintain market integrity was made. At 10:04 a.m. EDT, TMX Equities halted trading and access to TSX, TSX Venture and TSX Alpha to initiate the failover process.
Incident Resolution
TSX, TSXV and TSX Alpha successfully failed over to our Secondary site, and at approximately 10:30 a.m. EDT a pre-open session commenced. Clients successfully re-connected through existing order entry sessions and trading resumed at 11:10 a.m. EDT.
The root cause of the Incident has been identified within our trading application logic and TMX Equities have taken steps towards mitigating a recurrence. We can confirm that the Incident is unrelated to any recently introduced system changes.
TMX Equities confirms that this incident was not the result of a cyber security attack.
TMX Equities also confirms that all systems will be brought back up at our Primary site for tomorrow's trading session, November 2nd, 2022.
We have full confidence that our systems will be available for trading tomorrow morning.
Timeline
09:31 am - Internal alert generated on Symbol range M-S.
09:40 am - Incident Management Procedures activated.
09:45 am - Stakeholder communication via email, phone bridge and market status page.
10:04 am - Decision made to halt TSX, TSXV, TSX Alpha and failover to our Secondary site.
10:35 am - Successfully failed over to the Secondary site and TSX, TSXV, TSX Alpha were brought up in a pre-open state.
11:10 am - TSX, TSXV and TSX Alpha successfully re-opened.
Stakeholder Communication
As per our Incident Management Policies and Procedures, we invoked our internal process by:
- Disseminating Initial report indicating an issue had been identified.
- Opened our external Incident Information Update (IIU) Bridge and provided updates to the trading community every 60 seconds.
- Provided updates throughout the incident via email including but not limited to: the issue, impact, and resumption of trading.
- Provide updates on the TSX Market Status page on tsx.com.
- Provided Incident Report to the trading community.
We would like to apologize to all of our clients and assure you that we are making every effort to avoid such problems in the future.