Tokyo Stock Exchange, Inc. (TSE) would like to offer its sincere apologies to investors and the many others involved in its markets for the inconvenience caused by the Derivatives Trading System malfunction that occurred on July 22, 2008.
Below is a summary of the causes of the system malfunction, the TSE's response to the problem, the locus of responsibility, measures taken for preventing reoccurrences, and other related information.
1. Background
On Tuesday July 22, 2008 at 7:45 a.m., a malfunction occurred in several order book information distribution servers (below "servers") in the derivatives trading system, resulting in the failure of distribution of order book information to trading participants. As such, the TSE suspended auction trading in all derivative products in the derivatives trading system (Japanese Government Bond [JGB] futures, options on JGB futures, index futures, index options, and individual options) at 9:21 a.m.(*1) Later, the source of the problem was ascertained and removed, and the TSE resumed order entry for all products at 1:15 p.m. Auction trading resumed at 1:45 p.m.
The TSE received an order from the Financial Services Agency (FSA) Commissioner to provide a report on this matter pursuant to the provisions of Ar ticle 151 of the Financial Instruments and Exchange Act, and submitted the report to the FSA today accordingly.
*1 Auction trading in straight bonds was suspended at 12:30 p.m., the beginning of trading.
2. Outline of the trading system malfunction report
(1) Cause of malfunction
In the order book information distribution process, the derivatives trading system secures a specific amount of memory for editing such information designated to be distributed. There was a coding flaw within the program that secures this memory that was introduced when performance improvements to the order book distribution process were made, which resulted in a very limited amount of memory being available. When order book information requests exceeded the allocated memory area, the program automatically suspended processing of this information. The date of the malfunction, Tuesday July 22, 2008, was the day designated for the performance enhancements.
The coding flaw was as such: a change was made to the definition method for securing memory space at the design stage, but the contents of the change were misunderstood. In addition, the coding flaw was not identified during the development vendor's review of the program.
Moreover, over the course of acceptance inspections, the coding flaw effectively went unidentified despite numerous tests conducted by the TSE to verify the effects of the performance enhancements. This malfunction continued to go unidentified after conducting connectivity tests in which many trading participants partook as well.
In light of this, we believe that a strengthened framework for checking the program when the development vendor makes upgrades is necessary, and also that there is significant room for improvement in terms of the criteria for choosing points to test as well as the test implementation methods at both the development vendor and the TSE.
(2) TSE response
After the malfunction occurred, some servers were temporarily recovered through the rebooting process, but because the malfunction occurred once again, the TSE suspended all auction trading at 9:21 a.m.
The TSE immediately began working to ascertain the cause of the malfunction, and discovered that the program had a flaw in the processing logic that was created when the program was upgraded. As such, the exchange proceeded to revert the upgraded program back to the original version. Preparations for restarting operations were completed at 12:50 p.m., and the TSE determined that it would begin taking orders at 1:15 p.m. and begin auction trading at 1:45 p.m. The exchange notified involved parties of this fact, and resumed trading as scheduled at 1:45 p.m.
During this period, the TSE kept trading participants informed of the current status of events such as the occurrence of the malfunction, subsequent developments, and trading suspension and resumption through fax and other means, and also kept general investors informed through updates to the TSE website.
(3) Locus of responsibility
The TSE sincerely apologizes for the inconvenience caused to investors and related parties as a result of the system malfunction and the subsequent suspension of trading in the derivatives market, which occurred despite the TSE's responsibility to provide market functions in a steady manner, and also despite the measures implemented earlier this year after the previous system malfunction.
The direct cause of the malfunction is as outlined above, but the TSE also holds responsibility from our position of being accountable for managing and operating the market in a smooth and steady manner. We deeply regret we were unable to live up to this duty
(4) Disciplinary actions
Disciplinary actions for this incident are as follows:
- Reduction in executives' remuneration
The monthly salary of both Chairman Taizo Nishimuro and President/CEO Atsushi Saito will each be reduced by 30% for one month, and the monthly salary of both Senior Managing Director/COO Yasuo Tobiyama and Managing Director/CIO Yoshinori Suzuki will each be reduced by 20%. - Strict censure
Strict censure will be given to both Managing Director/CIO Yoshinori Suzuki and Masayuki Hirose, Director of Trading Systems in the IT Development Department
(5) Measures for preventing reoccurrence
The TSE will tirelessly strive to further bolster market fairness and reliability by promptly examining and steadily implementing the following preventative measures to ensure that a malfunction like this does not occur again in the future.
Both the TSE and the development vendor enacted a series of measures to prevent reoccurrences after the previous derivatives trading system malfunction, but based on the fact that these failed to prevent a reoccurrence, issues continue to persist. As such, the TSE will move forward with improvements by reviewing and strengthening previous efforts, placing an emphasis on the following 3 points:
- Organizational structure to check system upgrades
- When changes are made to function requirements and non-function requirements, a review of test point identification criteria and implementation methods with respect to testing changes other than direct logic changes.
- A fallback contingency plan for when system malfunctions occur (reversion to previous program).
Area for Improvement | Anticipated Implementation |
---|---|
1. Establishment of CIO
Special Mission Project Team Giving particular heed to the fact that a malfunction occurred in the derivatives trading system in February of this year, the TSE will establish the "CIO Special Mission Project Team" made up of members other than employees in charge of the derivatives trading system underneath Managing Director/CIO Yoshinori Suzuki. This CIO Special Mission Project Team will be able to look at this derivatives trading system malfunction and pinpoint problems from a different perspective, and consider response measures without being preoccupied with existing rules and frameworks. |
Complete examination by the end of August |
2. Review of
the TSE's test process flow Taking this malfunction into consideration, the TSE will once again review test identification methods in the test process flow, and will enhance the oversight mechanism for test point identification and implementation performed by the development vendor when upgrades are made. |
By the end of September |
3. External system audits The TSE will implement system audits conducted by external experts in order to ensure the effectiveness and objectivity of the measures taken to improve the system's reliability, stability and efficiency. |
By the end of September |
4. Enhancement of development vendor's verification
methodology for program updates The vendor will check all programs to verify that the memory securing glitch that was the cause of this system malfunction does not exist elsewhere. Meanwhile, the vendor will once again review the identification criteria for program unit tests, internal integration tests, and other test patterns performed when the programs are upgraded. |
By the end of August |
The development vendor will also set up a quality control team that will span all teams within its projects, and this team will perform inspections of all upgrades large and small. | By the end of August |
In addition to this, under the direction of the president of Fujitsu, Inc., a third-party process control team will be created and will perform process re-inspections. | By the end of September |
5. Strengthening of vendor
control With respect to the development vendor's detailed design, coding and internal test process flows, the TSE will create a system for necessary reviews and evaluating whether or not products are of a quality that can undergo TSE acceptance tests. |
By the end of September |
6. Consideration of measures to enable swift recovery
from system malfunctions Considering the fact that working to recover from the system malfunction required a considerable amount of time, the TSE will make improvements by considering methods and frameworks to enable more speedy system recovery. |
By the end of September |