Cutting Corners & Lack of Operational Protocols doomed Japan’s Hitomi Observatory

It had already been established that Japan’s Hitomi Space Observatory was lost in a complex, cascading chain of onboard failures.

Details were now shared on the proximate causes of the spacecraft’s loss, highlighting that Hitomi’s fate was sealed by the deliberate disabling of safety systems, a willingness to accept operational risk and a lack of operational protocols for a major change on the spacecraft’s thruster system.

The 2,700-Kilogram spacecraft spun itself into pieces on March 26 after a series of errors in its attitude and control systems fed incorrect information to the satellite’s computers, resulting in a spin-up that ultimately caused the solar arrays and optical bench to rip off. Hitomi dove into its deathly spin a little over a month after its successful launch when test observations of high-energy X-ray sources in the universe had just begun. The mission was declared lost on April 28.

In its detailed post-mortem investigation, the Japanese Aerospace Exploration Agency pieced together a complex chain of failures finding their cause onboard the spacecraft, its design and a lack of documentation governing the creation of commands sent to the vehicle.

JAXA only had very limited telemetry from the satellite after the occurrence of the attitude control problem on the satellite and had to rely on ground-based simulators to re-create what transpired aboard the spacecraft.

Hitomi’s long dance with death started at 18:01 UTC on March 25 when the spacecraft entered an attitude maneuver to point its four X-Ray telescopes from Crab nebula to the next observation target, Markarian205. At 19:09 UTC, the satellite exited from a period when Earth was blocking the field of view of its Star Trackers and the system completed a short acquisition before entering tracking mode – triggering a mode switch within the Attitude Control Flight Software.

The initialization of attitude control after a maneuver was set up to rapidly complete attitude determination in order to have the satellite pointed at an observation target as quickly as possible, maximizing observation time. To accomplish that, the Kalman-filter employed by the software was programmed to operate at higher gain at the point of initialization which can result in the Inertial Reference Unit (IRU) data jumping to a high bias rate.

This occurred at 19:10 UTC and was not yet a problematic situation as attitude data converged back towards the actual value when Star Tracker Data became available for processing. In previous cases, the time-integration algorithm quickly led to IRU biases converging to nominal values.

However, Star Tracker data dropped out at some point between 19:10 and 19:14 UTC when the system switched from Tracking Mode back into Acquisition mode.

This was due to only a few bright stars being available for tracking above a programmed window pixel size threshold. This problem had been known since early in the mission and an adjustment of window-pixel size was planned to be implemented before the end of satellite commissioning to avoid Star Tracker drop-outs.

With no more data from the Star Tracker (STT), the Attitude Control Flight Software only used the biased IRU data which had stalled at a body rate of 21.7°/hour when the spacecraft was actually in a stable posture.

Hitomi was outfitted with a pair of Star Trackers for redundancy, but the system was not programmed to operate the trackers in a hot-backup mode. In case of an anomaly with one STT, the software was programmed to disregard the input from that tracker and only use attitude estimates coming from the Attitude Control Flight Software which – by that point – was purely relying on erroneous IRU measurements.

Powering up the second STT could have saved the mission, but was suppressed to avoid lengthy attitude variations. When Star Tracker 1 finally re-entered tracking mode around 19:14 UTC, the difference between the onboard attitude solution and that provided by the Star Tracker was greater than one degree.

In this case, the spacecraft software instructed to simply reject STT data altogether.

The reasoning behind this was avoiding sporadic noise of the STT. Relying on IRU data only had been seen as an acceptable condition given the comparative accuracy and stability of IRU readings.

A scenario where IRU bias rates were fixed at high values was not foreseen. Additionally, no software implementation was made to transition Hitomi to safe mode in case of lengthy periods of STT data rejection because designers favored measures to be implemented through ground intervention (avoiding unnecessary safe hold periods).

The chosen approach placed more authority on the Mission Control Team that had to monitor the “non-update STT flag” and manually intervene in case of a longer rejection of Star Tracker Data. Hitomi sent clear indications of an attitude anomaly on three ground station passes at 20:49, 22:31 and 0:52 UTC, though no response came from the engineering team due to a lack of specific operational instructions for a continuous non-update flag.

As part of the post-mortem study, JAXA found that problems with the Star Trackers unexpectedly switching from Tracking into Acquisition mode had been known within a short time after launch.

As a result, the star trackers were simply placed into standby mode when stars were eclipsed by Earth. Updates of Star Tracker Parameters were to be implemented later in the mission.

With this known fault of the Star Trackers, the mission team chose the riskier path of continuing test observations instead of fine-tuning the Star Trackers first.

Another observation made in the failure investigation notes that attitude maneuvers during the early mission phase were carried out at the very end of the visibility window of Japan’s ground station network – eliminating insight into the progress of maneuvers and the satellite’s status.

The next fail-safe system that could have saved Hitomi, but was disabled, was the Coarse Sun Aspect Sensor. Normally, the sun sensors can be used to judge an abnormal attitude situation and then guide the satellite back into a sun-pointed orientation for a safe hold until instructions from Earth are available. Hitomi did not use data from the sun sensors in its Fault Detection, Isolation and Recovery system because of the possibility of unnecessary transitions into safe mode reducing observation time.

This decision was made because the sun sensors used on Hitomi only have a 20-degree linear field of view where 30° were needed for the typical spacecraft attitude range.

Without this safeguard in place, Hitomi could not enter a Reaction Wheel Safe Mode at which point the Attitude and Orbit Control Processor would have transitioned to a redundant unit and the IRU bias would have reset.

Essentially, designers allowed one attitude measuring device overruling all other systems, introducing a single point of failure.

With three layers of safety either missing or deliberately disabled, Hitomi began a multi-hour build-up of attitude rates.

The Reaction Wheels were commanded to counter the sensed 21.7°/h rotation, leading to the satellite spinning up. Momentum on the reaction wheels reached 112Nms according to telemetry from the spacecraft, close to the 120Nms design limit. The magnetic torquers were commanded to unload momentum from the wheels but the system was not effective since the satellite was already out of a controlled orientation.

Finally, Hitomi switched to Reaction Control System Safe Mode between 0:52 and 1:04 UTC on March 26 – several hours after it would already have done so if its attitude safety systems had been configured properly. By this time, the mission was not yet lost and easy recovery of attitude control was possible through the use of Hitomi’s thruster system.

It had already been uncovered that the spacecraft’s thruster parameters on that fateful March day were not appropriate for the vehicle’s configuration after deploying the 6.3-meter long Extendable Optical Bench (EOB).

EOB deployment occurred on February 28 and changed the spacecraft’s center of mass and moment of inertia – two extremely important properties when attempting to induce precisely controlled moments of torque in specific directions through the use of thrusters.

JAXA now revealed that no documentation or operational plans existed for the change of thruster parameters to reflect the new spacecraft configuration.

A support company was in charge of the operation and was only instructed on February 25 to deliver a new set of thruster parameters – technical details on how the parameters were changed were not shared between that company and JAXA.

The generation of thruster parameters was a process involving multiple steps, starting with a developer tool known as the RCS Drive Matrix Generation Tool which delivered a matrix of four rows and six columns.

This tool delivered a matrix containing six negative numbers while the next developer tool, the Thruster Parameter Table Generation Tool required these numbers to be entered as absolute values (omitting the minus).

The operator in charge failed to make this simple, but vital conversion. The operator had experience working with the tools but was doing this specific work for the first time. No manual for the process was available and no specific training was carried out.

After the invalid data was entered into the Parameter Generation Tool, two stages of verification failed. No ground simulation of the new parameters was run at the support company due to a miscommunication between staff and JAXA accepted the new thruster parameters without an operational readiness check.

The result of this long chain of design flaws, operational deficiencies and a willingness to accept greater risk was Hitomi spinning out of control, causing the solar arrays and optical bench to be flung from the spacecraft and prematurely end a promising mission of scientific discovery.

JAXA will continue its investigation to deliver corrective measures for future space projects to learn from Hitomi’s failure and avoid a similar fate.