SpaceX resorts to ‘Creative’ Testing in Falcon 9 Explosion Investigation

SpaceX is continuing to narrow down the root cause of the Falcon 9 explosion on September 1st atop its Florida launch pad that had proven to be the most complex failure mechanism encountered in the company’s 14-year history.

Tracing down the source of the destructive event which claimed the loss of the Falcon 9 and the Israeli AMOS-6 Satellite had proven difficult due to the rapid timeline of the initial blast that caused Falcon 9 to burst into a ball of fire eight minutes ahead of a scheduled test firing of its engines.

However, company officials said in the past week, tremendous progress is being made zeroing in on the root cause that, per current knowledge at SpaceX, is related to operational aspects as opposed to a fundamental flaw in the rocket’s design – good news for a timely return to flight of the Falcon 9.

SpaceX President Gwynne Shotwell spoke twice last week, at the APSSC Satellite Conference in Malaysia and the annual meeting of the National Academy of Engineering in Washington, reaffirming that the culprit lies within “business processes” and continuing to express hope that Falcon 9 can resume flying before the end of the year.

SpaceX confirmed on September 23 that the explosion was due to “a large breach in the cryogenic helium system of the second stage liquid oxygen tank.” Being more specific, Shotwell said the prime candidate for the failure was a helium pressure ‘letting go’ – industry speak for rupturing.

The COPVs, Composite-Overwrapped Pressure Vessels, are responsible for storing high-pressure helium which is heated up in flight and supplied to the fuel and oxidizer tanks to keep them at the proper pressure levels to ensure the vehicle’s stability and propulsion system operation. Falcon’s COPVs measure around 60 centimeters in diameter, stand 1.5 meters tall and a pressurized to over 350 bar.

On Falcon 9, COPVs on the first and second stage are installed within the Liquid Oxygen tanks and immersed in the sub-cooled oxidizer.

Submerging the helium bottles in the supercold oxidizer has the sole purpose of densifying the gas and thus filling a greater mass of pressurant into the bottles, reducing the number/size of COPVs needed to hold the necessary helium. However, this creates a challenging thermal environment for the COPV material.

Falcon 9 is not the only rocket featuring pressurant vessels* submerged in liquid oxygen – others include Orbital ATK’s Antares and the Russian Soyuz 2-1B and 2-1v, though none use LOX chilled to the temperatures SpaceX requires to achieve the high performance of the Falcon 9 FT.

Per Shotwell’s comments, SpaceX currently explores different operational deficiencies that could cause a COPV to fail in a dramatic way. This implies SpaceX is closely looking at the interaction between the helium COPV, its contents, the LOX in the surrounding tank and pressures within the vessel – creating a complex system with many variables (in addition to the fact that practical experience with densified LOX in rocketry is sparse).

The incident occurred at a time in the countdown when the second stage was being loaded with sub-cooled LOX and high-pressure helium. Interaction between the pressurized bottle filled with warm helium and the surrounding oxidizer is most critical when the COPVs are slowly submerged in the rising LOX level which is suspected to occur around this time in the count.

Potential causes of the COPV failure could include weakening of the structural integrity of the COPV due to the thermal difference between the metal liner on the inside subjected to the warm helium and the carbon fiber overwrap material on the outside in contact with the cold oxidizer. Acoustic phenomena resulting from the high-pressure helium being forced into the tank at a critical time during the LOX loading sequence are also on the table as possible scenarios that may have transpired.

The previous iteration of the Falcon 9 used Liquid Oxygen at boiling point temperature and began loading its tanks over three hours ahead of launch – permitting the COPVs to be fully chilled prior to applying high pressures. Falcon 9 FT enters LOX load on the second stage with just 19.5 minutes on the countdown clock followed by Helium load just over 13 minutes prior to launch – an aggressive tanking sequence unprecedented in the space launch business.

It is also understood that SpaceX was testing modifications to the countdown sequence on the Static Fire Test for the previous Falcon 9 mission with JCSat-16 to introduce window management capabilities for the FT version of Falcon 9 that initially had to launch very shortly after propellant loading finished in order to avoid the chilled propellants warming up inside the tanks. These modified countdown steps include adjustments to engine chilldown as well as the propellant and pressurant loading sequence.

It is possible that, with these seemingly minor adjustments to the sequencing of events, SpaceX has inadvertently designed a chain of events that overstressed the Helium bottles.

To gather data on the potential scenarios that can lead to a COPV failure, SpaceX has taken to its McGregor test site where, according to Shotwell, a lot of tests were being run on the COPVs. These tests, in all likelihood, are running different propellant and helium loading modes on highly instrumented tanks to learn about the critical chain of events in both loading sequences that overstress the COPVs.

Observant SpaceX fans passing by McGregor noted experimentation involving LOX tankers and helium supplies was underway in an open area of the facility.

Matching data from an instrumented, destructive test to the telemetry from the rocket gathered in the 93 milliseconds from the onset of the anomaly to loss of data can tell SpaceX whether their modified tanking sequence is indeed to blame for the unfortunate accident.

Identifying an operational deficiency as the root cause beyond any reasonable doubt would be the best case scenario for SpaceX, allowing them to return to flight relatively quickly compared to a design flaw requiring re-work of all existing and future Falcon 9 stages.

On the flip side, a purely ‘business-process’ related flaw can tell a lot about the ‘NewSpace’ culture ongoing at SpaceX – raising questions about other shortcuts taken to save money, cut processing time or increase launch-on-time probability and, as a result, taking hardware to or beyond design limits.

Whether SpaceX can return Falcon 9 to flight before the end of the year depends on the speed of the investigation as well as the status of the company’s launch facilities.

Space Launch Complex 40 was damaged in the September 1 explosion and requires a new Strongback to be built in addition to repairs on various ground systems – a multi-month process.

Launch Complex 39A at the Kennedy Space Center was being outfitted for Falcon 9 and Falcon Heavy missions beginning as soon as November, however, with Hurricane Matthew passing through the area last week, keeping that timeline may be a challenge. SpaceX confirmed no major damage occurred at LC-39A in the wake of Matthew, though further south, a payload processing building used by SpaceX suffered external damage.

SpaceX’s third launch pad, SLC-4E at Vandenberg Air Force Base in California, was unharmed in a recent wildfire but the infrastructure at the base took significant damage and repair work is just starting out in earnest.

*An earlier version of this article stated that “Falcon 9 is not the only rocket featuring COPVs submerged in liquid oxygen.” This incorrect statement was corrected to reflect that the examples of launch vehicles given in this article do use pressure vessels submerged in Liquid Oxygen, but do not employ Composite Overwrapped Pressure Vessels.