Two NASA Astronauts ventured outside the International Space Station on Tuesday as a quick-response team to deal with the failure of an external command and control unit that occurred just three days prior.
Jack Fischer and ten-time spacewalker Peggy Whitson spent two hours and 46 minutes outside the Station, successfully replacing the faulty computer box and installing a pair of communications antennas on the Destiny laboratory as an extra task.
Coming on the heels of the 200th ISS Spacewalk back on May 12, Tuesday’s EVA had to be put together in a hurry with only two days of lead time per contingency requirements that dictate an immediate EVA be performed in the event of an equipment failure that leaves the Space Station in a non-fault tolerant configuration. This failure occurred at 18:13 UTC on Saturday when the External Control Zone Multiplexer/Demultiplexer 1 (EXT-1 MDM) failed without prior warning.
ISS uses close to 50 Multiplexers/Demultiplexers that essentially built the Station’s brains and nervous system – processing data from thousands of sensors spread across ISS and transmitting commands sent by the crew or the ground to hundreds of actuators. The Station’s MDMs operate on three levels – three enhanced MDMs are used as the main controllers of all of the Station’s functions, a dozen local-level MDMs are responsible for routing commands and data from the user-level MDMs that each have specialized tasks and interface with the firmware controllers, sensors and effectors of the Station’s various systems.
The MDMs come in different sizes and basically consist of a stack of avionics cards including data processing cards, memory units and a series of input/output cards to connect to various systems and payloads.
The EXT-2 MDM that was in a hot-backup configuration took over control of the Station’s External Control Zone and there were no immediate impacts to crew safety and ISS operations after Saturday’s unexpected failure.
The two EXT MDMs provide commanding and telemetry return to a number of external systems, specifically the Secondary Electrical Power System, the Solar Alpha Rotary Joints, the Thermal Radiator Rotary Joint, the Thermal Control System and the Mobile Transporter. Given their importance, the local-level MDMs have a primary and a backup with a strong desire to keep both in an operational state for redundancy.
A failure of both EXT MDMs would have caused a loss of commanding and telemetry insight into the aforementioned systems plus other critical components such as two Control Moment Gyros, Rate Gyro Assembly 2 and the Floating Point Measurement Unit. Although the equipment would still receive power, a loss of both MDMs would cause the rotary joints to stop, impacting power generation and thermal control as well as CMG attitude control. As such, keeping both MDMs in working order is a priority for NASA and the EXT-1 failure set in motion immediate planning for a contingency EVA.
Mission Controllers meanwhile worked through a series of troubleshooting steps, cycling power to the MDM and attempting to send blind commands but the unit appeared failed for good. What caused the MDM to fail remains unknown and will be investigated once the unit or its avionics cards return to Earth for an engineering inspection to rule out any design flaws with this type of newer MDM unit.
The S0 truss segment launched in 2002 aboard Space Shuttle Atlantis already with its two External Control Zone MDMs installed. These MDMs operated without any major upsets for 12 years until EXT-2 failed in 2014 and was replaced with a spare unit in April of that year in a 96-minute spacewalk performed by Rick Mastracchio and Steve Swanson.
On March 24 and 30, EXT-2 and EXT-1 were replaced by NASA Astronaut Shane Kimbrough with upgraded ‘EPIC’ MDMs hosting Enhanced Processor & Integrated Communications for greater computing power plus Ethernet-interface support. Why a component rated for at least ten years of operation failed after only two months will be scrutinized by engineers to ensure the fault was not the result of any design flaw with the upgraded avionics cards that came with the EPIC upgrade.
Over recent years, some of the internal MDMs were switched over to the new EPIC standard to enhance the Station’s computing power, taking ISS from 1990s technology into the current millennium. Externally, the two EXT MDMs were the only ones upgraded to EPIC in order to support the Common Communications for Visiting Vehicle space-to-space communications system that will carry voice and data relay for future commercial crew vehicles.
With no joy troubleshooting the EXT-1 MDM, Mission Control teams began drawing up plans for a contingency EVA and Peggy Whitson received instructions to fabricate a new MDM in orbit – using an External MDM-16 housing and installing the necessary input/output, processing and memory cards per the EXT MDM’s requirements. The newly built MDM underwent diagnostic tests on board to ensure it was functional before moving it outside the Station. EVA procedure reviews and equipment preparations were completed on Monday and the crew reported they were ready to support Tuesday’s excursion.
The crew began setting up for their spacewalk at 7 UTC when Whitson and Fischer donned oxygen masks to begin the lengthy pre-breathe process and prepare their Extravehicular Mobility Units. Pressure inside the airlock was reduced to assist in the process of removing nitrogen from the spacewalker’s blood while getting into their suits followed by detailed EMU checks and 50 minutes of in-suit light exercise.
Once their suits were outfitted with SAFER jetpacks, the two crew members were sealed off inside the smaller crew lock portion of the Quest airlock for the methodical depressurization and suit leak checks. Tuesday’s EVA formally began at 11:20 UTC when the external hatch was open and the two astronauts switched their suits to battery power.
After the typical prerequisites of setting up tethers and inspecting their suits, the two spacewalkers parted ways as Peggy Whitson was headed up the CETA Spur to reach Face 1 of the S0 truss segment while Jack Fischer translated forward to the nadir side of the Destiny Laboratory for the installation of a pair of External Wireless Communications (EWC) antennas – a task left over from EVA-42 that had to be shortened due to an airlock hardware problem.
Arriving at S0, Whitson set up shop at the EXT-1 MDM residing on the zenith (upper) side of the S0 truss segment, reading out the old unit’s serial number as MDM-16E-0102 to double check she was getting ready for the removal of the correct MDM. The EXT MDMs measure 27 by 38 by 42 centimeters in size and weigh around 24 Kilograms – hosting blind-mate electrical/data connectors on their back side, a thermally conductive material on their bottom and three structural bolts & an Ethernet interface on their forward-facing side.
Whitson first disconnected a single NZGL (NASA Zero-G Lever Lock) connector to remove the MDM’s Ethernet connection before getting her Pistol Grip Tool to release a pair of secondary bolts and a central structural bolt to detach the MDM with relative ease.
For Jack Fischer, the EVA was originally planned to consist of a single task on the nadir side of the Destiny laboratory – bolting down a pair of handrails with EWC antennas attached to them and hooking up the antennas to the Station’s data system. Hand rails previously attached to the Lab were removed by Shane Kimbrough on an earlier EVA to make room for the new hand rails combined with EWC antenna.
Expanding the EWC coverage is necessary for the external HD camera assemblies that have begun deployment to the exterior of ISS; the Lab EWC antennas will become the primary receiving point for the four planned HD assemblies at camera ports 8,9, 3 and 13, routing 10Mbps video into the Station’s hard-line comm system for live downlink to the ground via Ku-Band.
Per his task, Fischer made use of his PGT to drive a pair of bolts on each of the hand rails to secure them in their respective positions in hand rails slots 261A and 262A.
For Peggy Whitson, all was smooth sailing until the spare MDM was to be bolted down in the EXT-1 slot. She confirmed there was no visible damage to the connectors in the receptacle and inspected the new MDM before sliding it into its soft-docked position.
When attempting to drive the central jacking bolt, the PGT returned low-torque readings. The initial belief was that the PGT’s wobble socket was at fault and Jack Fischer – who had just finished his bolting tasks with the EWC antennas – was called upon to hand over his PGT to Whitson, however, the bolt still would not engage properly.
Whitson was instructed to release the bolt and remove the MDM for an inspection which revealed several pieces floating away from the unit believed to be metal filings – suggesting the MDM was not seated properly when engaging the bolt. In response, Jack Fischer was dispatched to the airlock to fetch a cleaning tool and a GTEC bag while Peggy Whitson climbed into the S0 truss to be able to hold down the bag over the delicate blind mate connectors of the MDM receptacle while Fischer used a pressurized nitrogen cartridge to blow any debris out of the bolt holes.
The second try of installing the MDM appeared more promising but required some torque adjustments on the primary bolt and a series of attempts to drive the two secondary bolts to their expected turn and torque values, causing the EVA to fall behind the original timeline but eventually ending up with a firmly installed MDM. Next was the re-connection of the Ethernet cable to allow Mission Control to apply power to the MDM.
Mission Control was quick to report that the MDM had started up and headed into diagnostics mode which returned nominal results and allowed controllers to load the MDM software into the unit – fully restoring the Space Station’s external command and control system just two days and 21 hours after the initial failure emerged (for comparison, the contingency EVA after the 2014 EXT-2 failure came after 12 days of single-MDM operation).
For Jack Fischer, the last hour of the EVA was dedicated to hooking the two EWC antennas up to their cable harness – a straightforward but tedious task requiring him to route a series of connectors from the Lab-Node 2 interface area and secure the cables with wire ties.
Closing out the EVA in orderly fashion, both crew members made an inventory of their tools before climbing back into the airlock to hook their suits up to ISS commodities and close the Quest hatch. EVA-43 came to a close at 14:06 UTC after 2 hours and 46 minutes, achieving both its core objectives.
Tuesday’s EVA catapulted Peggy Whitson into third place on the all-time list, now with ten career EVAs for a total of 60 hours and 21 minutes, passing John Grunsfeld and Jerry Ross in terms of EVA time and matching Michael Lopez-Alegria for the record of most EVAs performed by a NASA astronaut. For Jack Fischer, his career EVA time now stands at 6 hours and 59 minutes accumulated on two spacewalks.
For ISS, Tuesday’s excursion marked the 201st EVA in support of ISS Assembly and Maintenance for a total of 1,250 hours and 41 minutes.
The next planned EVA for the ISS Program currently targets August 17 when Russian EVA operations will resume after a year-long gap, debuting a new type of Orlan Space Suit. This EVA will be conducted by Expedition 51/52 crew member Fyodor Yurchikhin and Sergei Ryazansky who is headed to ISS in late July.
The next U.S. spacewalk is not planned until late 2017 but the airlock and space suits are maintained in readiness for spacewalks needed on short notice to deal with external hardware failures as illustrated by Tuesday’s EVA-43, coming with only two full days of lead time.