I’m reading the biography of Gene Kranz, a name some of you might know from his presence as flight director at mission control (MCC) both for the Apollo 11 moon landing and for the Apollo 13 explosion. But, really, he was only one of seven or so mission control flight directors during the Apollo missions.

One take-home message from the book is the enormous number of failures on US space flights. The first trip around the Moon was a complete success, with only 26 equipment failures. Every single Mercury, Gemini and Apollo mission, even the unmanned ones, had equipment failures, often disasterous, more often nearly disasterous. One mission accorded a great success was an unmanned mission. For the first time a TV crew was filming the control center, this was a temporary control centre in the middle of a move from Cape Kennedy to Houston. The power requirements of the TV lights tripped the circuit breakers plunging the entire control centre into darkness and switching off all communications. They had been prepared for anything – except that. From then on power supplies were made double redundant and TV crews all had to supply their own power. It never happened again. In another case the Zanzibar remote control centre remained functioning under seige because of street rioting throughout the neighbourhood. In another case a medical doctor stepped in as mission control support, relaying data and commands when one of the top controllers was unable to talk because of car crash injuries just a few hours before.

The mission controllers, as well as the astronauts, had to be adrenaline junkies. Most of the top mission controllers had been jet fighter pilots or at least had had a near-death experience onboard a fast military plane before being hired. This is only one step down from the astronauts who were mostly test pilots for untried military aircraft. A training exercise for all controllers was to fly the space capsule simulator blindfolded. Not just flipping the correct switches they were asked to but also going through flight checklists in real time blindfolded. The mission controllers could have doubled as astronauts.

I have occasionally wondered how on earth a huge organisation like NASA was put together so fast. How do you get the best people in the best places, set up flight timelines, choose which readings to display on monitors and how, and all such details in very rapid time frame?

There are two key elements in how to do that, neither of which I’d thought of before. One is to not write job descriptions, or at most a job description is one line long. A person who applies is hired, given a mentor, given a heap of raw documentation to read (that looks like it is impossible to understand) and left alone. After a week of reading, talking and being an observer in meetings they have enough information to make their own job description, using their abilities and experience and by two weeks are making an active contribution. Gene Krantz, for instance, was familiar with checklists from his Korean War fighter pilot experiences so started off writing checklists. People who can’t take the strain or have less than perfect motivation are weeded out or leave of their own accord.

The second element is the bottom up approach. Mission operations staff quickly get seconded to contractors, where they learn the hardware drawings and specifications, then the hardware testing procedures. A lot of the technology was so new you couldn’t read about it. They called it “learning by doing”. They returned with bundles of drawings. “We studied the manufacturing and test data and then prepared schematics and performance plots on each of the spacecraft systems. This data was then used in out classroom studies, taught by mission controllers. When the schematics and training were completed, the controllers turned to flight procedures, then to mission rules.”

“Only after we thoroughly understood the design and operation of the spacecraft, did controllers focus on the Mission Control Centre, designing our displays and laying out our consoles. We were ready to start only when we trusted the data and trusted ourselves.”

Wow, what a way to work. But it would work, and did work.

I see now that the flexibility of staff appointements would have been a big advantage of the USA over Russia. The USA worked on the principle that no-one is irreplaceable. If one person is out of action for any reason, such as illness or car accident, someone else had to be ready to take over at a moments notice. This was not by having a defined understudy sitting in the wings doing nothing, but by people taking on multiple jobs and by instant promotions of people who had made themselves ready to take over in emergency.

mollwollfumble said:

Wow, what a way to work. But it would work, and did work.

I must find that book. Kranz is someone who i admire.

The staffing/development system you describe is also admirable.

Dwight Eisenhower said that the essence of good management is finding good people, and then letting them get on with the job.

Unfortunately, in what we might call real life, too many ‘managers’ would be terrified of the degree to which they’d lose ‘control’ over staff and their functions to even think about letting such things happen as you describe. There’d be too much risk of exposing the irrelevance of some managers.

Very flexible perhaps, but also a bit chaotic. Might be one of the reasons there were so many equipment failures.

Bubblecar said:

Very flexible perhaps, but also a bit chaotic. Might be one of the reasons there were so many equipment failures.

A key facet of the whole early NASA system is “time critical”. There was time to get it done. But there wasn’t time to get it right.

mollwollfumble said:

Wow, what a way to work. But it would work, and did work.

How could they design their displays and lay out their consoles without first understanding the intricacies of the spacecraft?

Ian said:

mollwollfumble said:

Wow, what a way to work. But it would work, and did work.

How could they design their displays and lay out their consoles without first understanding the intricacies of the spacecraft?

FWIW the consoles that you see in mission control weren’t exactly how they appeared to be. It’s very simple now but back in the 60’s they couldn’t create a screen that displayed the various labels next to the telemetry values. So what they did was have a screen that showed the values in the desired places on the screen, and above that a printed template was placed on top of the screen. Then a video camera was pointed at that assembly and the video from that was what we see on the mission control screens.
Bizarre but true.

Spiny Norman said:

Ian said:

mollwollfumble said:

Wow, what a way to work. But it would work, and did work.

How could they design their displays and lay out their consoles without first understanding the intricacies of the spacecraft?

FWIW the consoles that you see in mission control weren’t exactly how they appeared to be. It’s very simple now but back in the 60’s they couldn’t create a screen that displayed the various labels next to the telemetry values. So what they did was have a screen that showed the values in the desired places on the screen, and above that a printed template was placed on top of the screen. Then a video camera was pointed at that assembly and the video from that was what we see on the mission control screens.
Bizarre but true.

Interesting. Thanks

Looking back, NASA mission control’s greatest success, was also its greatest failure.

Apollo 13.

When there was a bang reported from the “Houston we have a problem”, mission control should have thought “explosion, shut off the valves”.

Instead, their first though was “what the heck happened?”, “could be an instrument failure”. They tried to diagnose it as an electrical/instrumentation problem.

It wasn’t until astronaut Lovel reported 17 minutes after the explosion that he’s looked through the window and seen that the spacecraft was venting something that mission control finally twigged to an explosion. That’s seventeen minutes of lost oxygen flowing out through an open valve.

The oxygen was vital for both electricity (which came from the fuel cells) and cooling (cooling water for the lunar module came from the fuel cells).

It was the contollers’ massive error during those 17 minutes that made survival such a touch-and-go situation.

The next error that was made was made by the controllers was in being late to see that the cooling water to the lunar module was a worse problem than the electrical power situation. The cooling water would run out before the electrical power would. It was that more than anything else that turned the astronauts into frozen popsicles.

Everything else the mission controllers did right. The decision to go around the Moon rather than jettison the Lunar Module and come straight home, for example. If they had jettisoned the Lunar Module then they would have had to fire up the Control Module’s main engine. It was only much later, shortly before reentry, that they confirmed that the Control Module’s main engine was severely damaged, probably fatally damaged in the initial explosion.

The mission controllers got medals for Apollo 13.

mollwollfumble said:

Looking back, NASA mission control’s greatest success, was also its greatest failure.

Apollo 13.

When there was a bang reported from the “Houston we have a problem”, mission control should have thought “explosion, shut off the valves”.

Instead, their first though was “what the heck happened?”, “could be an instrument failure”. They tried to diagnose it as an electrical/instrumentation problem.

It wasn’t until astronaut Lovel reported 17 minutes after the explosion that he’s looked through the window and seen that the spacecraft was venting something that mission control finally twigged to an explosion. That’s seventeen minutes of lost oxygen flowing out through an open valve.

The oxygen was vital for both electricity (which came from the fuel cells) and cooling (cooling water for the lunar module came from the fuel cells).

It was the contollers’ massive error during those 17 minutes that made survival such a touch-and-go situation.

The next error that was made was made by the controllers was in being late to see that the cooling water to the lunar module was a worse problem than the electrical power situation. The cooling water would run out before the electrical power would. It was that more than anything else that turned the astronauts into frozen popsicles.

Everything else the mission controllers did right. The decision to go around the Moon rather than jettison the Lunar Module and come straight home, for example. If they had jettisoned the Lunar Module then they would have had to fire up the Control Module’s main engine. It was only much later, shortly before reentry, that they confirmed that the Control Module’s main engine was severely damaged, probably fatally damaged in the initial explosion.

The mission controllers got medals for Apollo 13.

One reason they didn’t want to close those valves was that once closed they couldn’t be opened again. (I think)
And also once closed the power from the fuel cell(s) that the tank serviced would also be lost, leaving the capsule running on batteries before the crew could configure the craft for minimum power.

Spiny Norman said:

mollwollfumble said:

Looking back, NASA mission control’s greatest success, was also its greatest failure.

Apollo 13.

When there was a bang reported from the “Houston we have a problem”, mission control should have thought “explosion, shut off the valves”.

Instead, their first though was “what the heck happened?”, “could be an instrument failure”. They tried to diagnose it as an electrical/instrumentation problem.

It wasn’t until astronaut Lovel reported 17 minutes after the explosion that he’s looked through the window and seen that the spacecraft was venting something that mission control finally twigged to an explosion. That’s seventeen minutes of lost oxygen flowing out through an open valve.

…

One reason they didn’t want to close those valves was that once closed they couldn’t be opened again. (I think)
And also once closed the power from the fuel cell(s) that the tank serviced would also be lost, leaving the capsule running on batteries before the crew could configure the craft for minimum power.

Thanks for that.

During that 17 minutes before mission control realised that there was an explosion, they were still working on the assumption that a moon landing was possible.

Very different from Apollo 12, where the massive failure of the craft electrical system (discovered later due to a lightning strike) was mostly fixed in a couple of seconds when the crew comment from Conrad “what the hell was that” was followed by the now immortal “take the SCE to Aux” by flight controller electrical and communications specialist John Aaron. This recovered the spacecraft telemetry which showed that it was still on the correct trajectory. Flight saved.

In a very real sense, the landing of Apollo 11 was saved by a chap called Dick Koos. Koos wasn’t even on the crew or mission control. He was in charge of simulations. In the very last simulation before Apollo 11 took off, Koos programmed a Moon landing simulation with computer errors in it, including error 1201. The controller who was supposed to know all the computer error messages, because he had written most of the software, aborted the landing and was given a thorough earbashing because the computer errors weren’t critical. This controller did a thorough review of computer error messages after the simulation, and was abashed to learn that Koos was completely right. Actual first landing on the Moon, computer errors 1201 and 1202, landing continues.

The book starts out with “the four inch flight”. Mercury-Redstone 1 which was launched 21 Nov 1960.

“The launch countdown progressed without major incident.”

A few seconds after liftoff it became clear that the rocket was still on the launch pad. Soon it was realised that the Redstone rocket engine had shut down when the electrical umbilical detached, at a height of a few inches above the launch pad. The rocket settled back on the launch pad and the parachute deployed. The strong wind blowing into the parachute threatened to topple the rocket, which would have rather catastopic consequences. Ditto an automatic restart of the rocket engine would have catastropic consequences. The loss of the electrical umbilical meant that mission control had no way of sending a shutdown command.

Test your abilities as a mission controller. What did mission control decide to do?

a) Reattach the electrical umbilical so they could send a permanent shutdown command.
b) Send a truck out to drain the fuel off.
c) Cut a parachute cord to stop the rocket being toppled by the wind.
d) Shoot holes in the rocket to allow the fuel to drain out through the bullet holes.
e) Nothing.

The correct answer is “e) Nothing”. After checking that the wind was not expected to increase during the night they went home.

All options were brought up in a rapid emergency brainstorming session. Every other option was judged to be too dangerous. It can be difficult to do nothing when things are going very wroong. In this case, doing nothing meant that the rocket’s batteries went flat overnight, rendering it safe to remove fuel.

It can be difficult to deliberately do nothing when things have gone very wrong, but sometimes that’s the best thing to do.