NASA Office of Logic Design

NASA Office of Logic Design

A scientific study of the problems of digital engineering for space flight systems,
with a view to their practical solution.


SP-287 What Made Apollo a Success?

 

1. INTRODUCTION

 
By George M. Low
Manned Spacecraft Center

 

[1] On July 20, 1969, man first set foot on another planet. This "giant leap for man kind" represented one of the greatest engineering achievements of all time. This article and the others in this document describe and discuss some of the varied tasks behind this achievement.

We will limit ourselves to those tasks that were the direct responsibility of the NASA Manned Spacecraft Center: spacecraft development, mission design and mission planning, flight crew operations, and flight operations. We will describe spacecraft design principles, the all-important spacecraft test activities, and the discipline that evolved in the control of spacecraft changes and the closeout of spacecraft anomalies; and we will discuss how we determined the best series of flights to lead to a lunar landing at the earliest possible time, how these flights were planned in detail, the techniques used in establishing flight procedures and carrying out flight operations, and, finally, crew training and simulation activities-the activities that led to a perfect flight execution by the astronauts.

In short, we will describe three of the basic ingredients of the success of Apollo: spacecraft hardware that is most reliable, flight missions that are extremely well planned and executed, and flight crews that are superbly trained and skilled. (We will not discuss two equally important aspects of Apollo-the launch vehicles and launch operations. These elements, the responsibility of the NASA Marshall Space Flight Center and the NASA Kennedy Space Center, go beyond the scope of this series of articles. )

 

SPACECRAFT DEVELOPMENT

 Four aspects of spacecraft development stand out: design, test, control of changes, and interpretation of discrepancies. We can begin with them.

 

Spacecraft Design

The principles of manned spacecraft design involve a combination of aircraft-design practice and elements of missile-design technology: Build it simple and then double up on many components or systems so that if one fails the other will take over. Examples are ablative thrust chambers that do not require regenerative cooling; hypergolic propellants that do not require an ignition source; three fuel cells, where one [2] alone could bring the spacecraft back from the moon; series/parallel redundancy in valves, regulators, capacitors, and diodes so that neither an "open" nor a "closed" failure will be catastrophic.

Another important design rule, which we have not discussed as often as we should, reads: Minimize functional interfaces between complex pieces of hardware. In this way, two organizations can work on their own hardware relatively independently. Examples in Apollo include the interfaces between the spacecraft and launch vehicle and between the command module and the lunar module. Only some 100 wires link the Saturn launch vehicle and the Apollo spacecraft, and most of these have to do with the emergency detection system. The reason that this number could not be even smaller is twofold: Redundant circuits are employed, and the electrical power always comes from the module or stage where a function is to be performed. For example, the closing of relays in the launch vehicle could, in an automatic abort mode, fire the spacecraft escape motor. But the electrical power to do this, by design, originates in the spacecraft batteries. The main point is that a single man can fully understand this interface and can cope with all the effects of a change on either side of the interface. If there had been 10 times as many wires, it probably would have taken a hundred (or a thousand?) times as many people to handle the interface.

Another design question for manned flight concerns the use of man himself. Here again, we find no simple rule as to how man should interface with his machine. Generally, tedious, repetitive tasks are best performed automatically; and selection of the best data source to use, selection of control modes, and switching between redundant systems are tasks best performed by the pilot. In Apollo, the trend has been to rely more and more on automatic modes as systems experience has been gained. For example, computer programs for rendezvous were reworked to require far less operator input than had originally been planned, but the entire rendezvous sequence was designed so that the pilot could always monitor the automatic system's performance and apply a backup solution if deviations were noted. A tremendous amount of time and effort was spent to determine how the crew could best decide which data source to use and which of many redundant systems to rely on. This was always a basic mission-design consideration.

The concept of in-flight maintenance was discarded entirely as being impractical for flights with the specific purpose and duration of Apollo. In its place, more telemetry was added and full advantage was taken of the ground's ability to assess system performance, predict trends, and compare data with preflight test experience.

 

Apollo Test Activities

The single most important factor leading to the high degree of reliability of the Apollo spacecraft was the tremendous depth and breadth of the test activity.

There are two general categories of tests: (1) those made on a single prototype device (or on a few devices) to demonstrate that the design is proper and will perform properly in all environments and (2) those made on each flight item to assure that there are no manufacturing errors and that the item will function as intended. Both categories apply to individual parts, components, subsystems, systems, and entire [3] spacecraft. The first category includes development testing early in the design cycle and the very formal certification or qualification tests performed on test articles identical to the flight system. The second category covers acceptance testing.

Instead of reviewing the entire development and qualification test program, we can focus on only those tests involving complete spacecraft or boilerplates, as listed in table 1-I. Each of these tests taught us more about our spacecraft-their strengths and weaknesses. As a result of the thermal vacuum tests, the spacecraft withstood the translunar and lunar environments without a single thermal problem. Passive thermal-control modes were developed that required minimum crew inputs and gave a perfect thermal balance. The land-impact tests demonstrated that the command module could survive an emergency landing if wind velocity stayed within certain limits. These tests also led to the design of a new impact-attenuation strut for the astronaut couch. The strut allowed us to increase the permissible launch wind speed and thereby gave us more flexibility in an otherwise constrained launch window. Other tests brought equally significant results.


TABLE 1-I. DEVELOPMENT AND QUALIFICATION TESTS

[Full-scale spacecraft testing]

Escape motor flight tests

7

Parachute drop tests

40

Command module land impact tests

48

Command module water impact tests

52

Lunar module structural drop tests

16

Lunar module complete drop tests

5

Command and service module acoustic/vibration tests, hr

15.5

Lunar module acoustic/vibration tests, hr

3.5

Command and service module modal survey testing, hr

277.6

Lunar module modal survey testing, hr

351.4

Command and service module thermal vacuum tests, hr

773

Lunar module thermal vacuum tests, hr

2652

Service module propulsion-system tests, min

1474.5

Ascent- stage propulsion- system tests, min

153

Descent- stage propulsion- system tests, min

220


[4] Most important of all, the tests gave us a tremendous amount of time and experience on the spacecraft and their systems. Such experience-together with a detailed analysis of all previous failures, discrepancies, and anomalies-led us to the conclusion that we were ready to fly a lunar orbit with Apollo 8 and that we were ready to make a lunar landing with Apollo 11.

Acceptance testing played an equally important role. This testing starts with piece parts. Although Apollo was late in applying this rule, I believe that screened and burned-in electronic parts must be made a firm requirement. Next, each component, or black box, is tested before it is delivered, and again before it is installed in the spacecraft. Then, factory testing of the complete spacecraft begins. First, the wiring is wrung out, and individual subsystems are tested as installed. Then, groups of systems are jointly tested. Finally, the complete spacecraft, with all of its systems functioning, is run in an integrated test. All normal, emergency, and redundant modes are verified.

After delivery to the launch site, similar (when practical, identical) tests are performed. A major test at Cape Kennedy is a manned altitude-chamber run of each spacecraft. The final acceptance test, of course, is the countdown itself.

A most important facet of acceptance testing is environmental acceptance testing. The primary purpose of acceptance vibration testing and acceptance thermal testing is to find workmanship errors. To do this, the environment has to be severe enough to find the fault (e. g., a cold-solder joint), yet not so severe as to weaken or fatigue the component. Figures 1-1 and 1-2 show the levels selected for these tests in Apollo. These levels were picked on the basis of experience in Gemini and other programs. Each component type, of course, had to pass qualification tests under even more severe environments. Nevertheless, our environmental acceptance tests sometimes uncovered design faults (as opposed to workmanship faults) that had been missed in the qualification tests. The reason was that a single qualification test may have missed a marginal condition, which the large number of acceptance tests could catch.


Figure 1-1.- Vibration test level for acceptance.

Figure 1-1. Vibration test level for acceptance.

 


Figure 1-2. Thermal test level for acceptance.

Figure 1-2. Thermal test level for acceptance.

[5] We also considered environmental acceptance tests of complete spacecraft, but decided against this because the environment on most components, as mounted in the spacecraft, is not severe enough to bring out workmanship faults. The vibration levels on many components are one or two orders of magnitude less than those given in figure 1-1. (This conclusion would not be true for smaller, more compact spacecraft. ) Temperatures in the spacecraft generally remain constant because most electronic components are mounted on cold plates.

Figures 1-3 and 1-4 summarize the results of the Apollo environmental acceptance test program. Note that 5 percent of all components failed under vibration, and 10.3 percent of all components did not pass the thermal testing. Remember that these components were otherwise ready for installation in the spacecraft. By category, the failure modes look like those listed in table 1-II. If these tests had not been performed, and if these failures had occurred in flight, we very likely would still be waiting for the first manned lunar landing.

 


Figure 1-3. - Results of vibration acceptance tests for 11 447 tests of 166 different components.

Figure 1-3. Results of vibration acceptance tests for 11 447 tests of 166 different components.


Figure 1-4.  Results of thermal acceptance tests for 3685 tests of 127 different components.

Figure 1-4. Results of thermal acceptance tests for 3685 tests of 127 different components.


TABLE 1-II. HISTORY OF ENVIRONMENTAL ACCEPTANCE TEST FAILURES

Mode

Percent failed

.

Electrical

57.3

Mechanical

27.4

Contamination

11.5

Other

3.8

Total

100


Control of Changes

If the design has been verified and if a thorough test program has been completed, it should not be necessary to make any changes. Of course, this idealized situation does not exist in any program like Apollo where design, test, and flight often overlap and must be carried out at the same time. Changes may be required as a result of test failures, or another look at the design may identify a situation that could lead to a failure or to the inability to react to failure. Sometimes a more detailed definition of flight missions or operational use of the hardware itself leads to a requirement for change.

[6] Since it is not possible to eliminate all changes, we have to start with the premise that any change will be undesirable. That is, a change will void all previous test and flight experience and, no matter how simple, may have ramifications far beyond those identified by the initial engineering analysis.

Because changes must be made nevertheless, it becomes important to understand and to control them, no matter how small. In Apollo, we handled all changes through a series of Configuration Control Panels and a Configuration Control Board. The panels considered minor hardware changes early in the development cycle, as well as crew procedures and all computer programs. The Board considered more significant hardware changes, all hardware changes after spacecraft delivery, and procedures or software changes that could affect schedules or missions.

The Apollo Spacecraft Configuration Control Board met 90 times between June 1967 and July 1969, considered 1697 changes, approved 1341, and rejected 356. We had a low rejection rate because proposed changes were reviewed before they came to the Board, and only those deemed mandatory for flight safety were brought before it. The Board is chaired by the Program Manager, who also makes the final decision on all changes. The Board includes the directors of all major technical elements of the NASA Manned Spacecraft Center and the contractors' program managers.

We considered changes large and small. An example of a large change is the new spacecraft hatch that was incorporated after the fire. However, we reviewed in equal technical detail a relatively small change, such as a small piece of plastic to go inside the astronaut's ballpoint pen.

The Board was established to discipline the control of changes; but it was found to serve a much larger purpose: It constituted a decision-making forum for spacecraft developer and user. In reaching our decisions, we had the combined inputs of key people representing hardware development, flight operations, flight crews, safety, medicine, and science.

I have recently reviewed the results of the 90 Board meetings that preceded Apollo 11. Even with hindsight, I find few, if any, Board decisions that I would make differently today.

 

Closeout of Failures

Throughout Apollo, many discrepancies or failures occurred daily. The relationship may have been a close one (i. e., failures actually took place during testing of the next spacecraft to fly) or it might have been remote (i. e., a component identical to one used on Apollo failed on another program). In either instance the result was the same: The failure had to be understood and, if applicable, some corrective action taken. This might involve design change, re-inspection, or perhaps procedural change.

I will confine my remarks to anomalies that occurred during the first five manned Apollo flights. The number of anomalies for each mission are given in table 1-III. Note that, even though each of the flights was completely successful and met all its objectives, the number of anomalies went quite high. Perhaps this is the best proof of the validity of the Apollo design concept: The spacecraft were designed for mission success.


[7] TABLE 1-III. APOLLO FLIGHT ANOMALIES

Spacecraft

Number of anomalies

Command and service module

Lunar module

.

Apollo 7

22

-

Apollo 8

8

-

Apollo 9

14

12

Apollo 10

23

15

Apollo 11

9

13


Let us look at just one example. On Apollo 10, during several of the lunar orbits, a critical fuel-cell temperature started to oscillate significantly, as shown in figure 1-5. Normally, this temperature is steady, between 155° and 165° F. The oscillations encountered on Apollo 10 triggered the spacecraft alarm system, but otherwise were not detrimental. Yet, unless we understood their cause, we could not be sure that they would always be limited as they were in Apollo 10 and hence that they would not lead to a fuel-cell failure.

The closeout of these flight failures had to be done in the time available between the completion of one flight and the start of the next-a period usually only about 6 weeks long. Yet even these 6 weeks were not fully available, because hypergolic propellants were loaded into the spacecraft a month before launch, thereafter severely limiting ability to make spacecraft changes and to perform necessary retesting. Nevertheless, each of the failures listed in table 1-III was satisfactorily closed out before the next flight.

Our investigation revealed that small, isolated disturbances in fuel-cell temperature were often present, as figure 1-6 shows. Pratt & Whitney, North American, and NASA then performed a detailed stability analysis of the fuel-cell system, transfer functions were experimentally determined, and finally a complete fuel cell test was run to verify the results of the analysis. This work demonstrated that small, isolated disturbances could trigger an instability if the power loading ran sufficiently high and the temperature sufficiently low [8]. The analysis also showed that the amplitude of the oscillations would always be limited as it was in Apollo 10. With this information, it was possible to devise procedures to eliminate the oscillations, should they occur.


Figure 1-5. Apollo 10 fuel-cell temperature oscillations as they originally appeared in flight.

Figure 1-5. Apollo 10 fuel-cell temperature oscillations as they originally appeared in flight.

 


Figure 1-6. - Disturbance of Apollo 10 fuel-cell temperature as it was identified in the laboratory.

Figure 1-6. Disturbance of Apollo 10 fuel-cell temperature as it was identified in the laboratory.

The solution as described here probably sounds simple. Yet, a similar task, formulated as a research assignment, might have taken a year or more to complete. Here, closeout of the failure was done in weeks.

The fuel-cell anomaly was only one example of a discrepancy. The total task- handling all flight anomalies-was enormous; yet, it was completed before each flight.

 

FLIGHT MISSIONS

It is difficult to describe, to those not directly involved in the Apollo Program, just how much work went into operational activities. First, we had to decide the kinds of mission to be flown: What would be the best series of missions to achieve a successful manned lunar landing at the earliest time? Then these missions had to be planned in detail: How should each mission be designed to meet the largest number of operational and hardware objectives, even in the event of unplanned events? (Operational objectives are concerned with guidance, navigation, trajectory control, rendezvous, etc.; hardware objectives are concerned with the verification of each system or subsystem in the flight environment. ) Finally, plans had to be made for the execution of the mission: Detailed rules were evolved for every imaginable contingency; the proper flight-control displays were defined to permit instantaneous reaction to emergencies, and countless hours were spent in simulations of every conceivable situation.

 

Mission Definition

Early in 1967 the situation was as follows. Many development flights had taken place to test the launch-escape system under extreme conditions, to test the command module heat-protection system at speeds halfway between earth-orbital and lunar reentry velocities, and to put the guidance and propulsion systems through their preliminary paces. However, Saturn V had not yet been flown, reentry at lunar-return speeds had not yet been made, the lunar module had not yet been flown, and man had not yet been in space in Apollo hardware.

The flight-test program shown in figure 1-7 was then evolved through an iterative and flexible process that was changed as time went on to take the best advantage of knowledge about mission operations and hardware availability at any given time. The basic principle in planning these flights was to gain the maximum new experience (toward the goal of a lunar landing) on each flight without stretching either the equipment or the people beyond their ability to absorb the next step.


[
9]

Figure 1-7.- Buildup of Apollo mission capability.

Figure 1-7. Buildup of Apollo mission capability.

Too small a step would have involved the risk that is always inherent in manned flight, without any significant gain-without any real progress toward the lunar landing. Too large a step, on the other hand, might have stretched the system beyond the capability and to the point where risks would have become excessive because the new requirements in flight operations were more than people could learn and practice and perfect in available time.

Apollo 4 and 6 saw the first flights of Saturn V. Apollo 4 was almost letter perfect. Yet a repeat flight was planned and, in retrospect, proved to be very important. Serious defects in the Saturn propulsion system and in the spacecraft adapter, that were not apparent on Apollo 4, caused major failures on Apollo 6. These failures led to an extensive ground-test program and to hardware changes before the next flight of the launch vehicle. Apollo 4 also served to qualify the spacecraft heat shield under severe simulated lunar reentry conditions; the flight showed that the design was conservative.

Apollo 5 was an unmanned flight of the lunar module. The lunar module guidance system, both propulsion systems, and the all-important staging sequence between the ascent and descent stages functioned well.

The first manned flight of the command and service module came in October 1968, with Apollo 7. The spacecraft performed beyond all expectations in this 11-day flight. Each of the command and service module systems (except the docking system) was put through its paces without a significant malfunction.

[10] The decision to fly into lunar orbit on Apollo 8 came relatively late. It was made, on a tentative basis, in August 1968. At that time, the test experience with the command and service module had been very good. The lunar module schedule, on the other hand, was slipping; and the first manned lunar module exhibited the normal "first ship" difficulties during checkout at the NASA Kennedy Space Center. Also, a detailed analysis of results from the unmanned Lunar Orbiter Program had shown that navigation around the moon would present many unexpected computational difficulties.

For all of these reasons, it was decided that Apollo 8 should be a CSM-alone lunar-orbit flight. This decision was reaffirmed with the success of Apollo 7, and the die was cast for making man's first flight to the moon in December 1968.

In Apollo 9, both spacecraft, the lunar module and the command and service module, were tested together for the first time. First, all of the lunar module systems were tested in manned flight. Then methods for the following spacecraft operations were worked out and verified: communications between two spacecraft and the ground; tracking, guidance, and navigation; and rendezvous and docking. Also on Apollo 9, the extravehicular mobility unit (the lunar space suit and its life-support system) was tested in the actual space environment.

After Apollo 9 another decision had to be made: Were we then ready for a lunar landing, or was the step too big ? We decided that we faced too many remaining unknowns: performance of the lunar module in the deep-space environment, communications with the lunar module at lunar distances, combined operations with two spacecraft around the moon, rendezvous around the moon, and, of course, the lunar descent landing, surface operations, and ascent. In lieu of a landing, we planned to do as many of these tasks as possible on Apollo 10 without actually touching down on the surface of the moon.

The entire series of flights represented a step-by-step buildup, with each step leading closer to a lunar-landing ability. Our intent was to use the procedures developed on one flight on each subsequent mission. Changes were allowed only if they were essential for flight safety or mission success. By means of this buildup, we minimized the remaining tasks (descent, landing, surface operations, and ascent) that could be worked out only on the actual landing mission. The Apollo 11 crew was able to concentrate on these remaining tasks, to work them out in detail, and to carry them out with perfection.

 

Mission Planning and Execution

Once basic missions had been defined, each flight had to be planned in detail. The mission planner tries to fit into each flight the maximum number of tests of the hardware and the widest variety of operations. For example, he will develop a rendezvous profile for a single earth-orbital flight that involves all of the normal and abnormal rendezvous conditions which might be encountered around the moon- rendezvous from above, rendezvous from below, rendezvous with the lunar module active, rendezvous with the command and service module active, and rendezvous with varying lighting conditions. At the same time, the mission planner will try to exercise all of the propulsion systems and all of the navigation systems on both spacecraft.

[11] After mission plans come the mission techniques (by another name, data priority). Given two or three data sources (for trajectory control), which of the sources should be believed and which discarded? Limits for each system had to be determined, and logic flows for every conceivable situation had to be developed.

Finally, the flight controllers take over. They had participated, of course, in the mission-planning and mission-technique activities; but now they had to work out each step of the flight and anticipate every emergency situation that might arise. What is the proper action when one fuel cell fails? What if two fail? The answers to thousands of questions like these had to be derived in terms of the specific mission phase. A rendezvous radar failure before command and service module-lunar module separation dictates that the two vehicles should not be separated. The same failure after separation allows the mission to be continued because the risk of rendezvous without radar has already been incurred and will not increase in subsequent mission sequences. Each of these events was documented as a mission rule long before the flight, and mission rules were placed under "configuration control, " as was every other aspect of the Apollo system.

Flight controllers also worked out the best formats for their real-time displays. During the Apollo 11 descent to the surface of the moon, the flight controllers could watch, with a delay of only 6 to 10 seconds, the functioning of nearly every onboard system. They saw the rise in chamber pressure as the descent engine was throttled up to full thrust, and they could determine that the throttle-down occurred at the proper time. The flight controllers could also compare the descent trajectory from three data sources-two onboard guidance systems and the ground tracking system. With this information, a flight controller on the ground could tell the crew, nearly 250 000 miles away, to ignore the alarms from the onboard computer during the most critical portion of the descent, because the system was guiding the spacecraft correctly.

Many of the techniques used during the flight were developed during countless hours of simulations. Simulation is a game of "what-if's." What if the computer fails? What if the engine does not ignite? What if . . . ? The game is played over and over again. The flight controllers do not know what situation they will face on the next simulation. By the time of flight, they will have done simulations so often and they will have worked together as a team so long, that they can cope with any situation that arises.

Because the Apollo equipment has worked so well and because there have been so few contingency situations, one could conclude that much of the planning, many of the mission techniques, and much of the training were done in vain. But this is an incorrect conclusion. As a minimum, the state of readiness that evolved from these efforts gave us the courage and the confidence to press on from one mission to the next. Also, there were situations-the computer alarms during the descent of Apollo 11 and the lightning discharge during the launch of Apollo 12-that might have led to an abort if the team had been less well prepared and less ready to cope with the unexpected.

 

[12] FLIGHT CREW TRAINING

The first six Apollo manned flights carried 18 astronauts-all professional pilots, skilled and superbly trained. Altogether, they had flown on 18 flights in Mercury and Gemini before they flew in an Apollo spacecraft. Five had flown twice before, eight had flown once before, and five flew for the first time in Apollo.

Training for Apollo is not easy. Two highly sophisticated machines are involved, each far more complex than those in Gemini. The astronauts had to become expert in the workings of both spacecraft. They became computer programmers and computer operators, space navigators, guidance experts, propulsion engineers, fuel-cell-power managers, environmental-control-system experts-to mention but a few areas of expertise. Of course, they had to learn how to control and fly two spacecraft with vastly different handling qualities under conditions of launch, translunar flight, lunar-orbit flight, lunar landing, lunar launch, rendezvous, docking, transearth flight, and reentry.

The astronauts used a variety of training devices-high-performance airplanes to stay alert and sharp; a special dynamic launch simulator to practice manual takeover and abort modes; mission simulators to duplicate here on earth every spacecraft function and display under all possible conditions; partial-gravity simulations under water, in airplanes, and on a special servo-controlled device on the ground; a docking trainer; and a flying lunar-landing training vehicle that has a jet engine to take out five-sixths the gravity of earth so that the vehicle has the same flying characteristics as the lunar module has on the moon.

The astronauts also needed plans and procedures. Flight plans spelled out each step of the mission. Detailed "time lines" were developed for every function that had to be performed, minute by minute. Crew procedures and checklists were an adjunct to the flight plan. The step-by-step sequence for each spacecraft activity, each maneuver, each propulsive burn was worked out well in advance and was used again and again during practice and simulation.

Configuration control was as important in the astronaut training as in every other category. Simulators had to look just like the spacecraft to be useful, and last-minute spacecraft changes had to be incorporated in the simulators as well. Crew procedures that had worked well on one flight could not be changed, through "crew preference, " for the following flight.

Pete Conrad said that landing his Apollo 12 lunar module, after dust obscured the landing point, was the most difficult task he had ever performed. It took all of his 20 years of experience as a professional aviator, his previous work on two Gemini flights, his training for Apollo, and his knowledge and confidence in the Apollo spacecraft systems to make that landing a success.

 

[13] CONCLUDING REMARKS

Spacecraft development, mission operations, and flight crew activities-in reviewing these areas of Apollo, I see one overriding consideration that stands out above all the others: Attention to detail. Painstaking attention to detail, coupled with a dedication to get the job done well, by all people, at all levels, on every element of Apollo led to the success of what must be one of the greatest engineering achievements of all time - man's first landing on the moon. The reports which follow amplify this observation.

previouscontentsnext
 

Home - NASA Office of Logic Design
Last Revised: February 03, 2010
Web Grunt: Richard Katz
NACA Seal