Computers and Historical Research: Personal Reflections

Robert P. Swierenga

A paper delivered at Spring Arbor College, September 24, 1999, to the Regional History Meeting

When Professor Robert Eells asked me to speak about historians and computers, I was somewhat taken aback. Historians in recent years have shown little interest in the quantitative approach to history, which was so big in the 1970s and 1980s. My book, Quantification in American History: Theory and Research (Atheneum, 1970), is no longer a standard text in graduate history seminars, as it was for many years.

The old adage is true that every generation writes its own history. Interpretations and methodologies come and go. When I was in graduate school thirty-five years ago at the University of Iowa, I became a charter member of the "Iowa School" of historiography, which was led by three of my professors--William Aydelotte, Samuel Hays, and Allan Bogue. Aydelotte, now deceased, was a specialist in 19th century British political history and was the first historian to use computers. He analyzed the roll call votes of members of Parliament to uncover the various interest groups and voting blocs. Bogue did the same for the US Congresses during the Civil War years. Hays put US census records and election statistics of the nineteenth century into computer files and discovered that ethnocultural groups had very distinct and predictable voting patterns at the local level. This so-called ethnocultural interpretation of American political history took the profession by storm in the 1970s.

The Iowa School was part of a broader movement that came to be called social science history. This can best be defined as the study of past human behavior scientifically, that is, by compiling numeric data and analyzing it with social statistics, in order to test general theories of human behavior. This new approach is usually traced to the 1957 papers on the profitability of slavery by the young economic historians Alfred Conrad and John R. Meyer. Soon intellectual ferment gripped social scientists at midwestern universities such as Purdue, Iowa, Michigan, and Pittsburgh. Interestingly, social science history did not take root first in the prestigious Ivy League universities or at the University of Chicago, but at state schools.

The key organization for historians was the multi-million dollar, data collection project of the Inter-University Consortium for Political Research (ICPR) at Ann Arbor, founded in 1962. This institute made economic, census, and voting data available in electronic form for researchers, and it ran summer seminars that trained hundreds of historians in how to use such data. In the five years, 1969-1973, ICPR supplied historians and social scientists with 38 million "card images" (cases) from its Historical Data Archive. This was an impressive record indeed.

By 1974 social science history had reached professional maturity. One evidence of this fact is that the editor of the Journal of American History, the mouthpiece of the Organization of American Historians, asked me to write an article on the state of the art ("Computers and American History: The Impact of the 'New' Generation," LX [March 1974]: 1045-70). I noted how historians in fits and starts had made their peace with computer methods. They had "passed through the painful puberty phase" and were "reaching full maturity" (1070).

That same year, 1974, the "new historians," as they were known, were numerous enough to establish a professional society, the Social Science History Association. In 1976 the Association launched its official journal, Social Science History, which I co-edited for 15 years. The journal is now in its 24th year. The purpose of the SSHA, as spelled out in the first issue of its journal, was to improve "the quality of historical explanation by encouraging the selective use and adaptation in teaching and research of relevant theories and methods from the social science disciplines." Furthermore, the statement went on, SSHA was "a common enterprise within which historians seeking a more rigorous and consciously theoretical orientation in their discipline might join forces with other social scientists interested in longitudinal analysis."

Some of those other social scientists included the economic historians Robert Fogel and Stanley Engerman, whose book Time on the Cross: The Economics of American Negro Slavery (1974), was so controversial that it spawned a shelf of books in response, notably Herbert Gutman's Slavery and the Numbers Game (1975). Fogel later was awarded the Nobel Prize in Economic History for his work, along with Douglas North, another founding member of SSHA and an expert in organizational theory.

From the outset, SSHA was an umbrella organization of scholars with very diverse goals. Mine was to encourage historians to learn quantitative methods and computer programming skills so that they could included in their research projects numerical data sources like census and ship passenger lists, as well as the usual literary sources. This is what I did at Kent State University, where for 28 years I taught a course on quantitative methods in history that was required of all incoming graduate students. I also used computers and social statistics in my books on Iowa land history and Dutch immigration in the 19th century.

Many in the history profession were aghast at the young turks who would high-jack history and place it in the service of the social sciences. Jacques Barzun, an esteemed university professor at Harvard University, fired off a broadside in his book, Clio and the Doctors: Psycho-History, Quanto-History, and History (1974). Clio was the Greek muse of history. Although Barzun unfairly linked quantitative studies with psycho-histories like Erik Erickson's Young Man Luther and William Langer's The Mind of Adolf Hitler, Barzun rightly criticized the social scientists for their unacknowledged biases and their fondness for jargon. History needs no special language, Barzun declared, nor should it serve the purposes of special interest groups.

Barzun's critique was tame compared to Carl Bridenbaugh's presidential address to the American Historical Association in 1962, in which he admonished historians not "to worship at the shrine of the bitch-goddess, QUANTIFICATION." Gertrude Himmelfarb of Brooklyn College, another traditional narrative historian, attacked the pretensions of the new history, claiming that it devalued politics and demeaned reason by focusing on the essential irrationality of human life. All this soul searching was just because some historians wanted to study changes over time in group behavior rather than write biographies, institutional histories, and narrative accounts.

The computer revolution in history began in the hard sciences during and immediately after World War Two, and then it infiltrated the soft sciences in the 1950s and 1960s. I was fortunate to get in on the ground floor. As a graduate student at Iowa in 1963, I became involved in an Indian Claims case on behalf of the Sac and Fox and Iowa tribes in their law suit against the government in the Federal Court of Claims. The issue was some 12 million acres of prime Iowa lands that the government had literally confiscated from the Indians in the 1840s. The Indians were given about 10 cents an acre and the government turned around and sold it for an average price of $2.50.

My task was to help the lawyers representing the Indians to establish the fair market value of these lands as of the treaty date. The government allowed the petitioners to use the Books of Original Entry that the federal land offices compiled, which listed the prices paid by persons receiving the first patents from the government, as well as sales data from the county deed registers in the first years after white settlement when a free land market emerged.

Specifically, I collected purchase and sale data on thousands of tracts of land in nine sample counties in Iowa and then set about to calculate the rate of return on such frontier land investments, taking annual tax payments and management costs into account. Did the land dealers and investors make money or not? If they did, their collective experience confirmed that the fair market value of the land at the time was far above the paltry 10 cents an acre, or even the Congress price of $1.25.

To calculate the rate of profit on so many tracts of land, each with different holding periods and costs, I needed to use polynomial equations. I quickly saw that the task was virtually impossible without computers, which were then just becoming available for other than physics and math majors. I made an appointment with the director of the computer center and explained my research problem. He took pity on me and on his own time wrote a detailed "flow chart," which laid out the steps a computer programmer must take to do the calculations. I enrolled in a course in the computer language FORTRAN, which was then standard, and found a graduate student in engineering who would write a custom program under my direction.

Meanwhile, I had to sit at a key punch machine (a machine not unlike a typewriter) and enter the numbers for each tract of land, or "case," on a separate 80-column Hollerith or IBM card. Computers in the 1960s still needed data input devices in the form of cards (and later magnetic tapes) to get the information into the computer processor via a card reader or tape drive. It took six months and many trial runs to work the bugs out, but in the end the University computer, a primitive IBM vacuum tube contraption that filled a large room and required big time air conditioning because of the heat build-up, calculated profits on each of the thousands of tracts of land. I left the carton containing nearly 2,000 data cards at the computer center one afternoon, and by the next morning the results were in. How elegant the printout was. Computers were marvelous machines, I thought, even if they had to crank all night to do the calculations.

The research findings were so robust that they became the key chapter in my dissertation and the lead article in the top journal in the field, the Journal of Economic History ("Land Speculator 'Profits' Reconsidered: Central Iowa as a Test Case," XXVI [March 1966: 1-28). I had climbed the first rung of the academic ladder and was on the way up. The land investors profited greatly and so did I. The Indian claimants, incidentally, won a award from the government of $15 million.

With my success with land records, I decided to write another Iowa land book with the aid of computers; this was my book, Acres for Cents: Delinquent Tax Auctions in Frontier Iowa (1976). The county records this time were tax sale registers. I compiled data on a quarter of a million tracts of land sold for back taxes in sixteen counties.

I carried out the research on the tax sale book while teaching at Calvin College in 1965-68. At Calvin I stumbled on my next big data set in the college archives. These were some 21,000 records the Dutch government compiled from 1847 to 1880 on every immigrant family and single adult to depart the Netherlands for overseas destinations. It took several years to key punch this information onto IBM cards for computer analysis.

By the 1970s analysis of large data sets was made immeasurably easier by faster mainframe computers and the development of statistical software programs, such as SPSS (Statistical Package for the Social Sciences), which I use now in a PC environment. Computer mapping programs also became available to display the results geographically.

When I began computer-aided research at Iowa in 1963, historians using this tool were almost "nonexistent" (Theodore Rabb), and a survey in 1967 by the American Historical Association indicated that no more than two dozen faculty members had knowledge of social statistics as complex as multiple correlation and regression analysis. The first published article came out in 1963 from the work of William Aydelotte on the Corn Laws Parliament.

In the 1970s, however, the fear that computers would dehumanize history was lessening, and hundreds of graduate students took the plunge. Most gained a mastery of social statistics and computers by taking methods courses and seminars in sociology, political science, and economics departments. Many history graduate departments encouraged this outsourcing by encouraging graduate students to take courses in computer methods and social statistics as electives, cognates, or language options. The presumption was that statistics and computer languages were indeed "foreign languages" for history students, most of whom chose to study history because they were not good at mathematics. Little did they know . . . .

At the high point in the 1970s, books and articles based on the new methods made a major impact on social, political, and economic history. Yet, most historians remained as reluctant as ever to gain a minimal competence in quantitative research methods and social statistics. Nearly twenty years after the introduction of formal methods in history, "you could still get a Ph.D in history with no statistical training," lamented Theodore Rabb of Princeton University.

By 1981 half of the history departments nationwide offered at least one course in quantitative methods, but few required it. As a result, in any given year fewer than 20 percent of graduate students availed themselves of the opportunity, and among undergraduate history majors, many suffering from "math anxiety," the proportion was virtually nil. In retrospect, not to require such training of all graduate students was, in my opinion, almost a case of malpractice. And the failure to teach undergraduates the new methods had serious ramifications--it cut off the recruits.

By the early 1980s it was clear that the quantitative revolution in history had run its course. Several prestigious historians who had pioneered in the field in the 1960s expressed reservations. Bernard Bailyn of Harvard, in his presidential address to the American Historical Association, warned against elevating to a predominant place a "technical problem-solving" approach that is "severely vision-limiting."

Lawrence Stone of Princeton similarly worried about the computer tool becoming an end in itself. With his perceptive eyes trained on recent European historical scholarship, Stone noted approvingly that he detected the beginnings of a widespread shift away from numerical analysis and toward the revival of narrative history. Even among the new historians, such as the Annales school in France, mentalité had replaced quantification as the new kid on the block. This shift signals the end of an era, declared Stone. The quest for a scientific history failed, and practitioners are of necessity returning to older intuitive modes. Not only would the revival of narrative history restore a healthy balance among the various methodological genres or scholars, but the retreat from quantification toward story-telling and case studies would, said Stone, "shed more light with less trouble" ("The Revival of Narrative: Reflections on a New Old History," Past and Present 85 [Nov. 1979]: 19, 13).

Unanimity has never been a hallmark of the history profession, and the strong reservations of several notable scholars would not have been persuasive but for the ready acceptance by students and colleagues who were already skeptical of social science history. Clearly, by the late 1970s a "sense of disillusion" with quantitative methods had taken hold, noted Rabb. While no one wanted to deny the validity of counting, few history departments would revamp graduate training to include formal research methods. Rabb concluded: "The acceptance of quantification is grudging and limited, and its practitioners remain assured but isolated. The great hopes of the earlier days of the computer have apparently been disappointed; now there is a more limited role for those who wish to count" ("Coherence, Synthesis, and Quality in History, Journal of Interdisciplinary History, 12 [1981]: 315-32).

Why had the love affair between historians and computers gone sour? Was it simply a matter of changing fads? After all, by the early 1980s the initial excitement had worn off and computer-aided research had become a boring pastime of a few devotees. The play had now passed to case studies in the narrative mode that offered the recovery of meaning through "thick description," symbolism, and psychological analysis (Robert P. Swierenga, "Historians & Computers: Has the Love Affair Gone Sour?" OAH Newsletter [November 1984]: S2-S4).

One might also dismiss quantitative scholarship on the grounds that the multi-million dollar resource data banks had not yet produced definitive new interpretive syntheses. Indeed, the internecine squabbles among the practitioners, such as the "cliometrician's cockfight" over Time on the Cross, suggested that even the experts were confused and possibly misguided.

A spirit of pessimism pervaded the history profession at the time, as well. The general malaise was caused by falling enrollments and a lack of teaching jobs. The first students to abandon the profession were those with mathematical skills who could major in the social sciences where jobs were more plentiful. The diminishing pool of history graduate students thus included more people who were unable or unwilling to master cliometrics.

While faddishness, failed promises, flawed technical applications, and pessimism may partly explain the growing indifference among the 1980s generation of historians, there were more fundamental issues of political ideology and epistemology (the science of knowing, or how we know what we know).

Politically, from the outset many radical (i.e. Marxist) historians rejected quantitative approaches for its presumed conservative bias. (see David Landes and Charles Tilly in History as Social Science [1971], 14-15). Radical scholars would have none of serial data, statistical methods, and computer analysis. Quantitative data, they said, is inherently biased because it was created by the ruling class for their own purposes of domination. Statistical methods, likewise, were suspect because they often rest on positivist, neo-classical theories; and their handmaidens, computers, are instruments of the bourgeoisie. It wasn't a fluke that the Weatherman faction of Students for a Democratic Society (SDS) bombed the University of Wisconsin Computer Center in 1970, killing a physics graduate student who was working late into the night.

A few leftist historians, notably Lee Benson and Michael Katz, believed that computers can also serve radical scholarship. But Robert Berkhofer was essentially correct when he noted in 1983 that the ideological distance between radical and social science historians had widened rather than narrowed in the previous decade ("The Two New Histories: Competing Paradigms for Interpreting the American Past," OAH Newsletter 11, No. 2 [May 1983]: 9-12). In Berkhofer's picturesque phrase, "the two histories are not like two ships passing in the night upon the same sea of history; rather, they are like two ships sailing upon two quite different oceans, maybe at different times. . . . Their chances of communicating seem only slightly less remote than their chances of colliding."

The epistemological issue, I think, is at the crux of why historians no longer bother with quantitative data. Theodore Rabb first voiced this point in 1981, when he noted that at least since World War Two, historians did not share a common vision of their task, as they had done since the days of Heroditus and Thucydidies. "What is at stake is a profound epistemological question," said Rabb, "not just a disagreement over technique."

The reaction to quantification was rooted in the wider debate over the meaning of reality itself. On the one side are the materialists who seek to prove theory by statistical exactitude, replicable and precise methods, and generalized knowledge. On the other side are the anti-materialists who seek to restore the individual in history and to find meaning through the intuitive, persuasive study of ideas.

Given the "flight from materialism" and the rejection of science and technology by the "flower children" generation of the 1960s, it is no surprise, Rabb noted, that anti-materialists have won out. Perhaps, Rabb thought, "the relativism and uncertainty of the twentieth century world has caught up with us to such an extent that we consider even a small dose of unequivocal truth to be suspect, or more damning still, uninteresting." In a world of individual truths, one perspective is as valid as another. What is gained if economic historians labor for many years to prove that southern slave agriculture was 35 percent more efficient that northern family farms, or if social historians meticulously compare manuscript census lists to prove that over half of frontier Americans in any new community out-migrated within the first decade? The meaning of slavery or geographical mobility is apparent not in "hard" facts, but rather in the recovery of deeper levels of how the people felt, thought, and engaged in symbolic activities. The "culture of narcissism" (to use the phrase of Christopher Lasch) of the 1970s, clearly had a major impact on the study of history.

In the 1990s the deconstruction that is part and parcel of postmodernism has dealt a further blow to social science history. Indeed, it has undermined all historical methods. For postmodernists, the worldviews of individual researchers are no longer open to question. By worldviews, I mean the assumptions or "givens" that govern the framing of a scholar's research questions and determine the interpretation of their findings. Each worldview is self validating and equally true. The common ground of the scientific method, which historians have long held, has slipped away, in my opinion. Now historians and social scientists alike filter the past primarily through the lens of class, gender, or race.

Gene Edward Veith spelled this out clearly in a recent article in World magazine. "According to the worldview of postmodernism, truth is not a discovery but a construction. There are no absolutes or 'essences,' only social constructions that impose society's power relationships. Scholarship exists," Veith continues, "to 'deconstruct' what people take for granted as being good and true, unmasking an artifact, truth-claim, or institution as an act of cultural oppression" ("The APA does it again," World, 21 August 1999, 25).

So, for example, Farrakan's Nation of Islam has an Afrocentric worldview that makes Africa the mother of modern civilization and the American slave trade a Jewish-perpetrated holocaust. No amount of hard evidence, derived from voluminous ship records of the transatlantic trade of the nineteenth century, which show that few Jews owned slaving ships, makes a bit of a difference in challenging Farrakan's interpretation. His facts, buttressed by a whole library of books written by his followers at Malcolm X University in Chicago, are incontrovertible and not subject to debate or disproving by outsiders. No way can "white devils" understand his truth, just as he does not accept theirs.

Take another example from 1992, the 400th anniversary of Columbus' arrival on Santo Domingo. Columbus deserves no praise, said Russel Means, the American Indian leader; he "makes Hitler look like a juvenile delinquent." Hans Koning, a writer of Dutch decent, is equally close-minded: "It's almost obscene to celebrate Columbus because it's an unmitigated horror of history. We don't have to celebrate a man who was really--from an Indian point of view--worse that Attila." The National Council of Churches of Christ in the United States declared in righteous indignation: "For the descendants of the survivors of subsequent invasion, genocide, slavery, 'ecocide,' and exploitation of the wealth of the land, a celebration is not an appropriate observance of this anniversary."

This is pluralism with a vengeance. Two plus 2 no longer equals 4, it equals "whatever." My truth is as valid as your truth. In fact, we cannot even agree on the meaning of the words we use. The word holocaust for the Jews means the Nazi horror of the 1940s, but for Afrocentrists it means the slave trade of the 1840s.

The intellectual fallout of the "culture wars" has been to politicize history and the social sciences. There is no longer much place for scientific methods. It is a waste of time for graduate students to spend years gathering data from ship lists, census records, and land records, in order to discover broad patterns of human behavior. Other scholars will not accept your findings as valid anyway, unless they happen to fit into their reigning worldview of gender, race, or class. As Oscar Handlin observed: "When truth ceases to be an end in itself and becomes but a means toward an end, it also becomes malleable and manageable and it is danger of losing its character" (Truth in History, 1979, 414).

The developments of which I am speaking are reflected in the titles of recent books about our discipline. A sampling of publishers' lists bulge with titles such as: Silencing the Past: Power and the Production of History; In Face of Facts: Moral Inquiry in American Scholarship; Telling the Truth About History; and the chilling book, The Killing of History, by the Australian Keith Windschuttle (Mark Noll, "Intellectual History," Christianity & Culture [July/August, 1999]: 22-23).

Lest I leave the taste of sour grapes in your mouths, I want to end by expressing the hope that postmodernism too will pass. Until it does, everything past and present will be fragmentary and fictional. Quantitative history requires an acceptance of past reality as knowable, at least in part. The antireality of the postmodern conception of history is a road without traffic signs and guard railings. The sooner we get off that road the better.