THE CENTER FOR A PUBLIC ANTHROPOLOGY'S 2006 NATIONAL RANKINGS OF PUBLIC OUTREACH IN ANTHROPOLOGY DEPARTMENTS COMPARED WITH THE NATIONAL RESEARCH COUNCIL'S 1993/95 NATIONAL RANKINGS OF SCHOLARLY QUALITY IN THESE DEPARTMENTS
EXPLANATORY DETAILS ON RANKINGS
TOPIC OF ASSESSMENT
Public outreach was defined in the Public Anthropology Assessment as addressing social concerns in the broader world beyond the university. Assessors were provided with three categories of information on each department assessed: (1) the number and types of programs associated with a particular department that focused on public issues and public outreach; (2) the number and types of public outreach activities -- past and present -- that individual faculty members within a department chose to describe; and (3) following the example of ISI's Social Sciences Citation Index, the degree to which individual faculty members within a department were cited in prominent printed media.
The National Research Council (NRC) Assessment, in assessing the scholarly quality of a department, considered “the scholarly competence and achievements of the faculty” in a department (from Research-Doctorate Programs in the United States, 1995:124). Assessors used a list of the full-time faculty in a department to arrive at their assessments.
Public Outreach Assessment: The pool of potential assessors (or sample) included all full-time faculty within American anthropology departments listed in the American Anthropological Association’s (AAA) Guide. Because the assessment took more than two years to prepare, both the 2005 and the 2006 AAA Guides were used. The assessment included 394 schools. The anthropology departments and programs at these schools collectively had 3613 full-time faculty members.
Despite repeated efforts to correct "bounced emails" – emails that were returned from the receiver's server without passing through to the actual individual the email was sent to – "bounced emails" varied during the assessment from between 62 and 89. Taking a cautious approach, I used the lower "bounced" rate, 62, and subtracted it from 3613, leaving 3551. It made no sense to include in the sample people who never received the Assessment's emails and, hence, could not participate.
Unsubscribes, individuals who were deleted from the email list, numbered 363. Of these, roughly ½ deleted themselves and ½ were deleted by me. I deleted individuals, for example, who indicated they were not anthropologists (though listed in the AAA Guide) as well as individuals who indicated (in an email) they were no longer full-time faculty and/or affiliated with the department they were listed under. I included all 363 unsubscribes as part of the 3551 sample. One might have made a case for deleting them. But I decided to take a cautious approach in this regard.
I also left in the sample all emails that were returned because the recipient was described as "out of the office" (or some similar such phrasing). A close look at those who participated in the assessment indicated that some of those listed in the out of office category were indeed participating.
In brief, then sample size (or pool of potential assessors) was 3551. 1428 faculty participated in the Public Outreach Assessment. The participate rate for the assessment was thus 1428/3551 or 40.21%.
National Research Council (NRC) Assessment: University administrators at each of the 69 schools involved in the 1993/95 NRC project selected the sample of participants. In all, 268 were selected from the total number of anthropology faculty -- 1368 -- at these schools. In other words, 19.7% of the faculty (268/1363) were selected to participate of which half (134/268) actually completed the NRC Assessment. Thus, 9.83% of the total 1363 anthropology faculty at the 69 departments being assessed in the NRC project participated.
INFORMATION PROVIDED TO ASSESSORS FOR MAKING THEIR ASSESSMENTS REGARDING A DEPARTMENT'S RANKING
Public Outreach Assessment: Assessors examined the outreach programs associated with a particular department as well as the outreach activities of individual faculty within that department. Information on programs associated with a department was collected by the Center for a Public Anthropology and sent to each department chair for review. Information on individual faculty outreach activities was drawn from two sources. To have a quantitative measure across schools, the Center used the LexisNexis databases to assess to what degree individual faculty members within a department were cited in the public media. Data were gathered from both the General News/Newspapers and the General News/Magazines and Journals databases. In addition, in an effort to gather more qualititative data, faculty members within each department were offered space to record their present and past outreach activities.
National Research Council Assessment: Participants were presented with the names of the full-time faculty in a department. No other information was provided.
In the Public Outreach Assessment, participants were presented with four randomly selected departments (excluding their own) that they then ranked by positioning a department (through a click and drag operation on a web page) into a hierarchy with the top ranking department in public outreach highest and the lowest ranked department at the bottom. Participants could not place two schools at the same hierarchical level. Given the breadth of data assessors had to examine on each school, it was thought best to present each assessor only four departments for evaluation. (In a trial run, increasing the departments an assessor had to evaluate to five or six seemed overly burdensome to several assessors.)
In the NRC Assessment, participants were asked to grade each department on a five point scale. There were no technical restrictions on how many fives or fours a participant could record though the NRC instructions requested assessors to use five (the highest) for only 10% of the programs assessed. Each assessor ranked up to 50 anthropology departments.
To be included in the NRC Assessment a department had to have granted at least five doctoral degrees within the past five years. In 1993, when the NRC assessment began, there were 69 anthropology departments that fit this qualification. For the Public Outreach Assessment, I took the NRC list and added schools I perceived as probably fulfilling this “five in five” requirement. The selection was not based on hard data. I wanted to insure that several departments which had developed strong doctoral programs since 1993 were included in the assessment. Only the University of Rochester, part of the NRC assessment, was dropped from the Public Outreach Assessment. It had terminated its doctoral program and, hence, no longer fit the NRC criteria.
Of the 394 schools from which potential assessors were drawn, 323 (or 82%) of the schools had faculty members participate in the Public Outreach Assessment. The NRC Assessment drew potential assessors from all 69 schools being assessed. It did not indicate, however, which schools the 134 faculty who chose to participate in the NRC Assessment came from.
TIES BETWEEN SCHOOLS
One of the interesting differences between the two assessments is that the Public Outreach Assessment has more ties and, in its numbering, softens the separation between rankings compared to the NRC Assessment. The NRC assessment, for example, skips a ranking when two schools are tied; the Public Outreach assessment does not.
We suspect that at least part of the reason for the greater number of ties in the Public Outreach ranking derives from assessors using a four rather than a five point scale. Also, as one can see from the chart on the next page (Overlap Between Schools with Different Rankings in the 1995 Assessment), the NRC method of numerical ranking may overstate the rank order of various departments. There is considerable overlap among departments in respect to their confidence intervals.
Overlap Between Schools with Different Rankings in the 1995 Assessment
(from Research-Doctorate Programs in the United States, 1995:627)