Using the curriculum vitae for policy research: an evaluation of National Institutes of Health center and training support on career trajectories

Monica Gaughan 1 University of Georgia

1

[email protected]; College of Public Health, University of Georgia, Athens, GA 30302 th

An earlier version of this paper was presented at the 11 International Conference of the

International Society for Scientometrics and Informetrics, June 25-27, 2007, Madrid, Spain. The research reported here was supported by a contract from the National Institute of Child Health and Human Development (Barry Bozeman, Principal Investigator). Analysis was supported by a CAREER grant REC 0447878 from the National Science Foundation (Monica Gaughan, Principal Investigator). I thank my colleagues Barry Bozeman, Elizabeth Corley, Branco Ponomariov and Jan Youtie for their work throughout the project that made this study possible. Further, I am grateful for the able research assistance of Rebecca Arce, Timothy Atkins, Jason Epstein, Mary Feeney, Euiseok Kim, Dirk Libaers, Thomas Steiner, Alejandro Suarez, and Wesley Younger. The views reported here do not necessarily reflect those of the National Institutes of Health or the National Science Foundation.

1

Abstract Clinical research is a matter of vital concern to nations, yet there is widespread recognition that the institutions that support it may be somewhat limited. As a result, there has been enhanced attention to the mechanisms by which policy makers support such research, with particular attention to establishing research centers and fostering the training of clinical researchers. The impact of these relatively new activities on the scholarly career is yet to be explored. In this work, I use the curriculum vitae to study the careers of clinical scientists. Prospective, unobtrusive, complete, and available, the curriculum vitae contains such vital information as training, career timing and research characteristics. In the European context, it can also be used to study the effects of demographic indicators such as marital status, parity, and citizenship. The main limitation of using this vital source of data is the time and technological demands of reliably coding the data. In this study, I focus on the coding of CVs in light of model development and analytic demands to demonstrate how to use the curriculum vitae in multivariate survival analyses to study relevant policy outcomes.

Introduction Translational research—that which facilitates the transfer of scientific knowledge from bench to bedside—has emerged as a crucial component of the scientific research enterprise; the mechanisms by which training for this type of work should occur, however, are varied and not well understood (Nathan 2002). Multidisciplinary and multipurpose research centers and institutes create institutional contexts that enhance clinical research and extend scientific progress in human health research (Mallon and Bunton 2005). In 1990, the 101st U. S. Congress instructed the National Institutes of Health (NIH) to establish research centers to focus on the causes, diagnosis, and treatment of human infertility (US Congress 1990). The National Institutes of Health is the largest funding agency for biomedical research in the United States. The resulting centers of the National Cooperative Program for Infertility Research (NCPIR) joined other centers focused on human reproduction, Specialized Cooperative Centers Program in Reproduction Research (SCCPPR) to address, in part, the need for training in clinical and translational health research.

2

An important mission of these centers is to provide research training to doctoral, postdoctoral and preclinical researchers (DePaolo & Leppert 2002). The issue of developing the clinical research labor force is a matter of national importance, and the topic of a recent National Research Council/Institute of Medicine report that encourages creative approaches to supporting training (NRC 2006a). Center-based training is only one type of research training that the NIH supports. It is also possible to obtain doctoral and postdoctoral training through other mechanisms, which are targeted competitively to individual researchers rather than through the center funding mechanism (GAO 2002; Ley and Rosenberg 2005). Because of the shortage of reproductive health scientists, there is particular emphasis on the development of multiple training mechanisms in that field (Jaffe 1997; Longo, McClure and Jaffe 1999; McNellis and Alexander 1999). There is emerging evidence that such training arrangements are working to increase recruitment and retention to careers in academic medicine (Pion and Hammond 2005). It is therefore of some interest to U.S. policy makers to determine the relative effects of these two general mechanisms (individual compared to center-based) for training support on the research and careers of the researchers supported by federal funds. Several papers have been published discussing the organizational development of the National Institute of Child Health and Human Development (NICHD) reproductive research centers (Corley, Boardman and Bozeman 2006; Gaughan, Ponomariov and Bozeman 2007; Youtie, Libaers and Bozeman 2006). In this analysis, I examine the impact of affiliation with such centers on the career development of young trainees. Use of the Curriculum Vitae 1 to Study Policy Impacts

3

Scientific careers develop over long periods of time that include formal education, postdoctoral training, and moving through research and academic positions and institutions. As with any process that develops slowly and over a long period of time, the methodological challenge is to collect data that allows the study of causal processes in a valid and reliable way. Cross-sectional studies are limited in their ability to account for selection bias, and asking respondents to engage in retrospective reporting invites recall bias and respondent fatigue. An emerging research tradition has been using the curricula vitae of researchers to evaluate policy impacts. Curricula vitae can be reliably coded to reflect valid career constructs, and represent prospective, longitudinal records of scientific careers (Dietz et al, 2000). Because they are routinely maintained by scientists, respondent burden and recall bias is reduced. Recent work in Spain has shown the usefulness of obtaining program-generated curricula vitae to evaluate mobility dynamics predicted by European-level policy to have impacts on scientific productivity (Canibano, Otamendi and Andujar, 2008). In the United States, curricula vitae have been used to study center impacts on grants activity (Corley, Bozeman and Gaughan, 2003; Gaughan and Bozeman, 2002), commercial activity (Dietz and Bozeman, 2005; Lin and Bozeman, 2006), and collaboration and productivity (Lee and Bozeman, 2005). In France, the curriculum vitae has been used to study career transitions (Gaughan and Robin, 2004; Mangematin, 2000; Sabatier, Carrere and Mangematin, 2006). In this work, I use curricula vitae to test center impacts on training trajectories of young researchers. Research Design

4

This work is part of a larger program of research evaluation work conducted by the Research Value Mapping (RVM) program, which originated at the Georgia Institute of Technology and is now housed at the Consortium for Science Policy and Outcomes at Arizona State University. 2 The focus of the research was to evaluate the impact of National Cooperative Program for Infertility Research (NCPIR) and Specialized Cooperative Centers Program in Reproduction Research (SCCPRR) center involvement on their trainee affiliates in the early career. Center directors and administrators identified the names of trainees affiliated with their centers. The comparison group of trainees not affiliated with the focal centers was constructed using names provided by the NIH. Trainees at universities receiving other kinds of center training grants— mechanisms that preceded the NCPIR and SCCPRR Centers programs—were eliminated from the target population. This approach ensured that the comparison trainees doing reproductive health research were not affiliated with any of the target Centers, or with any other type of research Center during their training. The original data collection methodology was developed by Dietz and colleagues to examine National Science Foundation center affiliates (2000). Despite the passage of time, the Internet still is not a good source of unobtrusive CV data collection in the U.S. Therefore, as in other CV collection protocols, we used a variety of mechanisms for obtaining the most recent complete CV from the target trainees: by email, during interview scheduling for site visits, and on the site visits. More detail about the data sampling and collection and data management protocol is available in recently published work (Gaughan & Ponomariov, 2008). Using the Curriculum Vitae to Measure Constructs

5

We use the science policy and scientific careers literature to guide our choice of constructs, which include individual factors, education and employment histories, and publications and grants activity. The key construct of interest here is the impact of center affiliation on the early career. Although there are many studies of the importance of this rapidly developing aspect of universities (see Bozeman and Boardman 2004 for an historical overview), there are only a few studies looking specifically at impacts on the early career. For example, recent work by Boardman and Ponomariov (2007) on centeraffiliated scientists showed that tenure (in the US, associated with a more advanced career) was associated with a broader perspective on the what constitutes scientific productivity. In general, scientists affiliated with centers are more research productive than their exclusively discipline-based peers (Corley and Gaughan 2005). There is a well-developed research and policy literature in the United States documenting the persistent disadvantage associated with being a woman in science (for overviews, see National Research Council 2001, 2006 b, c; Long and Fox 1995). Similarly, there are a number of findings relevant to the role of educational and work experiences in the scientific career. In the United States, the postdoctoral position is widely associated with decreasing career velocity (NRC 2000). Furthermore, there is evidence that clinical research conducted by physicians is less likely to be funded by NIH than that conducted by researchers with the doctorate (Kotchen 2004). The positive role of collaborations has been established for publication productivity (Lee and Bozeman 2005). Dietz and Bozeman (2005) found that scientists with industrial experiences have higher patent productivity.

6

We used an ACCESS database to code and manage the data. We followed the reliability methodology established in previous work (Dietz et al. 2000), achieving a level of inter-coder reliability above .85 (Crittenden and Hill 1971). The final database includes the following indicators: Individual Education Employment Publications: Grants/Contracts:

Name, sex (national origin and citizenship when available) For all collegiate degrees and beyond: Degree type, degree year, degree institution, degree field For all jobs: Start year, end year, institution, title, type and disciplinary field For all peer-reviewed article publications since 1990: Publication year, number of authors, author order, journal For all grants and contracts: Start year, end year, total grant amount, grant source, activity code, Institute code, role on grant

I use these indicators to develop the constructs of interest in this evaluation. Specifically, I use education data to determine professional age (years since award of terminal degree) and whether the trainee is a medical doctor. Employment data is used to determine whether the person has ever held a postdoctoral position, ever had a nontraditional professional position (e.g., in government or industry), and for the dependent variable of the transition to an assistant professorship. Publications yield information about both productivity and tendency to collaborate with others. Finally, grants and contracts information establish transitions to first grant, first NIH grant, and first NIH grant as principal investigator. As already described, the sample development strategy is the source of the main independent variable: Center affiliation. To summarize, in the past, RVM used an Internet-based data-entry system. For the NICHD study, we converted to an ACCESS database design, which made the coding process more reliable and efficient, and analytic options more flexible. Most important,

7

we moved away from prior attempts to code every possible variable available on a CV. Instead, we relied on theoretical, evaluative, and analytic criteria to guide our choice of variable coding, significantly reducing the time and cost of coding the CVs. Descriptive Results In Table 1, I examine the overall mean, and mean differences between centeraffiliated and unaffiliated trainees. The first column indicates the parameter for the sample as a whole, while the columns on the right compare the center-affiliates with the unaffiliated controls. Figures shown in bold signify a statistically significant difference (two-tailed test, .05 or better) between the center-affiliates and the controls. In some important respects, the center trainees and comparison trainees are statistically identical. Slightly over one-half are male, and two-thirds have a Ph.D. In terms of grant milestones, the year in which those who receive a first grant (n=42), first NIH grant (n=36), and first NIH grant as PI (n=21) do not differ by center affiliation status. One quarter of trainees report any industry or government experience during the career. They publish at similar rates, somewhat over 1.5 articles per productive year. These articles are produced, over the time period studied, with about 4.5 coauthors. There also are systematic differences between center trainees and unaffiliated trainees—differences which should be taken into account when examining the impact of center affiliation on career trajectories. In particular, the unaffiliated trainee group is somewhat more experienced than the center trainees: they received their terminal degrees three years earlier (mean=1994) and started assistant professor positions three years earlier (mean=1995; n=31) than the center affiliated trainees. Not surprising, given their younger professional age, center trainees have published almost 10 fewer articles

8

over the period. Adjusting for professional age, there is no statistical difference in publication rates. Unaffiliated trainees average five and one-half more lifetime grants than center trainees, which is also related to their older career ages. The purpose of this bivariate analysis was to evaluate potential differences between the sample populations so that they could be controlled for in multivariate analysis. They should not be used for policy decisions, or for inferences about real differences between the two groups. Because the comparison group, on average, is older and more experienced, it is also more productive in a general way. It is therefore critical to model rates, or to adjust for professional age in all analyses. Career Trajectory Analyses Many multivariate productivity analyses rely on Ordinary Least Squares estimation; OLS is arguably the most commonly used multivariate technique used in policy analysis today—and for good reason. OLS is robust to violations of assumptions, and tends to yield unbiased, efficient estimators (Gujarati 1978). It is not without its limitations, however. The most important limitation with respect to one of the study’s theoretical questions is the difficulty of handling censored data using OLS estimation. Censoring occurs when a large percentage of respondents do not experience an event of interest. When this occurs, it violates the distributional assumptions underlying OLS estimation. This type of analytic problem is resolved by the use of survival analysis. The basic conceptual problem of survival analysis is to estimate the probability of some event occurring, given that the event has not occurred yet, at each point in the risk of experiencing the event (Allison 1995).

9

Among these trainees, I am interested in the impact of center affiliation and potentially other covariates on the likelihood that a trainee will experience an event in any period. I model the following events: time to assistant professor position, time to first grant, time to first NIH grant, and time to first NIH grant as a PI. Note that experiencing the “risk” or “hazard” is a positive development in a person’s career. I conceptualize career stage as time to permanent academic position or grant event. If center affiliation has an independent effect, then one will observe a difference in the hazard rates of entry or achievement. The output of these analyses is comparative survival plots (Allison 1995). A difference between the two groups calls for proceeding to multivariate hazard models to assess the additional effects of the control variables discussed in the productivity section. For other analyses using this method, see Gaughan and Robin 2004; Corley, Bozeman and Gaughan 2003; and Gaughan and Bozeman 2002. I use timing and event status data to construct hazard rates for entry into assistant professor or grant states. Of 29 unaffiliated trainees, 34% are censored, or do not enter the assistant professor rank. Of 47 Center affiliated trainees, three-quarters are censored: only 25% have become assistant professors during the observation period. The likelihood ratio test indicates that there is a significant difference between the two groups in propensity to enter an assistant professor position (Chi Square 8.82). To illustrate, Figure 1 shows the survival plots of the time to assistant professor by whether or not the trainee has any center affiliation. A survival plot is simply a graph of a life table: note the difference in survival rates between center affiliates and unaffiliated trainees. The lower curve—that of the unaffiliated trainees—indicates a lower survival rate. In this

10

case, lower survival is “good”: in every year of risk, the unaffiliated trainees experience the hazard (entry into an assistant professorship) at a higher rate than the center affiliates. Other survival plots (not shown) show similar significant differences in the distribution of outcomes for getting a first grant from any source (including NIH). Of 29 unaffiliated trainees, four-fifths have ever received a first grant, compared to 40% of the center affiliates. The likelihood ratio test indicates that unaffiliated trainees are significantly more likely ever to have been awarded a grant (Chi Square 8.47; p=.004). Turning to a consideration of NIH grants, among the 29 unaffiliated trainees, threequarters have received a NIH grant compared to 30% of center affiliates. The likelihood ratio chi-square of 12.07 indicates that unaffiliated trainees are significantly more likely to obtain NIH grants than center affiliates (p=.001). Finally, slightly more than half of unaffiliated trainees have experienced being principal investigators on NIH grants compared with only 13% of center affiliates. The likelihood ratio test of differences between the strata is significant, indicating that unaffiliated trainees and center affiliates differ significantly in their propensities to have achieved this career milestone (ChiSquare 8.79, p.=.003). In summary the results indicate that unaffiliated trainees are significantly more likely than center affiliates to experience the event at each career age. These differences are not surprising given the earlier bivariate findings that the trainee samples (affiliated and unaffiliated) differed significantly on several indicators. Because of these significant differences between the two groups, it is important to proceed to multivariate survival analysis to control for covariates that may explain the difference between the two groups. Cox Proportional Hazard Models

11

The Cox proportional hazard model models the instantaneous change in the hazard rate using maximum likelihood estimation. The hazard is the likelihood of experiencing the event in any time period (here conceptualized as years) given that the event has not yet occurred. The baseline hazard rate is a function of professional age (time since Ph.D.) and professional age squared. The “age” controls are typical in these models to adjust for the first increasing with time, and then decreasing with time, likelihood that the event will occur for any individual in his or her career. Once the baseline hazard is estimated, nested models control for the primary independent variable of center affiliation, with additional variables used as covariate controls. 2 Stated formally, the basic hazard model is represented as follows: Log (P(t)/(1 – P(t))= a(t) + B11 X1 + X2’ B12 + B13 X3 Where:

Log (P(t)/(1 – P(t))=hazard of entry to position a(t)=Vector of time X1=Center Affiliation X2= Vector of Demographic Controls X3=Collaboration Pattern

As in earlier bivariate analyses of survival plots, I evaluate four outcomes: time to assistant professor, time to first grant, time to first NIH grant, and time to first NIH grant as principal investigator. In the bivariate analysis of differences between trainee groups, there were significant differences in career age (as indicated by year of last degree, post-doctoral position starting date, and years of publication). The advantage of the event history framework is that it controls for differences in “age” in the model. To be specific, I define entry into the “risk” of experiencing the event as the year in which the researcher earned his or her terminal degree. The computational algorithm then

2

I used SPSS to estimate statistical models, but the event history/survival analysis models are available in most commonly distributed software packages, including SAS and Stata.

12

estimates the hazard of experiencing the risk in year one of the period, controlling for the specified covariates. Those who experience the event leave the risk set, and the estimation proceeds to the next time period of risk with those who remain. In this way, the careers of the two trainee groups are standardized by conceptualizing the events of interest in relation to a common reference point, the year of terminal degree (which all have experienced). Table 2 shows the survival analysis results for all outcomes. Entry into an Assistant Professor Position The first column reports the results of the hazard of entering an assistant professor position, which 41% of trainees have achieved during the study period. The baseline hazard rate indicated that each year increases the likelihood of entering such a position, and the time-squared variable indicates that eventually this likelihood becomes negative. This is a typical finding in survival analysis of career position transitions. Center affiliation has a negative effect in this model: Center-affiliated trainees are significantly less likely to become assistant professors, even controlling for other covariates. Stated in the more intuitively appealing odds-ratio, center-affiliated trainees are less than half as likely to become assistant professors as unaffiliated trainees. Other demographic and career event variables are not predictors in the full model. Finally, a trainee’s collaboration rate is associated with a lower likelihood of making the transition to assistant professor, while publication rate is associated with a higher likelihood of making the transition. Time to First Grants The second column of Table 2 reports the results of the Cox proportional hazard model of time to being awarded any grant. Column 3 reports the results for first NIH

13

grant, and the last column reports the results for first NIH grant as principal investigator. In these models, one does not observe the pattern of a first increasing, and then decreasing rate in the likelihood of being awarded a grant. Given the idiosyncrasies in grant success in careers, this is not a surprising set of findings. Indeed, the only consistent finding across the dependent variables is the strong, positive and statistically significant effect of publication velocity. Importantly, center affiliation has neither a positive nor a negative effect on grants timing. Discussion This paper has presented detailed information about how to proceed with a survival analysis of career data obtained from curricula vitae. Specifically, the structure of the data requirements are described, and the formal modeling approach are outlined. Using the Cox Proportional hazard model, I demonstrate that program effects of participation in a training program can be estimated. Such a methodological and estimation approach could be extended to answer other policy questions related to scientific careers. The results are limited to the population of early career age reproductive health researchers affiliated in some way with NIH training programs. While I feel strongly that the methodology is widely applicable to the study of scientists and engineers, inferences about the specific policy effects of centers should be limited to this population, and this particular policy intervention. The center supported trainees are less likely to become academics, but they are just as likely to obtain NIH grant support, suggesting that there may be other ways in which these two groups differ. Such analysis should rely on interview data, archival program information, and perhaps motivational factors derived

14

from formal or semi-structured interview protocols to develop. Although CV analysis is an important tool in research evaluation, researchers must not rely on it to the exclusion of other well-developed methodologies. I conclude by emphasizing the importance of relying on theoretical problems in developing data collection and coding strategies. To which population does the researcher want to generalize? What research questions are of interest? What constructs make up the research questions, and how might they be measured using data contained in CVs? Are there particularities of career behavior in this population that should be addressed (e.g., in this study, we had to create a MD/Ph.D. category for the first time)? My experience working with three major CV efforts is that there is no substitute for sound theoretical and methodological reasoning in CV work. The same is true for analytic work: research questions should drive the type of analysis that is conducted. In some cases, descriptive or bivariate analyses are all that is needed to illuminate career trajectories or policy processes. If the researcher wishes to conduct event history analyses like those presented in this paper, then he or she must ensure that both time to event and censoring variables are coded from the CVs. Again, the coding decision should be derived theoretically. For example, how do events unfold in a particular context (e.g. in the United States, “tenure” typically does not occur until 6 years after appointment to a professorship, while “tenure” is conferred in some European systems upon obtaining a professorship)? What are meaningful milestones (e.g. year of Ph.D., beginning of named chair, first national award)? Planning research questions (and all that entails) in advance will greatly improve the efficiency of CV collection and coding, and will help ensure that constructs of interest are available when it is time to do analysis.

15

TABLE 1: Descriptive and bivariate statistics of center and non-center affiliate trainees Sample N Demographic Gender Year of Last Degree Have Ph.D. Have M.D. Career Event Center Affiliation Post-doc Start Year Assistant Professor Start Year First Grant Year First NIH Grant Year First NIH PI Grant Year Any Nontraditional Productivity Years of publishing Total publications since 1990 Publication rate since 1990 Collaboration Total number coauthors Co-authorship rate Funding Total number of grants

Mean

Center Mean

Control Meana

76 76 76 76

.54 1996 .67 .33

.55 1997 .64 .38

.52 1994 .72 .24

76 62 31 42 36 21 76

.62 1995 1996 1994 1996 1997 .26

-1997 1998 1995 1996 1997 .26

-1993 1995 1993 1995 1997 .28

76 76 76

9.76 18.12 1.63

8.32 14.43 1.53

12.10 24.10 1.79

76 76

85.05 4.53

72.74 4.73

104.97 4.20

76

5.28

3.13

8.76

16

TABLE 2 Career Event Timing for the Trainee Sample of 75 Respondents Cox Proportional Hazards Model: Log Odds Coefficients Hazard Rate on Career Covariates and Controls Transition to: Assistant First Professor Grant

First NIH

First NIH PI

Baseline Professional Age Professional Age Squared

0.47* -0.02*

0.18 -0.01

0.13 -0.004

0.19 -0.01

Any Center

-0.85*

-0.44

-0.67

-1.22

Demographic Male MD

-0.32 0.07

-0.54 0.03

-0.88* -0.16

-1.07 0.54

Career Events Ever Postdoc Nontraditional Ever Grant

0.24 -0.40 0.54

0.37 0.38 0.18

0.48 0.52 0.25

0.17 0.05 0.48

Career Velocity Collaboration Rate Publication Rate

-0.32* 0.83***

-0.01 0.43***

0.06 0.51***

0.23 0.50*

-2 Log Likelihood d.f.

191.61 10

300.81 10

250.35 10

135.48 10

Note: *=p < .05; **=p<.01; ***=p<.001

17

FIGURE 1

Center Affiliates

Unaffiliated

18

References Allison, P D 1995. Survival analysis using the SAS system: a practical guide. Cary, NC: SAS Institute. Boardman, P C & B L Ponomariov 2007. Reward systems and NSF university research centers: The impact of tenure on university scientists’ valuation of applied and commercially relevant research. The Journal of Higher Education 78(1), 51-70. Bozeman, B, & Boardman, P C 2004. The NSF engineering research centers and the university-industry research revolution: A brief history featuring an interview with Erich Bloch. The Journal of Technology Transfer, 29(3–4), 365–375. Canibano C, J Otamendi and I Anujar 2008. Measuring and assessing researcher mobility from CV analysis: the case of the Ramon y Cajal programme in Spain. Research Evaluation, 17(1), 17- 31. Corley, E A, Boardman, P C & B Bozeman 2006. Design and the management of multi-institutional research collaborations: Theoretical implications from two case studies. Research Policy 35, 975-993. Corley, E A, B Bozeman & M Gaughan 2003. Evaluating the impacts of grants on women scientists’ careers: The curriculum vita as a tool for research assessment. In P. Shapira and S. Kuhlmann (Eds.), Learning from Science and Technology Policy Evaluation: Experiences from the U.S. and Europe. Cheltenham, UK: Edward Elgar Publishing. Corley, E A and M Gaughan. 2005. Scientists’ Participation in University Research Centers: What are the Gender Differences? Journal of Technology Transfer 30:371-381. Crittenden, K and R Hill 1971. Coding reliability and validity of interview data. American Sociological Review, 36, 1073-1080. DePaulo, L V and P C Leppert 2002. Providing research and research training infrastructures for clinical research in the reproductive sciences. American Journal of Obstetrics and Gynecology, 187, 1087-90. Dietz J and B Bozeman 2005. Academic careers, patents and productivity: industry experience as scientific and technical human capital. Research Policy, 34, 349367. Dietz, J, I Chompolov, B Bozeman, E Lane and J Park 2000. Using the curriculum vita to study the career paths of scientists and engineers: An exploratory assessment. Scientometrics, 49 (3), 419-442. Gaughan, M and B Bozeman 2002. Using curriculum vitae to compare some impacts of NSF research center grants with research center funding. Research Evaluation, 11, 17–26. Gaughan, M and B Ponomariov 2008 Faculty publication productivity, collaboration, and grants velocity: using curricula vitae to compare centeraffiliated scientists and unaffiliated scientists. Research Evaluation 17(2).

19

Gaughan, M and S Robin 2004. National science training policy and early scientific careers in France and the United States. Research Policy, 33, 569-581. General Accounting Office 2002. Clinical Research: NIH has Implemented Key Provisions of the Clinical Research Enhancement Act. Washington, DC: General Accounting Office. Gujarati, D N 1978. Basic Econometrics. New York: McGraw Hill. Jaffe, R B 1997. Is the academic physician-scientist an oxymoron in contemporary obstetrics and gynecology? American Journal of Obstetrics and Gynecology, 177, 892-893. Kotchen, TA, T Lindquist, K Malik and E Ehrenfeld 2004 NIH peer review of grant applications for clinical research. Journal of the American Medical Association, 291, 836-843. Lee S and B Bozeman 2005. The impact of research collaboration on scientific productivity. Social Studies of Science, 35(5), 673-702. Ley T J and L E Rosenberg 2005. The physician-scientist career pipeline in 2005: Build it and they will come. Journal of the American Medical Association, 294(11), 1343-1351. Lin M and B Bozeman 2006. Researchers’ industry experience and productivity in university-industry research centers: A “scientific and technical human capital” explanation. Journal of Technology Transfer, 31, 269-290. Long, J. S. and M. F. Fox (1995). Scientific careers: Universalism and particularism. Annual Review of Sociology, 21, 45-71. Longo, L D, M E McClure and R B Jaffe 1999. Reproductive physician-scientists for the twenty-first century. American Journal of Obstetrics and Gynecology, 181, 934-939. Mallon, W T and S A Bunton 2005. Research centers and institutes in US medical schools: A descriptive analysis. Academic Medicine, 80 (11), 1005-1011. Mangematin, V 2000 PhD job market: professional trajectories and incentives during the PhD. Research Policy, 29, 741-756. McNellis, D and D Alexander 1999. New opportunities for researchers in obstetrics and gynecology through programs supported by the National Institute of Child Health and Human Development. American Journal of Obstetrics and Gynecology, 181, 221-225. Nathan, D G 2002. Careers in translational clinical research—historical perspectives, future challenges. Journal of the American Medical Association, 287(18), 24242427. National Research Council 2007 Beyond bias and barriers: Fulfilling the potential of women in academic science and engineering. Washington, DC: The National Academies Press. ---------- 2006a Opportunities to Address Clinical Research Workforce Diversity Needs for 2010. Washington, DC: The National Academies Press. ---------- 2006b Biological, social, and organizational components of success for women in science and engineering. Washington, DC: National Academy Press. ---------- 2006c To recruit and advance: Women students and faculty in science and engineering. Washington, DC: The National Academies Press.

20

---------- 2001 From scarcity to visibility: Gender differences in the careers of doctoral scientists and engineers. Washington, DC: National Academy Press. ---------- 2000 Enhancing the postdoctoral experience for scientists and engineers: A guide for postdoctoral scholars, advisers, institutions, funding organizations, and disciplinary societies. Washington, DC: The National Academies Press. Pion, G M and C B Hammond 2005. The American Association of Obstetricians and Gynecologists Foundation Scholars Program: Additional data on research-related outcomes. American Journal of Obstetrics and Gynecology, 193, 1733-1739. Sabatier, M, M Carrere and V Mangematin 2006. Profiles of academic activities and careers: does gender matter? An analysis based on French life scientist CVs. Journal of Technology Transfer, 31, 311-324. United States Congress 1990. Departments of Labor, Health and Human Services, and Education, and related agencies appropriation bill. Report 101-127. Youtie, J, D Libaers and B Bozeman 2006. Institutionalization of university research centers: The case of the National Cooperative Program in Infertility Research Technovation 26, 1055-1063.

1

According to both the Oxford English and American Heritage Dictionaries, the correct spelling of the singular is curriculum vitae, and the plural is curricula vitae. 2 For Research Value Mapping Program information, see www.cspo.org/rvm/

21