Bibliographic Errors in Articles Submitted to Scholary Journals:
The Case for Research in the Schools

Anthony J. Onwuegbuzie
College of Education
University of South Florida
E-mail: tonyonwuegbuzie@aol.com

Vicki L. Waytowich
College of Education
University of North Florida
E-mail: VWaytowich@danielkids.org

            Articles published in scientific journals normally contain a list of references. The list of references serves a number of important scholarly functions. It provides credibility and support for the author’s statements, access to background information pertaining to the topic of interest, and a resource for the reader to find additional literature on the topic (Wyles, 2004). Readers of journal articles can follow up a cited reference to further their knowledge of existing research, to track down other related material, or to verify the assertions and claims made by the authors (Waytowich, Onwuegbuzie, & Jiao, 2006). To easily retrieve the cited works, it is essential that the references be accurate. Inaccurate references reflect poorly on the authors, devaluate the credibility of the article and the journal in which it appears, and also hinder the search for additional sources of related literature (Lok, Chan, & Martinson, 2001). Bibliographic errors in references can make it difficult for the reader to retrieve the cited works and to obtain, check, or verify information to which the text of a paper refers. Errors such as misspelled titles and author names could conceivably have the consequence of preventing important works from being retrieved, consulted, and recognized (Garfield, 1990). Certain errors such as incorrect publication year, volume, or page number make it particularly difficult to locate the cited work and to complete a bibliographic database search because the computer will not recognize an error or a misspelled name. Thus, reference errors can be a source of great frustration on the part of readers.

            Yet, errors in bibliographic citations are not recent problems. Citation errors can be traced back more than 100 years (Sweetland, 1989). The classic example of the bibliographic error is the 1887 Czech article written by Jaroslav Hlava, entitled “O uplavici”, which means “on dysentery” (Sassen, 1992). A German abstract later published the author’s name as O. Uplavici, consequently beginning 50 years of miscitations, culminating in 1938 as a result of research by Clifford Dobell (Sweetland, 1989). Even with the publication of various style manuals to guide the preparation of bibliographic citations, such as the Chicago Manual of Style (Chicago Manual, 2003), the Publication Manual of the American Psychological Association (APA, 2001), and the Modern Language Association (MLA) Handbook for Writers of Research Papers (Gibaldi, 2003), “citation errors continued to appear, as did an increasing number of complaints about them” (Sweetland, 1989, p. 293). In recent years, several studies have examined the bibliographic accuracy of citations.  These studies analyzed different types of citation errors in selected journals for a fixed time period to determine the prevalence of the bibliographic errors in these journals. The results have shown a high rate of errors ranging from 8% to 66.7%, with as many as 6% of the original articles being irretrievable (de Lacey, Record, & Wade 1985; Nishina, Asano, Mikawa, Maekawa, & Obara, 1995; Ngan Kee, Roach, & Lau, 1997; Roach, Lau, & Kee, 1997; Siebers, 2000; Siebers & Holt, 2000; Gosling, Cameron, & Gibbons, 2004).

            Until recently, studies of errors in bibliographic citations have tended to focus on published articles (Doms, 1988; Eichorn & Yankauer, 1987; Pundit, 1993; Ngan Kee, Roach, & Lau, 1997; O’Connor, 2002; Obermann, Mason, & Wilmes, 2002; Carroll-Johnson, 2004; Gosling, Cameron, & Gibbons, 2004; Spivey & Wilks, 2004; Wyles, 2004). However, little is known about the bibliographic error rates of articles before they become published. That is, to date, no study has examined bibliographic errors among articles submitted for possible publication to a journal. This was the purpose of the present study. Specifically, we investigated the manuscripts submitted to Research in the Schools, a nationally and internationally refereed journal, over a two-year period. Also of interest was to determine whether any of the following variables predict bibliographic citation errors: number of works cited in each article, number of authors of each article, gender of the first author, type of institution to which the first author belonged (i.e., Research I/Extensive vs. Research II/Intensive institutions of higher learning based on the Carnegie Classification of Institutions of Higher Education) (The Carnegie Foundation, 2000), and length of article. For the purpose of this inquiry, a bibliographic error was defined as a reference cited in the body of the article that was missing, incomplete, or inconsistent with the reference list.

Method

            The data for this study were obtained from the journal entitled, Research in the Schools (RitS). This journal was selected (i.e., criterion purposive sampling) because it represents a peer-reviewed journal that has a tradition of publishing quantitative, qualitative, and mixed methods studies (Daniel & Onwuegbuzie, 2004; McLean & Kaufman, 2003). Moreover, this journal has one of the widest scopes in the field of education, being devoted to research conducted in any educational setting from a public school (e.g., elementary, middle, high) to a training program conducted within an industry. Also, there are no age restrictions on the sample; the educational settings may include preschools, continuing education classes for adults, or adaptive skills courses in nursing homes. 

            RitS is a nationally and internationally refereed journal co-sponsored by the Mid-South Educational Research Association (MSERA) and the University of North Florida. Published twice yearly, RitS regularly reaches a large scholarly audience. The breadth of the journal’s authorship is evidenced by the varied geographical and professional backgrounds of scholars who have contributed to the journal and by the fact that academic libraries across the nation and abroad subscribe to it. RitS publishes original contributions in the following areas: (a) research in practice; (b) topical articles; (c) methods and techniques; (d) assessment; (e) educational policy, reform, and accountability; and (f) other topics of interest to educational researchers.

            Established in 1993, RitS has published 12 volumes that represent 23 issues by September 2005 (MSERA, 2005).  The current issue is Volume 12, Issue 1. The number of articles per issue has ranged from 6 to 9 (M = 7.86, SD = 0.71), yielding a total of 183 articles published in 12 years.

            RitS contains more than 50 editorial board members, of which a high proportion represents research methodologists. Each submitted article is assigned to five editorial board members. The editors of RitS ensure that every article submitted to the journal is reviewed by at least one research methodologist. Consequently, RitS provides an extremely rigorous review of articles. In fact, reviews of the submitted articles typically contain a minimum of 5 single-spaced pages and can contain as many as 25 single-spaced pages. RitS appears to be one of the very few journals that include a bibliographic error check of manuscripts as part of the review process.

            Consistent with our study purpose, we examined 52 articles submitted to RitS over a two-year period. This number of articles represents more than 50% of all the articles submitted to the journal over this period. For each of these submitted articles, the number of works cited in the body of the article that were missing, incomplete, or inconsistent with the reference list (i.e., bibliographic errors) were computed and entered into the Statistical Package for the Social Sciences (SPSS; SPSS Inc., 2005). Further, the number of works cited was documented. Also entered into this statistical database was a categorical variable that indicated whether the article was (a) accepted for publication, (b) given an opportunity to be revised and resubmitted, or (c) rejected. In addition, the length of the review of each article was recorded.

Results

            Of the 52 submitted articles, 7 were accepted for publication, 16 were given an opportunity to be revised and resubmitted, and 29 were rejected. Thus, the initial acceptance rate for these articles was 11.7%, whereas the initial rejection rate was 48.3%. The number of bibliographic errors ranged from 0 to 48, yielding a mean number of bibliographic errors of 5.87 (SD = 7.88). Only 7 out of the 52 submitted articles (i.e., 13.5%) contained a complete set of references. Of these 7 articles, 4 were accepted for publication.  The number of works included on each reference list of each article ranged from 8 to 60 (M = 31.10, SD = 15.68). Taking into account the total number of works cited in each submitted article, the mean proportion of bibliographic errors (i.e., bibliographic error rate) was 28.63% (SD = 27.37%).

            Articles that were rejected were then contrasted against those that either were accepted for publication or were given an opportunity to be revised and resubmitted. This dichotomous variable was then correlated with the number of bibliographic errors, yielding a statistically significant relationship (r = .25, p < .05) with a moderate effect size (Cohen, 1988). Alternatively stated, articles that were rejected (M = 7.62, SD = 9.78) contained statistically significantly more bibliographic errors (t = 2.02, p < .05) than did articles that were not rejected (M = 3.65, SD = 3.61), again with a moderate effect size (Cohen’s d = .51). Further, articles with more than three bibliographic errors were compared to articles with three or less bibliographic errors with respect to the editorial decision made. A subsequent chi-square analysis revealed that the former group of articles were statistically significantly more likely to be rejected (chi-square = 5.50, p < .05) than were the latter group, with an effect size, as measured by Cramer’s V, that was moderate (i.e., V = .33). Moreover, articles with more than three bibliographic errors were 4.01 (95% confidence interval = 1.22, 13.17) times more than likely to be rejected than were articles with three or less bibliographic errors.

            After applying the Bonferroni adjustment to control for the inflation of Type I error, a series of correlational analyses revealed that in each of these submitted articles, the number of bibliographic citation errors was not related to (a) the number of works cited (r = .20, p > .05) in each article, (b) the number of authors (r = .12, p > .05) of each article, (c) the gender of the first author (r = .18, p > .05), (d) the type of institution to which the first author belonged (r = .05, p > .05), and (e) the length of article (r = .11, p > .05).

            The length of reviews for each of the submitted article ranged from 4 to 24 single-spaced pages (M = 10.13, SD = 3.84). Interestingly, there was no relationship between the length of the reviews and whether or not the article was rejected (r = -.18, p > .05). Similarly, no association was found between the length of the reviews and the number of bibliographic errors (r = .03, p > .05).

Discussion

            An analysis of the bibliographic errors indicated that the authors’ articles, on average, contained more than 5 missing, inconsistent, or incomplete citations. Stated alternatively, the mean bibliographic error rate was 28.6%. This indicates that for approximately every 3 to 4 bibliographic citations (i.e., 100/28.6 = 3.49) contained in the reference lists of articles submitted to RitS, one of them represents some type of citation error. This finding is extremely disturbing because of the potentially serious consequences of bibliographic errors made by authors with regard to document retrieval, authenticating data, social factors, and credibility of the author (Faunce & Job, 2001; Hernon & Metoyer-Duran, 1992; Pandit, 1993; Spivey & Wilks, 2004; Sweetland, 1989).

            A statistically significant moderate relationship was found between the number of bibliographic errors and the decision made by the editor, with authors who made more than 3 bibliographic errors being more likely to have their articles rejected than their counterparts. This suggests that authors who commit bibliographic errors are more likely to have other major flaws (e.g., analytical errors) in their articles, leading to the rejection of their articles for publication. In other words, the carelessness demonstrated in these authors’ reference lists likely is symptomatic of an overall lack of attention to detail and to other aspects of the study and/or the manuscript. This carelessness in citing studies does not appear to improve as a function of the number of authors involved in preparing the manuscript. Indeed, the fact that co-authored manuscripts are just as likely as are single-authored manuscripts to contain bibliographic errors suggests that the multiple authors of these manuscripts are not being used efficiently with respect to checking the accuracy of bibliographic citations.  

            It could be argued that the high bibliographic error rate found among articles submitted to RitS represents an anomaly.  However, a preliminary analysis conducted by Onwuegbuzie, Waytowich, and Jiao (2005a) of a sample of articles submitted to American Educational Research Journal (AERJ), a top-tier journal, has revealed that the mean number of bibliographic errors is 8.00 (SD = 7.77). This mean number of bibliographic errors is even higher than that observed for RitS, although not statistically significantly higher (t = 0.67, p > .05). Furthermore, a preliminary analysis conducted by Onwuegbuzie, Waytowich, and Jiao (2005b) of a sample of articles submitted to Educational and Psychological Measurement (EPM), another top-tier journal, has revealed that the mean number of bibliographic errors is 3.00 (SD = 3.92). Although this is less than the mean number of bibliographic errors observed for RitS, it is not statistically significantly lower (t = 1.52, p > .05). Onwuegbuzie et al. (2005b) also documented a statistically significant and large relationship between the number of bibliographic errors and whether or not a manuscript is rejected (r = .52, p < .05), with articles containing more than 2 bibliographic errors being statistically significant more likely to be rejected (chi-square = 5.70, p < .05; Cramer’s V = .55) than manuscripts containing 2 or less errors. More specifically, articles with more than 2 bibliographic errors submitted to EPM were 12.50 times (95% confidence interval = 1.34, 116.80) more likely to be rejected than articles with 2 or less bibliographic errors.

            In any case, an important implication of the present findings, alongside the results pertaining to AERJ and EPM, is that interventions are needed to instill in authors the importance of avoiding bibliographic errors. Bearing in mind that citation error rates have been found to be even higher among doctoral students—with the citation error rates of doctoral dissertation proposals being as high as 90.91% (M = 33.45%, SD = 21.15%)—and that doctoral students likely will carry their tendencies to commit citation errors beyond their graduate programs (Waytowich et al., 2006), Several influential individuals can play an important role here, including college instructors, college advisors, and thesis/dissertation committee members. Journal editors also can play an important role here. Whatever interventions are used, it is essential that their effects on citation errors are monitored.
                                                           
                                                           References

American Psychological Association. (2001). Publication manual of the American &             Psychological Association (5th ed.). Washington, D.C.: Author.
Carnegie Foundation for the Advancement of Teaching. (2000). The Carnegie classification             of institutions of higher education (2000 ed.). Menlo Park, CA: Author. 
Chicago Manual of Style (15th ed.). (2003). Chicago: The University of Chicago Press.
Cohen, J. (1988) Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale,             N.J.:     Lawrence Erlbaum.
Daniel, L. G., & Onwuegbuzie, A. J. (2004). Editorial: Beginning the second decade
            of Research in the Schools. Research in the Schools, 11(1), 1-2.
de Lacey, G., Record, C., & Wade, J. (1985). How accurate are quotations and references in             medical journals? British Medical Journal (Clinical Research ed.), 291(6499),                         884-886.
Doms, C.A. (1988). A survey of reference accuracy in five national dental journals. Journal of
            Dental Research 68, 442-444.
Eichorn, P., & Yankauer, A. (1987). Do authors check their references? A survey of accuracy of
            references in three public health journals. American Journal of Public Health, 77(8),             1011-1012.
Faunce, G. J., & Job, R. F. S. (2001). The accuracy of reference lists in five experimental                     psychology journals. American Psychologist, 56(10), 829-830.
Garfield, E. (1990). Journal editors awaken to the impact of citation errors. How we control them             at ISI. Current Comments, 41, 5-13.
Gibaldi, J. (2003). MLA Handbook for Writers of Research Papers (6th ed.). New York:             Modern Language Association.
Gosling, C. M., Cameron, M., & Gibbons, P. F. (2004). Referencing and quotation accuracy in             four manual therapy journals. Manual Therapy, 9(1), 36-40.
Hernon, P., & Metoyer-Duran, C. (1992). Literature reviews and inaccurate referencing: An             exploratory study of academic librarians. College & Research Libraries, 53, 499-512.
Holt, S., Siebers, R., Suder, A., Loan, R., & Jeffery, O. (2000). The accuracy of references in             Australian and New Zealand Medical Journals. New Zealand Medical Journal,
           113 (1119), 416-417.
Lok, C. K.W., Chan, M.T.V., & Martinson, I. M. (2001). Risk factors for citation errors in             peer-reviewed nursing journals. Journal of Advanced Nursing 34(2), 223-229.
McLean, J. E., & Kaufman, A. S. (2003). A fond farewell editorial. Research in the Schools, 10             (2), 1-2.
Mid-South Educational Research Association (MSERA). (2005). Research in theSchools:             Index of articles published. Retrieved September 19, 2005, from                       http://www.msera.org/ritsindx.htm
Ngan Kee, W. D., Roach, V. J, & Lau, T. K. (1997). How accurate are references in the             Australian and New Zealand Journal of Surgery? Australian and New Zealand Journal             of Surgery, 67(7), 417-419.
Nishina, K., Asano, M., Mikawa, K., Maekawa, N., & Obara, H. (1995). The accuracy of             reference lists in Acta Anaesthesiologica Scandinavica. Acta Anaesthesiologica                     Scandinavica, 39(5), 577-578.
O’Connor, A. E. (2002). A review of the accuracy of references in the journal Emergency
            Medicine. Emergency Journal, 14(2), 139-141.
Obermann, M.H., Mason, N. M., & Wilmes (2002). Accuracy of references in general             readership nursing journals. Nurse Educator, 27(6), 260-264.
Onwuegbuzie, A. J., Waytowich, V. L., & Jiao, Q. G. (2005a). Citation errors in articles             submitted to scholarly journals: The case for American Educational Research             Journal. Unpublished manuscript.
Onwuegbuzie, A. J., Waytowich, V. L., & Jiao, Q. G. (2005b). Citation errors in articles             submitted to scholarly journals: The case for Educational and Psychological             Measurement. Unpublished manuscript.
Pandit, I. (1993). Citation errors in library literature: A study of five library science journals.             Library & Information Science Research, 15, 185-198.
Roach, V. J., Lau, T. K., & Kee, W. D. (1997). The quality of citations in major international             obstetrics and gynecology journals. American Journal of Obstetrics and Gynecology,             177 (4), 973-975.
Sassen, C. J. (1992). Citation accuracy in the journal literature of four disciplines: chemistry,
            psychology, library science, and English and American literature (English literature).             Unpublished doctoral dissertation, University of North Texas, Denton, Texas.
Siebers, R. (2000). The accuracy of references of three allergy journals. Journal of Allergy and             Clinical Immunology, 105(4), 837-838.
Siebers, R., & Holt, S. (2000). Accuracy of references in five leading medical journals. Lancet,             356 (9239), 1445.
Spivey, C. A., & Wilks, S. E. (2004). Reference list accuracy in social work journals. Research            on Social Work Practice, 14(4), 281-286.
SPSS Inc. (2005). SPSS 13.0 for Windows.  [Computer software]. Chicago, IL: SPSS Inc.
Sweetland, J. H. (1989). Errors in bibliographic citations: A continuing problem. Library            Quarterly, 59 (4), 291-304.
Waytowich, V. L., Onwuegbuzie, A. J., & Jiao, Q. G. (2006). Characteristics of doctoral             students who commit citation errors. Library Review, 55(3), 195-208..
Wyles, D. F. (2004). Citation errors in two journals of psychiatry: A Retrospective Analysis.                 Behavioral & Social Sciences Librarian, 22(2), 27-51.

You are invited to join AE Extra staff!
Send your ideas and/or writing sample to the Editor-in-chief:
Elizabeth Haller
Kent State University (e-mail: editoraee@hotmail.com)

Academic Exchange Extra invites reader response to any writings in this issue--especially articles advancing the scholarly debate of issues raised.