A Framework for Evaluating Scholarly Publications [VIDEO]
How to determine the quality of educational scholarship.
In academic professions, scholars desire to have their research articles published in professional journals that are widely read in their fields. However, multiple journals publish in most specialty areas. To impact a field of study, a scholar’s work must be published in an outlet with the greatest potential of influencing peers’ research. But how does one find out which journals are the most reputable for each field of study?
Rick West and Peter Rich, assistant professors in the Instructional Psychology and Technology Department in the McKay School of Education, developed a framework for evaluating scholarly publications that will soon be published in Innovative Higher Education, a scholarly journal dedicated to presenting fresh ideas in higher education. The proposed framework will help scholars determine a journal that will be most appropriate for their work and will help review boards more accurately assess a scholar’s publication. Without a well-defined rating system for journals, scholars rely on inconsistent criteria when comparing scholarly journals.
“Without clear criteria, many scholars guess at where they should submit their work, hoping others will approve of their decision,” West said. “It is critical to develop clear criteria for effectively evaluating the quality of publication outlets. Without such criteria, funding agencies and promotion committees are left to guess at how to evaluate a scholar’s portfolio.”
Many scholars tend to focus solely on a research journal’s impact factor from the Institute for Scientific Information when considering potential publication outlets. A journal’s impact is indicated in terms of how often the publication is cited by other scholars in the ISI database. If a journal ranks low in the citation count, experts do not highly value the publication. Articles that are published in journals with low citation counts are often overlooked and considered unimportant by experts and review boards. While impact factors are useful, they often underrepresent journals in specialty fields.
West and Rich believe that focusing on additional critical factors will help scholars to seek after publications with more balanced strengths and characteristics. The impact factor does not take into account the arduous selection processes of editorial boards, acceptance rates, or the academic journal’s reputation in a field.
To balance the journal evaluation process, West and Rich propose two additional criteria to accompany impact: rigor and prestige. West and Rich explain rigor as how strict the selection process is when work is submitted to be published in the journal. What are the journal’s acceptance rates? How good is the publication’s editor? How selective is the editorial board?
The professors explain prestige as the publication’s reputation. How do experts in the field regard the specific publication? If 10 experts were asked to indicate the most prestigious journal, where would the particular publication rank?
In addition, when considering impact, Rich and West believe scholars should consider more than just the ISI Impact Factor, looking at other measures of impact as well, such as Google Scholar citation numbers, adoption rates by end users, the circulation outreach of a publication, and other indications that the scholarship has impacted its target audience.
“Engaging in this discussion is critical,” Rich said. “If we cannot clearly articulate the criteria for determining the quality of our publication outlets, then others [i.e., promotion committees and funding agencies] will have to draw their own conclusions using metrics and criteria that may be less useful or even not applicable to, or representative of our disciplines.”
According to West and Rich, if scholars cannot effectively evaluate journals, important research may not influence scholars’ fields of specialization. When scholars evaluate a journal’s impact, rigor, and prestige collectively before seeking publication, scholarship will have a better chance of contributing to its target audiences and fields.
9 April 2012