Evaluating the Impact of Medical Research in a New Way

Scientific journals and research papers are evaluated by a metric known as their “impact factor,” which is based on how many times a given paper is cited by other papers. However, a new study from MIT and other institutions suggests that this measure does not accurately capture the impact of medical papers on health outcomes for all patients, particularly those in low- or middle-income countries.

To more fully capture a paper’s impact on health, metrics should take into account the demographics of the researchers who performed the studies and the patients who participated in them, the research team says. To that end, they have developed a metric that they call the “diversity factor.”

The new metric incorporates several factors, including the diversity of the paper authors (in terms of gender and geographic location), diversity of the patients studied, and how interdisciplinary the research team is. In a new study, the researchers evaluated more than 100,000 medical papers published in the last 20 years and found that most did not do well on this metric.

“The medical knowledge system is controlled by a very noninclusive group of academics, and it’s not diverse at all,” says Leo Anthony Celi, a senior research scientist at MIT’s Institute for Medical Engineering and Science, a physician at Beth Israel Deaconess Medical Center, an associate professor at Harvard Medical School, and one of the authors of the paper.

The researchers hope that their new study will generate more discussion of how to evaluate medical papers and make sure they are contributing to positive health outcomes for diverse populations, not just the groups who have traditionally led and been the subjects of medical studies.

Jack Gallifant, a physician at Imperial College London NHS Trust, led the new study, which appears today in PLOS Global Public Health. The authors also include researchers from institutions around the world, including Mbarara University of Science and Technology in Uganda, National Polytechnic Institute in Mexico, University of the Philippines at Manila, the University of Witwatersrand in South Africa, Handong Global University in South Korea, and King Hussein Cancer Center in Jordan, and representing the fields of public health, pharmacy, medicine, computer science, engineering and the social sciences.

“Blind spots” in medical knowledge

Celi and his colleagues began developing the new index in hopes of finding ways to document and combat the lack of diversity among authors of prominent medical publications. Most of these authors come from wealthy nations including the United States, and they are disproportionately white and male.

“How the medical knowledge system is designed has led to the fact that 80 percent of publications come from 20 percent of countries, and then the guidelines that are disseminated to treat diabetes, to treat hypertension, to treat cancer, are informed by trials and observational studies in those 20 percent of countries,” he says.

To quantify the extent of this problem, the researchers created an index based on several factors. One factor is the diversity of the authors of a study, including whether the authors are located in a high-income country or a low- and middle-income country. The researchers also used an algorithm that categorized the authors as male or female. This is important not only for promoting inclusion, but also because lack of diversity among authors of a medical study can result in failure to fully consider how a particular disease affects different groups of people, Celi says.

“What happens when all of the authors involved in a project are alike is that they’re going to have the same blind spots. They’re all going to see the problem from the same angle,” he says. “What we need is cognitive diversity, which is predicated on lived experiences.”

Another factor that the researchers incorporated is the diversity of departmental affiliations of the authors of a study. Under this metric, papers are given a higher score if they include authors from a wider range of disciplines. For example, a medical study that included doctors, nurses, and engineers would be rated more highly than one that included authors from just one of those fields.

The final metric that the researchers proposed is based on the diversity of traits of patients included in a given study, including sex, race-ethnicity, language, age, and geography. However, because many of the papers analyzed in this study were not open-access, the researchers could not get this information for many of the papers and so did not include it in their final analysis.

Celi emphasizes that the metrics chosen for this study should be considered only a starting point for measuring impact on health outcomes in a more equitable way.

“What we wanted is to start a public dialogue on this topic, and we also want the community to contribute and recommend what the metrics should be,” he says.

Tracking diversity

Once the researchers identified the criteria they wanted to analyze, they used the OpenAlex database, which aggregates information from many other databases of academic articles, to extract metadata on the authors of about 130,000 medical papers, from more than 7,500 journals, published between 2000 and August 2022.

They found, as they expected, that female authors were consistently underrepresented. Across all journals, male authors outnumbered female authors, but the representation of female authors has been increasing. In 2021, the ratio of female to male authors per publication was between 0.30 and 0.42 in the major journals.

The researchers also found a lack of geographic diversity. For papers published in 2021, there were more than 5 million authors from high-income countries, 1.5 million from upper-middle-income countries, about 470,000 from lower-middle-income countries, and just over 27,000 from low-income countries.

“It’s no surprise, but we wanted to quantify it. And we want to be able to track this over time, so we can tell if we are making headway,” Celi says.

The researchers found that overall, open-access journals had better scores on the diversity index than journals that are not open-access. Among the top 25 open-access and non-open-access journals (ranked by impact factor), the open-access journals had consistently higher proportions of female authors and authors from low and middle-income countries.

Many entities, including academic institutions, journals, and funding agencies can and should play a role in increasing the diversity of authors publishing in medicine, Celi says. Some funding agencies, including the National Institutes of Health, have created initiatives, such as the Artificial Intelligence/Machine Learning Consortium to Advance Health Equity and Researcher Diversity (AIM-AHEAD), that require research teams to be led by a principal investigator from institutions that aren’t as well-funded, or that historically serve minority communities.

“The funders, the universities, the journals, and the media too, to a certain extent, are all accountable for the problems that we’ve seen, and each one of them should innovate around operationalizing their mission and their vision,” Celi says.