Interview with Dr Sylvia Ortega

We have interviewed last week Dr. Sylvia Ortega Martinez, a passionate Spanish neuroscientist.

She got her international PhD in Neuroscience at Universidad Autónoma de Madrid (July 2013). Her research interest is the role of the new neuron formation in a specific brain area (known as Adult Hippocampal Neurogenesis (AHN)), as a key target in neurodegenerative and neuropsychiatric diseases. Indeed, her research experience in six different countries (Spain, USA, Germany, UK, France and Finland) and top institutions (Instituto Cajal, Washington University, Leibniz Institute for Age Research, Oxford University, Universite of Bourgogne, Turku Centre of Biotechnology and The University of Chicago) were focused on this brain process. From July 2017, she is working at The University of Chicago as a postdoctoral researcher in the laboratory of Dr. Sisodia. Her current project aims to elucidate the role of microglia in Alzheimer´s disease through its influence in AHN.

In addition, Dr. Ortega-Martinez is an enthusiastic of science divulgation. Indeed, she has participated in Clubes de Ciencia, Soapbox Science, and has given multiple interviews in newspapers or radio. She is grateful for the opportunity ARJ is given her to talk about science.

ARJ: Could you please do a brief introduction of your research?

Currently my project pursue to elucidate the role of microglia in Familiar early onset Alzheimer´s disease. Specifically, I am interested to understand how microglia affects the new neuron formation in the brain, or neurogenesis, in Alzheimer´s conditions. We already known that mice with specific mutations of Alzheimer´s disease shown lower neurogenesis compared with control mice, after environmental enrichment conditions. We are trying to understand now if microglia has an underlying role in this final output. This research could have a potential future impact in Alzheimer´s disease understanding.

Reproducibility of Research and Preclinical Validation: Problems and Solutions

Many scientific reports and literature present a considerable lack of reproducibility which causes high rates of failure when validating highly promising preclinical observations in clinical research. In these types of studies and especially in cancer research, there is a list of technical reasons that prevent from reproducing the experiments. The combination of all these reasons result in the publication of underdeveloped results, so it’s essential to identify all of them in order to make alternative plans and improve reproducibility.

Advances in biology and medicine have made possible the increase of life expectancy as well as an improvement in the quality of life. In this way, drug industries have been developing without stopping. Recently, the cost of drugs has increased considerably due to failed experiments researching new drugs. There are estimations that point out that 80% of phase II and 50% of phase III studies are failed in most cases because of a lack of efficacy. What is worse is that many published results limit reproducibility. Some reports reflect this problem. There is a research based on the published papers of a year in one high-impact journal which states that 50% of the results of some published clinical studies couldn’t be reproduced and only 33% of them could be reproduced to some extent. These facts, at the same time, weaken public trust in science and philanthropic support for research and without a doubt, they are the reason why the scientific progress is slow and there are some useless and failed experiments. This entire situation is caused in general by poor documentation trials, a selective publication of positive results, misinterpretation of technical noise and fabrication data. Organizations such as the NIH in the USA are promoting a responsible and reproducible research campaign as well as a list of recommendations. In addition, Nature published a new editorial policy in May 2013 in order to improve the quality of the experiments.

The current scientific field situation rewards positive results in regards to grants and funding, creating atension among true experimentation, integrity and the career interest of a study to transform the negative results into positive results. There is a study that found that 33% of researchers answering a survey admitted data manipulation and 70% of them reported questionable data manipulation by others. 

Recently, there is an increase of hypothesis-driven studies, which is one of the causes of cognitive biases. When it’s impossible to prove the scientist inquiries it’s necessary to reformulate the hypothesis. However, the scientists are focused on proving one’s own hypothesis and this is called by psychologists “confirmation bias.” Consequently, there is a tendency to think the experiment was wrong; instead of thinking the hypothesis was wrong.

Distributing information on good experimental design, appropriate statistical techniques and documentation could improve reproducibility of scientific experiments results. Nevertheless, these factors are not sufficient. In the following box there are some other recommendations looking for the same purpose.

Offering funding and grants to replicate published experiments and validate the results could be a good step towards the solution of this current problem. Investing in reproducibility research is the key to erase poor practices. A three-pronged grant system has been proposed to encourage laboratories to pay attention to document methodology and ensure reproducibility from the initial publication. In the following box there is a summary of this proposal.

According to these methods, changing the objectives from the hypothesis to the research question in the grant preparation process could be a very good way to reduce confirmation bias. All experiments start with an undefined question. However, many grant applications fail to state that question explicitly or as a brief part of the introductions whereas the hypothesis occupies an independent section of the application. This is the reason of this problem and they have to reestablish the priorities and strategy. 

Apart from that, negative results are as important as the positive ones despite a few publications that include fail experiments. Nowadays, there are more and more open-access journals like PLoS ONE and Scientific Reports that provide high visibility of result-neutral publications.

The transparency of the clinical trial process has increased substantially in the past 10 years since the International Committee of Medical Journal Editors (ICMJE) introduced a policy in 2005 that they would consider trials for publication only if the trial had been registered in the NIH clinical trial registry. This has made access to information about therapeutic clinical trials easier by making the synopses of trials available to the public.

Unfortunately, similar data repositories do not exist for basic and preclinical research data, possibly due to the difficulty to normalize these sorts of experiments.

Written by Dr. David Alcantara and Paula Ruíz for The All Results Journals. 

Is the “Bilingual Advantage” Skewed by Publication Bias?

In the past, research about bilingual speakers as an advantage for cognitive functions and executive control would be subjected to Second Opinion columns (such as this one).

Recently, a group of scientists from the University of Edinburgh and the University of Sassari- who themselves have published research that supports the idea that bilinguals have a cognitive advantage — have revealed evidence that bibliography on this topic may be skewed due to a bias toward publishing positive results.

Nevertheless, this study does not consider that the idea of a bilingual cognitive advantage is false but it teaches to always be skeptical about medical research.

The “File Drawer Effect” 

The authors of the Psychological Science paper say that at first, they thought that considering a bilingual cognitive advantage was a correct reflection of all the research they had submitted for publication enclosing just the positive results of the experiments.

This inclination of scientists to not publish their negative-result studies for publication is a well-known phenomenon known as the “file drawer effect.”

The researchers decided find evidence of the effect in the research literature. Firstly, they identified all the unpublished works about bilingualism and executive control presented in 169 important conferences between 1999 and 2012. They found 104. Of those, 38% had results that supported a bilingual advantage, 13% had mixed results that appeared to support a bilingual advantage, 32% had showed no executive control differences between monolinguals and bilinguals. Finally, 16% found no advantage at all. Fifty-two of those studies were finally included in 50 articles, which represents 68% of positive studies about bilingualism, 50% of the articles with mixed opinions but a positive tendency about bilingual advantage, 39% of the studies with mixed results that partly challenged the bilingual-advantage and 29% of research that found no differences between monolinguals and bilinguals or that reported a bilingual disadvantage.

According to detailed analyses, the size of the study or cognitive tasks are not the reasons why the studies are not published, a fact that the authors of the Psychological Science paper were unable to explain. They think researchers avoid publishing studies with mixed or negative results (the “file drawer effect”) or peer reviewers and editors may be rejecting studies that report those results more often than ones reporting positive results.

Staying Open to the Evidence 

Although all the scientific studies about bilingualism and executive control are not published, a cognitive advantage couldn’t be discarded. “We agree that bilingualism should be conceived, a priori, as a positive and desirable achievement,” they write, “[but] we are also convinced that educational and political debates addressing the relevance of bilingualism should not be promoted by ignoring null or negative results.”

Their findings also emphasize the ongoing need for the scientific community to “be more open to studies that challenge the existing theories, especially when these are not yet fully established.

All data, not just selected data that supports a particular theory, should be shared,” the researchers conclude, “and this is especially true when it comes to data regarding issues that have enormous societal relevance and implications, such as bilingualism.”

The abstract of the study is available on the Psychological Science website, but the study itself is behind a paywall.

Written by Dr. David Alcantara and Paula Ruíz for The All Results Journals.

One step more to combat publication bias

The European Federation of Pharmaceutical Industries and Associations (EFPIA) and The Pharmaceutical Research and Manufacturers of America (PhRMA) have recently bolstered their pledge to improve public health by embracing “Principles for Responsible Clinical Trial Data Sharing: Our Commitment to Patients and Researchers,” whose objective is for additional information to be accessible to all people who may be concerned.

These groups are ultimately looking for the blessing of patients by maintaining patient privacy and the veracity of regulatory systems as well encouraging investigation in biomedical research.

As said before, these commitments (not quite right word here) will be focusing on enlarging the total amount of information available in accordance with these aspects:

• Patient-level and study-level clinical trial data, as well as full clinical study reports and protocols from those clinical trials. (Medicines accepted in the United States and European Union will have to be shared with qualified scientific and medical researchers, protecting patient privacy and confidential commercial information.) In addition, scientists with such clinical trial data will be encouraged to publish their results. 
• Companies will publish an accurate summary of clinical trial results for patients who participate in them with the help of regulators. 
• Clinical Trials abstracts proposed to the Food and Drug Administration [FDA], European Medicines Agency [EMA], or national authorities of EU member states will be published once a new medicine or an indication has been approved. 
• Biopharmaceutical companies will also state their purpose and clinical trial results, even if the conclusion is not successful, for public disposal. 

These commitments show that clinical trial data results are essential for patients, healthcare and the economy. The Principles are available at http://transparency.efpia.eu/responsible-data-sharing and http://onphr.ma/18yru3e. These principles were instituted on January 1, 2014.

Written by Dr. David Alcantara and Paula Ruíz for The All Results Journals.

Why null results rarely see the light of day

The fact that experiments don’t always come to an end due to journals’ requirements is actually an important object of study. This problem, called ‘the file drawer effect’, is being carefully analyzed in order to provide a real insight into the issue.

A recent study started by a group of scientists from the Stanford University reveals that the majority of researchers reject the negative results, focusing on the positive ones, as the below graphic shows.

As we can see, more than 60% of the studies that gave null results were stored, while nearly all the studies providing strong results (96% approximately) were written up, which is the reason why literature is altered and we can find duplicated research.

To solve this problem, authors proposed gathering all data and studying designs in a public registry. This is a solution that has divided opinions. Most scientists agree and think that it could be useful, but there are others that are against this idea because it may introduces new biases.

One of the critical points in solving the problem of unacceptable results is the debate about the experiences related to medical studies because of their possible influence on people.

In this study, Neil Malhotra, a Stanford political economist, and two of his students have been investigating each subsidized research project since 2002, focusing on unpublished experiments and the granted ones. They have found that null results garner no interest from journals at all. The All Results Journals are the exception.

These detections suggested to Malhotra that it is all a matter of changing our thoughts on unsuccessful experiences. The statistics they considered show that only 42% of studies produce significant results and 62% of those are actually published. On the other side, 21% of studies are useless and about 65% of them are filed away.

Some scientists consider that publishing the results of these null experiments could avoid the waste of time and money and ultimately it could be a good way to prove that an experiment is as good as it seems, as well as discouraging researchers from altering their results in order to make them more publishable. They totally agree with the idea of a registry and they also approve of enclosing a preanalysis of the study to make the objective of their studies clear. There are also cautious opinions about the idea of publishing a plan, since it could show a lack of confidence in the scientist’s work. Others complain about the imposition of a registry, while others think it would be good to promote each study and lead the scientific community to adopt worthy rules.

Written by Dr. David Alcantara and Paula Ruíz for The All Results Journals.

A history of failure, serendipity and how the lack of funding can end it all

A recent article reflects how a failed experiment conducted in 1979 created a breakthrough in infant care research.

Sam Stein describes how a group of scientists at a research lab of pharmacology headed by Dr. Saul Schanberg were running tests on newborn rats to measure growth-related markers, when a failed experiment led them to rethink how to conduct the study.

Unknowingly, the results obtained by Schanberg and his team constituted the first step in a process that would see the upending of conventional wisdom when it came to post-natal care. A study done on rats became a study on humans, allowing premature babies to eat better and gain weight, along with achieving greater bone density through massage therapies. May be negative results are more important than we think.

Although, it also must have had something to do with serendipity, because Schanberg worked with Tiffany Field, a psychologist at the University of Miami School of Medicine, who had also been doing research on massaging prematurely born babies.

They put special emphasis on the need for immediate results; private enterprise would never have let them get the results they got. This could be another incentive for publication bias.

As Stain said, “The theories that his team stumbled upon by failure would save an estimated billion of dollars in medical costs and affect countless young parents’ lives.”

World powers like the United States have slipped to tenth place among economically advanced nations in global investment in research and development. About this it has been published in the prestigious journal Nature an article in which researchers affirm that "Policy makers of increasingly number of European countries, as well as the leaders of the European Union, have completely lost touch with the reality of world of scientific research.”

This is reflected in examples such as the situation of Italian universities, in which funding has been cut by 20% since 2009, resulting in a 100% reduction in funding for basic research projects; or in France, where 25% fewer new positions have been created at research centers. Even Germany is encouraging temporary recruitment of researchers.

“Spain has reduced its funding in civil R & D in a 40%, which means 40% less funding for research projects,” explains Moro-Martin, a researcher at the Space Telescope Science Institute in Baltimore (USA).

Schanberg died in 2009 but Field continued studying natal care. Her work has been widely cited in medical journals and newspaper articles, but the funding streams have run dry.

Three and a half decades later, they will be rewarded for their work. Their method is estimated to save $10,000 per infant — roughly $4.7 billion a year, and now she's faced with the prospect of dramatically narrowing the scope of her lifelong work, something that is at least a small amount surprising.

Original source

Written by Dra. Belén Suárez Jiménez for The All Results Journals

P-curve: A Way of Fighting Publication Bias?

I think we all agree that scientific journals usually do not publish results unless they are statistically significant. That is enough important to consider publication bias in order to estimate the true magnitude of a particular effect.

A variety of techniques have been developed for this goal. Some authors review publication bias correction tools that assume selective reporting based on p-values (probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true).

In general, considering the following thresholds:

- p<0.01 : very strong presumption against neutral hypothesis

- 0.01<p<0.05 : strong presumption against neutral hypothesis

- 0.05<p<0.1 : low presumption against neutral hypothesis

- p>0.1 : no presumption against the neutral hypothesis

Usually scientists publish significant effects (p<0.05) and spurn (on file-drawer) the rest.

A new paper has been recently published based on a p-curve where the authors suppose that if the true effect of something is X, and you do a bunch of studies, then statistical chance means that you'll get a range of results arrayed along a curve and centering on X.

According to the published study, when a studied effect is nonexistent, the p-curve is uniform, by definition. Moreover, for any given sample size, the bigger the effect, the more right-skewed the expected p-curve becomes. They say p-curve is more precise when it is based on studies with more observations and when it is based on more studies. Less obvious, perhaps, is that larger true effects also lead to more precision. This occurs because the p-curve’s expected shape becomes very right-skewed very fast as effect size increases, reducing the variance in skew of the observed p-curves. The skewness of p-curve and the statistical power of a test are very closely related. Both are a function only of the effect size and sample size.

The p-curves have a known shape so just looking at the small section of it allows you to estimate the size of the full curve. And this in turn allows you to estimate the true effect size just as if you had read all the studies, not just the ones that were published.

Although, to date, no technique exists that can eliminate this bias, we must point out that they have shown that the distribution of significant p-values can be analyzed to eliminate the impact of selective reporting of studies on size estimation.

Original source

Written by Dra. Belén Suárez Jiménez for The All Results Journals 

United for the negative results!

Considering that preclinical research provides the foundation on which many clinical trials are conceived, publication bias becomes a huge problem for the advancement of clinical research, leading to the revision of policies and guidelines on many occasions.

Recently a study conducted by Foster and Putos has been published on how the under-reporting of negative results in preclinical research distorts scientific knowledge and subsequently misguides clinical research.

Although publication bias is not considered scientific misconduct, it may be an even greater threat to science because it is more difficult to detect.

Preclinical research has made a special effort to improve the transparency of published studies, as the knowledge acquired in these disciplines are the basis on which clinical research priorities are set and evidence-based decisions are made. The non-publication of negative results makes up biomedical studies and decreases the validity of publications. That could be a factor responsible for the historically low rate of successful clinical translation from preclinical findings. An example of this is the nitrone-based drug NXY-059 to phase III clinical trials for the treatment of acute stroke, which benefits identified in preclinical studies failed in the clinical intervention.(1)

Non-publication of negative results in preclinical studies, besides involving a huge waste of money in clinical studies based on incomplete studies that do not lead to satisfactory results, also could involve harmful exposure of people who participate in these studies and therefore an increasing distrust in these studies. Not to mention animal studies, where it was recently estimated that 50% of research is never published and that this number may be far greater in for-profit organizations.

Foster and Putos reported some recommendations for achieving more transparent, efficient, and accurate reporting of preclinical research with a focus on strategies for increasing the publication of negative results. According to them, “educating all personnel involved in the publication process on the importance of communicating negative results will be instrumental for the publication of such findings.”

They propose initiatives like the ARRIVE (Animal Research: Reporting In Vivo Experiments) guidelines, CAMARADES (Collaborative Ap¬proach to Meta-Analysis and Review of Animal Data from Experi¬mental Studies), or the GSPC (Gold Standard Publication Checklist) to increase transparency of all preclinical studies submitted for peer-review.

In the commentary, they encourage both academic and non-academic institutions to organize conferences, seminars, and courses that teach researchers how to fully and accurately report their findings as another initiative to sensitize researchers and students, the heart of primary data generation, about publishing negative results.

Initiatives such as the first course in Journalology taught by the University of Ottawa, or the appearance of journals created for the sole purpose of publishing negative results like The Journal of Negative Results, The Journal of Negative Results in Biomedicine, and The All Results Journals go in the way of scientific progress.

(1) Bath PM, Gray LJ, Bath AJ, Buchan A, Miyata T, Green AR, et al. Effects of NXY-059 in experimental stroke: an individual animal meta-analysis. Br J Pharmacol 2009; 157(7):1157-1171.

Original source

Written by Dra. Belén Suárez Jiménez for The All Results Journals 


 

On the road of information disclosure

As we discussed in previous entries of our Blog, clinical trial results are being routinely withheld from doctors, researchers and patients, which undermines the ability to make informed decisions about treatments, even if regulators and the industry have made proposals to open up access.

It is quite worrying that only 50% of completed trial results are published as is concluded by an influential parliamentary committee after making a public report which examined the Department of Health's decision to spend £ 424m on stockpiling the flu drug Tamiflu, before writing off £ 74m Because of poor record keeping.

Research also suggests that trials which gave a favorable verdict are about twice as likely to be published as trials giving unfavorable results and “this is of extreme concern to this committee” as Richard Bacon, a senior member, said. Dr Fiona Godlee, editor-in-chief of the British Medical Journal, told the MPs that the journal had published very clear summaries of systematic reviews of data on individual medicines or classes of medicines where, "when you add together the published and unpublished evidence, you get a very different picture of the quality and effectiveness of those drugs".

For Richard Bacon “The department [of health] and Medicines and Healthcare products Regulatory Agency [MHRA] must make sure that clinical trials are registered and the full methods and results of all trials are available for independent wider scrutiny by doctors and researchers." Bacon continues on to say that if this is not done in a prospectively and retrospectively way, it will not cover the issue of access to the results of trials in the past which bear on the efficacy and safety of medicines in use today.

The non-profit Cochrane Collaboration conducted a review of 20 existing studies about Tamiflu, and found that it did not reduce influenza-related lower respiratory tract complications but did induce nausea. Study is currently being completed through full clinical study reports from manufacturer Roche and should be used by government, the MHRA and the National Institute for Health and Care Excellence to review the drug's use.

Bacon added: "There is still a lack of consensus over how well the antiviral medicine Tamiflu, stockpiled for use in an influenza pandemic, actually works. The case for stockpiling antiviral medicines at the current level is based on judgment rather than on evidence of their effectiveness during an influenza pandemic”. He thinks that “Before spending money in future to maintain the stockpile, the department needs to review what level of coverage is appropriate. It should look at the level of stockpiling in other countries, bearing in mind that the patent for the medicine runs out in 2016."

An MHRA spokesman said they would work with partners in the UK and in the EU to ensure greater transparency in the dissemination of clinical trials information.

Reference

Written by Dra. Belén Suárez Jiménez for The All Results Journals

Sources of negative results

If we look at the science blogs, scientific debates etc, it is increasingly common to find “the publication of negative results” as a central theme.

There are so many PhD students who venture into new areas of research and they should be aware of the stones they would find in the way. These obstacles slow not only their scientific progress but also the others researchers who decided to investigate the same scientific field.

The fact that these results are not considered good enough to be published in scientific journals of high impact, makes more than 60% of the results are not known. While it is relatively easy to publish positive findings (A correlates with B), it is almost impossible to make public the opposite – C doesn’t correlate with D – on the same topic, despite carefully designed and meticulously executed experiments and their brilliant analysis.

Many PhD students complete their doctoral thesis without brilliant results, and therefor without publication. Does this mean that the work they have developed is not interesting or couldn’t be useful to the scientific community? I definitely don’t think so.

And this is not the worst consequence since due to these results remaining unpublished, other students will finish their thesis with the same results and the same lack of publications, without mentioning the huge amount of money wasted in duplicated research that could have been employed in other topics.

Fortunately, as I said before, today there are a greater number of places for this phenomenon to be discussed and more and more scientific journals where negative results are published. Even now there are databases where all kinds of results can be deposited.

As you all know, since 2008 The All Results Journals has focused on recovering and publishing negative results. They firmly believe that these experiments should be taken into account as a vital key for the development of science. These negative results are the catalyst for real science-based empirical knowledge.

In addition to The All Results Journals, other journals that are committed to the publication of such results are:

f1000research.com: A part of digital biomedical publisher Faculty of 1000, F1000 research is an open access, peer reviewed journal. You can update newer versions of your paper as frequently as you like. And your first submission with negative findings 2013 can be published for free until September 30th.

PeerJ: an open access peer-reviewed journal. It charges a one off fee for an author, currently $99 for publishing of one paper. The papers are indexed in PubMed and other major databases.

Figshare: a free database, which allows uploading and sharing of data including papers. Each object has a unique Digital Object Identifier (DOI), which enables searching for and sharing the document.

ArXiv: a well-established database of preprints, mostly for math, physics and computational sciences, but they also include papers on quantitative biology. The papers are not peer-reviewed but they are curated to weed out pseudoscience and the papers can be found via Google Scholar.

The view that negative results are not worthless and make an important contribution to scientific knowledge is gaining momentum amongst researchers.

Do you know of other models or methods of publishing negative results?

Let’s publish negative results!

Written by Dra. Belén Suárez Jiménez for The All Results Journals.

Non-publication of Clinical Trials violates an ethical obligation towards study participants

Randomized clinical trials are a critical means of advancing medical knowledge. Clinical trials
depend on the willingness of participants to expose themselves to the risks of randomization, blinding, and unproven interventions, everything justified only by the knowledge of which society will benefit.

Until 1997 with the signing of the Food and Drug Administration Modernization Act in the United States, the protection of the interests of study participants after trial completion has received significantly less emphasis.

The registration of clinical trials serves an important role in protecting the interests of study participants after trial completion. Publication bias can distort the apparent efficacy of interventions.

A recent study has estimated the frequency of non-publication of large randomized clinical trials and, for unpublished trials, determined the frequency with which trial results are unavailable in ClinicalTrials.gov.

29% of large clinical trials remain unpublished five years after completion and, of those, 78% have no results publicly available. This means that an estimated 250,000 people (of course all volunteered and many even paid) have been exposed to the risks of trial participation without the societal benefits that accompany the dissemination of their results, worry the authors.

For the authors in the British Medical Journal, “willingness of people to expose themselves to risks” is due to society’s benefits and not because they are paid or may get a benefit from an experiment treatment. But when trial data remain unpublished, the societal benefit that may have motivated someone to enroll in a study remains unrealized.

Although US law requires that trials involving human participants now be registered on the largest clinical trial website, ClinicalTrials.gov. they believe that this legislation has been largely ignored. After searching in PubMed, Google Scholar, and Embase, 585 trials were identified with at least 500 participants registered with ClinicalTrials.gov and completed prior to January 2009 - before the mandatory requirement that came into effect September of 2009. The average time between study completion and the final literature search (November 2012) was 60 months for unpublished trials. Registry entries for unpublished trials were then reviewed to determine whether results for these studies were available in the ClinicalTrials.gov results database.

Of 585 registered trials, 171 (29%) remained unpublished. Of these, 133 (78%) had no results available in ClinicalTrials.gov. Non-publication was more common among trials that received industry funding (32%) than those that did not (18%).

They concluded that the lack of availability of results from these trials contributes to publication bias and also constitutes a failure to honor the ethical contract that is the basis for exposing study participants to the risks inherent in trial participation. Additional safeguards are needed to ensure timely public dissemination of trial data.

Original source

Written by Dr. Belén Suárez for The All Results Journals

Publication bias: what, why, how…?

For a long time, SACSIS has been talking about publication bias, but maybe there are people today who do not really know what it means. Recently it an interesting article has been published that may answer some questions on this topic.

Publication bias occurs when the publication of studies depends on the nature and direction of the results, so that published studies may be systematically different from those of unpublished studies.

Published literature is the main source of evidence for making clinical and health-policy decisions. The number of published studies has increased dramatically over time but it has been reported that about 50% of completed studies may still remain unpublished.

Since the first identified article with the term “publication bias” in 1979, the number of references that are potentially relevant to publication bias has considerably increased (Figure 1) and this increase may reflect the increased awareness of publication and related biases.

Bias may be introduced intentionally or unintentionally, consciously or unconsciously, into the process of research dissemination. The dissemination profile of research may be influenced by investigators, study sponsors, peer reviewers, and journal editors. For Song et al. according to surveys of investigators, the main reasons for nonpublication of completed studies included lack of time or low priority (34.5%), unimportant results (19.6%), and journal rejection (10.2%). Therefore, the nonpublication of studies was usually due to investigators’ failure to write up and submit to journals when the results were considered to be negative or nonsignificant.

Publication bias will result in misleading estimates of treatment effects and associations between study variables. Here the consequences of publication bias are considered separately for basic biomedical research, observational studies, and clinical trials.

Results of basic medical research are often used to support subsequent clinical trials. If the results of basic research are falsely positive due to biased selection for publication, subsequent clinical trials may waste limited resources and fail to confirm the published results of basic studies.

Also, in clinical trials publication bias has a direct impact on patients’ and populations’ health due to when the relative efficacy of a treatment is overestimated because of publication bias, health resources can be wasted by purchasing more expensive interventions, instead of cheaper alternatives, without a corresponding improvement in outcome.

Methods of avoiding publication bias, by identifying and including unpublished outcomes and unpublished studies, are discussed and evaluated. These include searching without limiting by outcome, searching prospective trials registers, searching informal sources, including meeting abstracts and PhD theses, searching regulatory body websites, contacting authors of included studies, and contacting pharmaceutical or medical device companies for further studies.

The choice of strategy to reduce the risk of publication bias depends on whether the aim is to tackle entire sets of missing studies, or whether selective/incomplete reporting of data by authors is considered to be the primary problem.

Trial registration, a process by which details about the design and conduct of a clinical trial are published, is considered to have both scientific and ethical implications, particularly in light of item 19 in the Declaration of Helsinki, which states “Every clinical trial must be registered in a publicly accessible database before recruitment of the first subject”.

Another way of reducing publication bias could be if journal editors moved away from the policy of giving greater priority to articles that were subjectively perceived as having greater novelty or importance, or significant findings. In this sense, some open access journals as The All Results Journals encourage authors to submit reports of negative or unexciting results.

That is the way!


Written by Dr. Belén Suarez  for The All Results Journals

 

Significance Bias in Animal Studies

John Ioannidis of Stanford and colleagues have published the paper “Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases” where they talk about results of animal biomedical experiments. These experiments generate valuable hypotheses that lead to the conduct of preventive or therapeutic clinical trials.

Studies have shown that the results of animal biomedical experiments fail to translate into human clinical trials; this could be attributed either to real differences in the underlying biology between humans and animals, to shortcomings in the experimental design, or to bias in the reporting of results from the animal studies.

They used a statistical technique to evaluate whether the number of published animal studies with “positive” (statistically significant) results is too large to be true. They assessed 4.445 animal studies for 160 candidate treatments of neurological disorders, and observed that 1.719 of them have a “positive” result, whereas only 919 studies would a priori be expected to have such a result. According to their methodology, only eight of the 160 evaluated treatments should have been subsequently tested in humans.

Ignoring the hypothesis that the "positive" results may be totally faked , this could be due to there are many analyses that can be performed, but only the analysis with the “best” results is presented resulting in potentially misleading findings or due to bias against publication of “negative” results (publication bias).

Ioanndis et al. showed (ref) a new concern about journals’ positive publication bias (non-publication of neutral/negative results) and points to problems elsewhere in the system – involving research paper authors, their institutions, funders and the journals.

They are convinced that publication and selective reporting biases may be diminished by pre-registering experimental animal studies. In this regard, access to the study protocol and also to raw data and analyses would allow verification of their results, and make their integration with other parallel or future efforts easier.

And why not?...it could be a beginning…

Written by Dr. Belén Suarez  for The All Results Journals

Publication bias promotes the invisibility of negative results from LAR

A recent survey by Dutch researchers indicates that Laboratory Animal Researchers of for-profit organizations estimated that only 10% of animal experiments are published. Respondents added that publication of processes and results or causes where no results were obtained, may increase scientific progress.

Many issues have been written and much has been said about publication bias. Now, Dutch Scientists wanted to prove it with a new study. In this case, they were convinced publication bias might also affect Laboratory Animal Research (LAR) but evidence is scarce so they made a survey to assess the opinion of laboratory animal researchers on the magnitude, drivers, consequences and potential solutions for publication bias.

Between 2000 and 3500 researchers from all animal laboratories in the Netherlands received the survey although only 474 responded. From that, 20 were excluded because of absence of data, leaving only 454 participants for the analysis.

Results show that the scientists in a for-profit environment themselves estimated that only 10 percent (5 to 50) of experiments are published. However these results change when looking at scientists in a not-for-profit institute. They estimated that about half (35 to 70) of all conducted laboratory animal experiments are published. Moreover, researchers in not-for-profit institutes reported that 80 percent (60 to 90) of their own work had been published against 10 percent (5 to 39) of the work of researchers in a for-profit environment.

Statistical non-significance and technical problems are considered to be the main drivers for non-publication. Supervisors, editors, and reviewers were all considered responsible for non-publication. Overall, respondents considered publication bias an important problem for LAR and for research duplication, literature syntheses and well-timed initiation of Phase-I clinical trials -- in humans in particular. One of the main conclusions extracted from the study shows that respondents thought mandatory publication of study protocols or results may help avoid unnecessary duplication, increase validity of literature syntheses and scientific progress, but at the cost of increased bureaucracy.

Despite these results, the experiment has several limitations. Among other reasons, authors do not know to which extent the results are representative for the Dutch LAR community, the survey was restricted to one country, only few researchers in for-profit organizations participated and study investigated researchers’ opinions, which in turn may not reflect the true rates of non-publication.

According to studies by Dr. Sena and colleagues, evidence from clinical research on humans suggests that between 46 and 67 percent of studies are not published and that in those published, positive findings are over-emphasized. Authors of this study admit Sena´s thesis: “non-publication is unethical since it deprives researchers of the accurate data they need to estimate the potential of novel therapies in clinical trials, but also because the included animals are wasted because they do not contribute to accumulating knowledge. In addition, research syntheses that overstate effects may lead to further unnecessary animal experiments testing poorly founded hypotheses”.

What can researchers do to increase the publication of negative results? Authors from this survey propose to prevent that study results have an effect on the editorial decision, to initially submit manuscripts without any results. Editors and peer reviewers would judge the importance of submissions through the background, hypotheses and methods sections. This would ensure that acceptance is not conditional on the results.

Until then, negative results appear increasingly in special journals, journal sections or repositories for negative results, such as The All Results Journals, the Journal of Negative Results in Biomedicine or Negative Results in Gynecological Oncology.

Written by Alejandro Balbuena for The All Results Journals. 

 

 

Is time to change the scientific review system?

 How do we perceive science? How do we get scientific results? Are reviews being made correctly? There are some complex issues that we try to analyze in this post.

First, we must emphasize that science is usually a news when it responds to the stimulus cause and effect. When any research gets positive or negative results, even if they are minimal, is when people know about it. But if the research had negative results that force to rethink some of the hypothesis is when the investigations disappear from the media landscape. Is the problem of publication bias which we have told many times at this blog.

This problem is affecting the publication of scientific research in generalist media. Although, there are some drawbacks when speaking of specialized media. Usually, scientific journals use a double check system or peer-review system. It is the most widespread standard to verify the veracity of the experiments. However, that does not mean it is the most accurate. Peers usually have the same level of knowledge of the author but they are not usually experts on the specific research to be judged to provide critical views about it. Furthermore, sometimes is difficult to get two independent reviewers to evaluate a manuscript without conflict of interest.

In this sense, it is interesting to remember the scandal of false claims by Dr Andrew Wakefield about the MMR vaccine to autism. A journalist, Brian Deer, was ordered to investigate on the research of Dr. Wakefield published in The Lancet in 1998. Deer himself said he had spoken with one of the reviewers of the original document and he had confessed that the reviewer had not read in detail the manuscript but he had approved it because he knew and respected author of the text, Professor John Walker-Smith. It was an event that was discovered in 2004 but The Lancet did not publish any rectification until 2010. Meanwhile, vaccination rates for measles, rubella or mumps were significantly reduced.

The essence of good peer review is generally practiced in scientific conferences. In them, each speaker face critical questions from the audience. It is the best way to review. Real time peer-review. However, these questions are generally not included in the abstracts. Instead, they publish a dossier with summaries from the research that is being presented. However, few people argue that, despite the effort to get the key concepts, summaries might be partial or even biased, specially when conferences are sponsored by a company, due to the close relationship between the company and the research presented.

There is another problem, which affect to people´s conception of scientific research and how people receive information through press release. After an investigation, the organization launches a press release to inform about the discovery. Generally scientific research can be reviewed but not press releases, giving, thus, partial messages to population while important concepts are silenced by non-specialize press which pay more attention to the statement than to verifiable scientific facts.

For every reason, we are witnessing a time when it is appropriate to ask whether the standard publishing model is valid and sufficient for the publication of scientific research. Now a new paradigm is opened with Internet and open access publications that allow further review of the publication with on-line commenting capabilities helping to increase the standards of peers evaluation.

Written by Alejandro Balbuena for The All Results Journals.

Publish or Perish ---The Current Trend in Scientific World

Words like “positive”, “significant”, “negative” or “null” are common in scientific jargon, but are obviously misleading, because all results are equivalent in science – as long as they have been produced by sound logic and established methods. Yet literature surveys have extensively documented an excess of positive results.

Papers with negative results are less likely to be published and are not encouraged within the scientific community. Various causes are known for this bias against negative results in science. Positive results make scientists happy and negative make them disappointed. Papers reporting positive results attract more interest from the community at large and this is a positive outcome for the researchers, particularly in their career. When confronted with negative results, scientists may be tempted not to publish these results. What happens to these missing negative results? They are presumed to be completely unpublishable or are somehow turned positive through selective reporting, post-hoc, re-interpretation and alteration of methods, analyses and data.

Negative results are virtually inevitable, unless the entire hypotheses tested were true, experiments were designed and conducted perfectly and the statistical powers available were always 100% (which is very rare – it is usually much lower). There is no doubt that negative results produced by a methodological flaw, which can be corrected or not be published at all. And it is likely that many scientists select or manipulate their negative (maybe the sample was too small or too heterogeneous, some measurements were inaccurate) and in most circumstances this might be nothing more than a “gut feeling”.

An unavoidable confounding factor that needs to be considered here is the quality and prestige of academic institutions, which is intrinsically linked to the productivity of their researchers. Indeed, official rankings of universities often include measuring publication rates. Separating this quality of institution effect from that of the bias induced by pressures to publish is difficult, because the two factors are strictly linked the best universities are also the most competitive and thus presumably the ones where pressure to produce are highest.

A journal’s name recognition determines (for some audiences) the quality of the science published within its pages. Some Hopkins research labs break out the champagne when a paper is accepted in Nature, Science or Cell. Others celebrate more soberly. Still, publishing a paper in a top-tier journal is always a high-five moment – for the authors and for the lab. This leads to a host of troubling questions. Has publishing in the so-called “top three” publications become not just optional but obligatory? Does career progress – here and elsewhere – increasingly depend not on what is published, but where? And what criteria are editors using to determine which articles are published in those journals anyway?

The current trend in the scientific field forces scientists to create publishable results. From an analysis it was concluded that researchers report the positive results for their experiments more frequently than negative results. Since papers are accepted by the review committees of various journals only if they report positive results that support an experimental hypothesis, producing a bias against negative results. Negative results are somehow unpublished or somehow turned into positive results by selective reporting and post hoc reinterpretation and alteration of methods, analyses and data.

Mixed opinions are seen on the public side about negative results, some say that these negative results are interesting only if they relate to some hypothesis that is widely believed or at least in general circulation. Negative results provide more information on which the future research can be based. For every study that didn`t work or didn`t produce positive results there may be a very well planned study. Negative results are not the ones where nothing is found, but where the evidence suggests that the hypothesis is wrong, this must have as much validity as the positive result. Hence making these results hard to find hinders the scientific process in the end and could lead into an inefficient research.

Competition is encouraged in scientifically advanced countries because it increases the efficiency and productivity of researchers. The flip side of the coin however is the conflict between their objectivity and integrity, because success of a scientific publication partly depends on its outcome. Survey suggests that a competitive research environment decreases the likelihood to follow scientific ideals and increase the likelihood to witness scientific misconduct. This increasing pressure will lead to scientific bias. However, no direct research has been done to connect the pressure to publish and the bias in the scientific literature, so the existence and gravity of the problem are still a matter of speculation and debate.

References: 
1. Publish-or-perish: Peer review and the corruption of science: David Colquhoun guardian.co.uk 
2. http://en.wikipedia.org/wiki/Publish_or_perish 
3. Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data: Daniele Fanelli : INNOGEN and Institute for the Study of Science, Technology and Innovation (ISSTI), The University of Edinburgh, Edinburgh, United Kingdom 


Written by Shalini P. Burra for The All Results Journals.

The case of the missing drug trial...

Publication bias. On one simple level it is about cherry-picking data, putting your best foot forward and presenting work in a positive light. On a much more real level it can result in a drug being licensed that doesn’t do exactly what it says on the box.

The story all began when lead author, Erick Turner, noticed that a database, initially set up by PhRMA (Pharmaceutical Research and Manufacturers of America) to “serve the valuable function of making clinical trial results for many marketed pharmaceuticals more transparent….” had mysteriously disappeared from the internet. This database was one Erick Turner and his collaborators had used and included in their recent PLoS Medicine paper. While the paper was in press they discovered it had disappeared. The PLoS paper describes the publication bias that exists in antipsychotic drug trials.

In the paper the researchers compare the data found in the literature against the data in the US Food and Drug Administration (FDA) database. Therefore being able to determine whether publication bias is present and go further to quantify it to assess the extent to which it inflates apparent drug efficacy. The benefits of a study like this are clear for all to see. If only positive results are published and seen in the literature, with negative results left in the bottom drawer, your drug -- one that acts only slightly better than a placebo will suddenly become the wonder drug. And it seems every “Big Pharma” company markets their latest drug as the new wonder drug.

But the question has to be asked; how much of the data have we actually seen? Such a reality can obviously affect clinical decision-making and patient outcomes. We are yet to see the dire consequences of these actions but the sleeping giant is there. The missing database, once found at http://www.clinicalstudyresults.org, was originally set up to prevent this sort of thing happening. It was supposed to act as a source to access the results of Phase III and IV studies, whether negative or positive. And in a bizarre twist of fate, the website exists no more. With no explanation as to why it was taken down or as to what becomes of the data that already existed on the site.

Erick Turner’s paper provides good reading, and helpful insight into the drug trial process. “In the US, all new drugs have to be approved by the FDA before they can be marketed. During this approval process, the FDA collects and keeps complete information about premarketing trials, including descriptions of their design and prespecified outcome measures and all the data collected during the trials. Thus, a comparison of the results included in the FDA reviews for a group of trials and the results that appear in the literature for the same trials can provide direct evidence about publication bias.”

It turns out that for this particular case, antipsychotics, the publication bias was not on a level previously seen as with antidepressants. Perhaps this could relate to the comparative market sizes for both types of drugs. The antidepressant drug market reached sales of almost $9.6 billion in 2008, and even doubling between 1996 and 2005. Antipsychotics, remaining the standard care for the treatment of mental disorders such as schizophrenia and bipolar disorder and generated $1.4 billion of revenue in 2010 in the top seven countries including the US.

The authors also note another possible reason; simply possibly because antipsychotics demonstrate superiority to placebo more consistently. Without increased access to regulatory agency data, publication bias will continue to blur distinctions between effective and ineffective drugs.

Written by Dr. Charles Ebikeme for The All Results Journals. 

Publication bias in antipsychotic trials

"Several negative studies of second-generation antipsychotic drugs were never published, leading to an exaggerated portrayal of the agents' effectiveness in the scientific literature", researchers said.

Of 24 registration trials involving eight products submitted to the FDA, four went unpublished in medical journals -- three of which found that the study drug's efficacy was equivalent to placebo or inferior to an active comparator, according to Erick H. Turner, MD, of Oregon Health and Science University in Portland, and colleagues.

Moreover, five of the 20 published trials "showed some evidence of outcome reporting bias," the researchers wrote online in PLoS Medicine.

Turner and colleagues also noted that the scale of the publication bias was relatively modest -- exaggerating the drugs' effectiveness relative to placebo or active comparators by a nonsignificant 8%. The weighted-average effect size in the published trials was 0.47 (95% CI 0.40 to 0.54), which declined only to 0.44 (95% CI 0.37 to 0.50) when the unpublished trials were included.

But overall, the researchers found the results concerning.

"Selective reporting of research results undermines the integrity of the evidence base, which ultimately deprives clinicians of accurate data for prescribing decisions," they wrote.

They also slammed the FDA for making it unnecessarily difficult for ordinary physicians to get the full picture from trial data submitted to the agency.

"At the present time, the FDA is not as transparent with its clinical trial data as it could be," Turner and colleagues wrote. "For example, we pointed out in 2004 that the reviews for several antipsychotic drugs were posted on the FDA website, but only for the original indication of schizophrenia and not for bipolar mania. More than seven years later, the mania reviews remain inaccessible."

For the current study, the researchers had to extensively maneuver the material posted on the FDA's website, which spread important data for a given drug across multiple, nonsearchable electronic documents.
Turner and colleagues examined the FDA archives for placebo-controlled premarketing trial data on aripiprazole (Abilify), iloperidone (Fanapt), olanzapine (Zyprexa), paliperidone (Invega), quetiapine (Seroquel), risperidone (Risperdal), risperidone long-acting injection (Risperdal Consta), and ziprasidone (Geodon).

The researchers then checked to see whether trials reported to the FDA were published in medical journals and, if so, how.

Trials were considered "published" if journal articles about them included enough data to be included in an ordinary meta-analysis, were in English, and "were reasonably discoverable by, and accessible to, the average clinician."

Of the 20 published trials, 15 had been rated by FDA reviewers as positive for the study drug whereas the remaining five were considered negative or questionable.

Only one of the four unpublished trials was deemed positive by FDA reviewers, the researchers found. The other three all found that the study drug was not significantly superior to placebo. Two of these were on aripiprazole and one tested ziprasidone.

The latter trial also included haloperidol as an active comparator, which proved to be statistically superior to ziprasidone.

However, the one unpublished study that favored the study drug over placebo also involved ziprasidone.

Turner and colleagues additionally found that, when trials were published, they were sometimes buffed up to show the drugs in a positive light, with unfavorable data ignored in summary descriptions or omitted altogether.

Four such reports dealt with iloperidone. "One efficacy issue that was apparent from the FDA review of iloperidone, but not from the corresponding journal articles, was that the drug frequently proved to be statistically inferior to active comparators," Turner and colleagues indicated.

They also noted that the iloperidone journal reports obscured trial findings that the drug was superior to placebo only in patients with schizoaffective disorder, not in those with schizophrenia only. Instead, the articles only reported results in the total patient population that included both diagnostic groups.

Also questioned by Turner and colleagues was the reporting of a quetiapine trial. An FDA reviewer had said it "provides marginal support for antipsychotic efficacy" and that "the data fall short of meeting the customary level of statistical proof."

In contrast, the corresponding journal article declared that the results showed "significant differences ... between treatment groups which favored [quetiapine] throughout the trial." Also, whereas the FDA review had noted no evidence of efficacy in patients who completed the scheduled treatment, this finding was not mentioned in the journal article.

One of the published ziprasidone studies also glossed over findings that did not support the drug's efficacy. For example, according to Turner and colleagues, an FDA review had noted that a 60-mg dose twice daily was superior to placebo only at certain time points and not on all outcome measures.

The journal report merely stated that the drug was superior to placebo at week four on two outcome measures, omitting mention of the lack of efficacy at other evaluations and on a third primary endpoint.

But the researchers also emphasized that the overall effect of such biases was modest, and less than other researchers had found for antipsychotic drugs (using a different set of drugs and trials). It was also less than was found in an earlier analysis of antidepressant trial reporting by Turner and a different group of colleagues.

Turner and colleagues acknowledged several limitations to the current study. The small number of trials available for analysis diminished the review's statistical power. They did not address safety endpoints, only efficacy.

Most importantly, the researchers noted that the FDA database on premarketing studies is not comprehensive. It may lack trials conducted outside the U.S. prior to starting the FDA approval process. Moreover, it does not include postmarketing analyses -- which "may represent the majority of trials conducted on a given drug," Turner and colleagues noted.

Secondary source

Why research coming from countries that do not have English as their mother tongue has less impact?

As a scientist born in a non English speaking country, I have asked myself this question in more than one opportunity. As I read not long ago, “many authors are convinced that the acceptability of their articles by indexed journals is based on parameters not related to quality or innovation.”¹

Is it true that our research may have less impact just because we are not English native speakers?

In order to try to answer this question and to open the discussion about this topic, I would like to share with you some of my personal thoughts and also the research recently published surrounding this area.

The potential existence of bias in scientific publishing is disclosed in the literature,^2,3 although on the majority of the occasions the decision not to accept a manuscript is attributed to external reviewers. There are several other factors related to publishing itself that are carefully discussed in an article that appeared in Neurología¹ this year, which I strongly recommend you to read.

Publication bias has been defined as “the tendency for certain kinds of studies, typically those showing a significant positive result in a clinical trial or an observational study, to receive more favorable publication decisions than equally well-conducted studies that report a negative or null result”.^4-6 In this sense, I believe that authors, editors and reviewers contribute to create this favoritism towards the publication of certain manuscripts and not others. Among the situations influencing the decision to accept or reject a manuscript we can find the authors’ nationalities, as well as the authors’ mother tongue, and the academic institution where the article comes from.¹

Regarding the language, I’m truly convinced that this is a major factor when the final decision of acceptance or rejection of a manuscript is taken. It is a fact that those journals that are not published in English tend to have a lower impact factor and less impact in the scientific community.^7-10

There is also evidence that bibliographic databases favor publications in English.¹ Articles in their own language are difficult to be accessed, therefore, less people read them and as a consequence, less people cite them.

You may also think that what I’m writing is a contradiction as I’m talking about editorial bias but I’m not writing this article in my mother tongue. However, I’m pretty sure more people are going to have access to this article if it’s written in English, which is basically what I need to spread this idea. I think the real question is whether or not we are willing to take actions when we think our manuscripts don’t receive a fair treatment during the peer-reviewing process and what we can do to change this.

The discussion is open and now it’s your turn to raise your voice and leave your comments here…

 References: 

1. Matías-Guiu, J.; García-Ramos, R.; Editorial bias in scientific Publications; Neurología, 2011, 26, 1 and references therein.

2. Begg C. B.; Berlin, J. A.; Publication bias; a problem in interpreting medical data; J. Roy. Stat. Soc. A., 1988, 151, 445.

3. Song, F.; Eastwood, A. J.; Gilbody, S.; Duley, L.; Sutton, A. J.; Publication and  related biases; Health Technol. Assess., 2000, 4, 1.

4. Sridharan, L.; Greenland, P.; Editorial Policies and Publication Bias. The Importance of  Negative Studies; Arch. Intern. Med, 2009, 169, 1022.

5. Olson, C. M.; Rennie, D. ; Cook, D. ; Dickersin, K. ; Flanagin, A. ; Hogan, J. W. ; Zhu, Q. ; Reiling, J. ; Pace, B.; Publication bias in editorial decision making, J. Am. Med. Assoc., 2002, 287, 2825.

6. Callaham, M. L.; Wears, R. L.; Weber, E. J.; Barton, C.; Young, G.; Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting; J. Am. Med. Assoc.,1998, 280, 254.

7. Jiménez-Contreras, E.; Delgado López-Cózar, E.; Ruiz-Pérez, R.; Fernández, V. M.; Impact factor rewards affect spanish research; Nature, 2002, 417, 898.

8. Winkmann, G.; Schlutius, S.; Schweim, H. G.; Citation rates of medical German-language journals in English-language papers –do they correlate with the impact factor, and who cites?; Klin. Monatsbl. Augenheilkd., 2002, 219, 72.

9. Winkmann, G.; Schlutius, S.; Schweim; H. G.; Publication languages of Impact Factor journals and of medical bibliographic databanks; Dtsch. Med. Wochenschr. 2002, 127, 131.

10. Gregoire, G.; Derderian, F.; Le Lorier, J.; Is there a Tower of Babel bias?; J. Clin. Epidemiol., 1995, 48, 159.

Written by Dr. Ana Bellomo for The All Results Journals.