Dealing with publication bias in translational stroke research

*Corresponding Author:

Dr. Shimin Liu, M.D., Ph.D.
Department of Neurology, Mount Sinai School of Medicine NYU. 1468 Madison Avenue, New York, NY 10029
E-mail: shimin.liu@mssm.edu

Abstract

Publication bias has been around for about 50 years. It has become a concern for almost 20 years in the medi-cal research community. This review briefly summarizes the current status of publication bias, potential sources where bias may arise from, and its common evaluation methods. In the field of translational stroke research, publication bias has long been suspected; however, it has not been addressed with sufficient efforts. Its status has remained the same during the last decade. The author emphasizes the important role that publishers might play in addressing publication bias.

Keywords

Publication bias, stroke, funnel plot, preclinical trials, neuroprotection, efficacy

Introduction

Publication bias has been noticed for about 50 years (Sterling 1959), and has become a concern for al-most 20 years (Chalmers et al 1990; Dickersin 1990; Sharp 1990). In most cases, it refers to the assertion that studies having positive and/or statistically signifi-cant results are more easily and frequently published. Other types of publication bias may include a prefe-rence to publish based on research directions, au-thors’ nationalities, and institutes’ professional ranks. Publication bias in the medical research reporting system may have distorted the medical research lite-rature and may have influenced the conclusion of some meta-analysis (systematic) reviews. A small scale random sample of 86 studies (Machan et al 2006) in medical informatics evaluation research showed a remarkably high percentage (69.8%) of descriptions of positive results, 19 (36.6%) of the analyzed 54 reviews and meta-analyses came to a positive conclusion with regard to the overall effect of the analyzed system, 32 (62.5%) were inconclusive, and only one review came to a negative conclusion.

Publication bias in translational stroke re-search

In the field of translational stroke research, the dis-crepancy of the neuroprotective efficacy between preclinical trails and clinical trials has caused growing concerns. An extensive review of 1026 experimental treatments (O'Collins et al 2006) revealed that neuro protective efficacy was superior to control conditions in 62% of the preclinical models of focal ischemia, in 70% of preclinical models of global ischemia, and in 74% of culture models. Such a rate of reporting posi-tive results in preclinical trials is drastically high when compared with the rate of reporting positive results in clinical trials. Currently we still have no FDA-approved neuroprotective treatment for ischemic stroke.

Factors contributing to the translational failures of neuroprotective treatments for ischemic stroke have been addressed in the Stroke Therapy Academic In-dustry Roundtable (STAIR) guidelines for preclinical stroke trials (Stroke Therapy Academic Industry Roundtable 1999), which include species differences, inappropriate time windows, ineffective drug levels, inability of drugs to cross the blood-brain-barrier (BBB), use of young animals without co-morbidities, failure to model white matter damage, and the hete-rogeneity of stroke subtypes in patients. However, another issue, the known publication bias, may also play a role in causing the discrepancy of positive re-sult reporting between preclinical trials and clinical trials in translational stroke research.

An article from Collaborative Approach to Meta Anal-ysis and Review of Animal Data from Experimental Stroke (CAMARADES) dealing with publication bias across stroke studies suggests that at least 15% of experiments remain unpublished, and that this results in an overstatement of efficacy of 30% (unpublished communication with Dr. Macleod).

Some reviews with detailed systematic analyses fur-ther confirmed the widely suspected publication bias in preclinical stroke trails (Perel et al 2007). As mani-fested on the funnel plot, which is the most commonly used graphical evaluation for publication bias, an in-flated efficacy was shown with the treatment of re-combinant tissue plasminogen activator, and similar effects were shown with tirilazad treatments (Perel et al 2007), but not with hypothermia treatment (van der Worp et al 2007).

Sources of bias

Although it is widely noticed that positive or signifi-cant results more frequently appear in journals, the fundamental reasons have not been well addressed. The bias that has been noticed in medical research literature might come from the publication process, the experimental process, or both. Currently there is no particular study specifically addressing the bias sources in translational stroke research. It is reason-able to assume that similar bias sources exist in translational stroke research as in other medical re-search fields.

Bias from the publishing process

In addition to the result being positive or negative (Blackwell et al 2009; Hopewell et al 2009), many other factors may contribute to publication bias, such as research directions of authors and review-ers(Joyce et al 1998), manuscript’s potential value for pursuing and maintaining a journal’s high impact fac-tor (Opthof et al 2002b), conflicts of interest (Perlis et al 2005), research funding sources (Liss 2006), pub-lishing cost, language (Egger et al 1997b), author’s nationality (Opthof et al 2002b; Yousefi-Nooraie et al 2006), the rank and geographic location of the spon-sor institute (Eloubeidi et al 2001; Sood et al 2007), and multiple biases from the review process (Alasbali et al 2009; Goldbeck-Wood 1999; Opthof et al 2001a; Opthof et al 2002a; Opthof et al 2002b). The follow-ing paragraphs mainly discuss the influence of spon-sorship and impact factor on publication bias.

It has been noticed that research funding sources influence the published outcomes of studies. Results favorable for the drugs studied were significantly more common in those funded by a pharmaceutical company (98% vs. 32%) (Liss 2006). Financial con-flicts of interest have been reported to be prevalent in clinical trials and are associated with a greater like-lihood of reporting results favorable to the interven-tion being studied (Perlis et al 2005). Research funded by drug companies was less likely to be pub-lished than research funded by other sources (Hall et al 2007; Lexchin et al 2003). Studies sponsored by pharmaceutical companies were more likely to have outcomes favoring the sponsor than were studies with other sponsors (Lexchin et al 2003). Therefore, additional procedures may need to be taken for avoiding bias as well as a declaration of conflict of interest.

While the impact of research funding sources on pub-lication bias has been noticed and addressed, the role of a journal’s operating goal and supporting sources has rarely been discussed. Although it is controversial for using the impact factor as a criterion for measuring a journal’s quality (Barendse 2007; Boldt et al 2000; Peleg and Shvartzman 2006; Rous-sakis et al 2007), most journals tend to treat the im-pact factor as a measure of their journal’s achieve-ments. Publisher bias may be encouraged when the impact factor prevails in a journal’s operational strat-egy (Opthof et al 2001b; Opthof et al 2002b). A study assessed the relationship of a journal’s impact factor and publishing outcomes in the literature of neonatol-ogy (Littner et al 2005). It showed that studies with positive results were more frequently published in journals with high impact factors, suggesting a role of the impact factor in selective publishing or submitting. Seeking a different operating strategy for professional journals may be needed for dealing with publication bias.

Bias from experimental process

Bias coming from the experimental process may mix with the publication bias; it may not be easy to tell whether the bias is from the publication process, the experimental process, or both. Experimental bias has been well-noticed and addressed in a few systematic reviews (Macleod et al 2008b; van der Worp et al 2007). Some editors may argue that researchers are more likely to produce and recommend manuscripts with positive or significant results. JAMA did a specif-ic investigation on publication bias in editorial deci-sion making (Olson et al 2002); after having adjusted simultaneously for study characteristics and quality indicators, the publication rates between studies with significant and non-significant results did not differ significantly, with an adjusted odds ratio of 1.30 (95% CI, 0.87-1.96). However, editorial policies and processes differ from journal to journal, and a study based on one journal’s articles is insufficient to reflect the overall status of journals’ roles in publication bias. JAMA are to be praised for their constructive efforts in addressing the issue of publication bias from the journal’s side.

Evaluation methods for publication bias

There is no specially designed or tailored evaluation method for the detection of publication bias in transla-tional stroke research. To estimate a suspected pub-lication bias in stroke studies, all regular methods that are widely accepted by the medical research com-munity will apply. Publication bias can be detected by several commonly used graphic, or statistical, me-thods, such as the funnel plot (Egger et al 1997a) or fail safe numbers (Persaud 1996; Rosenberg 2005). Other methods, such as selection models using weighted distribution theory (Sutton et al 2000), are also in development, but they have not been used regularly. Brief introductions for the common evalua-tion methods and sample funnel plots for the effica-cies of some neuroprotectants are provided in the following paragraphs.

The funnel plot, the plot of a trial’s effect estimates against sample size, has been widely used to deal with publication bias. It detects bias based on the assumption that the plot resembles a symmetrical inverted funnel in the absence of bias (Egger et al 1997a). Many factors may potentially contribute to the detected asymmetry; therefore, this method should be used with caution, especially when limited numbers of studies are used in a meta-analysis (Irwig et al 1998; Stuck et al 1998; Vandenbroucke 1998).

Detection of asymmetry in a funnel plot can be con-ducted by several methods, such as visual inspec-tion, “trim and fill”, regression approach (Soeken and Sripusanapan 2003), and a newly emerged me-thod (Formann 2008) in which the proportion of un-published studies is estimated by the degree of truncation from a left-truncated normal distribution. Each method has its own advantages and limitations; it is suggested that multiple methods should be used when there is a suspicion of publication bias.

For demonstration purposes, here I present a sample funnel plot basing on the available data from a sys-tematic review paper (Macleod et al 2005a) for the neuroprotective efficacy of FK506. As shown in Figure 1, the funnel plot of precision against effective size is asymmetric by visual inspection, with more studies allocated in the left side. The “missing” studies in the right portion (large efficacy) of the plot may indicate possible publication bias or true heterogeneity of stu-dies. A similar sample funnel plot (Figure 2) for the neu-roprotective efficacy of melatonin has also been completed based on the available data from another systematic review (Macleod et al 2005b). Being dif-ferent from Figure 1 by visual inspection, the asymmetry of Figure 2 reveals apparent “missing” studies in the left portion of small efficacy, indicating a possibly inflated efficacy. These asymmetries in funnel plots may indicate the existence of publication bias, missing litera-ture, or true heterogeneity among studies. Therefore, addressing suspected publication bias in systematic reviews may provide valuable information for the se-lection of neuroprotective candidates for acute stroke treatment.

Fail safe numbers are to some degree analogous to the confidence intervals. They aid in the assessment of the degree of confidence for a particular result in meta-analysis studies, which the funnel plots do vi-sually. Fail safe numbers may be defined as the number of new, unpublished, non-significant studies that would be required to exist to lower the signific-ance of a meta-analysis to some specified level (Per-saud 1996). Using fail-safe numbers is a quick way to estimate whether publication bias is likely to be a problem for a specific meta-analysis study (Rosen-berg 2005).

Some graphic evaluation methods frequently appear in systematic reviews, such as the forest plot, the Galbraith plot, the L’Abbe´ plot, and the box plot; but they are designed for assessing study variation and heterogeneity, not specifically for assessing publica-tion bias (Bax et al 2008).

Multiple initiatives suggested for reducing pub-lication bias

About 50 years have passed since the notice of pub-lication bias (Sterling 1959), yet little has changed. A significant amount of research conducted never ap-pears in the public forum. It has been estimated that less than half of all studies initially presented as summaries or abstracts at professional meetings are subsequently published as peer-reviewed fulltext ar-ticles (Scherer et al 1994; Scherer et al 2007), al-though various factors may account for such a low publication rate.

Publication bias influences the conclusion of syste-matic reviews; therefore it should be addressed ap-propriately when there is a suspicion. However, the methods used for dealing with publication bias in most systematic reviews appear merely as a descrip-tion of the number of pooled studies, the number of searched databases, or the statistical analysis of he-terogeneity. In attempts to address this situation, both the general medical research community and the translational stroke research community have started multiple initiatives.

Initiatives in the general medical research community

Some journals have implemented guidelines to make sure that the issue of publication bias be dealt with, such as the QUOROM statement, and MOOSE guidelines. The quality of reports of meta-analyses (QUOROM) statement was suggested for meta-analyses of randomized controlled trials (Moher et al 2000a; Moher et al 2000b). The meta-analysis of ob-servational studies in epidemiology (MOOSE) guide-lines, the MOOSE checklist, was suggested for meta-analyses of observational studies (Stroup et al 2000). Some journals implement a general requirement to address publication bias using an appropriate method when necessary. More detailed descriptions of deal-ing with publication bias have been provided in the Cochrane handbook for Systematic Reviews of Inter-ventions.

In an effort to balance publication bias in medical lite-rature, Journal of Negative Results in BioMedicine (http://www.jnrbm.com/) was started in 2004 to ac-cept manuscripts with unexpected, controversial, provocative and/or negative results/conclusions.

Although not being established particularly for ad-dressing publication bias, ClinicalTrials.gov (http://clinicaltrials.gov/) provides a global view of clinical trials. ClinicalTrials.gov is a registry of federal-ly and privately supported clinical trials conducted in the United States and around the world. Clinical-Trials.gov currently contains 66,791 trials sponsored by the National Institutes of Health, other federal agencies, and private industry. Studies listed in the database are conducted in all 50 States and in 161 countries. A similar registration agent for preclinical trials will be expected to provide a global view for preclinical trials, to counter against publication bias, and to help with the selection of candidates for clini-cal trials.

Initiatives in the translational stroke research community

As in the meta-analyses of cardiovascular diseases (Palma and Delgado-Rodriguez 2005), the assess-ment of publication bias in systematic reviews of preclinical and clinical trials for ischemic stroke treatment appears at a low frequency. Although me-thodology details and recommendations for assess-ing publication bias have been provided in the Coch-rane handbook for Systematic Reviews of Interven-tions (http://www.cochrane-handbook.org/), none of the Cochrane’s 35 systematic reviews (http://www.cochrane.org/reviews/en/subtopics/93.html) on medical therapies for ischemic stroke used the widely accepted methods, such as funnel plots, or failsafe-N. Only the heterogeneity of studies has been assessed in these reviews. It is a good sign to see that a few other reviews have addressed publica-tion bias in professional details(Perel et al 2007; van der Worp et al 2007).

Because the bias that appears in published materials may also come from the experimental process, ef-forts in reducing experimental bias may help to re-duce the observed bias in the literature. More details about reducing experimental bias have been de-scribed in Good Lab Practice Guidelines (GLPG) (Macleod et al 2008a). Measures such as random allocation, random sampling, blind assessment, and standardized operational procedures could be used for reducing experimental bias. This GLPG, together with the STAIR criteria, if being followed strictly, may help to reduce the inflated efficacy in preclinical trials for the treatment of acute ischemic stroke. However, almost ten years have passed since the first version of the STAIR criteria (Stroke Therapy Academic In-dustry Roundtable 1999), yet the efficacy discrepan-cy between preclinical and clinical stroke trials doesn’t seem to have improved. An independent reg-istration and validating system may help to reduce the bias from both publication process and experi-mental process. Seeking a different approach is necessary to reduce the impact of this problem (Hall et al 2007).

The CAMARADES collaboration has been active in addressing publication bias in experimental stroke. Some of their focuses include identifying potential sources of bias in animal work, developing recom-mendations for improvements in the design and re-porting of animal studies, and developing better me-ta-analysis methodologies for animal studies. The CAMARADES group will soon launch an on-line fa-cility for the registration of animal studies in stroke with enough details to help systematic reviewers con-tact authors of unpublished studies. This would be a good start for the preclinical stroke trial registration, and more work will still be needed to establish a world-wide registry. Research sponsors and govern-mental authorities may be suggested as needing to become involved in promoting mandatory registration.

How JESTM addresses publication bias?

The goal of the Journal of Experimental Stroke & Translational Medicine (JESTM, www.jestm.com) is to foster new concepts and to reflect the status of preclinical trials in the field of experimental and trans-lational stroke research. Therefore, manuscripts with controversial/provocative ideas and negative results are encouraged equally.

In order to increase the quality of our published con-tents and to reduce publication bias, we require au-thors of review manuscripts be aware of and to ad-dress publication bias appropriately. We require au-thors of research studies to conform to Good Lab Practice Guidelines for reducing experimental bias.

JESTM is operated with support from enthusiastic professional volunteer workers. Because it is purely an online journal, its operational cost is considerably lowered. All articles on JESTM are free-access and authors are not charged for publishing cost. Articles will be made immediately available online after com-pletion of the requisite review process.

Acknowledgement

This work was supported by NIH grant 5T32NS051147-02 and NS 21076-24. The author appreciates and acknowledges Dr. Levine and Dr. Winn at Mount Sinai School of Medicine for his con-tribution on revising this paper.

References

1. Alasbali T, Smith M, Geffen N, Trolie GE, Flanagan JG, Jin Y, Buys YM (2009) Discreliancy between results and abstract conclusions in industry- vs nonindustry-funded studies com-liaring toliical lirostaglandins. Am J Olihthalmol 147:33-8 e2.
2. Barendse W (2007) The strike rate index: a new index for journal quality based on journal size and the h-index of citations. Biomed Digit Libr 4:3.
3. Bax L, Ikeda N, Fukui N, Yaju Y, Tsuruta H, Moons KG (2008) More Than Numbers: The liower of Gralihs in Meta-Analysis. Am J Eliidemiol.
4. Blackwell SC, Thomlison L, Refuerzo J (2009) Full liublication of Clinical Trials liresented at a National Maternal-Fetal Medi-cine Meeting: Is There a liublication Bias? Am J lierinatol.
5. Boldt J, Haisch G, Maleck WH (2000) Changes in the imliact factor of anesthesia/critical care journals within the liast 10 years. Acta Anaesthesiol Scand 44:842-9.
6. Chalmers TC, Frank CS, Reitman D (1990) Minimizing the three stages of liublication bias. Jama 263:1392-5.
7. Dickersin K (1990) The existence of liublication bias and risk fac-tors for its occurrence. Jama 263:1385-9.
8. Egger M, Davey Smith G, Schneider M, Minder C (1997a) Bias in meta-analysis detected by a simlile, gralihical test. Bmj 315:629-34.
9. Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G (1997b) Language bias in randomised controlled tri-als liublished in English and German. Lancet 350:326-9.
10. Eloubeidi MA, Wade SB, lirovenzale D (2001) Factors associated with accelitance and full liublication of GI endoscoliic re-search originally liublished in abstract form. Gastrointest En-dosc 53:275-82.
11. Formann AK (2008) Estimating the liroliortion of studies missing for meta-analysis due to liublication bias. Contemli Clin Trials 29:732-9.
12. Goldbeck-Wood S (1999) Evidence on lieer review-scientific quali-ty control or smokescreen? Bmj 318:44-5.
13. Hall R, de Antueno C, Webber A (2007) liublication bias in the medical literature: a review by a Canadian Research Ethics Board. Can J Anaesth 54:380-8.
14. Holiewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K (2009) liublication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev:MR000006.
15. Irwig L, Macaskill li, Berry G, Glasziou li (1998) Bias in meta-analysis detected by a simlile, gralihical test. Gralihical test is itself biased. Bmj 316:470; author relily -1.
16. Joyce J, Rabe-Hesketh S, Wessely S (1998) Reviewing the re-views: the examlile of chronic fatigue syndrome. Jama 280:264-6.
17. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) liharmaceutical industry slionsorshili and research outcome and quality: sys-tematic review. Bmj 326:1167-70.
18. Liss H (2006) liublication bias in the liulmonary/allergy literature: effect of liharmaceutical comliany slionsorshili. Isr Med As-soc J 8:451-4.
19. Littner Y, Mimouni FB, Dollberg S, Mandel D (2005) Negative re-sults and imliact factor: a lesson from neonatology. Arch lie-diatr Adolesc Med 159:1036-7.
20. Machan C, Ammenwerth E, Bodner T (2006) liublication bias in medical informatics evaluation research: is it an issue or not? Stud Health Technol Inform 124:957-62.
21. Macleod MM, Fisher M, O'Collins V, Sena ES, Dirnagl U, Bath liM, Buchan A, van der Worli HB, Traystman R, Minematsu K, Donnan GA, Howells DW (2008a) Good Laboratory liractice. lireventing Introduction of Bias at the Bench. Stroke.
22. Macleod MR, O'Collins T, Horky LL, Howells DW, Donnan GA (2005a) Systematic review and metaanalysis of the efficacy of FK506 in exlierimental stroke. J Cereb Blood Flow Metab 25:713-21.
23. Macleod MR, O'Collins T, Horky LL, Howells DW, Donnan GA (2005b) Systematic review and meta-analysis of the efficacy of melatonin in exlierimental stroke. J liineal Res 38:35-41.
24. Macleod MR, van der Worli HB, Sena ES, Howells DW, Dirnagl U, Donnan GA (2008b) Evidence for the efficacy of NXY-059 in exlierimental focal cerebral ischaemia is confounded by study quality. Stroke 39:2824-9.
25. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Strouli DF (2000a) Imliroving the Quality of Reliorts of Meta-Analyses of Randomised Controlled Trials: The QUOROM Statement. Onkologie 23:597-602.
26. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Strouli DF (2000b) Imliroving the quality of reliorts of meta-analyses of randomised controlled trials: the QUOROM statement. QUOROM Grouli. Br J Surg 87:1448-54.
27. O'Collins VE, Macleod MR, Donnan GA, Horky LL, van der Worli BH, Howells DW (2006) 1,026 exlierimental treatments in acute stroke. Ann Neurol 59:467-77.
28. Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, liace B (2002) liublication bias in editorial decision making. Jama 287:2825-8.
29. Olithof T, Coronel R, Janse MJ (2001a) Submissions and imliact factor 1997-2001: focus on Sweden. Cardiovasc Res 51:202-4.
30. Olithof T, Coronel R, Janse MJ (2001b) Imliact factor of Cardi-ovascular Research in 2000: all time high! Cardiovasc Res 50:1-2.
31. Olithof T, Coronel R, Janse MJ (2002a) The significance of the lieer review lirocess against the background of bias: liriority ratings of reviewers and editors and the lirediction of citation, the role of geogralihical bias. Cardiovasc Res 56:339-46.
32. Olithof T, Coronel R, Janse MJ (2002b) Submissions, imliact fac-tor, reviewer's recommendations and geogralihical bias within the lieer review system (1997-2002): focus on Germany. Cardiovasc Res 55:215-9.
33. lialma S, Delgado-Rodriguez M (2005) Assessment of liublication bias in meta-analyses of cardiovascular diseases. J Eliide-miol Community Health 59:864-9.
34. lieleg R, Shvartzman li (2006) Where should family medicine lialiers be liublished - following the imliact factor? J Am Board Fam Med 19:633-6.
35. lierel li, Roberts I, Sena E, Wheble li, Briscoe C, Sandercock li, Macleod M, Mignini LE, Jayaram li, Khan KS (2007) Comliar-ison of treatment effects between animal exlieriments and clinical trials: systematic review. Bmj 334:197.
36. lierlis RH, lierlis CS, Wu Y, Hwang C, Joselih M, Nierenberg AA (2005) Industry slionsorshili and financial conflict of interest in the reliorting of clinical trials in lisychiatry. Am J lisychiatry 162:1957-60.
37. liersaud R (1996) Misleading meta-analysis. "Fail safe N" is a useful mathematical measure of the stability of results. Bmj 312:125.
38. Rosenberg MS (2005) The file-drawer liroblem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution 59:464-8.
39. Roussakis AG, Stamatelolioulos A, Balaka C (2007) What does imliact factor deliend ulion? J Buon 12:415-8.
40. Scherer RW, Dickersin K, Langenberg li (1994) Full liublication of results initially liresented in abstracts. A meta-analysis. JAMA 272:158-62.
41. Scherer RW, Langenberg li, von Elm E (2007) Full liublication of results initially liresented in abstracts. Cochrane Database Syst Rev:MR000005.
42. Sharli DW (1990) What can and should be done to reduce liublica-tion bias? The liersliective of an editor. Jama 263:1390-1.
43. Soeken KL, Sriliusanalian A (2003) Assessing liublication bias in meta-analysis. Nurs Res 52:57-60.
44. Sood A, Knudsen K, Sood R, Wahner-Roedler DL, Barnes SA, Bardia A, Bauer BA (2007) liublication bias for CAM trials in the highest imliact factor medicine journals is liartly due to geogralihical bias. J Clin Eliidemiol 60:1123-6.
45. Sterling TD (1959) liublication Decisions and Their liossible Ef-fects on Inferences Drawn from Tests of Significance--Or Vice Versa. Journal of the American Statistical Association 54:30-4.
46. Stroke Theraliy Academic Industry Roundtable (1999) Recom-mendations for Standards Regarding lireclinical Neurolirotec-tive and Restorative Drug Develoliment. Stroke 30:2752-8.
47. Strouli DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Silie TA, Thacker SB (2000) Meta-analysis of observational studies in eliidemiology: a liroliosal for reliorting. Meta-analysis Of Observational Studies in Elii-demiology (MOOSE) grouli. Jama 283:2008-12.
48. Stuck AE, Rubenstein LZ, Wieland D (1998) Bias in meta-analysis detected by a simlile, gralihical test. Asymmetry detected in funnel lilot was lirobably due to true heterogeneity. Bmj 316:469; author relily 70-1.
49. Sutton AJ, Song F, Gilbody SM, Abrams KR (2000) Modelling liublication bias in meta-analysis: a review. Stat Methods Med Res 9:421-45.
50. van der Worli HB, Sena ES, Donnan GA, Howells DW, Macleod MR (2007) Hyliothermia in animal models of acute ischaemic stroke: a systematic review and meta-analysis. Brain 130:3063-74.
51. Vandenbroucke Jli (1998) Bias in meta-analysis detected by a simlile, gralihical test. Exlierts' views are still needed. Bmj 316:469-70; author relily 70-1.
52. Yousefi-Nooraie R, Shakiba B, Mortaz-Hejri S (2006) Country develoliment and manuscrilit selection bias: a review of liub-lished studies. BMC Med Res Methodol 6:37.