- Open Access
Where there is no evidence: use of expert consensus methods to fill the evidence gap in low-income countries and cultural minorities
International Journal of Mental Health Systemsvolume 4, Article number: 33 (2010)
In both developing countries and in relation to cultural minorities there have been calls to scale up mental health services and for evidence-informed policy and practice.
Evidence based medicine
The evidence based medicine movement has had a major influence in improving practice. However, implementation of this approach has some major difficulties. One that has been neglected is the situation where there is no relevant evidence. This situation is more likely to occur for healthcare decisions in developing countries or for cultural minorities within developed countries, because resources do not exist for expensive research studies.
Consensus methods, such as the Delphi process, can be useful in providing an evidence base in situations where there is insufficient evidence. They provide a way of systematically tapping the expertise of people working in the area and give evidence that is readily applicable for a particular country and culture. Although consensus methods are often thought of as low in the hierarchy of evidence, consensus is central to the scientific process. We present four examples where the Delphi method was used to assess expert consensus in situations where no other evidence existed: estimating the prevalence of dementia in developing countries, developing mental health first aid guidelines in Asian countries, mental health first aid guidelines for Australian Aboriginal people, and modification of the concept of 'recovery' for Australian immigrant communities.
Consensus methods can provide a basis for decision-making and considered action when there is no evidence or when there are doubts about the applicability of evidence that has been generated from other populations or health system settings.
Mental disorders are common in all countries, with considerable variation in reported prevalence across countries, even when the same methods for estimation of prevalence are used . The treatment gap is wide everywhere, widest in developing countries [2, 3]. Among the reasons for failure of coverage is massive under-investment in mental health service provision , resulting in serious shortages of health facilities and skilled mental health professionals , reliance on outmoded mental hospitals and lack of community mental health services , and frequent neglect and abuse of the human rights of people with mental illness [7–9]. This developing country profile of population mental health and mental health services - variability in prevalence, under-investment in provision of appropriate and accessible mental health services, low rates of utilisation of mental health services, and failures of rights protection - is replicated in the case of cultural minorities, even in wealthy countries with relatively well-developed mental health systems [10, 11]. In both cases (developing countries and cultural minorities), there are legitimate concerns about the validity of 'western' diagnostic and treatment practices , and the applicability of 'western' models of mental health service . Also in both cases there have been calls for scaling up mental health services [11, 14] for new ways of working , and for evidence-informed policies and practice [14, 16].
Evidence based practice
Over the past 30 years, the evidence based medicine movement has had a major influence in improving practice. Evidence based medicine involves "integrating clinical expertise with the best available clinical evidence derived from systematic research" . This movement has led to improvements in the reporting and registration of clinical trials, the provision of systematic reviews of controlled trials (e.g. the Cochrane Collaboration) and the development and promotion of clinical practice guidelines. While the movement began in developed countries, there has also been advocacy for evidence based medicine in developing countries . It has been argued that the need for evidence based practice is vital in these countries because financial resources are scarce and need to be channelled to the best possible care options . In particular there is a need for evidence-informed decision making by policy makers, health system managers and others who responsible for shaping health systems and delivering services .
Despite its many successes, the evidence based medicine movement has encountered some significant barriers in changing clinical practice, including the difficulty that practitioners have in finding and interpreting the evidence, and in applying it in practice, due to organisational barriers, patient adherence and lack of quality practitioner education . There has also been criticism of the movement because of its emphasis on high quality evidence from randomised controlled trials, which tend to be carried out in ideal circumstances that may be remote from everyday practice. This has led to calls for more "practice-based evidence", where the evidence is gathered in real-life clinical settings and there is greater emphasis on the external validity of the evidence (generalisability) rather than on its internal validity (validity of causal inference) . An additional issue is the relative (compared to biomedical research) scarcity of health system research and evidence for policy and health system design [23–25].
Here we discuss an obvious, but neglected, barrier to evidence based practice--situations where there is no evidence . This barrier can be found for many areas of healthcare, but is more likely to be found in developing countries and for cultural minorities within developed countries.
Developing countries have limited resources to support research . Therefore, evidence is often imported from developed countries, raising the issue of applicability of this evidence in settings and from populations other than those in which it was generated. Where the evidence reflects universal biological processes (e.g. infection and immunisation), it will be possible to generalise, but where it involves social and cultural processes (e.g. persuading people to control infection or to immunise), it may not be. In psychiatry, where social and cultural factors are integral in making healthcare decisions and developing systems of care, much of the evidence from developed countries will be limited in generalisability. The same applies to cultural minorities within developed countries. Although it may be possible to develop an adequate evidence base for the mainstream population, the resources may not exist to do this for small indigenous or immigrant groups. In addition, immigrant and refugee communities that do not speak the host country language are frequently excluded from research because of the difficulties and expense of translation.
The development of a locally relevant evidence base using expert consensus is a valuable approach where other evidence is unavailable. While developing countries may not have the resources to carry out randomised controlled trials, population surveys, cohort studies or health service evaluations , they do have considerable local experience derived from practice. The same is true for cultural minorities in developed countries. There are formal methods of developing consensus from experts, such as the Delphi process, nominal group technique  and consensus conference methods , which can be used to harness this experience. These methods have acceptable construct validity  and reliability . By using these methods to develop an evidence base that can guide decisions, policy makers and practitioners can move beyond relying on their own experience and drawing on the accumulated experience of a larger, expert group.
As well as being feasible with limited resources, consensus methods overcome some of the limitations that have been identified with evidence based medicine. They embody the principle of practice-based evidence, provide results that are relevant to the local population and culture, and are readily implementable within the healthcare system. This approach fits with the call to improve practice by capitalising on accumulated practical experience and using this to develop better interventions.
The Delphi consensus method has been widely used in disparate fields to inform: policy-making [32, 33]; design of health services [34, 35]; development of diagnostic guidelines and protocols [36, 37]; developing mental health first aid guidelines for psychosis [38, 39], suicide [40–42], self-injury , and panic attacks ; and service and research priority-setting [44, 45]. In relation to dementia, the Delphi consensus method has been used to clarify issues and identify consensus in diagnosis and clinical assessment , treatment and management [47–49], services, outcomes , research, and to estimate prevalence  in settings where the necessary epidemiological studies have not been done.
There are several possible objections to the adequacy of expert consensus as a source of evidence. The evidence based medicine movement has been associated with a hierarchy of levels of evidence. Typically, the strongest form of evidence is held to be a meta-analysis of randomised controlled trials, with expert consensus ranked at the bottom of the hierarchy. However, while systematic reviews and meta-analyses are held to be the highest form of evidence, there is little agreement on the reporting of a critically important component of the systematic review process - the search methods used in carrying out systematic reviews . Even where relevant systematic reviews exist, there are often questions about the applicability of the findings of the systematic review to the task of choosing among policy or program options in particular circumstances .
If expert consensus is so weak, should we be proposing it as a strategy for contributing to informed decision-making? An important counter-question to ask is: how were some types of evidence assigned to be higher in the hierarchy than others? The answer is "expert consensus". Expert consensus is seen as adequate to validate hierarchies of evidence or indeed other components of the evidence based medicine enterprise, such as the CONSORT Statement for reporting randomised controlled trials  and principles for developing practice guidelines .
More generally, consensus has an important role in the scientific process. New theories gain ground as more members of the scientific community see a new theory as giving a better account of the evidence than older ones . The role of consensus can be seen in the past rise of scientific developments like continental drift and prions as infectious agents, but also in contemporary challenges such as climate change .
A paradox in recommending consensus in order to make progress is that consensus may hinder progress and promote no longer useful or even harmful practices. Consensus is an expression of the prevailing political, social, cultural, scientific world view of the time. There may, for example, be a broad consensus within society with which scientists do not agree, or there may be a consensus among experts that produces undesirable outcomes . The validity or probity of consensus-based decisions can be called into question, as has famously been done recently in relation to the work of the Inter-Governmental Panel on Climate Change (IPCC) which, as a global body of 800 climate scientists, relies on reaching consensus .
The issue is not whether consensus per se is weak or strong, because it is integral to how science works, but rather what foundation of evidence the consensus rests on. A consensus based on a set of randomised controlled trials  may well be better than a consensus based on personal clinical experiences. What we are proposing here is not that expert consensus is as good a source of evidence as a meta-analysis of trials, but that it is a better source of evidence than the experience of a single individual, which is the alternative when there is no other evidence available. Expert consensus may well turn out to be wrong. Indeed, the history of science indicates that it inevitably will. However, it presents a better basis for guiding action, because group consensus will generally produce better judgements than any individual's judgement . Expert consensus methods are a step on the road to informed decision-making. When better forms of evidence become available, these should of course be relied upon.
Examples of how expert consensus can be used
To illustrate the use of expert consensus methods in situations where there is no evidence, we describe four projects that have used the Delphi process to fill evidence gaps in developing countries and with cultural minorities.
Estimating the prevalence of dementia
Studies of the prevalence of dementia are major research undertakings. Estimates of prevalence rates are available from well-conducted studies in most developed countries, but for the rest of the world there is limited or no evidence. To overcome this lack, Alzheimer's Disease International carried out a Delphi study with a panel of 12 international experts . The panel members were provided with a systematic review of the published studies and, on the basis of those studies and additional information of possible relevance such as development status and child and adult mortality in the regions, were asked to estimate prevalence for every WHO world region for 5-year age groups from age 60 onwards. "When published information is scarce, experts can make inferences using other data from comparable contexts" . The consensus estimates of prevalence led to an estimate that 24 million people had dementia in 2001 and that 60% of them lived in developing countries, with this rising to 71% by 2040. These prevalence estimates are currently the best available basis for making policy and planning services. In particular, they raise awareness of the need for planning in relation to the rapid rise in number of dementia cases in developing countries. The purpose of the study was to generate estimates that are "the best currently available basis for policymaking, planning, and allocation of health and welfare resources" .
Mental health first aid in developing countries
Mental health first aid is the help provided to a person who is developing a mental health problem or is in a mental health crisis. A Mental Health First Aid training course has been developed in Australia and has spread to many other countries . Mental health first aid guidelines have been developed for English-speaking countries and used as a basis for the Mental Health First Aid training curriculum , but these may not be generalisable to countries with very different cultures and health systems. As a low-cost solution to this problem, psychosis first aid guidelines were developed using the Delphi method with clinicians from a wide range of Asian countries . Because guidelines need to be specific to particular countries, more recent work has used the Delphi process to develop suicide first aid guidelines for India , the Philippines and Japan, using the consensus of panels of clinicians from each of these countries.
Mental health first aid with Aboriginal Australians
A related example concerns Aboriginal Australians, who comprise less than 3% of the Australian population, but are a priority for action because they are disadvantaged in both physical and mental health. A Mental Health First Aid training program has been developed for Aboriginal and Torres Strait Islander peoples and found to be culturally acceptable . However, the first aid actions recommended in the training were based on a modification of those for the mainstream Australian community. To develop more culturally appropriate mental health first aid, a Delphi study was carried out with an expert panel of Aboriginal mental health professionals . Panel members were presented with statements about possible first aid actions and were encouraged to suggest other actions not already covered by the survey content. The endorsed statements, covering depression, psychosis, suicidal thoughts and behaviours, deliberate non-suicidal self-injury, trauma and loss, and cultural considerations in giving mental health first aid, were used to write guidelines and a revised curriculum for Aboriginal and Torres Strait Islander Mental Health First Aid training .
The construct of 'recovery' in immigrant communities
Mental health policy, particularly as it relates to the mental health NGO sector that provides rehabilitation, social support, housing, employment and related services, increasingly requires that recovery principles should underpin service design and delivery. However, the origins of the recovery movement are in western Anglophone countries, and recovery principles reflect the cultural commitment of these countries to values such as privacy and confidentiality, individual autonomy, and self-determination. The relevance of these underlying values to many immigrant communities in Australia, which tend to place greater value on interdependence, collective decision-making, and authority of the senior member of the family, has been called into question, therefore challenging whether the current recovery construct can be applied in a multicultural society. This study  used the Delphi method, with senior staff of a major NGO constituting the expert panel, to explore ways in which the construct of recovery may need to be modified to make it more relevant for a culturally diverse population of clients. The study also focused on ways in which a modified conception of recovery can inform the design and delivery of mental health services by the organisation. It is an example of use of the consensus method to inform organisational change for service improvement.
Evidence is most important as a basis for decision-making and action, such as medical treatment of an individual, decisions about health policy, or actions such as investment in one rather than another form of service delivery. It is clear that such decisions will be made and actions embarked upon whether or not there is evidence or whether the decision-maker or practitioner is familiar with such evidence as does exist. Consensus methods can provide a basis for decision-making and considered action when there is no evidence or when there are doubts about the applicability in a particular setting of evidence that has been generated from other populations or health systems. The issue is not whether consensus methods provide evidence that is as good as other ways of generating evidence, but whether the evidence generated by using such methods is better than no evidence or inapplicable evidence.
Beyond questions of the availability or quality of evidence, expert consensus methods have a particular advantage when applied in fields that are neglected and that require urgent action, such as mental health system development in low-income countries and for immigrant and refugee communities. The process of identifying consensus shifts the focus away from areas of disagreement, which frequently are many, to areas of agreement. If key stakeholders, such as government decision makers, service users, as well as clinicians and researchers, are involved in the expert panel considering a particular issue, this can facilitate informed decision-making and action.
Kessler RC, Haro JM, Heeringa SG, Pennell BE, Ustun TB: The World Health Organization World Mental Health Survey Initiative. Epidemiol Psichiatr Soc. 2006, 15 (3): 161-166.
Wang PS, Angermeyer M, Borges G, Bruffaerts R, Tat Chiu W, G DEG, Fayyad J, Gureje O, Haro JM, Huang Y: Delay and failure in treatment seeking after first onset of mental disorders in the World Health Organization's World Mental Health Survey Initiative. World Psychiatry. 2007, 6 (3): 177-185.
World Health Organization: mhGAP: Mental Health Gap Action Programme: Scaling up care mental, neurological, and substance use disorders. 2008, Geneva: World Health Organization
Chisholm D, Flisher AJ, Lund C, Patel V, Saxena S, Thornicroft G, Tomlinson M: Scale up services for mental disorders: a call for action. Lancet. 2007, 370 (9594): 1241-1252. 10.1016/S0140-6736(07)61242-2.
Jacob KS, Sharan P, Mirza I, Garrido-Cumbrera M, Seedat S, Mari JJ, Sreenivas V, Saxena S: Mental health systems in countries: where are we now?. Lancet. 2007, 370 (9592): 1061-1077. 10.1016/S0140-6736(07)61241-0.
Lin CY, Huang AL, Minas H, Cohen A: Mental hospital reform in Asia: the case of Yuli Veterans Hospital, Taiwan. Int J Ment Health Syst. 2009, 3 (1): 1-10.1186/1752-4458-3-1.
Minas H: Mentally ill patients dying in social shelters in Indonesia. Lancet. 2009, 374 (9690): 592-593. 10.1016/S0140-6736(09)61206-X.
Irmansyah I, Prasetyo Y, Minas H: Human rights of persons with mental illness in Indonesia: more than legislation is needed. Int J Ment Health Syst. 2009, 3 (1): 14-10.1186/1752-4458-3-14.
Minas H, Diatri H: Pasung: Physical restraint and confinement of the mentally ill in the community. Int J Ment Health Syst. 2008, 2 (1): 8-10.1186/1752-4458-2-8.
Minas H, Klimidis S, Kokanovic R: Depression in multicultural Australia: Policies, research and services. Australia and New Zealand Health Policy. 2007, 4: 16-10.1186/1743-8462-4-16.
Minas H: Proposal for a Victorian Mental Health and Cultural Diversity Taskforce. New Paradigm: The the Australian Journal on Psychosocial Rehabilitation. 2009, 11 (1): 37-42.
Minas H, Klimidis S, Tuncer C: Illness causal beliefs in Turkish immigrants. BMC Psychiatry. 2007, 7: 34-10.1186/1471-244X-7-34.
Watters E: Crazy like us: the globalization of the American psyche. 2010, New York: Free Press
Lancet Global Mental Health Group: Scale up services for mental disorders: a call for action. Lancet. 2007, 370 (9594): 1241-1252. 10.1016/S0140-6736(07)61242-2.
McPake B, Mensah K: Task shifting in health care in resource-poor countries. Lancet. 2008, 372 (9642): 870-871. 10.1016/S0140-6736(08)61375-6.
Saraceno B: Mental health systems research is urgently needed. Int J Ment Health Syst. 2007, 1 (1): 2-10.1186/1752-4458-1-2.
Straus SE, Sackett DL: Using research findings in clinical practice. BMJ. 1998, 317 (7154): 339-342.
Siddiqi K, Newell J, Robinson M: Getting evidence into practice: what works in developing countries?. Int J Qual Health Care. 2005, 17 (5): 447-454. 10.1093/intqhc/mzi051.
Garner P, Kale R, Dickson R, Dans T, Salinas R: Getting research findings into practice: implementing research findings in developing countries. BMJ. 1998, 317 (7157): 531-535.
Oxman AD, Lavis JN, Fretheim A, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP) 16: Using research evidence in balancing the pros and cons of policies. Health Res Policy Syst. 2009, 7 (Suppl 1): S16-10.1186/1478-4505-7-S1-S16.
Haynes B, Haines A: Barriers and bridges to evidence based clinical practice. BMJ. 1998, 317 (7153): 273-276.
Green LW: Making research relevant: if it is an evidence-based practice, where's the practice-based evidence?. Fam Pract. 2008, 25 (Suppl 1): i20-24. 10.1093/fampra/cmn055.
Oxman AD, Lavis JN, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking?. Health Res Policy Syst. 2009, 7 (Suppl 1): S1-10.1186/1478-4505-7-S1-S1.
Lewin S, Oxman AD, Lavis JN, Fretheim A, Garcia Marti S, Munabi-Babigumira S: SUPPORT tools for evidence-informed policymaking in health 11: Finding and using evidence about local conditions. Health Res Policy Syst. 2009, 7 (Suppl 1): S11-10.1186/1478-4505-7-S1-S11.
Lavis JN, Boyko JA, Oxman AD, Lewin S, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking. Health Res Policy Syst. 2009, 7 (Suppl 1): S14-10.1186/1478-4505-7-S1-S14.
Oxman AD, Lavis JN, Fretheim A, Lewin S: SUPPORT Tools for evidence-informed health Policymaking (STP) 17: Dealing with insufficient research evidence. Health Res Policy Syst. 2009, 7 (Suppl 1): S17-10.1186/1478-4505-7-S1-S17.
Razzouk D, Sharan P, Gallo C, Gureje O, Lamberte EE, de Jesus Mari J, Mazzotti G, Patel V, Swartz L, Olifson S: Scarcity and inequity of mental health research resources in low-and-middle income countries: a global survey. Health Policy. 94 (3): 211-220. 10.1016/j.healthpol.2009.09.009.
Jones J, Hunter D: Consensus methods for medical and health services research. BMJ. 1995, 311 (7001): 376-380.
Compton S, Lang E, Richardson TM, Hess E, Green J, Meurer W, Stanley R, Dunne R, Scott-Findlay S, Khare RK: Knowledge translation consensus conference: research methods. Acad Emerg Med. 2007, 14 (11): 991-995.
Cross H: Consensus methods: a bridge between clinical reasoning and clinical research?. Int J Lepr Other Mycobact Dis. 2005, 73 (1): 28-32. 10.1489/1544-581X(2005)73[28:CMABBC]2.0.CO;2.
Hutchings A, Raine R, Sanderson C, Black N: A comparison of formal consensus methods used for developing clinical guidelines. J Health Serv Res Policy. 2006, 11 (4): 218-224. 10.1258/135581906778476553.
Syed AM, Hjarnoe L, Krumkramp R, Reintjes R, Aro AR: Developing policy options for SARS and SARS-like diseases - a Delphi study. Glob Public Health. 2010, 1-13.
O'Loughlin R, Kelly A: Equity in resource allocation in the Irish health service. A policy Delphi study. Health Policy. 2004, 67 (3): 271-280.
Leang S: The impact of the HIV epidemic on health services in Cambodia: a Delphi study. Asia Pac J Public Health. 2008, 20 (Suppl): 141-147.
Toward JI, Ostwald SK: Exploring mental health service needs for the elderly: results of a modified Delphi study. Community Ment Health J. 2002, 38 (2): 141-149. 10.1023/A:1014595121309.
Graham B, Regehr G, Wright JG: Delphi as a method to establish consensus for diagnostic criteria. J Clin Epidemiol. 2003, 56 (12): 1150-1156. 10.1016/S0895-4356(03)00211-7.
Maarsingh OR, Dros J, van Weert HC, Schellevis FG, Bindels PJ, van der Horst HE: Development of a diagnostic protocol for dizziness in elderly patients in general practice: a Delphi procedure. BMC Fam Pract. 2009, 10: 12-10.1186/1471-2296-10-12.
Langlands RL, Jorm AF, Kelly CM, Kitchener BA: First aid recommendations for psychosis: using the Delphi method to gain consensus between mental health consumers, carers, and clinicians. Schizophr Bull. 2008, 34 (3): 435-443. 10.1093/schbul/sbm099.
Jorm AF, Minas H, Langlands RL, Kelly CM: First aid guidelines for psychosis in Asian countries: A Delphi consensus study. Int J Ment Health Syst. 2008, 2 (1): 2-10.1186/1752-4458-2-2.
Colucci E, Kelly CM, Minas H, Jorm AF, Chatterjee S: Mental Health First Aid guidelines for helping a suicidal person: a Delphi consensus study in India. Int J Ment Health Syst. 2010, 4: 4-10.1186/1752-4458-4-4.
Kelly CM, Jorm AF, Kitchener BA, Langlands RL: Development of mental health first aid guidelines for suicidal ideation and behaviour: a Delphi study. BMC Psychiatry. 2008, 8: 17-10.1186/1471-244X-8-17.
Kelly CM, Jorm AF, Kitchener BA, Langlands RL: Development of mental health first aid guidelines for deliberate non-suicidal self-injury: a Delphi study. BMC Psychiatry. 2008, 8: 62-10.1186/1471-244X-8-62.
Kelly CM, Jorm AF, Kitchener BA: Development of mental health first aid guidelines for panic attacks: a Delphi study. BMC Psychiatry. 2009, 9: 49-10.1186/1471-244X-9-49.
Lee MS, Hoe M, Hwang TY, Lee YM: Service priority and standard performance of community mental health centers in South Korea: a delphi approach. Psychiatry Investig. 2009, 6 (2): 59-65. 10.4306/pi.2009.6.2.59.
Yuen PP, Gould D, Cheng MY: Rationing hospital services in Hong Kong: priority setting by clinicians using the Delphi method. Health Serv Manage Res. 2002, 15 (1): 1-13. 10.1258/0951484021912789.
Perry M, Draskovic I, van Achterberg T, van Eijken M, Lucassen P, Vernooij-Dassen M, Olde Rikkert M: Development and validation of quality indicators for dementia diagnosis and management in a primary care setting. J Am Geriatr Soc. 58 (3): 557-563. 10.1111/j.1532-5415.2010.02726.x.
Verkade PJ, van Meijel B, Brink C, van Os-Medendorp H, Koekkoek B, Francke AL: Delphi-research exploring essential components and preconditions for case management in people with dementia. BMC Geriatr. 10: 54-10.1186/1471-2318-10-54.
Jones JA, Brown EJ, Volicer L: Target outcomes for long-term oral health care in dementia: a Delphi approach. J Public Health Dent. 2000, 60 (4): 330-334. 10.1111/j.1752-7325.2000.tb03344.x.
Ferris LE, Norton PG, Dunn EV, Gort EH, Degani N: Guidelines for managing domestic abuse when male and female partners are patients of the same physician. The Delphi Panel and the Consulting Group. JAMA. 1997, 278 (10): 851-857. 10.1001/jama.278.10.851.
Ferri CP, Prince M, Brayne C, Brodaty H, Fratiglioni L, Ganguli M, Hall K, Hasegawa K, Hendrie H, Huang Y: Global prevalence of dementia: a Delphi consensus study. Lancet. 2005, 366 (9503): 2112-2117. 10.1016/S0140-6736(05)67889-0.
Sampson M, McGowan J, Tetzlaff J, Cogo E, Moher D: No consensus exists on search reporting methods for systematic reviews. J Clin Epidemiol. 2008, 61 (8): 748-754. 10.1016/j.jclinepi.2007.10.009.
Lavis JN, Oxman AD, Souza NM, Lewin S, Gruen RL, Fretheim A: SUPPORT Tools for evidence-informed health Policymaking (STP) 9: Assessing the applicability of the findings of a systematic review. Health Res Policy Syst. 2009, 7 (Suppl 1): S9-10.1186/1478-4505-7-S1-S9.
Schulz KF, Altman DG, Moher D: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. Trials. 11: 32-10.1186/1745-6215-11-32.
Shekelle PG, Woolf SH, Eccles M, Grimshaw J: Developing clinical guidelines. West J Med. 1999, 170 (6): 348-351.
Kuhn TS: The structure of scientific revolutions. 1970, Chicago: University of Chicago Press, 2
Oreskos N: The scientific consensus on climate change. Science. 2004, 306: 11-32.
Schmiedebach HP, Priebe S: Social psychiatry in Germany in the twentieth century: ideas and models. Med Hist. 2004, 48 (4): 449-472.
Principles Governing IPCC Work. [http://www.ipcc.ch/pdf/ipcc-principles/ipcc-principles.pdf]
Sniezek JA, Henry RA: Accuracy and confidence in group judgment. Organizational Behavior and Human Decision Processes. 1989, 43: 1-28. 10.1016/0749-5978(89)90055-1.
Kitchener BA, Jorm AF, Kelly CM: Mental health first aid manual. 2010, Melbourne: Orygen Youth Health Research Centre, 2
Hart LM, Jorm AF, Kanowski LG, Kelly CM, Langlands RL: Mental health first aid for Indigenous Australians: using Delphi consensus studies to develop guidelines for culturally appropriate responses to mental health problems. BMC Psychiatry. 2009, 9: 47-10.1186/1471-244X-9-47.
Kanowski LG, Jorm AF, Hart LM: A mental health first aid training program for Australian Aboriginal and Torres Strait Islander peoples: description and initial evaluation. Int J Ment Health Syst. 2009, 3 (1): 10-10.1186/1752-4458-3-10.
Farnan S: Recovery principles relevant to people with mental illness from culturally and linguistically diverse backgrounds using the psychosocial rehabilitation services of Mental Illness Fellowship Victoria. 2010, University of Melbourne
The authors wish to acknowledge the thoughtful comments of reviewers. The revised manuscript is improved as a result of those comments.
The authors declare that they have no competing interests.
HM and AJF contributed equally to the conception and writing of this paper. Both authors read and approved the final manuscript.