Skip to main content

Creation of consensus recommendations for collaborative practice in the Malaysian psychiatric system: a modified Delphi study



There is strong evidence that collaborative practice in mental healthcare improves outcomes for patients. The concept of collaborative practice can include collaboration between healthcare workers of different professional backgrounds and collaboration with patients, families and communities. Most models of collaborative practice were developed in Western and high-income countries and are not easily translatable to settings which are culturally diverse and lower in resources. This project aimed to develop a set of recommendations to improve collaborative practice in Malaysia.


In the first phase, qualitative research was conducted to better understand collaboration in a psychiatric hospital (previously published). In the second phase a local hospital level committee from the same hospital was created to act on the qualitative research and create a set of recommendations to improve collaborative practice at the hospital for the hospital. Some of these recommendations were implemented, where feasible and the outcomes discussed. These recommendations were then sent to a nationwide Delphi panel. These committees consisted of healthcare staff of various professions, patients and carers.


The Delphi panel reached consensus after three rounds. The recommendations include ways to improve collaborative problem solving and decision making in the hospital, ways to improve the autonomy and relatedness of patients, carers and staff and ways to improve the levels of resources (e.g. skills training in staff, allowing people with lived experience of mental disorder to contribute).


This study showed that the Delphi method is a feasible method of developing recommendations and guidelines in Malaysia and allowed a wider range of stakeholders to contribute than traditional methods of developing guidelines and recommendations.

Trial registration Registered in the National Medical Research Register, Malaysia, NMRR-13-308-14792


Collaborative practice is defined by the World Health Organisation as: “when multiple health workers from different professional backgrounds work together with patients, families, carers and communities to deliver the highest quality of care” [1]. Collaborative practice is a broad term, which includes collaboration between the patient and healthcare staff, collaboration between the different members of the multidisciplinary team, collaboration between primary care and mental health staff and collaboration between healthcare staff and other members of the community. The complexity of severe mental problems mean that the good care is normally team based, with several different professions working together to help the patient. In the care of people with mental health problems, collaboration between service users and healthcare providers allows them to work towards common goals and this partnership has been shown to be one of the most efficacious components of many treatments [2,3,4,5,6]. Evidence that some of these collaborative models are more effective than usual care is strong, particularly collaborations between primary care and mental health services, where more than 80 randomised controlled trials have demonstrated clear improvements in outcomes with no increase in costs [7].

Developing collaborative practice in Malaysia was considered important because qualitative research showed that interactions were often hierarchical, rather than collaborative [8]. This was sometimes having a negative effect on patient care, for example nurses not telling doctors if they believed a treatment plan would not work and patients not discussing with their doctors if they had stopped medication due to side effects. Service provision was often siloed, for example there was little communication between the psychiatric hospital and community health clinics [8, 9]. There was also a large treatment gap of more than 90% in primary care [10]. Most collaborative models of care were developed in Western cultures. Cultural factors affect the way that people work together, and a model of care developed in a Western setting may not be the best model of care for an Asian context. These models are complex, with many elements and it is difficult to know what the most important elements for effectiveness are.

Developing complex models of care and interventions requires a different approach to developing simple interventions. Consensus methods offer a way of creating guidelines where the proposed intervention is complex and in situations where there is no strong evidence [11, 12]. Consensus methods include Delphi methods, nominal group techniques and the consensus development conference [12]. Delphi methods have been used extensively in higher-income countries and some low-income countries as a way of improving mental health services [13, 14] as well as a way of developing culturally appropriate responses [15]. The Delphi method involves sending a panel a series of items to rate and comment on. The panel never meet in person and all their ratings and comments remain anonymous. After the first round, the panel are sent the ratings and comments of the other panel members and the process is repeated in a series of rounds until consensus is reached [16]. In this study the Delphi method was chosen as a way of generalising recommendations developed by a local hospital level committee so that they would be useful for the whole of Malaysia. The Delphi method was chosen because it has been shown to be a reliable way of reaching a consensus [17], the anonymity makes it easier for people to express ideas if there is any perceived hierarchy and because we wished to get opinions from different geographical areas of Malaysia, where face-to-face meetings can be difficult and expensive.

This exercise aimed to answer the question: ‘What will improve collaboration in the Malaysian Mental Health System?’ The aim was to find a common vision of what would be effective, whether or not it was implementable at the current time, in view of the fact that Malaysia is a rapidly developing country and has very different levels of resource availability across the country [18]. A clear view of a desirable future is helpful in creating momentum and allowing people to work together.


Framing the research question and creating the first set of recommendations

The research question had originally come from a qualitative research project, which explored collaboration in the mental health system in Sabah, Malaysia. The first set of recommendations were produced by a local level committee in a psychiatric hospital, who met face to face, in the state of Sabah on the island of Borneo. This committee consisted of mental health staff and service users and had a total of eight meetings to produce the first set of recommendations. The committee were asked to create a set of recommendations for the hospital, based on qualitative research that had been conducted looking at collaborative practice in the hospital [8, 19]. Information from searches of research databases was also regularly presented at these meetings, where research questions arose from the discussion, focusing mainly on systematic reviews and meta-analyses (searches included the Cochrane Library, Embase, MEDLINE, Web of Science, Scopus, Psychinfo and Google Scholar). All these meetings were recorded and all but one of them was transcribed and coded, using the coding template derived from the original grounded theory study. The meeting minutes and items for the recommendations were created from this thematic analysis and then discussed at the next meeting. Some of the recommendations were implemented in the hospital and the results discussed at the meetings. Further discussion of the functioning of this committee and implementation of the recommendations will be discussed in separate papers.

Formation of the panel

People with a special interest in mental health systems and collaborative practice in Malaysia were identified. Most of the people suggested for the Delphi panel had been originally suggested by the hospital level committee, mentioned above, but others were found through reviewing the literature and by referral of other people recruited to the panel. Patient and carer representatives were identified through a snowballing mechanism, whereby opinion leaders were asked if there were any patients or carers who were active in support groups or in advocacy. Invitations to participate in the Delphi panel were sent to a total of 36 people, using a mixture of What’s App messaging, emails, phone calls and face to face discussions. The people invited included psychiatrists, psychologists, counsellors, psychiatric nurses, primary care doctors, public health professionals, NGO representatives and service users.

Determination of the expert panel size

It was aimed to get approximately twenty members of the panel, in order to ensure that all members could be communicated with individually if needed. A panel size of approximately twenty has reasonable stability and scores have been shown to correlate well with larger panel sizes [13].

Creating the questionnaire

An initial questionnaire was developed using Google forms about the model of collaborative care developed in Sabah. A five-point Likert scale was used to assess each of the items. This questionnaire was piloted by sending to two members of the research team (SF, DF) who were not involved in the development of the questionnaire and were not from a mental health background and a psychiatrist who was not involved in the study or in the panel.

Information provided to panel members to aid their judgements

The items were proceeded by an introduction to the concept of collaborative practice and a brief explanation of the results of the qualitative research. Most of the items were accompanied with a brief explanation or a link to other materials. All of the additional material had been agreed upon by the hospital level committee. The accompanying material was designed to be informative without creating a large reading burden for the members of the Delphi panel.

Administering the questionnaire

Three Delphi rounds were conducted February 2018, July 2018 and February 2019. The questionnaire was sent to the panel of experts, providing an anonymous on-line mechanism for them to review and comment on the collaborative care model. Panel members were given approximately 1 month to complete each round, with several reminders sent during this period. Panel members were not reimbursed for their time. Panel members who completed round one, but did not complete round 2, were given the opportunity to take part in round 3, after informing them about previous results.

Analysing rounds and providing feedback to the panel

An a priori decision was made that items were considered to have reached consensus if no members of the panel disagreed with an item and if the interquartile deviation was less than one [20]. The items were changed or removed if consensus had not been reached and the panel asked to re-rate the items and comment on them in the next round. The panel were sent anonymised ratings and comments from the previous round, which were displayed before asking the panel members to rate the item again. Where consensus had been reached, but comments were made suggesting minor changes to wording, the changes were made, and the panel was asked to comment. The panel was also asked to suggest any additional items during the first round of the Delphi process, and these were rated during the second round. The original hospital level committee discussed changes to the items before sending out for the final round as a way of reducing biases. The Delphi panel were also asked to rate and make comments about the process and changes were made to the process of subsequent rounds based on these comments.

Reporting results

Written comments were imported into NVivo version 10. Initial open coding was conducted, followed by amalgamation of codes into higher level categories. WS conducted the initial coding and SF also examined the raw data and checked agreement with the codes. Comparison between patient/NGO comments and healthcare staff comments was conducted using a matrix coding query. The area of difference are highlighted in this paper in order to better understand how service user input into guideline development is important. Reporting was done using the CREDES statement for Delphi studies [21].

Ethics and consent

The study was approved by the Medical Research and Ethics Committee, Ministry of Health Malaysia (NMRR-13-308-14792). All hospital level committee members signed written informed consent forms to agree for the recording of the meetings to be used for the purpose of research. All Delphi panel members agreed to participate electronically, after personalised contact (through emails, messages and sometimes phone calls) to explain the process.


This paper will focus mainly on the Delphi panel, the functioning and effectiveness of the hospital level committee will be discussed in a separate paper.

Composition of the hospital level committee and the Delphi panel

Table 1 shows the composition of the hospital level committee and the Delphi panel. There were 33 people who had attended the meetings of the hospital level committee, who were all based in one psychiatric hospital in Sabah. Twenty-two people agreed to take part in the Delphi panel, from different parts of Malaysia, and different institutions, including hospitals, universities and government institutes. Ten people did not reply, one declined because they felt it was not related to them and one suggested another person. Eighteen completed round one, 11 completed round two and 14 completed round three (three participants who did not complete round two subsequently completed round three). Four more patients were recruited during the first round, suggested by one of the Delphi panel members, who felt there were inadequate numbers of patients in the panel and was a member of a social media-based support group.

Table 1 Composition of the hospital committee and Delphi panel (participants that completed at least one round)

Quantitative analysis of items by the Delphi panel

Table 2 shows a list of items, together with the mean response for each round and the interquartile deviation (IQD). Where items had reached consensus in the first round, there are no round 2 results shown. The items were categorised under themes: autonomy, relatedness, resources, collaborating outside the hospital and decision making. At the end of round one, 39 items had been endorsed and ten rejected. The ten rejected items were rewritten and nine new items were added before round two (see Fig. 1). All items except for two had reached consensus after two rounds. The two items which had not yet reached consensus were rewritten and then a third round conducted, with only these two items to rate. One of these items was the title. For this item, the respondents were given a choice of three titles and asked to give the preferred title and asked if they were acceptable. The chosen title was the preferred title and was considered acceptable by all participants. The full version of the recommendations can be found in Additional file 1. A full account of how the committee changed the items is in Additional file 2.

Table 2 Means and interquartile deviations of items
Fig. 1
figure 1

Flow diagram showing number of statements in each round

Analysis of comments from Delphi panel

Most of the comments that both service users and staff made were broadly supportive of the guideline statements. The differences between the ways that staff and service users commented is highlighted.


All the respondents agreed with the need to empower patients and staff. Some service users described feeling intimidated and the difficulties that patients sometimes had in expressing themselves with doctors. Three new items were added after round one after suggestions from the panel, about ways to choose a meeting chair, giving leadership opportunities to staff and ensuring there was time for questions at the end of meetings.

Relatedness and continuity of care

Seeing a regular doctor appeared to be particularly important to service users. Service users commented extensively about this, illustrating with stories of difficulty in the system due to problems in continuity of care.

D8: I have experience seeing the same doctor from 2011 to 2014 and it really helps me a lot! Now I have to see different doctors at every visit, and I feel lazy to tell my stories again and again. The communication is just superficial, I tell the surface stories and the doctor gives surface suggestions. No chance to explore further. After all, why share so much if I may not see him again?

D18: As a patient, I felt truly disconnected from my treatment plan because it was handled by different doctors. After moving to a hospital that assigns patients to the same doctor throughout their treatment process, I began to feel a sense of connection. My doctor knows my story from the beginning, so I didn’t have to keep repeating the same story. Repeating my story to different doctors when I was unstable prevented me from seeking help because I had the idea that nobody cares.

Staff also agreed that this was important, but some had concerns about whether it was feasible for patients to regularly see the same doctor. Having a ‘primary nurse’ on the ward (a nurse case manager who cares for a patient throughout the length of their stay) also appeared to be more important to service users than staff.


Staff regularly mentioned resources, particularly not having enough staff in the system, not having enough time and how lack of resources made it difficult to implement some of the collaborative practice interventions. Service users focussed on the quality of staff and the problems associated with having undertrained staff, some commenting on how bad experiences with staff could hamper recovery.

Service users emphasised how people that had used the system were a useful resource and several of them commented on how they wanted to contribute, for example in producing educational materials. Service users placed a high value on help from people who had gone through similar experiences and some commented that service users could understand better than hospital staff. Three additional items were added after round two, which were related to peer support workers.

Working with people outside the healthcare system

All comments agreed that there was a need for better collaboration with people outside the hospital. There was concern about working with Bomoh (traditional healers) from some panel members:

D11: Ambiguity will arise when we work with Bomohs. Does this mean that we accept what they are doing? Does this mean that we from the scientific “community” agree with supernatural existence as what is practiced by Bomohs? Clear rules should be set before working with unregistered authorities. Because it can always backfire on us.

Some described out that the contribution of people outside the health system needed to be valued more and this may require a change in attitude from some of the staff.

D16: The doctors and hospital staff must be open to this idea and must realize that everyone has their own area of expertise and knowledge. Sometimes because doctors and those from the medical field feel or regard themselves as the ‘experts’ it may hinder from us achieving this goal as they might not want to listen or take in the opinion of others.

Decision making

This was the area with the most initial disagreement, particularly about the involvement of other people in decision making. Some panel members were concerned that involving other people in decision making might cause problems in confidentiality, be impractical and delay decision making The wording was changed to make it clear that this should be done with the permission of the patient, was optional and the time spent should be proportional to the difficulty and implications of the decision being made. There were also some concerns about patients that may not be able to make decisions for themselves, so further clarification was given on this in Additional file 2: Appendix S1. Additional items were added about defining the problem and setting goals, following suggestions from the panel.

Ratings and comments on the delphi process

Table 3 shows the ratings of panel members, regarding the Delphi process. These ratings were done at the end of round one.

Table 3 Participants experience of the Delphi process

The comments on the Delphi Process were generally positive, with panel members glad to have been given the opportunity to take part.

D6 (staff): It was a good opportunity to learn from other professionals as well as patients and caregivers.

Some members described feeling confused by some of the items and that not enough context was given to them in round one. A further introduction with more context was written for round two.

Service users described feeling empowered by the process, felt that their voices were being heard and that they were contributing.

D8: I feel I am involved in nation-building and we are all working towards a better Malaysia, better society and better standard of living. Process is long but it is unavoidable. It is good that you give us a reasonable timeframe to allow us to take part according to our pace.

Some commented that certain professions were missing (e.g. family medicine specialist, social worker) and some members felt that the balance between service users and professionals was not enough:

D16: Thank you so much. I do feel that this is a great way to get our voices heard. However, the mix is not balanced hence the answers will always lean to a medical model rather than a social model and will again fall back to what the mental health professionals feel, think and want and do not fully represent what the patients and carers fully need and want.


This process has created a set of recommendations, which aim to improve the general environment of the psychiatric system so that collaboration is more likely. The recommendations include ways to empower and improve autonomy, improvements in continuity of care and ways to enhance and make the best use of scarce resources. This is particularly important in Malaysia, where care is often fragmented [9] with low continuity of care [8], decision making is often hierarchical rather than collaborative [8], resources are limited [18, 22] and the treatment gap is large [10]. It is possible to implement many of these recommendations with the existing levels of resources. Although some of these recommendations are not feasible in many areas of Malaysia with the current level of resources, this exercise allowed consensus to be reached on what was desirable. Many of these recommendations already have empirical evidence to support them, which is briefly reviewed below.


Responses by Delphi panel members demonstrated how low levels of autonomy negatively affects patient care. A large meta-analysis of 184 studies based on self-determination theory showed patient autonomy to be associated with mental health and physical health outcomes. This effect is likely to be motivated by perceived competence, whereby patients that feel more in control of their lives are more likely to feel competent in the management of their health [23]. The effects of interventions that aim to improve autonomy have been found to be greater in marginalised and disempowered groups such as people with low income or education levels [24, 25].


The responses from service users on the Delphi panel highlight how inadequate continuity of care (e.g. patients seeing a different doctor on each visit) has a negative effect on care. Continuity of care is well studied and has been shown to be related to improved health outcomes [26], improved satisfaction [26, 27], improved cost effectiveness [26], decreased hospitalizations [27], decreased emergency department visits [27], and increased probability of receiving preventive services [27], particularly in patients with chronic diseases. A study using the French National Health Insurance database to follow up 14515 people with mental disorders for 3 years found relational continuity of care was related to reduced risk of death in people with mental disorders [28]. A UK longitudinal study examining the relationship between continuity of care in 5552 individuals with severe mental illness over 11 years showed that people with lower continuity of care had worse outcomes, with a large effect size of 1.75 (Cohen’s d) for the relationship between continuity of care and patient outcome [29]. Relatedness among staff is also important and a UK study of over 7000 health staff in 400 different healthcare teams showed that working in well-functioning teams led to lower levels of staff stress, lower death rates and higher levels of innovation [30].


The original qualitative research showed that the level of resources limited collaboration, including the mental health and collaborative competencies of staff and service users, time and physical resources [8]. Meta-analytic evidence shows that interventions that improve the competence of healthcare staff, including educational meetings [31] and training in patient centeredness [32], improve patient outcomes. These interventions are more effective if the training involves mixed didactic and interactive elements [31]. Training of mental healthcare staff also reduces burnout [33] and the use of restraint [34]. Empathy training is effective in improving empathic responding in healthcare staff [35].

Psychoeducation programs that improve patient understanding of their illness improves patient outcomes, including compliance, relapse and satisfaction with services [36]. Individualised care planning improves the ability of patients with chronic illnesses to manage their condition as well as reducing depression [37] and qualitative evidence suggests that many patients value and use written care plans [38]. Discharge planning processes reduce the length of hospital stay, readmissions and improves patient satisfaction [39]. Handheld records have been shown to improve communication and patient knowledge in other disciplines [40, 41]. There is evidence that peer support interventions can improve patient outcomes, including reducing inpatient service use [42]. Research in Malaysia has shown that approximately 20% of patients have unvoiced needs following a doctor’s appointment [43] and a waiting room intervention led to reductions in unvoiced needs [44]. Waiting room interventions that help patients to identify their informational needs improve aspects of the consultation, including asking questions, patient satisfaction and pre-consultation anxiety [45]. Waiting room poster displays and educational brochures are read by patients in other contexts [46, 47], but there is currently little research into how these interventions affect health outcomes.

In the Malaysian setting a typical outpatient appointment lasts approximately 5–10 min and committee and panel members discussed how this resulted in patients feeling rushed. There is evidence in primary care that longer appointment times improve the detection of psychiatric problems [48]. There is little research into the optimum appointment length in psychiatry. In the US setting, appointments with psychiatrists are often reduced to a 15 min ‘medication check’, with the expectation that the patient will be seeing another professional for psychological interventions. This has led to dissatisfaction from both psychiatrists and patients and concerns that care is substandard [49]. In the Malaysian setting, patients are normally not seeing any other professional for psychological treatment and most patients are seen by inexperienced medical officers, rather than psychiatrists, so longer times are likely to be needed to provide adequate care.

Participants in this research commented on the way that physical infrastructure affected collaboration, for example the institutional feel of the wards reducing the sense of autonomy. Building design influences the way that people interact with each other [50,51,52] and architecture has the potential to increase or reduce the sense of power imbalance [53]. In a psychiatric setting, meeting spaces need to feel private and psychologically safe for patients and staff [53]. Hospital information systems can also improve collaboration in healthcare settings [54], improve communication between healthcare providers and service users [55] and improve accessibility of healthcare, leading to improved patient outcomes [56,57,58,59].

Collaboration with people outside the hospital

Communities play a large role in mental health care in lower and middle income countries [60, 61] and partnership with communities is a strategy that has been successfully employed in Malaysia to improve mental health [62]. Training community members to provide initial help to people with mental disorders helps improve confidence, intention to help others and helping behaviours, however it is not yet clear whether these programs help improve mental health outcomes in people with mental health problems [63,64,65]. A systematic review of religious interventions concluded that they were effective [66] and in Malaysia religious professionals and traditional healers sometimes refer patients to services if they feel that the problem is a mental health concern, rather than a spiritual issue [19]. Interagency collaboration is considered best practice in the field of mental health, but the evidence that it is effective in improving patient outcomes was considered weak by a Cochrane review [66]. This is likely to be due to the complexity of these kind of interventions, where conducting a randomised control trial is difficult. However, a systematic review has shown that interagency collaboration has been shown to lead to better child welfare outcomes where there is parental drug use [66]. There is strong evidence that collaborations between primary care and specialist mental health staff are effective in treating people with mental disorders [7]. Treating patients with common mental disorders in primary care rather than secondary care has been recommended by the World Health Organisation for many years, since primary care is more accessible and acceptable to patients [61].

The decision-making process

The process for shared decision making and problem solving that we have recommended has similarities and differences with processes for doctor-patient relationships previously described in the literature [67,68,69,70,71,72,73]. The step of ‘identifying stakeholders’ is unusual in models of doctor-patient decision making, since most models only concern the doctor and patient. In Malaysia decisions are frequently made outside of the doctor-patient dyad, with family and other community members often involved in decision making, even after the patient has left the doctors office [19]. Programs that aim to improve shared decision making have been shown to improve patient satisfaction and collaboration with the treatment process, but most studies do not show improvements in symptoms or behavioural outcomes [24, 74, 75].

Complex multicomponent interventions

It appears that complex programs, which involve several of the components of collaborative practice (e.g. programs that increase patient education, autonomy and relatedness together), have an effect on more outcomes than programs which only introduce one component (e.g. only training in shared decision making) [7, 24, 76]. It is possible that the components of collaborative practice work synergistically, where several elements working together have a greater effect.


There are limitations to the way that we carried out this study. The first was that some groups were only represented in the original hospital committee, but not in the Delphi panel, particularly nursing and allied health staff. Service users were also underrepresented, particularly in the original hospital level committee and the service users on the committee are unlikely to have represented the views of all service users. Secondly, there was quite a large drop-out rate from the professionals in the Delphi panel. This is likely to reflect the reality of working in mental health in Malaysia, where it is extremely difficult to fulfil core work duties within working time and there is rarely time to do anything outside of core duties. This may have improved if panel members had been paid an honorarium for their time.


This study has shown that the Delphi method is a feasible method of making recommendations in mental health in Malaysia. A search of the literature did not reveal any other studies using this method in mental health in Malaysia, other than one pan-Asian study [77]. This method used minimal budget and has led to a more diverse group of people being involved in forming recommendations than is the case with traditionally used methods of decision making, which often only involve people in positions of power, who live in a small geographical area. This study attempted to give voice to those who have traditionally been left out of decision making. The World Health Organisation recommended in 2001 that “Communities, families and consumers should be included in the development and decision-making of policies, programmes and services” [78] and formal collaborations with service users is one of the WHO quality indicators [79]. This is not currently common practice in Malaysia [80] and is not one of the quality indicators commonly used [81]. This study has demonstrated the usefulness of involving consumers in forming recommendations, in that the perspective that they gave, and their priorities were different from the priorities of people that worked in the system. This was particularly the case with continuity of care, which appeared to be high priority for service users. This study demonstrated some of the difficulties of recruiting service users to committees in a setting where patriarchal attitudes to patients are prevalent [82], service user involvement is not common practice and there are still very few consumer groups. However, the process highlighted the existence of informal social media-based groups of mental health service users that are now growing and empowering users, one of which was eventually used to help recruit the patients and carers to the Delphi panel.

Future directions

It is hoped that over time some of these recommendations will be implemented and incorporated into quality indicators of the Malaysian healthcare system. Further research is now needed into the effectiveness of some of these recommendations, in the context of Malaysia. Systems based research in lower and middle income countries is currently lacking, but was rated as high priority in study of researchers and stakeholders in low and middle income countries [83]. Research in lower and middle income countries is needed, particularly into the effectiveness of patient and staff empowerment, shared decision making, improving relatedness in the system, written care plans and information, increasing the provision of certain types of training and collaborating with the wider system.


This study sought to build evidence on interventions which will help to improve patient care through improving collaborative practice in Malaysia. This has shown that the modified Delphi method is a feasible method in Malaysia and led to participation of a more diverse group of people than traditional methods of decision making. It also demonstrated the importance of involving service users and the challenges in doing this when it is not yet part of the culture. These recommendations could potentially be part of level III evidence [84], in the formation of clinical practice guidelines for complex systems level interventions, where higher level evidence is currently weak in Malaysia.

Availability of data and materials

The anonymised, raw data from the Delphi Committee and the changes made to each item are available in Additional files 1, 2.


  1. World Health Organization. Framework for action on interprofessional education & collaborative practice. practice. Geneva: Department of Human Resources for Health; 2010. Available Accessed 9 May 2012.

  2. Wampold BE. How important are the common factors in psychotherapy? An update. World Psychiatry. 2015;14(3):270–7.

    PubMed  PubMed Central  Google Scholar 

  3. Howgego IM, Yellowlees P, Owen C, Meldrum L, Dark F. The therapeutic alliance: the key to effective patient outcome? A descriptive review of the evidence in community mental health case management. Aust N Z J Psychiatry. 2003;37(2):169–83.

    PubMed  Google Scholar 

  4. Nakagami E. Working alliance, hope, and functional outcome for individuals with schizophrenia: mechanisms of influence. Los Angeles: University of Southern California; 2009.

    Google Scholar 

  5. Hopkins M, Ramsundar N. Which factors predict case management services and how do these services relate to client outcomes? Psychiatr Rehabil J. 2006;29(3):219–22.

    PubMed  Google Scholar 

  6. McCabe R, Bullenkamp J, Hansson L, Lauber C, Martinez-Leal R, Rössler W, et al. The therapeutic relationship and adherence to antipsychotic medication in schizophrenia. PLoS ONE. 2012;7(4):e36080.

    CAS  PubMed  PubMed Central  Google Scholar 

  7. Woltmann E, Ph D, Grogan-kaylor A, Ph D, Perron B, Ph D, et al. Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: systematic review and meta-analysis emily. Am J Psychiatry. 2012;169(18):790–804.

    PubMed  Google Scholar 

  8. Shoesmith W, Awang Borhanuddin AFB, Pereira EJ, Nordin N, Giridharan B, Forman D, et al. Barriers and enablers to collaboration in the mental health system in Sabah, Malaysia: towards a theory of collaboration. BJPsych Open. 2020;6(1):1–10.

    Google Scholar 

  9. Shoesmith WD, Sawatan W, Abdullah AF, Fyfe S. Leadership and evaluation issues in interprofessional education in Sabah, Malaysia. Leading research and evaluation in interprofessional education and collaborative practice. London: Palgrave Macmillan UK; 2016. p. 193–212.

    Google Scholar 

  10. Bin Abdullah AF. Mental health of primary care attendees in Kota Kinabalu, Sabah. Parkville: University of Melbourne; 2018.

    Google Scholar 

  11. Minas H, Jorm AF. Where there is no evidence: use of expert consensus methods to fill the evidence gap in low-income countries and cultural minorities. Int J Ment Health Syst. 2010;4(1):33.

    PubMed  PubMed Central  Google Scholar 

  12. World Health Organization. Decision-making for guideline development at WHO. In: WHO handbook for guideline development, 2nd edn; 2014. p. 201–14. Accessed 6 May 2020.

  13. Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49(10):887–97.

    PubMed  Google Scholar 

  14. De Silva SA, Colucci E, Mendis J, Kelly CM, Jorm AF, Minas H. Suicide first aid guidelines for Sri Lanka: a Delphi consensus study. Int J Ment Health Syst. 2016;10(1):1–10.

    Google Scholar 

  15. Hart LM, Jorm AF, Kanowski LG, Kelly CM, Langlands RL. Mental health first aid for Indigenous Australians: using Delphi consensus studies to develop guidelines for culturally appropriate responses to mental health problems. BMC Psychiatry. 2009;9:1–12.

    Google Scholar 

  16. Jones J, Hunter D. Consensus methods for medical and health services research. Br Med J. 1995;311(August):376–80.

    CAS  Google Scholar 

  17. Hutchings A, Raine R, Sanderson C, Black N. A comparison of formal consensus methods used for developing clinical guidelines. J Heal Serv Res Policy. 2006;11(4):218–24.

    Google Scholar 

  18. Guan NC, Lee TC, Francis B, Yen TS. Psychiatrists in Malaysia: the Ratio and Distribution. Malaysian J Psychiatry. 2018;27(1):1–9.

    Google Scholar 

  19. Shoesmith WD, Borhanuddin AFBA, Yong Pau Lin P, Abdullah AF, Nordin N, Giridharan B, et al. Reactions to symptoms of mental disorder and help seeking in Sabah, Malaysia. Int J Soc Psychiatry. 2017;64:49–54.

    PubMed  Google Scholar 

  20. Rayens MK, Hahn EJ. Building consensus using the policy Delphi method. Policy Polit Nurs Pract. 2000;1(4):308–15.

    Google Scholar 

  21. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706.

    PubMed  Google Scholar 

  22. Yoon CK. Mental health services where there is no psychiatrist: my experience in Sabah. Asia-Pacific Psychiatry. 2010;2(3):128–128.

    Google Scholar 

  23. Ng JYY, Ntoumanis N, Thøgersen-Ntoumani C, Deci EL, Ryan RM, Duda JL, et al. Self-determination theory applied to health contexts: a meta-analysis. Perspect Psychol Sci. 2012;7(4):325–40.

    PubMed  Google Scholar 

  24. Samalin L, Genty J-B, Boyer L, Lopez-Castroman J, Abbar M, Llorca P-M. Shared decision-making: a systematic review focusing on mood disorders. Curr Psychiatry Rep. 2018;20(4):23.

    PubMed  Google Scholar 

  25. Durand M-A, Carpenter L, Dolan H, Bravo P, Mann M, Bunn F, et al. Do interventions designed to support shared decision-making reduce health inequalities? A systematic review and meta-analysis. PLoS ONE. 2014;9(4):e94670.

    PubMed  PubMed Central  Google Scholar 

  26. Sans-Corrales M, Pujol-Ribera E, Gené-Badia J, Pasarín-Rua MI, Iglesias-Pérez B, Casajuana-Brunet J. Family medicine attributes related to satisfaction, health and costs. Fam Pract. 2006;23(3):308–16.

    PubMed  Google Scholar 

  27. Cabana MD, Jee SH. Does continuity of care improve patient outcomes? J Fam Pract. 2004;53(12):974–80.

    PubMed  Google Scholar 

  28. Hoertel N, Limosin F, Leleu H. Poor longitudinal continuity of care is associated with an increased mortality rate among patients with mental disorders: results from the French National Health Insurance Reimbursement Database. Eur Psychiatry. 2014;29(6):358–64.

    CAS  PubMed  Google Scholar 

  29. MacDonald A, Adamis D, Craig T, Murray R. Continuity of care and clinical outcomes in the community for people with severe mental illness. Br J Psychiatry. 2019;214(5):273–8.

    PubMed  Google Scholar 

  30. Borrill CS, Carletta J, Carter AJ, Dawson JF, Garrod S, Rees A, et al. The effectiveness of health care teams in the National Health Service Report. Aston Centre for Health Service Organization Research; 2001.

  31. Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O’Brien MA, Wolf F, et al. Continuing education meetings and workshops: Effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009;2009(2):CD003030.

    PubMed Central  Google Scholar 

  32. Dwamena F, Cm G, Jorgenson S, Sadigh G, Sikorskii A, Lewin S, et al. Interventions for providers to promote a patient-centred approach in clinical consultations. Cochrane Libr. 2012;12(12):CD003267.

    Google Scholar 

  33. Dreison KC, Luther L, Bonfils KA, Sliter MT, Mcgrew JH, Salyers MP. Supplemental material for job burnout in mental health providers: a meta-analysis of 35 years of intervention research. J Occup Health Psychol. 2018;23(1):18–30.

    PubMed  Google Scholar 

  34. Ye J, Xiao A, Yu L, Guo J, Lei H, Wei H, et al. Staff training reduces the use of physical restraint in mental health service, evidence-based reflection for China. Arch Psychiatr Nurs. 2018;32(3):488–94.

    PubMed  Google Scholar 

  35. Teding van Berkhout E, Malouff JM. The efficacy of empathy training: a meta-analysis of randomized controlled trials. J Couns Psychol. 2016;63(1):32–41.

    PubMed  Google Scholar 

  36. Xia J, Merinder L, Belgamwar M. Psychoeducation for schizophrenia. Cochrane Database Syst Rev. 2011;6:1–170.

    Google Scholar 

  37. Coulter A, Entwistle VA, Eccles A, Ryan S, Shepperd S, Perera R. Personalised care planning for adults with chronic or long-term health conditions. Cochrane Database Syst Rev. 2013;2013(5):CD010523.

    Google Scholar 

  38. Palmer VJ, Johnson CL, Furler JS, Densley K, Potiriadis M, Gunn JM. Written plans: an overlooked mechanism to develop recovery-oriented primary care for depression? Aust J Prim Health. 2014;20(3):241–9.

    PubMed  Google Scholar 

  39. Gonçalves-Bradley DC, Lannin NA, Clemson LM, Cameron ID, Shepperd S. Discharge planning from hospital. Cochrane Database Syst Rev. 2016;1:74.

    Google Scholar 

  40. Nguyen M, Lennox N, Ware R. Hand-held health records for individuals with intellectual disability: a systematic review. J Intellect Disabil Res. 2014;58(12):1172–8.

    CAS  PubMed  Google Scholar 

  41. Lester HE, Allan T, Wilson S, Jowett S, Roberts L. A cluster randomised controlled trial of patient-held medical records for people with schizophrenia receiving shared care. Br J Gen Pract. 2003;53(488):197–203.

    PubMed  PubMed Central  Google Scholar 

  42. Chinman M, George P, Dougherty RH, Daniels AS, Ghose SS, Swift A, et al. Peer support services for individuals with serious mental illnesses: assessing the evidence. Psychiatr Serv. 2014;65(4):429–41.

    PubMed  Google Scholar 

  43. Low LL, Sondi S, Azman AB, Goh PP, Maimunah AH, Ibrahim MY, et al. Extent and determinants of patients’ unvoiced needs. Asia Pac J Public Health. 2011;23(5):690–702.

    Article  PubMed  Google Scholar 

  44. Low LL, Azman A, Sararaks S, Maimunah A, Goh P, Ibrahim M, et al. Reducing patients’ unvoiced needs—the Malaysian experience. International Forum On Quality and Safety in Health Care, Berlin, 2009.

  45. Kinnersley P, Edwards A, Hood K, Cadbury N, Ryan R, Prout H, et al. Interventions before consultations for helping patients address their information needs (Review). Cochrane Database Syst Rev. 2010.

    Article  Google Scholar 

  46. Devroey D, Moerenhout T, Borgermans L, Schol S, Vansintejan J, Van De Vijver E. Patient health information materials in waiting rooms of family physicians: do patients care? Patient Prefer Adherence. 2013;7:489–97.

    PubMed  PubMed Central  Google Scholar 

  47. Montazeri A, Sajadian A. Do women read poster displays on breast cancer in waiting rooms? J Public Health (Bangkok). 2004;26(4):355–8.

    Google Scholar 

  48. Hutton C, Gunn J. Do longer consultations improve the management of psychological problems in general practice? A systematic literature review. BMC Health Serv Res. 2007;7:71.

    PubMed  PubMed Central  Google Scholar 

  49. Cruz M, Roter DL, Cruz RF, Wieland M, Larson S, Cooper LA, et al. Appointment length, psychiatrists’ communication behaviors, and medication management appointment adherence. Psychiatr Serv. 2013;64(9):886–92.

    PubMed  Google Scholar 

  50. Penn A, Turner JS, Penn A. Can we identify general architectural principles that impact the collective behaviour of both human and animal systems? Phil Trans R Soc B. 2018.

    Article  PubMed  Google Scholar 

  51. Bernstein ES, Turban S, Bernstein ES. The impact of the ‘open’ workspace on human collaboration. Phil Trans R Soc B. 2018.

    Article  PubMed  Google Scholar 

  52. Arya D. So, you want to design an acute mental health inpatient unit: physical issues for consideration. Inpatient Psychiatry. 2011;19(2):163–7.

    Google Scholar 

  53. Gillespie R. Architecture and power: a family planning clinic as a case study. Health Place. 2002;8:211–20.

    PubMed  Google Scholar 

  54. El-kareh R, Gandhi TK, Poon EG, Newmark LP, Ungar J, Lipsitz S, et al. Trends in primary care clinician perceptions of a new electronic health record robert. J Gen Intern Med. 2008;24(4):464–9.

    Google Scholar 

  55. Burke RP, Rossi AF, Wilner BR, Hannan RL, Zabinsky JA, White JA. Transforming patient and family access to medical information: utilisation patterns of a patient-accessible electronic health record. Cardiol Young. 2020;20(October):477–84.

    Google Scholar 

  56. Wani D, Malhotra M. Does the meaningful use of electronic health records improve patient outcomes? J Oper Manage. 2018;60:1–18.

    Google Scholar 

  57. Dayal P, Hojman NM, Kissee JL, Evans J, Natale JE, Huang Y, et al. Impact of telemedicine on severity of illness and outcomes among children transferred from referring emergency departments to a children’s hospital PICU. Pediatr Crit Care Med. 2016;17(6):516–21.

    PubMed  Google Scholar 

  58. Mattos SDS, Mourato FA, De Araújo JSS, Moser LRDN, Hatem TDP, Albuquerque FCDL, et al. Impact of a telemedicine network on neonatal mortality in a state in Northeast Brazil. Popul Health Manag. 2018;21(6):517.

    PubMed  Google Scholar 

  59. Faruque LI, Wiebe N, Ehteshami-Afshar A, Liu Y, Dianati-Maleki N, Hemmelgarn BR, et al. Effect of telemedicine on glycated hemoglobin in diabetes: a systematic review and meta-analysis of randomized trials. CMAJ. 2017;189(9):E341–64.

    PubMed  PubMed Central  Google Scholar 

  60. Kohrt BA, Asher L, Bhardwaj A, Fazel M, Jordans MJD, Mutamba BB, et al. The role of communities in mental health care in low- and middle-income countries: a meta-review of components and competencies. Int J Environ Res Public Health. 2018;15:1279.

    PubMed Central  Google Scholar 

  61. van Ginneken N, Tharyan P, Lewin S, Rao G, Meera S, Pian J, et al. Non-specialist health worker interventions for the care of mental, neurological and substance-abuse disorders in low- and middle-income countries (Review). Cochrane Database Syst Rev. 2013.

    Article  PubMed  Google Scholar 

  62. Lasimbang HB, Eckermann E, Shoesmith WD, James S, Igau AEB, Iggau O, et al. Alcohol toolkit: empowering sabah indigenous communities to reduce alcohol related harm. Borneo J Med Sci. 2019;13(3):11–8.

    Google Scholar 

  63. Morgan AJ, Ross A, Reavley NJ. Systematic review and meta-analysis of Mental Health First Aid training: effects on knowledge, stigma, and helping behaviour. PLoS ONE. 2018;13:e0197102.

    PubMed  PubMed Central  Google Scholar 

  64. Morgan AJ, Fischer JA, Hart LM, Kelly CM, Kitchener BA, Reavley NJ, et al. Does Mental Health First Aid training improve the mental health of aid recipients? The training for parents of teenagers randomised controlled trial. BMC Psychiatry. 2019;19:99.

    PubMed  PubMed Central  Google Scholar 

  65. Hadlaczky G, Hökby S, Mkrtchian A, Carli V, Wasserman D. Mental Health First Aid is an effective public health intervention for improving knowledge, attitudes, and behaviour: a meta-analysis Mental Health First Aid is an effective public health intervention for improving knowledge, attitudes, and behaviour. Int Rev Psychiatry. 2014.

    Article  PubMed  Google Scholar 

  66. Hayes SL, Mann MK, Morgan FM, Kelly MJ, Weightman AL. Collaboration between local health and local government agencies for health improvement. Cochrane Database Syst Rev. 2012.

    Article  PubMed  Google Scholar 

  67. Towle A, Godolphin W. Framework for teaching and learning informed shared decision making. 1999;319:766–71.

    CAS  Google Scholar 

  68. Elwyn G, Edwards A, Kinnersley P, Grol R. Shared decision making and the concept of equipoise: the competences of involving patients in healthcare choices. Br J Gen Pract. 2000;50(460):892–9.

    CAS  PubMed  PubMed Central  Google Scholar 

  69. Simon D, Schorr G, Wirtz M, Vodermaier A, Caspari C, Neuner B, et al. Development and first validation of the shared decision-making questionnaire (SDM-Q). Patient Educ Couns. 2006;63(3):319–27.

    CAS  PubMed  Google Scholar 

  70. Harnett T. Consensus-oriented decision-making: the CODM model for facilitating groups to widespread agreement. Gabriola Island: New Society Publishers; 2011.

    Google Scholar 

  71. Elwyn G, Frosch D, Thomson R, Joseph-Williams N, Lloyd A, Kinnersley P, et al. Shared decision making: a model for clinical practice. J Gen Intern Med. 2012;27(10):1361–7.

    PubMed  PubMed Central  Google Scholar 

  72. Elwyn G, James P, Grande SW, Thompson R, Walsh T, Ozanne EM. Developing CollaboRATE: a fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Educ Couns. 2013;93(1):102–7.

    Article  PubMed  Google Scholar 

  73. Pouliot S, Kryworuchko J, Dunn S, Stacey D, Le F. Patient Education and Counseling Shared decision making models to inform an interprofessional perspective on decision making: a theory analysis. Patient Educ Counsel. 2010;80:164–72.

    Google Scholar 

  74. Aubree Shay L, Lafata JE. Where is the evidence? a systematic review of shared decision making and patient outcomes. Med Decis Mak. 2015;35(1):114–31.

    Google Scholar 

  75. ​Stacey D, Légaré F, Lewis K, Barry M, Bennett C, Eden K, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2017;4:1–242.

    Google Scholar 

  76. Shortell SM, Poon BY, Ramsay PP, Rodriguez HP, Ivey SL, Huber T, et al. A multilevel analysis of patient engagement and patient-reported outcomes in primary care practices of accountable care organizations. J Gen Intern Med. 2017;32(6):640–7.

    PubMed  PubMed Central  Google Scholar 

  77. Jorm AF, Minas H, Langlands RL, Kelly CM. First aid guidelines for psychosis in Asian countries: a Delphi consensus study. Int J Ment Health Syst. 2008;2:1–5.

    Google Scholar 

  78. World Health Organization. Mental Health- New Understanding, New Hope. The World Health Report 2001. Geneva; 2001.

  79. World Health Organization. Mental Health Atlas 2017. Geneva: World Health Organization; 2018.

    Google Scholar 

  80. Semrau M, Lempp H, Keynejad R, Evans-Lacko S, Mugisha J, Raja S, et al. Service user and caregiver involvement in mental health system strengthening in low- and middle-income countries: Systematic review. BMC Health Serv Res. 2016.

    Article  PubMed  PubMed Central  Google Scholar 

  81. Malaysian Healthcare Performance Unit. Malaysian Mental Healthcare Performance: Technical report 2016; 2016. Accessed 6 May 2020.

  82. Crabtree SA. Medication, healing and resistance in East Malaysia. Ment Heal Relig Cult. 2005;8(1):17–25.

    Google Scholar 

  83. Sharan P, Gallo C, Gureje O, Lamberte E, Mari JJ, Mazzotti G, et al. Mental health research priorities in low- and middle-income countries of Africa, Asia, Latin America and the Caribbean. Br J Psychiatry. 2009;195(4):354–63.

    CAS  PubMed  PubMed Central  Google Scholar 

  84. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the U.S. preventive services task force: a review of the process. Am J Prev Med. 2001;20(3S):21–35.

    CAS  PubMed  Google Scholar 

Download references


We would like to thank all of the members of the Hospital Mesra Bukit Padang Collaborative Practice Committee and the Delphi Committee: Adam Abdullah, Ahmad Qabil Bin Khalib, Segeran Ramodran, Atiqah Chew Abdullah, Brenagempli Anak Jampong, Cassie Ellena Chew, Cornelius John, Emmanuel Joseph Pereira, Lee Shea Wah, Ng Boon Seng, Norhayati Nordin, Razifah Adbul Rahman, Shahlini Selvarajoo, Sughashini Subramaniam, Ebba Gondilou, Geo Allen George, Julius Jalani, Lee Sang Choon, Mary Malanjun, Matron Nurzelah Abdullah, Ong Kok Loong @ Mohd Benyamin Abdullah, Rewajoh Lahai, Zaidin Endut, Rommy B. Ramlee, Nurmina Nawali, Sharifah Aini Yusoff, Sidik Singkak, Kulnah Ambor, Masnah Ajis, Umi Izzatti Saedon, Tang Poh Yee, Siti Hazrah, Nurul Nadia Ismail, Marhani Midin, Low Lee Lan, Abdul Kadir Abu Bakar, Anita Abu Bakar, Ang Kim Teng, Amalina Abdullah, Ahmad Rostam Md Zin, Abdul Rasyid Sulaiman, Shazwani Rhosky binti Fadzir, Wan Zakirah bt. Wan Zakaria.


The funding for the qualitative research that formed the initial phase of this project was supported by a grant from the Ministry of Health, Malaysia (no. NMRR-13-308-14792).

Author information

Authors and Affiliations



WS: designed the research protocol, wrote the proposal, recruited and organised both committees, drafting the paper and submitting the paper. SHC: helped organise the hospital level committee, undertook regular discussion and reflection on findings, reviewed and critically appraised the final article. BG: was a PhD supervisor for this project, undertook design of the research protocol, regular discussion and reflection on findings, reviewed and critically appraised the final article. DF: PhD supervisor, undertook design of the research protocol, regular discussion and reflection on findings, reviewed and critically appraised the final article. SF: PhD supervisor for this project, undertook design of the research protocol, regular discussion and reflection on findings, analysis of the data, reviewed and critically appraised the final article. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Wendy Shoesmith.

Ethics declarations

Ethics approval and consent to participate

See “Methods” section.

Consent for publication

We would like to thank the Director General of Health Malaysia for permission to publish this paper.

Competing interests

The author declares that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

The full guidelines.

Additional file 2.

Detail about how items were modified and Delphi panel comments for each round.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shoesmith, W., Chua, S.H., Giridharan, B. et al. Creation of consensus recommendations for collaborative practice in the Malaysian psychiatric system: a modified Delphi study. Int J Ment Health Syst 14, 45 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: