Background: South Africa is challenged with high levels of unemployment, comprising many people with low levels of education and also individuals who have never held a job before. Despite having many vulnerable participants, interventions aimed at the unemployed generally exclude psychosocial training and are methodologically weak.

Aim: The JOBS programme, a scientifically sound intervention, has been developed specifically to help people affected by unemployment to cope with the psychological effects. As a means of applying such a programme in South Africa, this study aimed to develop a framework based on the insights gained on the implementation and evaluation of the JOBS programme.

Methods: The study comprised a systematic review of literature regarding the JOBS intervention and its derivatives (n = 34).

Results: The results revealed that populations similar to the unemployed in South Africa had benefitted significantly regarding re-employment and mental health outcomes.

Conclusion: Suggestions derived from the literature included aiming the programme at the most vulnerable unemployed in South Africa: the youth and long-term unemployed. Furthermore, expanding the programme by adding an entrepreneurial component may yield positive results, considering the lack of employment opportunities in South Africa.

Keywords: JOBS programme; employment interventions; systematic literature review; unemployment; South Africa.

Introduction

South Africa is facing an unemployment crisis: currently 29.1% of people in South Africa are jobless (Stats SA 2019). While statistics may indicate the magnitude of the problem at hand, they fail to depict the nature and severe impact of unemployment. Unemployment is not only associated with societal and economic ramifications; it also has serious psychological consequences for those who are unemployed (see Strandh et al. 2014; Wanberg 2012).

Numerous interventions have been implemented to alleviate unemployment (Independent Evaluation Group 2013; McCarthy 2008). Despite the benefits of evidence-based practices (see Heckman, Lalonde & Smith 1999; Ravallion 2008), it is startling to see that most vocational interventions are consensus-based – implying that the interventions are based on what the stakeholders think is necessary, without the supporting evidence to prove what is really needed (Marais & Matebesi 2013). Considering the urgency of unemployment, the shortage of evidence-based practices in South Africa is a concern.

One profound example of a scientifically sound employment initiative is the JOBS programme (Caplan et al. 1989). The JOBS intervention seeks to enhance the employability of jobseekers by equipping them with the necessary job search, social and problem-solving skills to support them in their job search efforts. Several factors contribute to the achieved outcomes. Two of the strengths of the programme are its strong theoretical foundation and empirically tested evidence (Vinokur & Price 2015). Likewise, the comprehensive protocol guiding the programme contributes significantly to successful dissemination undertakings (Curran, Wishart & Gingrich 1999). Extensive evidence of the effectiveness of the JOBS programme has previously been reported (Price & Vinokur 2014).

Due to the encouraging results, several JOBS derivatives have been implemented in other countries. These programmes include the Työhön Job Search Programme in Finland (Vuori et al. 2002), the Jobs in China programme (Price & Fang 2002), the Job-Search programme in Israel (Shirom Vinokur & Price 2008), the Winning New Jobs (WNJ) programme in Ireland (Barry et al. 2006) and the JOBS intervention in the Netherlands (Brenninkmeijer & Blonk 2011). Although these programmes have proven to be reliable in different economic contexts (Vinokur et al. 1995a), they have been implemented mainly in developed countries, except for China. The lack of evidence-based practices in South Africa, together with the successful replication of the JOBS programme, creates an opportunity to explore whether such an intervention can be tailored to suit the South African context.

Much effort has been devoted to developing materials that explicitly explain the procedures and dynamics of the JOBS programme (Curran et al. 1999). Yet investigating literature pertaining to the execution and subsequent outcomes of the JOBS programme may assist further dissemination. A greater understanding of typical components of an intervention – implementation and evaluation – can be used to serve as guiding principles for application and assessment in the South African context. Based on the above statement of the research problem, the objectives of this research were:

Research design

Research approach

A systematic literature review was done to achieve the objectives of this study. A systematic review identifies the main scientific contributions relevant to a specific topic by conducting extensive literature searches of published and unpublished studies (Tranfield, Denyer & Smart 2003). This review aimed to identify literature containing information about the JOBS programme and variations of it. Transparent and reproducible procedures were used to enhance the quality and outcomes of the review process.

Research method

Targeted body of literature

Before starting with the review, the founders of the JOBS programme and the web page of the Michigan Prevention Research Center (MPRC) were consulted to obtain information regarding the JOBS programme and its dissemination. Electronic searches were undertaken to allocate articles identified on the web page. Next, a search was conducted to ascertain whether possible articles were excluded. Databases such as Google Scholar and EBSCOhost (Academic Search Premier, Africa-Wide Information, American Doctoral Dissertations, PsycARTICLES and PsycINFO) were utilised to find the relevant articles.

Search terms and selected criteria

It was anticipated that the articles worth including in the review would refer to the JOBS programme in the articles themselves. Therefore, numerous searches were conducted by including the authors involved (as obtained from the dissemination page) with the term Job Opportunities and Basic Skills programme (e.g. AUTHOR: Vuori; IN-TEXT: ‘JOBS program*’). Because another programme, called the JOBS Program, yielded additional results when searching for ‘Jobs Program*’, it was necessary to include the various authors. The search string consisted of two search concepts joined by the Boolean operator AND; the second string contained a list of authors joined by the Boolean operator OR. The following search string was entered in the databases: [1] IN-TEXT: ‘jobs program*’ [2] AUTHOR: ‘Barry’, ‘Caplan’; ‘Choi’; ‘Kessler’ ‘Price’; ‘Schul’; ‘Van Ryn’ ‘Vinokur’ and ‘Vuori’.

To prevent the omission of essential articles, a complementary search was performed. Reference lists of the selected articles were reviewed for more relevant publications. During the process, it became evident that there were indeed articles that did not refer to the JOBS programme in their content. Therefore, an additional search was conducted using the authors involved from the different variations of the JOBS programme and each of the different JOBS variations (e.g. AUTHOR: ‘Barry’; IN-TEXT: ‘Winning New Jobs’). The second search string again consisted of two search concepts joined by the Boolean operator AND; the first string contained a list of names for variations of the JOBS programme and the second a list of authors, with both strings joined by the Boolean operator OR. The following search string was entered in the databases: (1) IN-TEXT: Jobs in China Project, Job-search Intervention, Työhön Job Search Program, or Winning New Jobs (2) AUTHOR: ‘Blonk’, ‘Brenninkmeijer’, ‘Choi’, ‘Donaldson’, ‘Fang’, ‘Friedland’, ‘Ling’, ‘Shirom’, and ‘Turner’.

Gathering the data

Conference proceedings and papers to which access was limited or where no full-text papers were available were excluded. Price and Vinokur (2014) mention that the JOBS programme has previously been executed in Sweden and South Korea; however, the literature seemed limited and unavailable. Furthermore, the JOBS programme has also been implemented in organisational and school contexts. Considering that the circumstances of the participants are not the same, these studies were omitted. The inclusion and exclusion criteria narrowed the scope of this review. Finally, 34 articles met all the inclusion criteria (Figure 1).

Analysis and presentation of the data

Implementation and evaluation aspects were studied to gain a better understanding of the JOBS programme and its derivatives.

Implementation is described as the process of putting a plan into action to achieve objectives (Miller, Wilson & Hickson 2004). To ensure sufficient programme fidelity and to effectively replicate the JOBS, it seemed necessary to study the various components involved in executing such a programme. Evaluation can be described as the determination of the merit, worth and significance of an area of interest using criteria directed by specific standards for purposes of decision-making (Richards & Schmidt 2002). Evaluation practices are a crucial component in the success of evidence-based programmes (Jané-Llopis et al. 2005). To develop a framework aimed at guiding the implementation and evaluation of the JOBS programme in South Africa, the following aspects of the papers included in the systematic literature review were studied (Table 1).

Assessment of methodological quality

An additional reviewer – a researcher involved in the broader project – was consulted to ensure methodological quality. After the duplicates had been removed, both the researcher and reviewer were involved in selecting studies to remain in the systematic review based on their abstracts and full content.

Ethical considerations

Ethical clearance was obtained from the Humanities and Health Research Ethics Committee (HHREC), North-West University, with ethical clearance number: NWU-HS-2018-0006.

Results

The literature review comprised 34 studies. The predetermined criteria, as per Table 1, guided the systematic review. The main aspects regarding the implementation and evaluation of the JOBS programme and variations of it were tabulated. The summarised findings are reported below.

Implementation

This section contains information about the context in which the various programmes were implemented as well as participant and programme specifics.

Context

The JOBS programme has been implemented in numerous states and countries, in all of which the context has differed to some degree (see Appendix 1). The unemployment rates, as reported in the studies included, varied from 4% to 20% (Brenninkmeijer & Blonk 2011; Vuori et al. 2002). Additionally, the unemployed seemed to be supported by welfare benefits, also safeguarding them against severe financial hardship. Unemployment grants differed, depending on the social policies of the countries involved (Vinokur & Price 2015).

Participants

Biographical variables

According to Vinokur and Price (2015), benefits of the JOBS programmes did not seem to be distributed equally to all the participants. Some findings are reported below; yet it should be noted that only a few of the studies included mentioned the impact of demographic variables on the intended outcomes. Findings regarding participants’ demographic variables were not always consistent and sufficient to substantiate these relationships. Compared to other demographic variables, education had the strongest impact on the outcomes of the job search interventions.

Gender: Female participants seemed to suffer greater economic losses and experience more difficulties regaining employment (Vinokur et al. 2000). Yet women generally benefitted more from the programme than their male counterparts (MPRC 2003). They were more likely to (1) obtain employment six months after the intervention (Shirom et al. 2008), (2) score higher on participant engagement (Caplan et al. 1989), (3) experience positive group participation (Vuori et al. 2005) and (4) participate voluntarily (Vuori et al. 2002).

Age: Unemployed people between the ages of 16 and 65 were generally the targeted population. The mean age of participants in the studies included was 36 (Standard deviation [SD] = 9). Vuori et al. (2005) state that younger participants were usually more positive and found employment more easily than older participants, but showed a higher tendency of non-participation (Van Ryn & Vinokur 1992; Vinokur et al. 2000; Vinokur, Price & Caplan 1991a). The training programme seemed to also have a positive impact on older participants regarding improved job search skills and increased self-confidence (Price & Choi 2001).

Level of education: The majority of participants in the programmes involved had a secondary level of education (equivalent to 12 years of schooling; mean of all the studies: 40.65%). Participants with higher levels of education more often gained in terms of obtaining re-employment (MPRC 2003), increases in job search self-efficacy (Choi, Price & Vinokur 2003), a lower likelihood of major depressive episode diagnosis (Vinokur et al. 2000) and non-participation (Caplan et al. 1989; Van Ryn & Vinokur 1992; Vinokur et al. 2000), and higher levels of voluntary participation (Vuori et al. 2002). Despite these positive findings, in some other studies, it was evident that the programme also clearly yielded mental health benefits and economic benefits for those less educated and most disadvantaged in terms of the job market (Price & Choi 2001; Vinokur, Price & Schul 1995b).

Duration of unemployment: The JOBS programme was originally designed to prevent further deterioration in mental health among the unemployed and was not specifically intended to deal with potential problems associated with long-term unemployment (Caplan et al. 1989). Therefore, the majority of the initial studies included only those who had been unemployed for less than 13 weeks (Caplan et al. 1989; Price et al. 1992; Van Ryn & Vinokur 1992; Vinokur et al. 1991a, 1991b, 1995a, 2000; Vinokur, Price & Caplan 1996; Vinokur & Schul 1997). However, because the long-term unemployed were reported as the most vulnerable, more recent developments included participants who had been unemployed for longer periods (Brenninkmeijer & Blonk 2011; Malmberg-Heimonen & Vuori 2005; Price & Choi 2001; Reynolds, Barry & Gabhainn 2010; Vuori et al. 2005).

Population, sample size and recruitment

Population: Eligibility criteria required individuals to be aged between 16 and 65. Prospective screening questionnaires were used in some studies to determine participants’ risk score for poor mental health (Vinokur et al. 1995b). However, those who showed major signs of mental illness, serious psychosocial problems or behavioural problems or who scored extremely high on depression symptoms were omitted from the study (Brenninkmeijer & Blonk 2011; Vinokur et al. 1995a).

Sample size: Most of the studies included were conducted as a part of large-scale field experiments, ranging from 1087 to 3402 participants. Smaller-scale studies ranged from 125 to 672 participants. Sample sizes did not seem to influence the results achieved or the sustainability of the programmes. It was rather the use (or lack) of effective methods that seemed to have an impact on the outcomes (Price & Vinokur 2014).

Recruitment: The primary method used to recruit participants was through recruiters who approached individuals eligible for employment benefits while collecting grants at employment offices. In studies including an experimental and control group, participants were told about the two programmes on job-seeking methods. One programme was described as a workshop consisting of five half-day sessions (the experimental condition); the other was described as a self-guided programme, in which participants received a booklet with job search information (the control condition). To prevent biases, participants had to show no preference for a type of intervention (experimental or control condition; Caplan et al. 1989; Vinokur et al. 1995a).

Voluntary or enforced participation: Participants from some programmes had to participate in the job search workshop to qualify for welfare benefits (Brenninkmeijer & Blonk 2011; Lee & Vinokur 2007). When studying the impact of voluntary or enforced participation, a Finnish study showed that enforced participation did not increase re-employment; however, it impaired the positive mental health impacts of the programme. Further analysis demonstrated that enforced participation in job search training decreased re-employment among the longer-term unemployed workers (Malmberg-Heimonen & Vuori 2005).

Dropout rates: In the US programmes, on average, 59% (varying by 5%) of participants failed to show up for the intervention (Caplan et al. 1989; Vinokur et al. 1995a). Consequently, dropout rates could be anticipated and, therefore, twice as many participants were recruited and allocated to intervention groups in the Israeli study (Shirom et al. 2008). In the Malmberg-Heimonen and Vuori (2005) study, it was surprising to find that response rates did not differ much among the enforced, voluntary and control groups (94%, 92%, and 91%).

Programme

The information presented below was derived from the JOBS training manual (Curran et al. 1999).

Delivery

The JOBS programme entails two main processes. On the one hand, job search skills (the actual content of the programme) are taught to participants, while, on the other hand, empowerment of the participants, by applying the programme’s underlying principles in the method of delivery, is the true underlying mission of the workshop. The following aspects guided the method of delivery used by the trainers:

Referent power: Moderate self-disclosures shared by facilitators create an environment in which participants feel safe to reveal their concerns and experiences. These also contribute to creating an atmosphere of unconditional acceptance and to enhancing feelings of being normal and valued.

Guiding behaviour: Specific positive feedback is given to participants to reinforce positive behaviour. Strategies used to generate positive feedback include active listening, observation and reflecting on what participants have shared as a means of showing participants that they are valued.

Inoculation against setbacks: The group is encouraged to identify potential setbacks and difficulties in the job search process. Strategies are developed to overcome the identified challenges and, consequently, participants realise that their problems can be solved. Participants are asked to commit themselves to action by selecting and vowing to undertake a solution most appealing to them.

Social support: Social support forms an integral part of the underlying processes, as exercises are specifically designed to create opportunities for facilitators and participants to support each other. An environment where participants are unconditionally accepted is created. Such a safe environment contributes to participants feeling comfortable to express their opinions and reveal their feelings.

Active leaning: The learning process relies greatly on participants’ knowledge and skills. Participants acquire job search skills by using active learning methods, elicited using group discussions and brainstorming sessions.

In contrast to traditional top-down, trainer-focused training methods, the JOBS programme relies heavily on its individual-focused approach. The delivery principles mentioned above contribute to the strong individual-focused approach. Principles are continuously applied and integrated and form the basis on which the content is delivered.

Content

The programme consists of five sessions. During the first session, participants discover their job skills; the second session focuses on dealing with obstacles related to employment; the third session is used to introduce participants to some job search techniques; the fourth session covers topics such as curriculum vitae writing and preparing participants for job interviews; during the fifth session participants rehearse skills acquired throughout the week. The workshop concludes with a certificate ceremony, during which facilitators boost participants’ confidence by highlighting their strengths and skills and providing each participant with a sincere and inspiring message.

The JOBS protocol describes the programme processes meticulously. Yet these processes are flexible and can be altered, depending on the needs of the groups, without losing the intended effects of the programme. The majority of the disseminated versions of the JOBS programme were implemented strictly according to the protocol. The content differed in terms of minor language, cultural, procedural and scheduling changes to suit different contexts. To maintain the standard of the JOBS programme, all materials were piloted and approved. It is worth mentioning that, when the protocol was somewhat neglected, it was reported that the programme was less successful in achieving the intended outcomes (Shirom et al. 2008).

Incentives

Participants were rewarded monetarily for participation or each returned questionnaire (varying between $5.00 and $15.00, depending on the currency of the country). In cases where questionnaires were not returned, an additional amount was issued on the completion of their questionnaires. This incentive was reported to result in a substantial increase in response rates (about 20%; Shirom et al. 2008). Participants in the JOBS programme and Netherlands JOBS programme also received a certificate of participation for completing the programme (Brenninkmeijer & Blonk 2011; Caplan et al. 1989).

Researchers obtained higher response rates when offering incentives: in cases with relatively high dropout rates, no mention of rewards or incentives was evident (Barry et al. 2006; Reynolds et al. 2010; Shirom et al. 2008). The same finding was, however, not true in the WNJ California studies, which managed to retain approximately 70% of their participants, seemingly without the use of incentives (Choi et al. 2003).

Facilitators

Pairing: Teams consisting of one male and one female trainer are prescribed by Curran et al. (1999) to complement each other well. An untested assumption existed that a pair of trainers reduced deviation from the principles of the JOBS programme. However, the assumed benefits of having male–female pair facilitators have not yet been tested. Benefit-cost research could determine whether the cost of using two trainers, rather than one, is outweighed by the benefits that are generated (Price et al. 1998).

Prerequisites: Facilitators were generally social workers, labour advisors, educational counsellors or high school teachers. It was suggested that facilitators ought to be skillful in working with people (public speaking and communications backgrounds). Because trained individuals (that is, mental health professionals, such as counsellors or clinicians) might execute strongly embedded techniques not necessarily consistent with unemployment-related counselling methods, professional training was not a prerequisite (Caplan et al. 1989).

Programme-related training: Facilitators had to undergo extensive formal training. The content of the training covered understanding of group processes, theoretical foundations of the programme and extensive rehearsal in the form of pilot studies. The duration of training varied from 6 to 30 days (48 h–240 h). The reason for the extensive training was that facilitators were not only conveyors of information, but also experts in navigating the group processes, with the ability to connect emotionally with the participants and facilitate interactions in a group setting. To promote conformity, trainers’ performance was evaluated by trained supervisors.

Duration of the programme

Some of the findings yielded by the original JOBS trial encouraged the revision of the programme, which consequently led to the development of the JOBS II intervention (Vinokur et al. 1995a). The first version (the JOBS I programme) spanned eight 3 h sessions, over a two-week period (four mornings per week; Caplan et al. 1989). To increase programme efficiency and the attendance of participants, meeting hours were reduced by 30%, delivered over five 4 h sessions in a one-week period in the JOBS II (Vinokur et al. 1995a). The majority of disseminated versions of the JOBS programme continued to apply the programme following the JOBS II protocol. In some groups, the Finnish programme was delivered over four days, as the first day was used to deal with recently laid-off workers’ negative emotions (Vuori et al. 2002) – an illustration of how the programme can be altered to meet the needs of the group, without affecting the outcomes.

Group sizes

Guidelines of the JOBS programme suggest groups consisting of 12–20 participants (Curran et al. 1999). There were exceptions, where the groups ranged from three to 110 participants per group (median = 11; Malmberg-Heimonen & Vuori 2005). Although only a few studies reported on the impact of group sizes, larger groups seemed to have more negative experiences than smaller groups (Vuori et al. 2005).

Venue of training

Venues such as community centres, school classrooms, churches and union halls, easily accessible to participants, were mostly used. Venues had to be large enough to accommodate 25 people and furnished with movable chairs, arranged in a semicircular layout. Such a layout was reported to be most effective in delivering the group intervention (Curran et al. 1999).

Stakeholder involvement

Crucial to the success of the WNJ programme in Ireland was that the developers of the original JOBS programme were involved from the outset and contributed to obtaining buy-in from strategic stakeholder agencies. Despite a substantial initial investment of resources for demonstration, neither the WNJ in California nor the JOBS in China project continued beyond their initial stages, as commitment of resources for continuation was not offered by the government or other stakeholders. Therefore, the success of programme dissemination depended considerably on the involvement of, and support received from, stakeholders (Vinokur & Price 2015).

Evaluation

This section is comprised of information regarding evaluation of the processes and the impact of the JOBS programme.

Methodology

Data collection method

Self-administered questionnaires were used to assess participants’ attitudes, intentions, various behavioural components and experience of the workshop (Van Ryn & Vinokur 1992). In cases of unreturned questionnaires or where participants failed to show up for the workshop, telephonic interviews were conducted (Barry et al. 2006).

Research design and data collection intervals

Randomised field study designs were used to investigate the intervention effect between experimental and control conditions (Caplan et al. 1989; Vinokur et al. 1995a; Vuori et al. 2002). Programmes that made use of a randomised field study design had three to four interval times, namely pre-intervention (two weeks before the programme), post-intervention (directly after the programme), post-post-intervention (between two and six months after the programme) and long-term follow-ups, varying from 12 to 32 months after the intervention (Barry et al. 2006; Brenninkmeijer & Blonk 2011; Vinokur et al. 1991a). Other programmes only tested pre-intervention and post-intervention to determine the impact of the programme (Lee & Vinokur 2007; Shirom et al. 2008).

Process evaluation

To determine the internal validity and the strength and integrity of the JOBS programmes, two types of analysis were generally conducted. These process measures consisted of testing the integrity of randomisation and strength of the programme (Vinokur et al. 1995a).

Effectiveness of randomisation

The first check to determine the validity of the programme was to determine whether the statistical analyses were conducted on a randomised (true) experimental design. This was established by comparing the demographic and other tested variables of the experimental and control conditions at baseline to identify possible differences. In cases where differences were found, these variables were controlled for in further analyses (Vinokur et al. 1995a).

Manipulation checks, integrity and strength of the intervention

The second test was to test the strength and integrity of the intervention through self-reported questionnaires at the end of each session. Participants were asked to evaluate their experience of facilitators and the programme. These evaluations were used to determine whether various intervention elements had been implemented and had operated as designed (Vinokur et al. 1995a). Participants who scored high on these measures also reported higher levels of internal control and job-seeking self-efficacy (Choi et al. 2003), decreases in depression and anger, and increases in self-esteem and quality of life (Caplan et al. 1989). Also, trainer skills (one of the evaluated variables) exhibited during group interactions contributed to increased re-employment, even at the 12-month follow-up (Reynolds et al. 2010).

Two additional methods were used to ensure the quality of the programme and a high level of trainer adherence to the protocol. Firstly, members of the research team frequently observed programme trainers: after each session, constructive feedback was given to trainers. Secondly, the facilitators met weekly to discuss skill-related topics they encountered during their sessions (Vinokur et al. 1995b).

Impact evaluation

Outcomes

The positive outcomes of the JOBS programme were documented amply. Below are some of the most prominent findings related to the two core objectives of the JOBS programmes: prevention of poor mental health and promotion of re-employment, and other post-hoc outcomes.

Prevention of poor mental health: Participation in the intervention resulted in increased self-esteem, self-efficacy and social assertiveness among participants; consequently, participants also showed improved psychological and mental health and well-being (Lee & Vinokur 2007; Reynolds et al. 2010). Furthermore, long-term effects of the programme revealed that participants experienced lower symptoms of depression (Price et al. 1992; Vuori & Silvonen 2005), improved self-esteem (Reynolds et al. 2010) and an enhanced ability to deal with setbacks. A noteworthy finding is that participants screened for showing higher risk for depression seemed to benefit the most in terms of mental health and re-employment outcomes (Vinokur et al. 1995b).

Promotion of re-employment: Several programmes demonstrated increased rates of re-employment, with an average of 46% after the intervention, compared to the control group, with an average of 18% (Brenninkmeijer & Blonk 2011; Caplan et al. 1989; Donaldson 2012; Shirom et al. 2008; Vuori et al. 2005). Programme participants also showed higher motivation to persist in job search efforts (Caplan et al. 1989), were employed in better jobs (in terms of earnings and job satisfaction) (Vinokur et al. 1991b), were employed faster, had less recurring episodes of unemployment (Vinokur & Price 2015) and experienced reduced economic hardship after being employed (Barry et al. 2006). Results remained over time, as long-term effects of the programme revealed that participants, compared to their counterparts, experienced higher re-employment (Brenninkmeijer & Blonk 2011). Another crucial finding is that both the Työhön and the Netherland’s JOBS programmes confirmed the effectiveness of the intervention to help even the more vulnerable long-term unemployed gain employment (Brenninkmeijer & Blonk 2011; Vuori et al. 2002).

Consequential outcomes: Finally, the JOBS programme demonstrated substantial cost-benefit effectiveness because the higher earnings led, on average, to higher tax revenues and decreased welfare grants for governments (Vinokur et al. 1991b).

Discussion

The purpose of this study was to review literature regarding the JOBS programme and variations of it, with the intention of developing a framework that could guide the successful implementation and evaluation of the JOBS programme within the South African context. To gain a better understanding of the components related to the implementation of the JOBS programme, the contexts in which the programme have previously been implemented and the targeted population, as well as aspects regarding the programme, were studied. Based on the findings of the systematic review, as well as context-specific matters, a framework is proposed for the implementation and evaluation of the JOBS programme within the South African context (see Appendix 2).

In terms of contextual differences between developed countries (where the JOBS programme has previously been implemented), and developmental countries (e.g. South Africa), some differences are crucial to consider when implementing an employment programme, such as the JOBS intervention. While the unemployment rates of the involved developed countries averaged 12%, more than 27% (37.3% when including those who have stopped looking for employment; Stats SA 2018) of South Africans are currently unemployed. Moreover, it has been reported that 69% of these individuals have been unemployed for longer than a year (Stats SA 2018). In South Africa, unlike the other countries, unemployment grants safeguarding people from financial hardship are not available. Also, the unemployed are generally situated in rural areas isolated from major economic activity. With limited job opportunities, jobseekers feel discouraged, and deprived of a chance to compete in the labour market (Du Toit et al. 2018). Fortunately, the JOBS programme is specifically designed to deal with such conditions, yet it remains important to be cognisant of the impact of contextual factors on potential participants’ state of mind.

With regard to participant-related matters, the reviewed literature showed that young and old, educated and less educated participants had previously benefitted from the JOBS interventions. However, it is important to note that South Africa has a youth unemployment rate of 52% (aged between 15 and 24; Stats SA 2018); 62% of the unemployed population have never even held a job before (Stats SA 2017); and 57% of South Africans have an education of less than matric (Grade 12). Therefore, although the unemployed in general could benefit from the programme, it is suggested to target vulnerable populations, such as younger, less educated and long-term unemployed individuals, as it may yield promising results.

Furthermore, participants from previous studies were reached at employment services offices. Because unemployment grants are not available in South Africa, participants cannot be reached on a large scale in a similar way. Therefore, different strategies of reaching the intended population should be considered. Suggestions include making use of newspaper and radio advertisements, and government agencies working with jobseekers, or working with youth and community leaders. One programme in particular tested the effectiveness of forced versus voluntary participation. Findings revealed that enforced participation did not increase re-employment and impaired the positive mental health impact of the programme (Malmberg-Heimonen & Vuori 2005). Giving participants the autonomy to participate voluntarily in the programme seems to yield more positive benefits. This may be an important finding for policymakers, as a precondition for receiving unemployment grants is often enforced attendance of a job search programme. Yet responsibility also rests with workshop trainers to be particularly devoted in creating an environment to which participants choose to return.

Considering the possibility that participants showing a preference to partake in employment programmes may be somewhat more intrinsically motivated, at the same time, it is those who show a higher risk of depressive symptoms that may benefit more (Vinokur et al. 1995a). Thus, careful attention should be paid to recruitment measures, ensuring that both the motivated and those who may be at risk of depressive symptoms are reached through recruitment methods, as they are equally important in achieving intended programme outcomes. Similarly, some programmes made use of screening questionnaires to identify participants at risk of poor mental health (Vinokur et al. 1995b); those who scored exceptionally high on depression symptoms or showed major signs of mental illness were omitted from the programme (Brenninkmeijer & Blonk 2011). As previously mentioned, given that many of the unemployed in South Africa may be severely discouraged, it is recommended to refrain from screening participants to identify high-risk cases, as it may result in the exclusion of participants who may benefit from the programme.

The next implementation aspect investigated related to programme-related matters. In line with previous adaptations of the JOBS programme, it is suggested to tailor the content of the manuals and activities to better suit the context and to increase cultural acceptability (Barry et al. 2006; Brenninkmeijer & Blonk 2011). Due to slow economic growth and the lack of skills in specific disadvantaged populations, changes in conditions of obtaining a job may be difficult (Vinokur & Price 2015). A solution that may fill both of these voids could be to consider fostering an entrepreneurial mindset among programme participants. People working in the informal sector often have a lower education level (although not lower wages) compared to those employed in the formal sector (Kim 2002), which may be a suitable solution within the South African context.

Furthermore, the ability to facilitate and understand group processes, build feelings of competence and create an environment of unconditional acceptance were essential requirements for facilitators. Yet education levels of the facilitators were not reported as particularly important. Due to the great demand for social work in South Africa, a shortfall of qualified social workers exists, which often results in employing people at social services offices who are less skilled and experienced (Collin 2017). Failure to grasp the importance of, and means of executing, the principal components of the programme may be problematic for the successful execution of the programme. Consequently, involving trainers knowledgeable and experienced in this area, while at the same time having the ability to relate to participants, should be considered. These may typically include individuals with higher degrees, coming from a similar background, who can also serve as role models for participants. Additionally, trainers should have the ability to adopt an individual-focused training approach, aimed at the enhancement of active learning among participants, instead of taking on the traditional role of teacher.

A noteworthy lesson was that the success of the programme lies greatly in the adherence to the designed protocols, as fewer of the anticipated outcomes were achieved when the protocol was neglected (Shirom et al. 2008). The majority of workshops included between 12 and 18 participants per group, as it was effective and economical. Ideally, delivery was guided by two training facilitators, as two were more capable of monitoring the behaviour and reactions of participants (Vinokur & Price 2015). Also, five half-day sessions, compared to longer two-week sessions, seemed to be more effective in keeping participants engaged. A vital lesson could be learned from the Finnish study that allowed for a debriefing day. During this session former appointed employees had an opportunity to deal with negative emotions caused by their dismissals (Vuori et al. 2005). Providing participants with such a venting opportunity may have made them more receptive to the programme.

Lastly, the founders of the JOBS programme strongly advised involving an effective champion, advocating for the programme at the policy level from the outset. It was also suggested that service delivery agencies be included that were open to applying innovative initiatives. Furthermore, a continuous flow of resources and funding seemed fundamental to the success and sustainability of large-scale programmes (Price & Vinokur 2014). In the South African context, economic development departments in local governments, supported by training providers, could act as champions of the JOBS programme.

This study also explored three elements (methodology, process and impact) related to the evaluation of the JOBS programme. The investigated studies were either conducted with a randomised field or quasi-experiment design as the chosen methodology, with self-reported questionnaires as the main data collection method. Considering the effectiveness of these designs in reporting the effectiveness and changes over time, it is suggested to use a similar approach. Furthermore, attrition was a pervasive problem experienced by most of the studies. However, offering incentives and recruiting more participants due to anticipated dropouts yielded higher attendance rates (Caplan et al. 1989; Shirom et al. 2008).

Aspects contributing to the process evaluation of the intervention included randomisation and manipulation checks of the studies included. To ensure internal validity, comparisons between the control and experimental groups’ demographic and other variables were tested for possible biases. Furthermore, the strength and integrity of the various interventions were assessed by means of self-reported questionnaires at the end of each workshop. Several benefits can be gained from delivering a programme that is valid and reliable. Firstly, as mentioned earlier, adhering to programme protocols is strongly recommended, as the intended outcomes are achieved through reliable practices. Secondly, ensuring that participants experience the programme positively has previously been shown to increase engagement and, consequently, has led to other outcomes, such as decreased depression and anger, increased internal control, job-seeking self-efficacy and self-esteem (Vinokur et al. 1995a; Vinokur & Schul 1997).

With regard to the JOBS programme’s impact, one of the most significant findings was the beneficial re-employment outcomes for those who had been unemployed for a moderate length of time (longer than a year; Brenninkmeijer & Blonk 2011; Vuori et al. 2002). Likewise, findings from examined literature also showed beneficial mental health and re-employment outcomes, particularly for high-risk participants (Vinokur & Schul 1997). These findings are valuable as it was found in a South African study that approximately 70% of the unemployed population was categorised as desperate or discouraged (Van der Vaart et al. 2018). The unemployed in both clusters generally came from poor socio-economic backgrounds, had relatively low levels of education, had limited opportunities for odd jobs or temporary employment, and were quite pessimistic. Given the capability of the JOBS programme to produce significant outcomes for high-risk participants, it appears that it could hold valuable outcomes, also for those who have been unemployed for long periods and may be truly discouraged.

Limitations and recommendations

Some limitations of this study need to be considered. Firstly, only peer-reviewed articles and book chapters that were written in English were included in the current study. Since the JOBS programme has been implemented in the Netherlands, Israel, Finland and China, where other official languages occur, the possibility of excluding potential articles exists. Secondly, access to some articles (Jobs in China project and Työhön trainers’ manual) was limited, or they could not be found, resulting in their omission from the review (i.e. Fang & Ling 2001; Mäkitalo, Tervahartiala & Saarinen 1997; Price 2001). In the third place, due to the nature of intervention studies, it is possible that only studies yielding significant results were published. Although all versions of the JOBS programme known to the developers were reported, it is possible that there may be unpublished efforts. Consequently, meaningful lessons that could have been learnt from these papers were not available. However, much effort was invested in systematically searching for and including all possible studies. Lastly, the study did not include articles where the JOBS programme had been applied in work-to-school and organisational contexts. Although these programmes may have yielded valuable findings, these studies were omitted, as the aim of this study was to specifically focus on the most effective methods to assist the unemployed.

Conclusion

This study reviewed literature about the JOBS programme as a means of extending our knowledge of applying such a job search intervention in a South African context. Therefore, core aspects regarding the implementation and evaluation of the JOBS programme and variations of it were investigated. Specifically, implementation features such as contextual factors, participant characteristics and programme aspects were studied, while evaluation features included impact and process evaluation components.

Evidently, the success of the JOBS programme largely depended on following the protocol. Thus, studying the previously performed methods and outcomes of the JOBS programme, in various contexts, may serve as a valuable guideline to prescribe possible best practices. The integration of included literature and important aspects regarding the South African context produced a framework that could be valuable in the implementation and evaluation of the JOBS programme in South Africa.

Acknowledgements

This work was supported by the Experiences of Unemployment Research Project funded by the Flemish Interuniversity Council – University Development Cooperation (VLIR-UOS). We are truly grateful for this opportunity.

This publication was based on the PhD thesis of R.P. H.D.W., S.R. and A.V.d.B. were co-authors as well as supervisors of the project. R.B. made conceptual contributions to the manuscript.

Flemish Interuniversity Council – University Development Cooperation (VLIR-UOS), ZEIN2013PR397.

The main aspects regarding the implementation and evaluation of the JOBS programme and variations of it were tabulated. A summary table can be requested from the first author. The literature review comprised 34 studies; these articles can be requested from the first author.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References

Barry, M., Reynolds, C., Sheridan, A. & Egenton, R., 2006, ‘Implementation of the JOBS programme in Ireland’, Journal of Public Mental Health 5(4), 10–25. https://doi.org/10.1108/17465729200600028

Brenninkmeijer, V. & Blonk, R.W., 2011, ‘The effectiveness of the JOBS program among the long-term unemployed: A randomized experiment in the Netherlands’, Health Promotion International 27(2), 220–229. https://doi.org/10.1177/1069072716657534

Caplan, R.D., Vinokur, A.D., Price, R.H. & Van Ryn, M., 1989, ‘Job seeking, reemployment, and mental health: A randomized field experiment in coping with job loss’, The Journal of Applied Psychology 74(5), 759–769. https://doi.org/10.1037/0021-9010.74.5.759

Choi, J.N., Price, R.H. & Vinokur, A.D., 2003, ‘Self-efficacy changes in groups: Effects of diversity, leadership, and group climate’, Journal of Organizational Behavior 24(4), 357–372. https://doi.org/10.1002/job.195

Collin, F., 2017, ‘NGOs trained to fill in for social workers’, Times Live, 21 November, viewed 21 February 2019, from https://www.timeslive.co.za/news/south-africa/2017-11-21-ngos-trained-to-fill-in-for-social-workers/

Curran, J., Wishart, P. & Gingrich, J., 1999, JOBS: A manual for teaching people successful job search strategies, University of Michigan, Institute for Social Research, Michigan Prevention Research Center, Ann Arbor, MI.

Donaldson, S.I., 2012, ‘Evaluation of the winning new jobs program’, in S.I. Donaldson (ed.), Program theory-driven evaluation science: Strategies and applications, pp. 93–112, Routledge, New York.

Donaldson, S.I., Gooler, L.E. & Weiss, R., 1998, ‘Promoting health and well-being through work: Science and practice’, in X.B. Arriaga & S. Oskamp (eds.), Addressing community problems: Psychological research and interventions, pp. 160–194, Sage, Thousand Oaks, CA.

Donaldson, S.I. & Gooler, L.E., 2002, ‘Theory-driven evaluation of the Work and Health initiative: A focus on winning new jobs’, American Journal of Evaluation, 23(3), 341–346.

Du Toit, M., De Witte, H., Rothmann, S. & Van den Broeck, A., 2018, ‘Unemployment experiences in context: A phenomenological study in two townships in South Africa’, Journal of Psychology in Africa 28(2), 122–127. https://doi.org/10.1080/14330237.2018.1454575

Fang, L. & Ling, W., 2001, Jobs in China: A seven city project, Institute of Psychology, National Academy of Sciences, Beijing.

Heckman, J.J., LaLonde, R.J. & Smith, J.A., 1999, ‘The economics and econometrics of active labor market programs’, in O. Ashenfelter & D. Card (eds.), Handbook of labor economics, pp. 1865–2097, Elsevier, Amsterdam. https://doi.org/10.1016/S1573-4463(99)03012-6

Independent Evaluation Group, 2013, Youth employment programs: An evaluation of World Bank and international finance corporation support, World Bank Publications, Washington, DC.

Jané-Llopis, E., Barry, M., Hosman, C. & Patel, V., 2005, ‘Mental health promotion works: A review’, Promotion and Education 12(2), 9–25. https://doi.org/10.1177/10253823050120020103x

Kim, B.Y., 2002, ‘The participation of Russian households in the informal economy. Evidence from the VTsIOM data’, Economics of Transition 10(3), 689–717. https://doi.org/10.1111/1468-0351.t01-1-00130

Lee, S.J. & Vinokur, A.D., 2007, ‘Work barriers in the context of pathways to the employment of welfare-to-work clients’, American Journal of Community Psychology 40(3–4), 301–312. https://doi.org/10.1007/s10464-007-9144-x

Mäkitalo, M., Tervahartiala, T. & Saarinen, M., 1997, Työhön Työhöohjelma Ohjaajan käsikirja [Työhön program trainers’ manual], Finnish Institute of Occupational Health, Helsinki.

Malmberg-Heimonen, I. & Vuori, J., 2005, ‘Activation or discouragement: The effect of enforced participation on the success of job-search training’, European Journal of Social Work 8(4), 451–467. https://doi.org/10.1080/13691450500314178

Marais, L. & Matebesi, Z., 2013, ‘Evidence-based policy development in South Africa: The case of provincial growth and development strategies’, Urban Forum 24(3), 357–371. https://doi.org/10.1007/s12132-012-9179-4

McCarthy, P., 2008, South Africa’s ‘door knockers’: Young people and unemployment in metropolitan South Africa, Centre for Development and Enterprise, Johannesburg.

Michigan Prevention Research Center (MPRC), 2013, The JOBS project for the unemployed: Update, Michigan Prevention Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI.

Miller, S., Wilson, D. & Hickson, D., 2004, ‘Beyond planning: Strategies for successfully implementing strategic decisions’, Long Range Planning 37(3), 201–218. https://doi.org/10.1016/j.lrp.2004.03.003

Price, R.H., 2001, ‘Winning new jobs in China. Preface’ in L. Fang & W. Ling (eds.), Jobs in China: A seven city project, Institute of Psychology, Chinese Academy of Sciences, Beijing, Republic of China.

Price, R.H., Choi, J.N., & Lim, S., 2006, ‘Beyond the iron rice bowl: Life stage and family dynamics in unemployed Chinese workers’, in M. Warner & G. Lee (eds.), Unemployment in China, pp.123–142, Routledge, London, UK.

Price, R.H., Choi, J. & Vinokur, A.D., 2002, ‘Links in the chain of adversity following job loss: How economic hardship and loss of personal control lead to depression, impaired functioning and poor health’, Journal Occupational Health Psychology 7(4), 302–312. https://doi.org/10.1037/1076-8998.7.4.302

Price, R.H. & Fang, L., 2002, ‘Unemployed Chinese workers: The survivors, the worried young and the discouraged old’, International Journal of Human Resource Management 13(3), 416–430. https://doi.org/10.1080/09585190110111459

Price, R.H., Friedland, D.S., Choi, J. & Caplan, R.D., 1998, ‘Job loss and work transitions in a time of global economic change’, in X. Arriaga & S. Oskamp (eds.), Addressing community problems, pp. 195–222, Sage, Thousand Oaks, CA.

Price, R.H., Van Ryn, M. & Vinokur, A.D., 1992, ‘Impact of preventive job search intervention on the likelihood of depression among the unemployed’, Journal of Health and Social Behavior 33(2), 158–167.

Price, R.H. & Vinokur, A.D., 2014, ‘The JOBS program: Impact on job seeker motivation reemployment, and mental health’, in U. Klehe & E.A.J. van Hooft (eds.), Oxford handbook of job loss and job search, pp. 575–590, Oxford University Press, Oxford, UK.

Ravallion, M., 2008, Evaluation in the practice of development, Working paper No. 4547, World Bank Publications, Washington, DC.

Reynolds, C., Barry, M.M. & Gabhainn, S.N., 2010, ‘Evaluating the impact of the winning new jobs programme on the re-employment and mental health of a mixed profile of unemployed people’, The International Journal of Mental Health Promotion 12(2), 32–41. https://doi.org/10.1080/14623730.2010.9721812

Richards, J.C. & Schmidt, R., 2002, Longman dictionary of applied linguistics and language teaching, Longman, Harlow.

Shirom, A., Vinokur, A. & Price, R., 2008, ‘Self-efficacy as a moderator of the effects of job-search workshops on re-employment: A field experiment’, Journal of Applied Social Psychology 38(7), 1778–1804. https://doi.org/10.1111/j.1559-1816.2008.00369.x

Statistics South Africa (Stats SA), 2017, Quarterly labour force survey: Quarter 1, 2017, viewed 21 January 2020, from www.statssa.gov.za/publications/P0211/P02111stQuarter2017.pdf.

Statistics South Africa (Stats SA), 2018, Quarterly labour force survey: Quarter 1, 2018, viewed 21 January 2020, from www.statssa.gov.za/?p=11129.

Statistics South Africa (Stats SA), 2019, Quarterly labour force survey: Quarter 3, 2019, viewed 21 January 2020, from www.statssa.gov.za/?p=12689.

Strandh, M., Winefield, A., Nilsson, K. & Hammarström, A., 2014, ‘Unemployment and mental health scarring during the life course’, The European Journal of Public Health 24(3), 440–445. https://doi.org/10.1093/eurpub/cku005

Tranfield, D., Denyer, D. & Smart, P., 2003, ‘Towards a methodology for developing evidence-informed management knowledge by means of systematic review’, British Journal of Management 14(3), 207–222. https://doi.org/10.1111/1467-8551.00375

Van der Vaart, L., De Witte, H., Van den Broeck, A. & Rothmann, S., 2018, ‘A psychosocial typology of the unemployed in South Africa’, South African Journal of Psychology 48(2), 179–192. https://doi.org/10.1177/0081246317721600

Van Ryn, M. & Vinokur, A.D., 1992, ‘How did it work? An examination of the mechanisms through which an intervention for the unemployed promoted job-search behavior’, American Journal of Community Psychology 20(5), 577–597. https://doi.org/10.1007/BF00941773

Vinokur, A.D. & Price, R.H., 2015, ‘Promoting reemployment and mental health among the unemployed’, in J. Vuori, R. Blonk & R. Price (eds.), Sustainable working lives. Aligning perspectives on health, safety and well-being, pp. 171–186, Springer, Dordrecht.

Vinokur, A.D., Price, R.H. & Caplan, R.D., 1991a, ‘From field experiments to program implementation: Assessing the potential outcomes of an experimental intervention program for unemployed persons’, American Journal of Community Psychology 19(4), 543–562. https://doi.org/10.1007/978-1-4615-0565-5_3

Vinokur, A.D., Price, R.H. & Caplan, R.D., 1996, ‘Hard times and hurtful partners: How financial strain affects depression and relationship satisfaction of unemployed persons and their spouses’, Journal of Personality and Social Psychology 71(1), 166–179. https://doi.org/10.1037/0022-3514.71.1.166

Vinokur, A.D., Price, R.H., Caplan, R.D., Van Ryn, M. & Curran, J., 1995a, ‘The jobs I preventive intervention for unemployed individuals: Short-and long-term effects on reemployment and mental health’, in L.R. Murphy, J.J. Hurrell Jr, S.L. Sauter & G.P. Keita (eds.), Job stress interventions, pp. 125–138, American Psychological Association, Washington, DC.

Vinokur, A.D., Price, R.H. & Schul, Y., 1995b, ‘Impact of the JOBS intervention on unemployed workers varying in risk for depression’, American Journal of Community Psychology 23(1), 39–74. https://doi.org/10.1007/BF02506922

Vinokur, A.D. & Schul, Y., 1997, ‘Mastery and inoculation against setbacks as active ingredients in the JOBS intervention for the unemployed’, Journal of Consulting and Clinical Psychology 65(5), 867–877. https://doi.org/10.1037/0022-006X.65.5.867

Vinokur, A.D. & Schul, Y., 2002, ‘The web of coping resources and pathways to reemployment following a job loss’, Journal of Occupational Health Psychology 7(1), 68–83. http://doi.org/10.1037/1076-8998.7.1.68

Vinokur, A.D., Schul, Y., Vuori, J. & Price, R.H., 2000, ‘Two years after a job loss: Long term impact of the JOBS program on reemployment and mental health’, Journal of Occupational Health Psychology 5(1), 32–47. https://doi.org/10.1037/1076-8998.5.1.32

Vinokur, A.D. & Van Ryn, M., 1993, ‘Social support and undermining in close relationships: Their independent effects on the mental health of unemployed persons’, Journal of Personality and Social Psychology 65(2), 350–359. https://doi.org/10.1037/0022-3514.65.2.350

Vinokur, A.D., Van Ryn, M., Gramlich, E.M. & Price, R.H., 1991b, ‘Long-term follow-up and benefit-cost analysis of the jobs program: A preventive intervention for the unemployed’, Journal of Applied Psychology 76(2), 213–219. https://doi.org/10.1037/0021-9010.76.2.213

Vuori, J., Price, R.H., Mutanen, P. & Malmberg-Heimonen, I., 2005, ‘Effective group training techniques in job-search training’, Journal of Occupational Health Psychology 10(3), 261–275. https://doi.org/10.1037/1076-8998.10.3.261

Vuori, J. & Silvonen, J., 2005, ‘The benefits of a preventive job search program on re-employment and mental health at 2-year follow-up’, Journal of Occupational and Organizational Psychology 78(1), 43–52. https://doi.org/10.1348/096317904X23790

Vuori, J., Silvonen, J., Vinokur, A.D. & Price, R.H., 2002, ‘The Työhön job search program in Finland: Benefits for the unemployed with risk of depression or discouragement’, Journal of Occupational Health Psychology 7(1), 5–19. https://doi.org/10.1037/1076-8998.7.1.5

Vuori, J. & Vinokur, A.D., 2005, ‘Job-search preparedness as a mediator of the effects of the Työhön job search intervention on re-employment and mental health’, Journal of Organizational Behavior 26(3), 275–291. https://doi.org/10.1002/job.308

Wanberg, C.R., 2012, ‘The individual experience of unemployment’, Annual Review of Psychology 63, 369–396. https://doi.org/10.1146/annurev-psych-120710-100500

Appendix 1

The context of the countries and states in which the JOBS programme have been implemented.

Appendix 2

The overall findings regarding the implementation and evaluation of the JOBS programme.

Original Research

A systematic literature review of the implementation and evaluation of the JOBS programme: A suggested framework for South Africa

Rachéle Paver, Hans De Witte, Sebastiaan Rothmann, Anja Van den Broeck, Roland Blonk

Abstract

Introduction

Research design

Research approach

Research method

Targeted body of literature

Search terms and selected criteria

Gathering the data

Analysis and presentation of the data

Assessment of methodological quality

Ethical considerations

Results

Implementation

Context

Participants

Biographical variables

Population, sample size and recruitment

Programme

Delivery

Content

Incentives

Facilitators

Duration of the programme

Group sizes

Venue of training

Stakeholder involvement

Evaluation

Methodology

Data collection method

Research design and data collection intervals

Process evaluation

Effectiveness of randomisation

Manipulation checks, integrity and strength of the intervention

Impact evaluation

Outcomes

Discussion

Limitations and recommendations

Conclusion

Acknowledgements

Competing interests

Authors’ contributions

Funding information

Data availability statement

Disclaimer

References

Appendix 1

Appendix 2