Lacking Accountability and Effectiveness Measures: Exploring the Implementation of Mentoring Programs for Refugee Youth

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

General Note:

For reasons of honesty and respect and to provide the author with helpful feedback, I would like to make my reading and evaluation of the draft as transparent as possible: First of all, I consider the chosen focus on organizational aspects and questions of steering and governance of youth mentoring programs for refugee youth in complex vertical networks to be extremely important. I support the statement on the research gap in mentoring studies fully, which is particularly evident in lines 186-192. The article is also clearly structured and provides a great deal of essential background information for understanding the specific regional/local phenomenon. However, I have comments and suggestions for improving the following points:

1. Presentation of the focus and anchoring of the research in scientific discourse:

I see two different ways of understanding the article and its underlying study.  In my first reading, the article primarily considers the inter-institutional forms of cooperation / action coordination in a kind of multi-level governance process between heterogeneous actors, in the given case public-nonprofit relationships, which are described as a ‘vertically complex network’ in the context of a project implementation with different power potentials. From this view, the article is therefore primarily about organizational questions of management and (reciprocal) steering mechanisms in the context of policy processes. In this light, the phenomenon of ‘Youth Mentoring for Refugees’ is primarily considered as an example or test case. The second interpretation is that the problem presented in the article arose in the practical field of the participating organisations or resulted from an engagement of the researcher with it, possibly in the context of an ongoing study, and the author then came to the conclusion that the questions of the empirical field were best examined and theorized in the context of an organizational consideration.

In my opinion, both are permissible and, as a critical mentoring scholar, I definitely welcome an organisational approach. However, the article gives the impression at key points that it is primarily about these mentoring programs – in the sense of the specific level of a mentoring organization or an initiative that then matches mentors and mentees in order to help them “integrate”. I suggest that it should be made clearer in the abstract (lines 4 to 18) that this is not the case, or not the main point of the study, and that the study fairly more offers a comprehensive consideration of inter-organizational coordination in the context of multi-level processes of providing support measures for young refugees, merely illustrated by the example of youth mentoring programming. Subsequently, the reference to the state of the art (Part 4: ‘Relevant Research’) does not  need to – or must not –  refer to the general mainstream discussion on the quality and precision of youth mentoring. This is because many of the studies mentioned here (e.g. Raposa et al. 2019) do not address such multilevel issues of service provision and performance, but rather seek to ‘measure’ effects at the individual level (usually that of the mentees), by using psychological concepts and quantitative pre-post designs. However, this state of the art in mentoring studies does not contribute anything to the research question posed here. In contrast to what is mainly researched, a critical examination of the results from broader organizational studies that examine the complex structure of ‘vertical networks’ is missing – especially when it is stated that there are hardly any or no results available on this perspective in the field of YM.                         

In light of this, the title could then also be formulated more precisely – and less strong value judgements (such as ‘Lacking accountability and effectiveness’) would then be appropriate, which, in my opinion, sound a little sensational.

2. Explanation of what is meant by YM in the US context

Even though the complex multi-level structure, which the article examines on 3 of 4 levels (levels 2 to 4), is described quite well, an introductory remark is missing as to what is actually meant by ‘Youth Mentoring’. In the US and Canada, the phenomenon is widespread, but not globally renown under this name. For example, in Europe, the terms ‘social mentoring’ and ‘mentoring for social inclusion’ are increasingly being used. To give the reader orientation I suggest that at the beginning (Section 1: Introduction), the core elements associated with a youth mentoring program in this specific case should be mentioned, at least briefly (presumably: YM here means a program or an organized initiative that matches and supports a volunteer, usually older or more experienced person (mentor) with a younger person or a person who has not yet been resident in the local area for so long (mentee), in a direct one-to-one relationship in which regular personal meetings take place over a period of several months – or something similar).

3. Relevant literature and careful reference of the empirical results to this in the discussion section

I have already written above that up to now the article does not present central findings from research on inter-organizational cooperation and on service provision in the context of welfare policies. Against the background of such research – e.g., organizational studies on social services and welfare policy - the empirical findings are relatively unsurprising. For example, it has been known for a long time since organizational studies researched into on the creation of ‘quasi-markets’ in the 1980s and 1990s that for organizations providing funding or channeling tax money to others in the chain (i.e. in the case in which private or not-for profit agencies work as intermediary organisations on behalf of the state) is always also concerned with the outflow of funds and transaction costs, i.e. the timely disbursement of funds to the specific providers at the end of the service chain. In this context – for example, when looking at theories on ‘louse coupling’ in heterogeneous actor networks – value judgements or strong statements such as that it is a matter of a ‘lack of program overview and control’ (quote from the article) should then be critically reconsidered. Based on the evidence presented in this study  an alternative interpretation could be that the actors, against the background of their respective reference systems, act extremely competently. The statement that the author makes implicitly and explicitly, namely that there is a lack of control to enable a ‘genuine’ and ‘correct’ service provision (which includes normative ideas about effectiveness and impact at the level of the service recipients at the end of the chain, where the young refugees find themselves), can maybe turn out differently.

4. Clear statement of the focus and limitations of ‘Discussion and Implications’ (last section)

As a rule, research, at least when it is more fundamentally designed and aims to produce ‘new knowledge’, aims to classify and discuss its own empirical results in the last part against the background of the state of the art, which is presented in advance. However, this only happens to a limited extent in part 8 ‘Discussion and Implications’. At the same time, several recommendations are given here that are aimed more at the actors in the examined field of practice (e.g. lines 456 to 459, lines 481 to 482). I think that in any case a scientific value must be pointed out and, in my opinion, a transfer of findings to practice is optional – an add-on. In any case, however, the positioning and intention of the author should be clearly stated. Furthermore, it should be reconsidered whether practical suggestions for action, such as how individual actors should ‘behave’ of what they should do, can really be implemented by these organizations. In my opinion, this is rather questionable in view of an organizational understanding of the phenomenon. Furthermore, the author's conclusions are strongly based on the assumption that all that is needed to help refugees is to design the right ‘effective programs’. However, we know that as soon as such ideas of effectiveness and measurement are presented to organisations in a service chain, they react reflexively to them, especially if their influx of resources is dependent on it. In the end, you end up with programs that produce “effective, measurable results” in line with the notion of effectiveness that is being validated. Whether this is ultimately in the interest of the end users (young refugees or mentees) remains questionable. So if a rather normative notion of effectiveness is introduced here and is considered the horizon for value judgements, then this should also be made transparent. This is because one could also come to completely different conclusions and ideas - based on organizational knolwedge: that it is ‘quite good’ to keep the various actors ‘at a distance’ (e.g. by means of ‘boundary objects’), so that each system is relatively autonomous in its own actions (e.g., the final delivery of a mentoring program). This would, for example, give youth mentoring programs the opportunity to use appropriate, reflective instruments for measuring “effectiveness” along progressive professional ideas and standards, e.g., by using participatory and action-oriented instruments.

5. Final remarks

I consider the empirical results of the thematic analysis to be quite exciting, especially for the phenomenon of youth mentoring in vertical networks or systems of action coordination. This could also advance the state of the art for the specific phenomenon of ‘youth mentoring programs for refugees’ – including internationally, provided that the writing is aimed more at an international, non-exclusively US readership. However, I would recommend a more comprehensive and thorough examination of studies and approaches that fit the chosen focus and approach, especially insights into the coordination of action in complex networks of heterogeneous actors, insights into inter-organisational governance in quasi-market-based systems for the provision of welfare state services and services for refugees and the like.

 

Author Response

Responses to Reviewer 1

Dear Reviewer,

Thank you for taking the time to read the manuscript and provide such thoughtful and helpful feedback. I have made several changes to the manuscript based on your feedback and recommendations. You made several points that have helped me gear this manuscript towards a wider international audience and have surely resulted in added value.

1. Presentation of the focus and anchoring of the research in scientific discourse:

I see two different ways of understanding the article and its underlying study.  In my first reading, the article primarily considers the inter-institutional forms of cooperation / action coordination in a kind of multi-level governance process between heterogeneous actors, in the given case public-nonprofit relationships, which are described as a ‘vertically complex network’ in the context of a project implementation with different power potentials. From this view, the article is therefore primarily about organizational questions of management and (reciprocal) steering mechanisms in the context of policy processes. In this light, the phenomenon of ‘Youth Mentoring for Refugees’ is primarily considered as an example or test case. The second interpretation is that the problem presented in the article arose in the practical field of the participating organisations or resulted from an engagement of the researcher with it, possibly in the context of an ongoing study, and the author then came to the conclusion that the questions of the empirical field were best examined and theorized in the context of an organizational consideration.

In my opinion, both are permissible and, as a critical mentoring scholar, I definitely welcome an organisational approach. However, the article gives the impression at key points that it is primarily about these mentoring programs – in the sense of the specific level of a mentoring organization or an initiative that then matches mentors and mentees in order to help them “integrate”.

Thank you for giving the manuscript such a thorough read. I have indeed studied the case of implementation of education related programs for refugee children from an organizational perspective. But as an educational policy scholar, my goal was to stay connected to the population for whom I am doing this work, the refugee children, and how the programs (which may be falling short since qualitative measures of outcomes are missing or do not get considered) may shape their integration into the country.

I suggest that it should be made clearer in the abstract (lines 4 to 18) that this is not the case, or not the main point of the study, and that the study fairly more offers a comprehensive consideration of inter-organizational coordination in the context of multi-level processes of providing support measures for young refugees, merely illustrated by the example of youth mentoring programming.

Thank you for your suggestion, I have incorporated your feedback by adding lines 7-9 in the abstract.     

Subsequently, the reference to the state of the art (Part 4: ‘Relevant Research’) does not  need to – or must not –  refer to the general mainstream discussion on the quality and precision of youth mentoring. This is because many of the studies mentioned here (e.g. Raposa et al. 2019) do not address such multilevel issues of service provision and performance, but rather seek to ‘measure’ effects at the individual level (usually that of the mentees), by using psychological concepts and quantitative pre-post designs. However, this state of the art in mentoring studies does not contribute anything to the research question posed here. In contrast to what is mainly researched, a critical examination of the results from broader organizational studies that examine the complex structure of ‘vertical networks’ is missing – especially when it is stated that there are hardly any or no results available on this perspective in the field of YM.    

Thank you for your important feedback. In the light of your comments, I have added to the Relevant research section and deleted the parts which discussed the youth mentoring programming itself. As to including an examination of vertically complex systems in public service, the studies are scares and mostly theoretical in nature and I have included those in the “Theoretical Framework” section.                  

In light of this, the title could then also be formulated more precisely – and less strong value judgements (such as ‘Lacking accountability and effectiveness’) would then be appropriate, which, in my opinion, sound a little sensational.

Thank you for your feedback. I do not disagree with your comment, however, I believe that these programs and the populations they serve are overlooked consistently, especially in organizational contexts. Maybe a little sensation is what we need for systematic change to occur.

2. Explanation of what is meant by YM in the US context

To give the reader orientation I suggest that at the beginning (Section 1: Introduction), the core elements associated with a youth mentoring program in this specific case should be mentioned, at least briefly (presumably: YM here means a program or an organized initiative that matches and supports a volunteer, usually older or more experienced person (mentor) with a younger person or a person who has not yet been resident in the local area for so long (mentee), in a direct one-to-one relationship in which regular personal meetings take place over a period of several months – or something similar).

This is really helpful and I have not thought about it. I have added a referenced description of youth mentoring. Please see lines 149-152.

3. Relevant literature and careful reference of the empirical results to this in the discussion section

I have already written above that up to now the article does not present central findings from research on inter-organizational cooperation and on service provision in the context of welfare policies. Against the background of such research – e.g., organizational studies on social services and welfare policy - the empirical findings are relatively unsurprising. For example, it has been known for a long time since organizational studies researched into on the creation of ‘quasi-markets’ in the 1980s and 1990s that for organizations providing funding or channeling tax money to others in the chain (i.e. in the case in which private or not-for profit agencies work as intermediary organisations on behalf of the state) is always also concerned with the outflow of funds and transaction costs, i.e. the timely disbursement of funds to the specific providers at the end of the service chain. In this context – for example, when looking at theories on ‘louse coupling’ in heterogeneous actor networks – value judgements or strong statements such as that it is a matter of a ‘lack of program overview and control’ (quote from the article) should then be critically reconsidered. Based on the evidence presented in this study  an alternative interpretation could be that the actors, against the background of their respective reference systems, act extremely competently. The statement that the author makes implicitly and explicitly, namely that there is a lack of control to enable a ‘genuine’ and ‘correct’ service provision (which includes normative ideas about effectiveness and impact at the level of the service recipients at the end of the chain, where the young refugees find themselves), can maybe turn out differently.

Thank you so much for your discussion and understanding of my findings. I agree with the statement that “an alternative interpretation could be that the actors, against the background of their respective reference systems, act extremely competently,” which is why I highlight that both organizations were measuring outcomes differently and there is a need for more input from the lead organization on defining outcomes for each organization implementing YM. Indeed, the timely disbursement of funds is important to the funder, however in addition to that, a funder that issues a grant every year, should then establish key performance measures for grantees to meet. Especially in this case where the monitoring of the grantees is the responsibility of middle level organizations (as they currently ensure compliance with federal requirements because the leading agency requires that). My proposal for increased involvement from the lead agency at least in terms of defining outcomes does not require recurring dedication of resources. On the contrary, it might become easier for middle level agencies to monitor the implementers if they are measuring outcomes based on a set checklist, rather than receiving and interpreting several different forms of performance measures from different grantees of the same program.

4. Clear statement of the focus and limitations of ‘Discussion and Implications’ (last section)

As a rule, research, at least when it is more fundamentally designed and aims to produce ‘new knowledge’, aims to classify and discuss its own empirical results in the last part against the background of the state of the art, which is presented in advance. However, this only happens to a limited extent in part 8 ‘Discussion and Implications’. At the same time, several recommendations are given here that are aimed more at the actors in the examined field of practice (e.g. lines 456 to 459, lines 481 to 482). I think that in any case a scientific value must be pointed out and, in my opinion, a transfer of findings to practice is optional – an add-on.

Thank you for your important feedback. Please see changes to the discussion section (lines 453-463 and 482-484).

In any case, however, the positioning and intention of the author should be clearly stated.

Please see lines 430-433 where I have added positionality statements.

Furthermore, it should be reconsidered whether practical suggestions for action, such as how individual actors should ‘behave’ of what they should do, can really be implemented by these organizations. In my opinion, this is rather questionable in view of an organizational understanding of the phenomenon. Furthermore, the author's conclusions are strongly based on the assumption that all that is needed to help refugees is to design the right ‘effective programs’. However, we know that as soon as such ideas of effectiveness and measurement are presented to organisations in a service chain, they react reflexively to them, especially if their influx of resources is dependent on it. In the end, you end up with programs that produce “effective, measurable results” in line with the notion of effectiveness that is being validated. Whether this is ultimately in the interest of the end users (young refugees or mentees) remains questionable. So if a rather normative notion of effectiveness is introduced here and is considered the horizon for value judgements, then this should also be made transparent. This is because one could also come to completely different conclusions and ideas - based on organizational knolwedge: that it is ‘quite good’ to keep the various actors ‘at a distance’ (e.g. by means of ‘boundary objects’), so that each system is relatively autonomous in its own actions (e.g., the final delivery of a mentoring program). This would, for example, give youth mentoring programs the opportunity to use appropriate, reflective instruments for measuring “effectiveness” along progressive professional ideas and standards, e.g., by using participatory and action-oriented instruments.

Thank you so much for your feedback. I can see that one could reach completely different conclusions and form other opinions of my research, which are beyond the scope of my work in this paper. However, in my paper, I plan to elaborate on key takeaways and suggestions, which I find most relevant to the current issues in serving refugee populations.

Furthermore, based on the helpful feedback, I have revised the language of policy and programmatic recommendations for international audience.

5. Final remarks

I consider the empirical results of the thematic analysis to be quite exciting, especially for the phenomenon of youth mentoring in vertical networks or systems of action coordination. This could also advance the state of the art for the specific phenomenon of ‘youth mentoring programs for refugees’ – including internationally, provided that the writing is aimed more at an international, non-exclusively US readership. However, I would recommend a more comprehensive and thorough examination of studies and approaches that fit the chosen focus and approach, especially insights into the coordination of action in complex networks of heterogeneous actors, insights into inter-organisational governance in quasi-market-based systems for the provision of welfare state services and services for refugees and the like.

Thank you so much for your feedback. I have made many changes to incorporate the valuable feedback.

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the opportunity to review the paper titled “Lacking Accountability and Effectiveness Measures: Exploring the Implementation of Mentoring Programs for Refugee Youth”. The current paper analyses how a federally funded program funded by the YM grant is implemented in one Texas city. Overall, this study is interesting and is a relevant contribution to the literature on refugee resettlement while focusing on younger individuals. However, there are a few flaws in the manuscript that need to be addressed. I have provided some comments below that I hope will help the authors.

 Introduction

The introduction does a great job of outlining what the paper will focus on and the questions the paper intends to address.

There was some repetition in the first few sections that can be streamlined to prevent redundancies.

 Methods

The methodology was clearly explained and appropriate for this type of study. There were a good number of participants, and one can see how they can speak about the issue being studied. Additional key informants could help future papers allow for more perspectives.

 Results

The results were presented in a logical fashion.

 One main issue to be addressed is the block quotes. There needs to be some indentation since they can get lost in the flow of the paper and it makes for a difficult read, making the reader go back and see if they were reading a quote. For example, in theme 2 the third paragraph is a block quote but is not delineated as such. Same with the paragraphs starting on line 357 and 381.

 Discussion

The discussion and conclusion were clearly stated and outlines the significance of the research and its contribution to the larger discussion about refugee resettlement.

Author Response

Responses to Reviewer 2

Thank you so much for your consideration of my paper. I appreciate your time and feedback.

Introduction

There was some repetition in the first few sections that can be streamlined to prevent redundancies.

Thank you for your important feedback, I re-read the paper and given the limitation of space and in the interest of giving enough background to readers, I have decided to keep it as is.

Results

One main issue to be addressed is the block quotes. There needs to be some indentation since they can get lost in the flow of the paper and it makes for a difficult read, making the reader go back and see if they were reading a quote. For example, in theme 2 the third paragraph is a block quote but is not delineated as such. Same with the paragraphs starting on line 357 and 381.

Thank you for bringing my attention to this issue. I have addressed it.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

I can see that a number of the recommendations from my peer review have been incorporated. In this respect, the article has improved and is more publishable. However, I still do not consider the contribution to the state of scientific research to be particularly high. I have therefore continued to answer the question “Contritubion to Scholarship” with “low” because the article is still aimed at gaining knowledge with a view to practice. Above all, recommendations are formulated for the actors involved in the practice examined.
In my opinion, the responsibility for making a decision on this lies with the editor. The publisher MDPI does not make it clear in its explanations of the “aim and scope” what exactly the journal is aiming at. Nor did the editorial team or editorial board provide any comments of their own in response to my comments. I will therefore refrain from making a final decision, but will state “Accept in present form” as the final verdict so that the editor can decide for themselves.