Lacking Accountability and Effectiveness Measures: Exploring the Implementation of Mentoring Programs for Refugee Youth

Abstract

A high proportion of refugees coming into the U.S. are under the age of 18, and many continue to struggle to attain basic educational qualifications which may lead to viable economic opportunity. Recently, the Office of Refugee Resettlement (ORR) began issuing youth mentoring (YM) grants to support the integration of refugee youth. This paper explores the decision-making processes of resettlement organizations in a vertically complex network of organizations which implemented the YM programs for refugee youth in one large city of Texas. Through a series of semi-structured interviews with program implementers, findings suggest that resettlement agencies remain powerless in making key programmatic decisions, including implementation of the programs and designing accountability and effectiveness measures, which can truly evaluate the program quality. Other findings suggest that across the network of organizations involved in implementation of the YM programs, compliance with federal regulation is prioritized over accountability. The lack of program effectiveness and accountability measures have implications for one of the most vulnerable population, refugee youth. The main recommendation for policy and practice argues for the lead public agency’s increased involvement in defining key outcomes for the program recipients in a vertically complex network.

1. Introduction

The United States (U.S.) has world’s first and largest federal refugee resettlement program which was established under the Refugee Act of 1980. Since then, the U.S. has resettled more than 3.7 million refugees and continues to resettle more each year (Baugh 2022). The government-sponsored resettlement activities are carried out under the Office of Refugee Resettlement (ORR) which in turn relies entirely on nine voluntary agencies (volags) and resettlement agencies for all resettlement related activities. In addition to initial resettlement support such as housing, job provision, and enrolling children in schools, these organizations also provide a wide array of other support services to adult refugees, including job training, English language learning, and legal services. Recently, research has criticized the hyper-focus of refugee resettlement on economic self-sufficiency (Mairaj 2022; Breitkreuz and Williamson 2012; Renfroe 2020), which inevitably results in the refugee resettlement program’s resources being focused on refugee adults’ economic outcomes. As evidenced by the annual refugee report presented to congress, only refugee adults’ outcomes are documented and reported by the ORR. Since a major proportion of those resettled are children and young adults, research shows that many continue to struggle years after resettlement to attain basic educational qualifications, such as a high school diploma, that may lead to better socioeconomic outcomes (Hos 2016; McKinnon 2008; Shakya et al. 2010). The ORR made available the Youth Mentoring grant, which is aimed at supporting the integration of refugee youth (ORR 2018). Similar to most adult programs, the programs under the YM grant are mostly implemented through nonprofit resettlement agencies (Mairaj 2022).

Even though it is a federally funded program administered by the ORR, little is known about the program’s implementation processes, outcomes, and its effectiveness, as the ORR does not report any outcomes on programs implemented under the YM grant (Mairaj 2022). This study fills a gap in understanding how multiple nonprofit organizations work together to implement programs under the YM set-aside, how they make program-related decisions, and they report to the ORR. Texas presents a favorable setting for such an analysis as it resettles one of the highest numbers of refugees in the U.S. (Ward and Batalova 2023).

Through analysis of the Youth Mentoring (YM) set-aside program—an ORR program aimed at supporting refugee youth—this study seeks to describe the inner workings of local resettlement agencies and nonprofit organizations in one large Texan city as they implemented federally funded refugee programs. The study aims to understand how the local resettlement agencies made decisions to implement the YM program and ensure effectiveness and accountability. In particular, the study posed the following research questions:

In a complex implementation structure of federally funded refugee YM programs in Texas,

(1) 

how are programs first established,

(2) 

how is program effectiveness measured and reported, and

(3) 

who is held accountable for program success?

Through the concept of vertical complexity in a network of organizations, I explore the decision-making processes of local resettlement agencies in one large Texan city in their implementation of youth programs. Through a series of targeted interviews with actors from resettlement agencies and other nonprofit organizations in Texas, I seek to demystify the role of one of the oldest public–private resettlement structures in the world and show how the organizational practices shape youth mentoring programs for most vulnerable refugee youth.

2. Theoretical Framework

This study is framed by the concept of vertical complexity, which describes the organizational layers and hierarchy that exist in a service delivery model within a network (Tolbert and Hall 2016). In a vertically complex system, decision-making power and supervisory duties are divided between member organizations on the basis of their position in the hierarchy, whereas in a horizontally complex network, tasks are divided between member organizations on the basis of organizational skills and service-area expertise (Vermeiren et al. 2021). In vertical complexity, as more layers of formal authority are established, decision-making is distributed between the top or lead public agency and the implementers of the programs (Bratton and Chiaramonte 2006).

Multiple service implementation structures including public–nonprofit networks are vertically complex, and decision-making processes may vary across different networks. For the purposes of this research, I consider the refugee resettlement network in the United States as a public–nonprofit network, where the ORR is the lead public agency. Within most public–nonprofit networks, the leading public agency is often responsible for tasks such as network administration, facilitating member organizations in achievement of goals, and seeking and controlling access to government funding (Vermeiren et al. 2021). Contrastingly, in other such networks, decision-making is based on consensus between participating member organizations without the interference of one leading agency. In both cases, vertically complex networks may experience a considerable number of challenges in decision-making, including slower and poorer decision-making and limited decision-making abilities of lower-level organizations (Pugh et al. 1968), in this case the resettlement agencies.

Another challenge of a vertically complex structure is accountability, which is often determined by the history of collaboration between the network participants and the leading agency (Vermeiren 2018). Past positive collaboration allows lower-level organizations to trust the leading organization to take full responsibility for the decisions that are made in the network. Past negative collaboration, on the other hand, elevates the role of “network coordinators” to a facilitating or coproducing governance role so the network can achieve higher levels of accountability (Vermeiren 2018). Network coordinators are especially important to ensure effective functioning, as they share the decision-making burden of the top decision-makers and facilitate the flow of information and how it is processed across the network (Vermeiren et al. 2021). For this study, I explore a vertically complex public–nonprofit network where the ORR is the lead public agency. Although the local resettlement agencies in this network implement most resettlement-related activities across the United States, little is known about the decision-making processes, measures for program effectiveness, or the systems in place to ensure accountability.

3. Background

The Refugee Act of 1980—created to provide support to the large influx of refugees resettled in urban areas during the Southeast Asian refugee crisis borne of the conflict in Vietnam—was the last and most comprehensive reform in refugee resettlement policy (Tempo 2008). With the passage of the Refugee Act, the U.S. government adopted the United Nations’ definition of a refugee1 and formalized refugee resettlement by establishing the ORR under the U.S. Department of Health and Human Services (Holman 1996). Amid the increased privatization of public services across most levels of government during the 1970s and 1980s, the Refugee Act was crafted to rely heavily on public–private partnerships to deliver public goods such as food, housing, education, and healthcare to the growing refugee community.

The structure of public–private partnerships providing goods and services to refugees post-resettlement is vertically complex. At the highest level is the lead public organization, the ORR. The ORR provides program funding to the state, or to a state designee, when the state does not choose to participate in federal refugee resettlement as is the case in Texas. Currently, a replacement designee, or RD (which is an affiliate resettlement agency) serves in the state’s capacity in Texas. The focus of this study is on the vertical complexity of one particular program funded by the ORR, the Youth Mentoring (YM) set-aside, as executed in one large Texan city which is also a major resettlement destination for refugees.

3.1. Youth Mentoring Set-Aside

Refugee Support Services (RSS) is one of the programs administered by the ORR and predominantly funds programs to support the employability of refugee adults, such as job development training, vocational training, and skills recertification (ORR n.d.). The YM set-aside program, a recent addition to RSS, has been receiving funds specifically ear-marked to support the educational and social integration of refugee youth since 2019 (ORR 2018). I focus on the YM set-aside because of the promise it offers to refugee youth—perhaps the most vulnerable group within the refugee population.

3.2. YM Set-Aside in Texas

For this study, I focus on the vertically complex network of the YM set-aside program in one large Texan city that is a major resettlement destination for refugees. Because the state withdrew from the federal refugee program in 2016 and replacement designees were selected to serve in its place, Texas provides an opportunity to explore the ramifications of this policy decision. Before the state withdrew, it employed four RDs to oversee resettlement activities in Texas.2 Upon the receipt of such grants as the YM set-aside under RSS, these designees oversee the disbursement of funds to qualifying resettlement agencies. How and why organizations are selected for grants is unknown. The RDs serving in place of the state add to the vertical complexity of the network of organizations used for service delivery, further separating the federal government and the state from service delivery and, subsequently, any direct accountability.

4. Review of the Relevant Research

This study explores the decision-making, measurement, and accountability involved in the delivery of YM services to refugee youth under the YM set-aside program. Youth mentoring, specifically in the context of the U.S., is defined as the practice of assigning a mentor to a young individual who will contribute to the youth’s personal development and help with improving educational outcomes (Mentorink 2024). The inception and growth of the Big Brothers Big Sisters of America mentoring program led to the popularity and mass adoption of YM programs across many sectors in the U.S.; at-risk youth or young people from marginalized communities were especially targeted for such programs (Allen et al. 2006; Preston et al. 2019). For the refugee community specifically, research has suggested that mentoring can facilitate acculturation and social integration and may promote academic and school engagement (Koyama and Ghosh 2018; Oberoi 2016).

All of the programs funded by the ORR to support refugees’ social integration or economic outcomes are implemented through similar vertically complex organizational networks, where the ORR, as the lead public organization, sits at the top and several organizational layers away from the actual program (Mairaj 2022). Milward (2014) refers to it as the hollow state, where clear accountability for program implementation disappears and any potential command and control relationship ceases to exist. Consequently, recent research on ORR-managed programs for refugees has criticized the limited state and federal oversight of programs (Afkhami and Gorentz 2019). On the other hand, research has also documented the existing financial accountability- and compliance-related burdens on the nonprofit organizations in such networks, which may lead to inadequate services provided to refugees. While exploring the organizational structure of a refugee resettlement agency in California, Fee (2019) found that there were extensive reporting burdens placed upon the organization involved with providing services to refugee adults.

For the nonprofits implementing public services, there is an increasing demand for demonstrating performance accountability, alongside other dimensions of accountability such as financial accountability and legal accountability (Smith 2014). While exploring the impact of YM programs, DuBois and Karcher (2013) highlighted the lack of evaluation against key outcomes and the lack of research on whether the benefits for youth are sustained at later points in their development. Another study evaluated a YM program designed specifically for adolescent refugee girls and found that 8 of 14 program outcomes identified as important indicators of program success were not assigned specific, measurable indicators, leaving the results inconclusive (Rao 2014).

Research exploring the mechanisms behind the implementation and overall effectiveness of programs that support refugee youth and children is incredibly limited. While recent research on ORR-funded programs supporting refugee youth in and outside of schools has acknowledged the involvement of volags and resettlement agencies in service delivery (Mairaj and Callahan 2022; Cancino and Cruz 2019; Koyama and Ghosh 2018; Mendenhall and Bartlett 2018), these studies do not fully explore program decision-making, accountability, or effectiveness. This study seeks to fill the gap in the literature by illustrating how one public–nonprofit network of organizations led by the ORR worked together to implement the YM programs for refugee youth and what roles they assumed in this network to ensure the implementation of quality programs.

5. Methodology and Methods

I used a single-embedded case study design to study the refugee resettlement structure in one large city in Texas and the ways in which federally funded YM programs were implemented. In particular, I identified subunits for analysis within the original case (Yin 2018). Several distinct organizational levels comprised the subunits that formed the entire case, as shown in Figure 1. Specifically, I explored Level 2, the replacement designee (RD), as well as the organizations at Level 3 (local resettlement agencies) and Level 4 (local subcontracted nonprofits with expertise in youth mentoring). At the time of the study, the organizations at Levels 2 through 4 were directly involved in subcontracting for or actively implementing refugee educational programs. The large Texan city selected for this study is home to one of the largest refugee communities in the United States and was served by the affiliates of four of the nine national volags. These two key characteristics offer reason to believe that considerable refugee resettlement-related activities and subsequent decision-making took place in the city, making it not only a suitable but also an interesting case to study.

Within the focal city, four prominent resettlement agencies (Level 3) were identified and two were chosen because they were grantees for the city’s YM services and also subcontracted with two other local nonprofit organizations (Level 4) to implement the programs germane to the inquiry. Figure 1 highlights how the grant was disbursed through the vertically complex structure.

5.1. Participants

I used convenient purposeful sampling for the organizations in Levels 2, 3, and 4, initially approaching most participants through existing contacts I had established while interning at the local school district’s refugee services office and through LinkedIn.

Table 1. Overview of participants and organizations.

Elite Participants
S. No.	Pseudonym	Organization Type	Organization Level	General Position
1	Alexis	Replacement Designee (RD)	L2	Director
2	Bea	Resettlement Agency (Local Affiliate of National Volag)	L3	Director
3	Becca	Resettlement Agency (Local Affiliate of National Volag)	L3	Manager
4	Bianca	Resettlement Agency (Local Affiliate of National Volag)	L3	Manager
5	Blair	Resettlement Agency (Local Affiliate of National Volag)	L3	Director
6	Bella	Resettlement Agency (Local Affiliate of National Volag)	L3	Manager
7	Camila	Sub-contractor (of Resettlement Agency)	L4	Director
8	Clara	Sub-contractor (of Resettlement Agency)	L4	Director
Key Informants
	Pseudonym	Organization Type	Organization Level	General Position
1	Zorro	NA	NA	Retired
2	Zara	VOLAG	L2	Director

provides an overview of the participants and their distribution across the three organizational levels included in the study. I assigned pseudonyms to all the participants. The column “General Position” in Table 1 does not identify participants’ specific positions or titles at their respective organizations; rather, I assigned equivalent and more generic position titles to protect participants’ confidentiality. For example, the title “manager” entails positions ranging from a program manager to a program coordinator, and the title “director” refers to positions ranging from executive director to program director. All elite participants were women and had completed an undergraduate degree, and most held a master’s degree. One of the key informants was male and the other female; both possessed extensive experience at various levels of the refugee resettlement structure in the United States.

5.2. Data Collection

It is important to note the researcher’s positionality here as a Muslim, cisgender woman of color and an outsider to the refugee resettlement structure under study. Therefore, semi-structured elite interviews and key informant interviews were chosen as the main method of data collection to generate insight and discussion. Elite interviews are an especially relevant method of data collection when individuals or employees interviewed have considerable decision-making power over policies or program implementation (Natow 2020). For the purposes of this research, I identified all employees of resettlement agencies and those of subcontracted nonprofit organizations as elites, as they all possessed decision-making power on their level and influenced service implementation (Natow 2020). Furthermore, I identified and recruited the key informants based on their extensive experience in the refugee resettlement sector.

The key informants included in this study had extensive experience working at different organizational levels within the refugee resettlement structure in the United States. Notably, I interviewed the key informants twice during the course of the study to triangulate findings: first after the majority of data had been collected, when a few overarching themes materialized; and then following the data analysis at a point when themes were identified. Both elite participants and key informants were asked questions regarding decision-making while implementing YM programs. I further utilized key informants to ensure trustworthiness of the data, codes, themes, and findings of a larger study of the nonprofit resettlement structure in Texas. In the second interview with the key informants, I discussed my insights, identified themes and my overall perceptions of the data, and, upon their assurances, finalized the themes. To further ensure trustworthiness, I utilized member checking and carefully triangulated insights and findings among all participants. In addition, I reviewed publicly available documents issued by the ORR, such as policy memos, policy letters, grant and set-aside announcements, and the annual reports to triangulate and enhance my findings.

5.3. Data Analysis

I employed thematic analysis as my primary analytic tool (Clarke and Braun 2014; Nowell et al. 2017). In my research, I used codes as groups to meaningfully organize the data (Braun and Clarke 2006) and capture its interesting features (Nowell et al. 2017). In the first cycle of coding, I inductively generated codes, and following the first round, I repeated the coding cycle a second time using both inductive and deductive methods to ensure proper identification and grouping. At this stage, new codes were also developed, and some previous codes were either combined or were sub-grouped. I generated the four themes inductively by grouping codes together; the most important consideration for a theme was to assess whether it fit into the story about the entire data set in relation to the research question.

6. Limitations

The greatest limitation to the present study lies in the lack of ORR representation as the leading agency in the vertically complex network. Despite multiple attempts, I was unable to secure the participation of an ORR representative. Instead, I captured the role of the federal government, the ORR, via the perceptions of the non-state actors working in organizations charged with implementing educational programs for young refugees.

7. Insights and Findings

In the following section, the findings are organized by theme to tell the multifaceted story of decision-making in the YM program implementation processes of one of largest refugee-receiving cities in the U.S., as studied in 2021–2022.

7.1. Theme 1: Funding Motivates Program Development

Once the ORR determines the availability of funds for a specific program, it makes those funds available through an announcement that broadly outlines how the funds can be used. “It was a policy memo that came out from ORR saying you’ve been awarded X dollars for this specific purpose and reason, it is now on you to determine how you wish to execute or utilize those dollars,” said RD director Alexis. She continued to explain that the ORR offers few specific details about program content and deliverables, leaving program design largely to the lower-level organizations that will implement the program: “There were four sort-of areas: academic, civic engagement, vocational, and career laddering … and, again, they didn’t have to do all four; they could self-select or they could identify [other areas]”.

Lower-level organizations (resettlement agencies and subcontractor nonprofits) appeared to enjoy significant freedom in designing programs to support refugee youth under the educational support grants. One interpretation could suggest that these lower levels were granted this freedom because they were most closely involved with refugees and may have been most aware of unmet community needs. However, the evidence suggests a different causal pathway: these agencies developed programs in direct response to the available funding rather than community needs. For example, Blair, a resettlement agency director, discussed their longstanding awareness of the need for a youth program: “Refugee youth mentoring program is an ORR program; they really determine what’s going to be funded and what’s not, but we also at the ground level did see a need for this and we have seen a need for this for many years”. Despite acknowledging an apparent longstanding need for a program dedicated to older teens and young adults, Blair suggested that such programs were only designed and implemented in response to the funding announcement.

Prior to the YM funding announcement, resettlement agencies did not prioritize forming community ties to implement youth mentoring programs; it was only following the YM set-aside funding announcement that they began to develop and implement such programs. Two resettlement agency actors discussed how the youth mentoring program became a part of their portfolio of services for refugees: Manager Bella noted that “youth mentoring was our newest one back in 2019. They [RD] gave us … a proposal … that says, hey, there’s some new funding for this grant, do y’all want to apply for it?” Bea, a director, added, “They [RD] were the ones who said that we have funding for youth mentoring and if you want to, apply”. This behavior of resettlement agencies suggests that the refugee resettlement structure was restrictive and that they were not as independent in terms of implementing and designing new programs as one might expect from the decentralized structure and lack of program oversight and control.

7.2. Theme 2: Checking for Compliance Versus Accountability

The RD acting in lieu of the state of Texas for this Texan city was the recipient of YM grants. My findings suggest that this RD assumed the role of network coordinator in this vertically complex structure in three ways: It acted as the steward, mediator, and catalyst. As steward, the RD facilitated collaboration within the network, and as mediator, it facilitated collaboration and arbitrated information and documentation exchanges between the ORR and resettlement agencies (Ansell and Gash 2012). As catalyst, the RD helped resettlement agencies identify and utilize value-creating opportunities (Ansell and Gash 2012). Alexis discussed the RD’s role as mediator: “The money [YM] needs to be programmable to meet a myriad of employment, education, ELL, whole bunch of needs and needs to be programmed by the state [in Texas, the RDs] out to providers to meet the needs for up to five years”. Alexis said that the RD not only mediated the flow of dollars but also the information exchange between the ORR and resettlement agencies. In its role as an intermediary, the RD outlined for resettlement agencies what types of programs could be funded with the available funds. In this way, the RD shared the burden of the leading organization, the ORR, by defining for the resettlement agencies what was “programmable” with the awarded funds.

In addition to facilitating the funding and directives from the ORR to resettlement agencies, the RD also provided technical programmatic support and oversight, which included handling all of the programmatic reporting by the resettlement agencies. When discussing the guidelines that the RD provided to resettlement agencies to ensure accountability and transparency for the YM programs, Bea, a resettlement agencies director, discussed the availability of a program manual that outlines who can be provided services for certain grants and what the programs can look like under specific grants:

They [RD] do have one program manual, which is very helpful. It is … intended to, you know, at any level in the agency, somebody should be able to pick it up to know who is eligible [for services], what kind of documentation they [refugees] need, what kind of services they [refugees] can be enrolled in, and what those services look like as well like how extensive or not. They are the program guidelines by each program. It also discusses accountability and reporting structure, so what does our documentation need to look like, minimum standards [e.g.,], when do we have to report what we’re doing [and] the progress we’re making to the RD.

The above quote highlights the key role that the RD has taken in this structure to ensure compliance and accountability for lower-level organizations. Zara, one of the key informants who served as a director at a major national volag, described her role as also being a network coordinator while managing grants and grantees for refugee services:

I manage, like, the compliance for all of our federally funded programs, and my job entails monitoring for compliance through various methods, like organizational assessments to see how our partners [Level-3 resettlement agencies] are doing, where we can better support them.

When asked what “monitoring for compliance” means, Zara responded: “So, realistically speaking, it’s essentially [that] we prioritize compliance for what the federal government’s program standards are. That’s the baseline, [e.g.,] what our partners [are] doing, what they’re required [to do per] the federal programs guidelines”. In this way, the RD is ensuring that the resettlement agencies meet basic accountability criteria, which only entails meeting compliance standards provided by the ORR. This evidence suggests that the RD’s concerns are centered on ORR’s baseline requirements and that the desire to ensure the effectiveness of the programs is not a prioritized task for the RD, arguably because it is also not an ORR priority. Therefore, compliance with federal regulations overshadows accountability for program success.

7.3. Theme 3: Autonomy in Evaluating Outcomes

My findings suggest that since the ORR, as the grantor of these funds, does not require qualitative outcomes, it falls to the organizations implementing the programs to determine what outcomes to measure and report. Alexis, an RD director, summarized the lack of outcomes measured for YM: “For ORR, they currently do not request qualitative measures or real true outcome measures; they want quantity measures”. On the other hand, my data show that Level 4 organizations—those ultimately implementing the YM programs—conducted intermittent pre- and post-program surveys to measure YM program outcomes and effectiveness. Camila, a local nonprofit (subcontractor) director, discussed the different output and outcome measures assessed by their youth mentoring program:

There [are] the outputs that we’re collecting: how many mentors are enrolled, how many mentees are enrolled, how many received training, how many matches [with mentors] were made. So, you’ve got those kind of output metrics. We [also] look at things like how many demonstrated a change in attitude about various aspects, like being in another country or their attitudes about how they feel about school or academic success. So we’re capturing those kinds of outcome measures by surveys; [we] pre- and post-test them to measure, you know, what the difference is when they first came into the program and then, 3–6 months into the program, how they’re responding to it.

Clara, another local nonprofit (subcontractor) director, added: “We’ll do a pre-test question and a post-test question to measure the impact of that activity [or for a workshop] only for the group of kids that were there that day”. Here we see that, when subcontracted, local nonprofits measured not only the quantity of impact (i.e., counts of the numbers of participating students) but also the quality of the program (i.e., how it shapes experiences and outcomes for refugee youth participants). Another detail to note here is that while both local nonprofit organizations were implementing programs under the same grant, they were measuring their outcomes differently. Camila’s nonprofit measured the impact of the program over time, while Clara’s nonprofit measured the impact of one singular activity. This disaccord in measurements of outcomes under the same grant and grantor highlights how organizations are given significant autonomy over measuring program effectiveness and deciding on which outcomes to measure.

Once the local nonprofits collected these outcomes, they shared them with their main contractor (resettlement agency): “What we report to [them] gets wrapped into their report that goes to the [RD],” said Clara. From a structural perspective, local nonprofits collected measures of the quality and quantity of program effectiveness and shared those reports with their contracting resettlement agency. The resettlement agencies then made their own report including the outcomes reported by local nonprofits along with their own outcomes; this report was then sent to the RD.

As for what the resettlement agencies reported to the RD, manager Bianca said: “We’ve proposed that we are going to serve 40 clients total. Twenty for the in-school cohort and 20 for the out-of-school cohort. The other outcome is the [number] of clients who actually have access to a mentor”. Since this was a new program for this particular resettlement agency, Bianca said they struggled with measuring some of the critical aspects of the programming: “We’re still trying to figure out how to truly measure things that are difficult to measure, like the emotional skills or self-awareness, social awareness, acculturation [etc.]”. Through this quote, two things are clear: The first is that the both the resettlement agencies and local nonprofits do not receive guidance from either the RD or the ORR on how to implement qualitative outcome measures; instead, they are given autonomy over the selection and implementation of measures of effectiveness. The second is that, even when subcontracting local nonprofits who possess YM expertise, resettlement agencies are not using these partnerships to address critical issues related to refugee youth programming.

Since the measures related to the effectiveness of YM programming are inconsistent across the subcontracted local nonprofits included in this study, and resettlement agencies rarely advised them on program content or measuring outcomes, I wondered if the continuation of grant funding depends on the effectiveness or performance of the subcontractors. I found that although the RD can grant any nonprofit or school district to carry out activities under the YM set-aside program, there appears to be little to no will to organize competition for a grant. Regarding the choice of contractors for YM programs, RD director said: “In order to get the funding out within a timely manner, it had to be those that were already contracted [resettlement agencies],” which suggests that the ease of transactions was prioritized over quality implementation. Therefore, subcontractors—the same organizations that develop and implement the programs—are tasked with determining what counts as quality implementation, and then designing and measuring the set outcomes.

8. Discussion and Implications

I am a Muslim, female, woman of color, who was born and raised in Pakistan, where I also taught refugee girls in a resettlement community. My research as an educational policy researcher fundamentally stems from these identities as I strive to strengthen the systems of resettlement globally to better accommodate the needs of refugee children.

The purpose of this study was to bring forth the inner workings of one city’s network of organizations that implemented federally funded programs for refugee youth and the processes they undertake to ensure effectiveness and accountability for the programs. Overall, the findings point to some problematic trends in the federally funded programs implemented to support the integration of refugee youth, from their inception to monitoring and evaluation. The findings also affirm some of the existing research on the functionality of vertically complex organizational networks. The findings under the first and second theme reaffirm prior research (Tolbert and Hall 2016) and illustrate the ways in which lower-level organizations in vertically complex networks are limited to the application of predefined rules and criteria rather than engaging in any actual decision-making. Findings under the first theme suggested that resettlement agencies were reluctant to implement youth-focused programs even when they saw the need for it first-hand. Furthermore, the second theme highlighted the ways in which the ORR-provided guidelines were held up across the network, and that compliance with said guidelines was prioritized over measures of program effectiveness.

Historically, resettlement organizations have worked most closely with refugee communities and are therefore uniquely positioned to determine both the need and design of any given program. However, an important aspect of resettlement programming may be deterring resettlement agencies from implementing new or innovative programming; it is a reimbursement-based system. This finding in particular directly relates to the dilemmas of nonprofit service providers; as they eventually grow and start to receive governmental funding, many lose the fundamental purpose of their existence—responsiveness to their community of interest (Smith 2014). I found that even when working with high needs populations, such as refugee youth, resettlement agencies often did not want to implement programs that the ORR may not approve as an allowable cost, thereby refusing to reimburse money already spent on a program. Given that, I recommend that lead public agencies in vertically complex networks should allow the lower-level nonprofits greater freedom in programmatic decision-making and provide avenues for flexible funding. When receiving flexible funding, which may be programmable to meet variable needs of the community, the nonprofits can once again thrive as sites of community engagement rather than scrambling to meet the compliance requirements of the funders.

Moreover, I found that “accountability” was distributed across the organizational levels: The top three organizational layers, starting with the ORR (Level 1), monitored the ones below them for compliance with federal guidelines. These findings point to the problems that arise when the leading agency—the ORR in this case—prioritizes compliance over accountability and effectiveness of the programs. Furthermore, the findings also showed how the ORR did not receive any qualitative measures of the outcomes, as the ORR does not specify any qualitative outcomes that programs must meet. The local nonprofits implementing YM programs were measuring outcomes that they deemed relevant to the program. Research on YM programs specifically warns that the selection of an assessment approach to measure program outcomes may substantially affect its observed impact, especially when the outcome measures are developed by the evaluators themselves (Raposa et al. 2019). The delivery of YM programs may look very different under two different local nonprofits. Therefore, the measurement of effectiveness across set outcomes is key to ensure quality in programming that addresses the most critical needs of the refugee youth. I found that, since outcomes or program effectiveness measures were not being consistently measured across all grantees, an organization’s previous performance with the grant was not a consideration for continuation of the award. In this case, relational contracting was at play where a pure market-like competition to win the grants was made unlikely. While Milward and Provan (2003) argue that in relational contracting rebidding the contract may frequently disincentivize the providers to invest in the infrastructure in the long run, they also warn that it can degenerate into collusion. Since the performance of the subcontractor did not dictate the distribution of grant money, the RD, and subsequently the resettlement agencies, rarely engaged in monitoring the subcontracted local nonprofits for performance.

Therefore, I argue that involvement of the leading agency in identifying and defining program goals against which the nonprofits can measure the outcomes will help ensure the implementation of quality programming. While many scholars have put forth guidance on designing outcomes for YM programs (Oberoi 2016; Raposa et al. 2019), I further argue for the inclusion of long-term surveys that note workforce outcomes and the completion of high school/GED for refugee youth who received services under the YM grants. The government’s involvement in drafting performance measures when nonprofits deliver services is not novel. Carlson et al. (2010) studied the impact of Oregon’s state-mandated performance measure reforms on nonprofit human service organizations. The study found that as the nonprofits included more outcome-focused and quantifiable performance measures in their evaluations, they not only improved their overall performance but also increased capacity for their future performance (Carlson et al. 2010). The case in Oregon is a great example of government-led reform to ensure that taxpayer money spent on social services delivers results that speak to the quality of the services as well. Therefore, I recommend that the field of refugee youth services pay attention to the functions related to the delivery of YM programs and how to measure their quality.

Furthermore, the insights from this study present some of the ways in which resettlement organizations and networks make decisions regarding the implementation of programs for youth—programs that significantly shape their integration into the United States. The study brings attention to the world’s first formal government-sponsored refugee resettlement program and highlights how at least one city’s implementation overlooked the quality of the programs for the young refugees by prioritizing compliance. As numbers of refugees, asylees, and displaced persons rise globally (International Rescue Committee 2023), more attention needs to be paid to designing effective programs that support better socioeconomic futures for refugees. As my findings revealed that the data collected on youth programming in this case was inconsistent across organizations and did not ultimately reach the ORR—the lead agency and funding entity—I posit that there is a need for data-related strategizing at all organizational layers involved. I argue for the collection and dissemination of performance measures and data on refugee support programs that can provide evidence of the effectiveness of such programming so that other organizations, and even other countries, can learn from it and replicate it.

9. Conclusions

In addition to expanding the existing literature, this study contributes significantly to building an understanding of the factors associated with the integration of refugee youth as they are shaped by nonprofits. My study situates a marginalized and often overlooked population—refugee youth—at the center of the discourse on designing effective YM programs with measurable outcomes that are implemented using a structure with clear accountability for program quality. Schools rely heavily on the local resettlement agencies and other local nonprofits to provide educational and learning support when dealing with refugee students and their families (Mairaj and Callahan 2022). Therefore, it is important to explore the effectiveness of these organizations’ programs and to hold the entities involved accountable for their success.