Courage, Honesty, and Evaluation in the Apprehensive University

Abstract

A consistent question in education is how to evaluate the degree to which universities and their programs are meeting the claims they make on their webpages and other materials, which entice students and faculty alike to join their collegiate community. Misalignments between what is promised and what is provided harm all community members but have disproportionate effects on students of color. It is therefore an ethical imperative for the higher education sector to undertake system wide evaluations because of the ever-rising financial and emotional costs of graduate education. For educators and administrators alike, this means systematically interrogating data to identify unseen patterns, challenge assumptions, and ask both critical and highly uncomfortable questions; for educators, this may include a truthful assessment of our own practices and assumptions. We propose drawing from the field of program evaluation and using theory-driven evaluation as a specific framework to understand graduate education process and outcomes. This conceptual paper links together existing literatures and is augmented by the authors’ reflection and dialogue about their experiences designing and implementing graduate education across several institutions. We end with a call for courage and honesty in carefully evaluating graduate education for the betterment of all students, faculty, and administrators.

1. Introduction

It is difficult to imagine a university, degree-granting program, or professional development opportunity operating in contemporary times without an online presence. University and program webpages are a trove of information and promises about what will happen should the reader apply and enroll, describing courses, sequences, and timelines, alongside curated descriptions of the job opportunities held by select alumni. A potential problem is that someone who peruses program webpages engages with a portrayal of graduate education that may not resemble the actual experience of the participants, even though the webpages have been reviewed and approved, and the official curricular documents represent an agreement between the student and university that has been codified by the Office of the Registrar. One consistent question in education is how to evaluate the degree to which universities and their programs are meeting the claims they make on their webpages, which entice students and faculty alike to join the collegiate community.

The importance of evaluating educational programs and institutions is not a hypothetical thought experiment. Many students enter higher education with aspirations to obtain knowledge and skills that prepare them for careers in which they can “do good” for their communities and society [1,2]. Within the social sciences, liberal arts, and educational disciplines, this often necessitates a graduate degree to be competitive for paid work opportunities. However, the increasing financial and emotional costs of graduate education [3,4,5], misalignment between course offerings and the program’s stated goals [6,7], and restricted access for communities of color [8,9] make revisiting the promised processes and impacts of graduate education an ethical priority grounded in dignity, equity, and inclusion. For educators and administrators, this means systematically interrogating data to identify unseen patterns, ask questions, challenge assumptions, and be proactively reflective, such as the processes described by Estela Bensimon and Alicia Dowd in their work on the Equity Score Card [10]. This process could result in both individual and collective sacrifice and reinvestment of resources, an honest assessment of facts, data, and assumptions, as well as the uncovering of uncomfortable aspects of institutional and programmatic policies and practices. For educators individually, this may include a truthful assessment of our own practice and finding ways to sit with our discomfort without shifting immediately to solutions or deflecting responsibility. However, graduate education can and must undergo a systematic reckoning to address the tensions between the status-quo and the differentiated needs of today’s students, as well as to mitigate the misalignment between graduate programs’ implementation and the outcomes they advertise.

Calls to reform higher education are not new though, and previous efforts have yielded minimal benefits for the students (for review, see Cassuto & Weisbuch [11], Dickeson [12], or Rhodes [13]). Evaluation theory [14] suggests that may be because previous reform efforts have attempted to influence only small aspects of a large and complex system; instead, evaluation theory applied to higher education suggests entire aspects of the system must be interrogated, discussed, and acted upon if it is to deliver on its promises. In this piece, we recommend formal program evaluation processes as an actionable step in the quest to reimagine a more equitable and sustainable graduate education system. We first describe our terms, positionalities, and the processes we used to craft this paper. We then describe the discipline of program evaluation and why the evaluation process would benefit graduate education. From there, we describe five key areas of graduate education to focus the evaluative exercise, offering critical questions to aid in reimagining. We conclude with a call to honesty and courage, highlighting who needs to be involved and where this evaluative intervention might occur. With this framing, we hope to illustrate the power of evaluation to facilitate reflection, uncover patterns of abuses, and generate transformative solutions for a future of higher education that is rooted in dignity and equity.

2. Terminology

Throughout this work we use the terms “institutionally disadvantaged” and “differentiation” frequently. By “institutionally disadvantaged” we are referring to students who have been systemically under-resourced and underserved and are, thus, often pushed to the margins. Quite often, these students are excluded based on race, socioeconomic class, disability, gender, or other aspects of their identity. We use “institutionally” to ensure the onus and accountability is placed on the institution.

We use the term “differentiated” to indicate the multitude of ways educational experiences can support students based on their collective and individual needs, including tailored student supports and services that are needed to reimagine the future of graduate education. We believe differentiation is not “extra” support and services; it is creating and equitably offering wider arrays of supports to assist students on their unique and varied graduate education journeys, as well as dramatically reimagining the ways in which graduate education positions itself in the larger societal landscape.

3. Authors’ Positionality and Process

3.1. Positionality

Our individual and collective positionality regarding graduate education is multifaceted and complicated. We represent different graduate and undergraduate disciplines (e.g., program evaluation, psychology, sociology, health, k-12 education, and modern languages), ethnic and cultural backgrounds (e.g., Latina, white), career stages (new Ph.D., early career, and later-career), areas of practice (administrator, faculty, practitioner), and experience as educators and administrators. Some of us were expected to go to college and graduate school as a baseline, whereas others were not expected to succeed after high school. These different perspectives, which we have tried to reconcile, have helped us approach the critical topic of higher education differently than others have before us. We are similar in our methodological and analytical orientations (mixed) and that we believe in the promise of higher education, if not in the way it has been conceptualized and delivered. Further, we believe that we, as educators, practitioners, and university staff, have a role and responsibility to reflect critically on our educational outcomes and processes. We recognize that many of the challenges observed and experienced by the most senior member of the team are akin to the challenges experienced by the most junior. We agree that we have a responsibility to listen to our students, to learn from our mistakes and missteps, and to do better.

3.2. Process

Our process for developing this work involved individual and group self-reflection [15], dialogue about our experiences participating in and delivering graduate education across several institutions, critical interrogation of our beliefs and assumptions, and a visioning process similar to appreciative inquiry paired with evaluation for organizational development [16,17] with a common goal: understanding what graduate education currently is, and what could be possible.

We individually reflected on and journalled about our experiences participating in graduate education to discern our most salient experiences and the challenges therein; those of us that had (re)designed and implemented graduate education programs also reflected on these experiences. We brought these reflections to initial meetings to discuss what seemed to work and where the limitations in these experiences were; these dialogues were purposefully structured so that all could speak, discuss, push the conversation forward, as well as revisit previous topics for more nuance and understanding. The goal was not to reach consensus, though we often found ourselves in agreement; the goal was to share, be heard, and collectively invest in thoughtful conversations. Following these initial discussions, we read Cassuto and Weisbuch [11] and used their framework to further contextualize our own experiences. The theoretically grounded evaluation model we used to organize our treatment of graduate education was selected based on several authors’ doctoral-level preparation in program evaluation, the model’s congruence with complex topics, and the degree to which the model conceptually organized the constructs posited by Cassuto and Weisbuch [11].

During and following the ongoing reflections and dialogue, we each selected specific components about which we felt most compelled to write. We shared drafts, offered insights for each other, and challenged each other’s assumptions. In this way, all authors took responsibility for reflecting, writing, and editing across each of the sections.

A note for our readers: This piece is written about higher education in general with a particular focus on graduate education, and while it draws upon deidentified examples from our lived experiences, the purpose is not to shame or complain; it is to critique what has been, and build towards a more hopeful, equitable, and sustainable future. Readers that personally identify with or react negatively to the scenarios may wish to consider guided reflections to process their responses.

4. Graduate Education as an Ecosystem of Programs

Rhodes [13] begins his discussion of higher education with the phrase “every institution needs a focus; no institution can prosper if it neglects its core business” (p. 84), then clarifying that while universities and businesses both benefit from focus on their mission and goals, a university is different from a business in spirit, goals, and implementation, leaving the reader to wonder “what is the core business of the modern research university?”. After positing that the core function of the university is knowledge creation and dissemination, Rhodes describes the undergraduate curriculum as a potpourri; an intimidating breadth of choices and options that simultaneously lacks both an effective core curriculum as well as guidance for students. He is more positive but still unsparing in his assessment of professional and graduate education, suggesting that although they are popular and meant to combine intensive specialist training that aligns with industry standards, they are difficult to define, measure, and compare in terms of processual or outcomes-based quality, especially in fields that lack explicit boundaries or standards of practice.

Indeed, graduate education is highly varied, individualistic, loosely structured, and is often dramatically different between institutions, departments, and even between individual faculty within a single focus area [12,13]. These characteristics may benefit select individual students and faculty, but may not benefit most participants, could indeed be harmful, and lead to the waste of many human and instrumental resources [11,12,18]. Simultaneously, there is a lack of clarity surrounding the proximal and distal impacts of graduate education, which combine with significant conceptual and process misalignments between its implementation and desired impacts, thereby leading to a difficult ecosystem for everyone involved. Therefore, rather than piecemeal efforts to reform individual aspects of graduate education, we advocate for the systematic evaluation of graduate education itself. This would help to discern what the purpose(s) of graduate education should be for institutions and programs, explore if these purposes are being enacted, identify which aspects are ineffective or detrimental, and both reimagine and build towards a healthier organizational system.

We find it helpful to consider higher education as an ecosystem that supports many “programs”—graduate education being one of them—implemented across a host of sites. Nested within each site are collections of individual programs that are arranged by college, department, and track. A “program”, as defined by the American Evaluation Association’s Joint Committee on Program Evaluation Standards [19] (p. xxiv) is:

• A set of planned activities
• Using managed resources
• To achieve specific goals
• Related to specific needs
• Of specific, identified, participating human individuals or groups
• In specific contexts
• Resulting in documentable outputs, outcomes, and impacts
• Following assumed (explicit or implicit) systems of beliefs (diagnostic, causal, intervention, and intervention theories about how the program works)
• With specific, investigable costs and benefits.

This definition of a program is helpful because it applies structure onto a system that has historically resisted the delineation and description of its parts. The definition also provides space for individual actors to consider the whole educational system, and it explicitly separates the planned, purposeful aspects of graduate education (e.g., courses, benchmarks) from the planned-for-some aspects (e.g., research and/or teaching experiences) from the unplanned-but-lauded aspects (e.g., coaching, mentorship, career guidance). Further, it provides space to consider if graduate education is working well for all participants, or even being implemented consistently.

5. Evaluation as a Tool for Disrupting Graduate Education’s Status Quo

It is easy to suggest that graduate education should be evaluated. After all, without evaluation, how could stakeholders—individuals and groups affected by the program [19]—know if a program is working, for whom it is good, under which circumstances it is working, and how it might be improved [20]? Higher education institutions and programs often claim to engage in “evaluation”; it is true that they have a preponderance of available perspectives and ideas that have been gleaned through processes such as Quality Improvement (QI), Leadership Development activities, Departmental Retreats, and External Reviews, as well as by drawing upon other perspectives that could be based either in empirical data or on stakeholders’ philosophical orientations 12. These activities are sometimes paired with assessments and data collection tools but are consistently hindered by a lack of accountability in how the processes and results will be either used or shared with stakeholders along the way [12]. This makes it easy for educators and leaders to ignore, discount, or assign the activities to a working group or committee for “follow up”. While this might be called “evaluation” by some, data collection or analysis alone is not evaluation [20,21]; formal evaluation is a holistic process by which systematically connecting program activities with the outcomes of a program can be realistically expected to be achieved under realistic—not optimal or best-case—political, economic, environmental, and humanistic conditions.

6. Evaluation Purposes and Processes

Evaluators do their work for many reasons, including understanding the merit, worth, and significance of a program [22]; determining the degree to which a program is meeting the needs of a community or group of people [23]; aligning the program’s activities with reasonable and desirable outcomes [20,24,25]; or promoting individual, group, organizational, and systems learning [26,27]. When program evaluation is performed as a learning-oriented inquiry process [27], it typically helps organizations realistically examine their current offerings and align them with the longer-term group, community, and system-wide impacts they hope to achieve, such as promoting social betterment [28] and stakeholder empowerment [29]. Many—though not all—evaluation professionals also view their practice as having an ethical imperative to dismantle the historical power dynamics by using strategies that promote equity [30]. In this way, evaluation is especially suited to serve as an intervention to disrupt the status quo of the graduate education system.

There are many approaches to evaluation (see Shadish, Cook, & Leviton [31]; Alkin [32]; Russ-Eft & Preskill [17]), but most evaluators follow six general steps outlined by the Centers for Disease Control and Prevention [33] (Figure 1). This model suggests that evaluators must engage stakeholders, describe the program being evaluated, focus the evaluation’s design, collect credible evidence, analyze data and justify conclusions, and work to ensure use of the evaluation’s process and products. These activities, however, may take different forms based on the stakeholder needs, interest and engagement, resources, and the evaluators’ ethical or theoretical orientations [34].

One omnipresent tool of contemporary evaluation practice is a logic model or other visual representation of the program; more comprehensive logic models would include what evaluators call the action model, implementation model, and the change model. Chen’s theory-driven evaluation model [14] suggests that while evaluators often focus on the change model (i.e., how the program activities should lead to the desired outcomes), the action model is critical to understanding how the program was developed and how it relates to the needs and interests of the program participants and recipients. The practical link between the action and change models is the implementation model, which refers to the fidelity or variance to which the program was implemented. These models build from each other (see Figure 2), and assumptions or problems in early areas can cascade into later areas. For example, one could imagine a graduate program that has been developed and implemented over years or decades that is no longer relevant to contemporary knowledge bases and skills (failure located in the action model). Or, a program might be developed to reflect contemporary topics and standards, but could be under-resourced, implemented haphazardly, or outsource critical knowledge bases to other departments in ways that make the desired outcomes implausible (failure located in the implementation model). Similarly, a program could be developed based on contemporary knowledge and skills and implemented regularly but be taught using pedagogies or course sequences that do not align with the desired outcomes for individual courses or the entire program (failure located in the change model).

Note: Source is Chen [14]; Permissions for reproduction granted through STM.

We are unrepentant advocates for evaluation in graduate education, and we also recognize that, like many other people, faculty and administrators are often hesitant of engaging in evaluation. In addition to concerns that the evaluation will be performed poorly or be used for political purposes [35], research has attributed this reluctance to engage in evaluation to anxiety or misunderstandings and stereotypes about evaluation itself [36]. Indeed, stakeholders may be loath to hear that their graduate programs need realignment, or even complete reconceptualization. Further, they may be reluctant to take individual and group responsibility for their program gaps or failures—though they have no qualms about taking credit for its successes—or even claim “academic freedom” as a reason to not formally evaluate their graduate education programs, especially in relation to advising, teaching, leadership, curriculum, and other dimensions of quality education. However, academic freedom does not include topics such as curriculum, advising, student benchmarks, admissions, and attrition [37], making them both viable and critical elements in an evaluative framework.

7. Framing the Evaluation: Interrogating the Dimensions of Graduate Education

To realistically apply Chen’s theory-driven model of evaluation to understanding graduate education, it is important to conduct evaluative exploration of multiple processes and outcomes in graduate education. Cassuto and Weisbuch’s dimensions of graduate education are a comprehensive and approachable framework to apply an evaluators’ perspective on graduate education, and though their analysis is rooted in the Liberal Arts, their dimensions of quality may transcend disciplinary contexts [11]. They suggest graduate education be interrogated and reformed along five mutually supportive dimensions: (1) career diversity, (2) admissions and attrition, (3) student support and time to degree, (4) the curriculum and its benchmarks, and (5) advising. These categories form the structure for the rest of our discussion, and though we discuss these topics as independent, they are interdependent, further bolstering the utility of a comprehensive evaluative framework [14]. We also note that these foci intersect with but do not attend to related issues of faculty behavior towards each other, department and college hiring, tenure and review processes, or the intersections of the three [38,39,40,41], making them important topics for future discussion.

8. Career Diversity

The term “career diversity” suggests a plurality of ways in which students and alumni can apply their learning in practice. Not all students that engage in higher education can or want to become faculty, but contemporary educational culture is engineered against supporting students that want to pursue applied careers. This can take the form of active or passive discouragement, along with behaviors that “otherize” students interested in applied careers such as highlighting only alumni that have acquired tenured/tenure-track faculty positions. This bias towards academic careers is also evident when taking into account calls from higher education administrators for its students and faculty to do “work that matters’’ via public and civic engagement [42,43], though this work is not rewarded in the higher education system.

More challenging is the faculty’s general lack of awareness about career differentiation and the wide variety of careers available to students [44,45], as well as the lack of responsibility that faculty take in ensuring that the content and processes of their courses align with the knowledge, skills, and abilities desired in the contemporary job market. As a result, students interested in non-academic careers are often outsourced to university-level supports, though graduate students are—by definition—participating in high-level, differentiated educational experiences, and their needs are—by definition—different from the needs of undergraduate students. Indeed, it may be difficult for graduate students to be holistically counseled by Career Services employees who are well-intentioned but not knowledgeable about the careers and opportunities within specific fields.

A potential solution to the problem of career diversity is the deliberate insourcing of career knowledge and career networks, perhaps by a full- or part-time professional and administrative staff (p/a), or even by an advanced departmental Graduate Assistant whose role could be developed and supported with either discretionary strategic or research-aligned funds. Or faculty could get creative in how they help students approach careers. For example, LaVelle, Dighe, and Sarode [46] describe an educational process by which they require students in their courses to submit job ads weekly in lieu of taking formal attendance; the data from the job descriptions are then used to contrast course content with desired program, course, and professional outcomes. The job descriptions are maintained and available for individual career counseling to future students, as well as serving as empirical evidence for available jobs for prospective students; indeed, few things are more impressive to a prospective student than seeing a massive binder of current and novel job opportunities and being given permission to review it at their leisure for inspiration.

9. Admissions and Attrition

If a prospective student were to ask “what are reviewers looking for in a successful graduate school application” how would faculty and administrators respond? Most would give the standard responses such as strong GRE scores, strong GPA, letters of recommendation, and perceived fit, but these answers are neither specific enough to be particularly helpful to potential applicants, nor used consistently to guide admissions decisions. This “institutional unmentionable” prompted Posselt [47] to examine what criteria were important to graduate admissions decision makers. Her data disclosed that, as a matter of efficiency, committees often only discuss those files in which faculties’ initial ratings diverge from each other. While this might be celebrated in some circles because it approximates data triangulation and interrater agreement (see Tashakkori & Teddlie [48] for discussions on triangulation), this process also allows for faculty to easily ignore many student applications based on numeric criteria (e.g., GRE) that might not be the most important indicators of potential and completion, thus artificially placing a limited number of applicants in the top pool. Decisions are then made based on subtle perceived distinctions between applicants based on implicit criteria that were not considered with the other applicants, leading to an inequitable admissions process.

Although Educational Testing Service (ETS) state that Graduate Record Examination (GRE) and other standardized tests should not be the sole or primary decision-making point [47] (p. 71), faculty have done so in the past and likely continue to use them as such. Some institutions have begun eliminating the GRE as an application requirement [49], but we wonder what—if anything—will replace it. If the replacement is quantitative, will it be susceptible to similar conceptual and methodological flaws as the imperfect test(s) they replaced? If qualitative, will faculty invest the time and emotional energy to fully engage with it? However institutions move forward, we see that the crux of the matter is intentionality and transparency: when eliminating or reducing the priority of admissions tests, how have faculty intentionally rethought, restructured, and made admissions processes better and more transparent? This is no small question for an essential process that leads directly to the determination of financial aid packages, perks, and other supports which have not historically been distributed equitably [47].

In a parallel process, admissions processes have been working towards inclusivity, which subsequently manifests in the awarding of better-than-before financial aid to diverse students. Funders and institutions alike recognize that some students have been disadvantaged because of issues of race, class, gender, (dis)ability, and have set aside funds to support these students. However, to be eligible for these resources, students must often reveal intimately personal stories about themselves, their identities, and their experiences with often traumatic life events to position themselves as resilient, gritty, and worthy of support. An unintended consequence of this has been a shift in what faculty perceive as “meritorious”. Merit refers to the characteristics in individuals that deserve recognition, and we are concerned that legitimate efforts to expand admissions and financial aid have inadvertently made faculty into voyeurs; students, particularly institutionally disadvantaged students, are asked to provide statements that enter them into the “sweepstakes of the most disadvantaged” which faculty then judge. We note that students that are not institutionally disadvantaged do not have to reveal these stories and can imagine the righteous outrage that would be sparked by requiring privileged, mainstream students to expose their privilege and reveal the ways their school, life, finances have all been taken-for-granted. We would never ask that of mainstream students. Yet, we award scholarships by asking how people are disadvantaged rather than by what they have accomplished.

Similarly, Diversity Statements appear to be a slippery slope for institutions and faculty as they review students for admission. There is rarely an explanation for the goal of the statement or the perceived value it brings to an application dossier, and there is certainly no evidence of consistency on how those statements are reviewed or perceived [47] (pp. 59–65). Instead, reviewers bring their personal judgements about if a student “really” understands diversity in the same way as the faculty and has experience with it in their life. To be clear, we believe that diversity in the collegiate community is good, as is supporting students; our concerns are in the processes used and the outcomes they produce. Evaluating admissions processes considering both the aspirational goals of the program and the curriculum itself improves these processes; indeed, faculty have autonomy to change the way admissions are conducted so they are be conducted carefully and thoughtfully, with awareness of the faculty role(s) within the system.

Conversely from student application and matriculation, attrition is an underdiscussed aspect of graduate education. Students are often informed at the beginning of their experience that approximately 50% of the people that start a Ph.D. will not finish, and the student response is—understandably—“well, that will not be me” [11,13]. On the one hand, we recognize some attrition is normal in years-long processes like education, but on the other, we wonder what faculty and programs know about the students that withdraw. If information about student withdrawals is systematically collected, is it ever analyzed to identify areas in which the program has responsibility? Or do the students simply disappear from enrollment rosters? We posit that there is institutional safety in pleading ignorance about why students withdraw. As difficult as it would be to receive and hear honest answers about faculty, courses, curriculum, and the institution, as well as to take responsibility for (in)actions and hubris, we owe it to our former, current, and future students to find and interrogate data about student attrition so that systematic issues can be identified and resolved.

10. Student Support and Time to Degree

Graduate students harbor numerous identities that contextualize their graduate school experience; spouses, parents, caregivers, and full-time professionals, and others adopt these identities over the course of their studies. For many students, graduate education is not just an intellectual exercise in the pursuit of knowledge, but a critical way to elevate their professional ambitions, provide for their families, and make positive impacts in their communities. However, due to the high emotional and financial costs, the anticipated burden of future student loan payments, as well as the sizable time investment, graduate education is not accessible to all individuals. People of color are often excluded from graduate education, and even when they do gain access, many are under-resourced financially and emotionally by their programs.

Some individual faculty are adept at providing support for students, though there is a wide chasm of practice between the excellent and the unsatisfactory. The degree to which individual faculty’s differentiated support permeates the graduate program is unclear, but we note that faculty who provide quality support quickly become overloaded with students, and unsatisfactory faculty are rewarded with students that will succeed regardless of any level of faculty support. Similarly, students that need the most differentiated support are sometimes trapped within cycles of requesting for “time to degree extensions” and rising tuition fees, while those who require very little attention are ushered through.

We wonder how different things would look if graduate education programs conducted an evaluative needs assessment to determine what kind(s) of differentiated supports institutionally disadvantaged students need to achieve the promise of graduate education. Money would be a factor, though we note that the services and support that institutionally disadvantaged students need to succeed (e.g., career supports, writing supports, research and teaching opportunities) would indeed benefit every student. We posit that it is through evaluative processes that we can identify, clarify, and realign programs to ensure programs provide differentiated student support that correspond with stated student need.

Similarly, it is also our experience that the time to degree serves as a particularly problematic area of graduate education. For many, the number of years spent earning a doctorate is a badge of courage akin to battle scars; the longer it takes, the more bragging rights you earn; or the quicker they move through, the more focused or agile you are. While we acknowledge that all students need varying times to complete a degree—the authors of this article spanned between three and ten years post-masters—there are some harmful practices that can trap students within graduate education programs. Faculty, as they pursue their own research agendas or the ephemeral promise of tenure, may implicitly captivate students with the promise of publications or fellowships—where fellowships are available—while compensating at below market hourly rates. Students might call this an extractive process. Faculty will disagree, of course, framing the publications and work experience as important for students’ professional development. Both perspectives can be true simultaneously, and it is incumbent on the faculty to be honest with students about expectations and progress towards degrees, and that student progress be differentiated to support their personal and professional goals.

These illustrative examples suggest how the idiosyncratic processes of graduate education admissions often overpromises, under-delivers, and ultimately betrays student trust. An evaluation of this realm of graduate education would surface critical process-based questions, forcing graduate educators to reflect on their role in a system that has historically both supported and taken unfair advantage of students. A more nuanced evaluation would also examine the ways the system does or does not work equitably for all. We contend that a support system that works for 85% of the students but works against the needs of the remaining 15% needs reform.

11. Curriculum and Its Benchmarks

Curriculum is a contested concept and educational space [12]. Educators often think of curriculum as the courses they create and teach; however, they may not consider the curriculum as an additive whole rather than as individual pieces such as the formal series of required courses, supplemental and supporting courses, and elective courses. Scholars see additional elements in curricula, such as the hidden curriculum and its navigation [50], physical and emotional places of inclusion or exclusion [51,52,53], a space to consider and develop cultural competency, as well as the space that either preserves the status quo or moves a discipline. We think of curriculum as a very specific aspect of Chen’s framework for evaluation: it is the core, actual activities that the educational institution provides, and it frames the knowledge base, skills, and values that the disciplines find important. Because curriculum is the core experience in which all students engage, graduate programs or departments without a core curriculum—or one that is under-developed—must be regarded suspiciously, and the faculty and administrators should be interrogated forcefully about this problem. Graduate education is both the pursuit of knowledge and the development of practical skills valued in the labor market [13], so faculty should constantly be developing, refining, and questioning the curriculum they implement, meaning the core should be a recurring topic of discussion and action.

One issue that faculty struggle with is considering and articulating what the core of a program or discipline should include. Discussions on core curriculum sometimes happen because faculty anecdotally assess how courses support student progress, because of the onboarding or departure of faculty, or because student complaints have reached the ears of higher administration. However, thoroughly evaluating and aligning the curriculum with student progress and needs may not happen. Simply stated, it is a very difficult endeavor, and requires faculty to move from individual responsibility for a course to collective responsibility for the totality of the curriculum. We note that tenured faculty have job security for life, and that they have a responsibility to courageously interrogate their curriculum and make changes where appropriate.

For example, one of the most beneficial and difficult exercises we ever engaged in was to address a curriculum and its individual classes from the perspective of students with disabilities. In this exercise, the instructor strips a syllabus down to only the essentials, the core of what a student needs to learn. This exacting process is repeated for all classes and sequenced, which reveals a multitude of truths and assumptions, including the topics the instructor(s) are passionate about, are philosophically linked with, or have simply been accumulated over a long period of teaching. Focusing on the core is critical from a disability standpoint because educators may find that some non-core course requirements are a barrier for some students, inclusive of both (dis)ability as well as students that are institutionally disadvantaged. Further, uncovering the essential elements of a course makes alignment across the program and departmental curriculum clear in terms of what is included, but also in what is missing and must be shouldered elsewhere.

Benchmarks for student progress should strongly and purposefully align with the actual curriculum. However, there is often misalignment between benchmarks and course experiences, and students pay the price in time, money, confusion, and unnecessary emotional stress. As with other aspects of graduate education, the curriculum often reflects individual organizational demands such as individual faculty careers, responses to too few faculty supporting a program, outsourcing critical courses such as inquiry methods and data analysis, inattention to what students need rather than what faculty wish to do, or fear of challenging the status quo. Benchmarks are created, and guidelines for students to attain those benchmarks are usually structured, but the degree to which the courses and benchmarks are aligned is an empirical question all programs must answer.

For example, many doctoral programs require students to pass a preliminary exam or essay in their first two years of graduate school, which is supposed to be based on students’ mastery of foundational concepts and principles. In reality, however, the preliminary exam is often a test of the students’ ability to write and/or read the minds of their faculty, leading to confusion for students on how to prepare for the exam, and for faculty about how to assess it. For us, the exam process should both test and reinforce the key principles that every student should know, and if/when students perform poorly, faculty should reflect on their responsibility in the holistic educational process; examples might include the content and pedagogy that informed the course design and implementation of the core classes, or (in)consistency in faculty expectations for student performance. Instead, even good faculty place responsibility wholly on the student without considering their role(s) in the curricular ecosystem that contributed to the students’ performance.

Similarly, in a graduate student-driven departmental quality improvement project, #Thrive, graduate students reported a lack of transparency and clarity in most areas of their progression. From this graduate student-led project, we learned that students would value creating safe spaces in the formal curriculum in which students could share their experiences, lessons learned, and insights about departmental benchmarks. In addition, they believed the curriculum should be a key space in which students could find connections with faculty, staff, other students, their program, and even the department. We note that the graduate students found occasions when connection and inclusion existed with single courses or individual faculty, but this was not intentionally built into the curricular design. In addition, students clearly articulated that they lacked the research and teaching experiences and skills needed in their development for their desired career paths. The place where these skills and experiences are developed is often connected to available resources such as assistantships, which is complicated when considering traditional and limited support. The required curriculum, and particularly the core of the curriculum, is the place to intentionally build what students recognize as essential to their success, including service-learning courses, experiential learning courses, mentored internships, and experiences with teaching and building capacity. We view these requests as reasonable, and if any of these student expectations and hopes are not reasonable, then it is incumbent on the faculty and administration to be clear about what is and what is not possible.

12. Advising

Advising is one of the most contentious topics in graduate education. It is not covered by academic freedom, few faculty are formally taught how to do it, and it is based on a relationship rooted in a power imbalance between faculty and students. It is an element of graduate school that adds color and depth to the broad strokes provided by the curriculum. Many prospective graduate students are convinced to attend a particular graduate school by the promise of a supportive relationship with the advisor, only to feel cheated in a perceived bait-and-switch dynamic as they find that their advisors are—to put it kindly—human.

We have seen potential students be assured of close relationships with their advisors, access to faculty research opportunities, and be dazzled with tales about students and faculty developing lifelong relationships, both personal and professional, complete with dinners at their advisor’s home. Sometimes, we are even the person telling prospective students these tales, and they can be true, but it may be unfair to expect all faculty to have close relationships with their students, and unfair to students to promise it. While there may indeed be dinners for some students, as well as conferences and networking, more common are unanswered student emails, laments, and promises of feedback on draft papers. We are sympathetic to both students and faculty in this regard. Advisors are often saddled with unrealistic numbers of advisees, teaching requirements, service requirements, and often daunting publication expectations which may or may not actually end up mattering during promotion or tenure processes.

At its core, this is an issue of student trust, expectations, and what is reasonable given the context. A student’s choice to enter graduate school is a life-altering decision, laden with risk and sacrifice. When a student commits to a graduate program, they are putting their trust in the individual and collective faculty to deliver on the assurances they have been given through program webpages and discussions with faculty. Thus, faculty have an ethical obligation to meet the promises they make to all their students, not just a select few. If that is not possible, then faculty and administrators need to articulate and set more realistic connection expectations so that students do not feel betrayed, faculty understand what is expected of them, and both are equipped for the relationship ahead.

We find that an important question for faculty to consider through an evaluation process is whether they want to be coaches or mentors for their graduate students. Both are critically important but are distinct concepts with differing anticipated outcomes. Coaching is a human development process that involves structured, focused interactions and the use of appropriate strategies, tools, and techniques to promote positive and sustainable change in individuals and groups [54] (p. xxix). Coaching is mostly task-oriented, focusing on topics such as clearly outlining program requirements and creating timelines to degree completion. Conversely, mentoring is focused on providing guidance, motivation, emotional support, and role modeling [55]. Effective mentors need to be prepared to offer differentiated support to institutionally disadvantaged students and be honest about the support they can feasibly provide to their students.

We appreciate the distinctions because it helps us realize that (1) faculty can be coaches for some and mentors for others, and (2) coaches can be assigned to a student, but students select their mentors. Indeed, faculty often haphazardly—and without preparation—assume the role of a mentor, which may be what the students are expecting based on the information they received as prospective students. There may be a misalignment in the reality of the relationship and the expectations of it, and the consequences of it can be everything from switching advisors to leaving the programs entirely, saddled with tens or hundreds of thousands of dollars in debt. An evaluative lens could offer insights here by discerning what differentiated services students need, what types of advising supports they view as beneficial, and then aid in determining what skills faculty need to properly mentor and coach.

13. An Invitation for Honesty, Courage, and Evaluation

Graduate education in its contemporary educational ecosystem does not have to change [56], and individual university systems are unlikely to do so unless there is individual and collective motivation for the systematic assessment of the system’s inputs, processes, outcomes, and impacts, such as the ones described by Cassuto and Weisbuch [11] and organized using systematic frameworks such as Chen’s theory-driven approach to evaluation [14]. The fact is that depending on the framing of the process, evaluation can be unnerving and provoke anxiety [36], and it is understandable why individuals and groups would resist due to a range of reasons (e.g., entrenchment, anxiety, failures of leadership), including the administration’s misunderstanding of the faculty’s expertise and the faculty’s own challenges bridging their theoretical knowledge and addressing the issues in their own programs and systems. Yet, the status quo is not acceptable either; it is both unethical and completely changeable. There must be another way forward.

We posit that in the context of graduate education, a large part of systemic evaluation’s value is in its ability to facilitate intentional reflection through the contextually bound inquiry of questions from stakeholders: students, faculty, administration, and staff. Through an evaluation process, a plurality of stakeholders would be called upon to interrogate data, recognize patterns, and sit with discomfort. Evaluation also offers processes to translate reflections into actions through sibling fields such as program design, implementation, monitoring, and organizational development and change. We recognize that data cannot solve all the problems in graduate education, but by remembering that each datum represents a person and their lived story, it can stoke the flames of critical reflection.

At its best, evaluation pulls stakeholders together around process and action. We advocate for a plurality of participation in the evaluation process to reimagine graduate education; it takes all levels of the system—administrators, department leadership, faculty, staff, alumni, and students—to engage in the evaluation process to truly promote transformational change [12]. Each set of stakeholders has unique levers of control within their reach that are vital for systems change. However, we collectively view departmental faculty as needing to take a major role in any evaluative process that aims to disrupt the graduate education system. Although faculty often protest about their lack of autonomy to make substantial changes in the system, these individuals do indeed have control over many of the promises of high-quality processes and outcomes being made to current and prospective students. Our experiences also reveal that faculty are prone to dispersing responsibility for their role in the dysfunctional system. All too often, faculty think they can only be accountable for what they personally teach, failing to hold their colleagues, departmental chairs, and deans accountable.

As a collective, we have an obligation to do better. Engaging meaningfully in an evaluation process may be a first step in addressing the misalignment in higher education implementation and proximal and distal impact. We view this piece as both a call-out of faculty for their role in perpetuating an oppressive graduate education system as well as an invitation to take the necessary steps to honestly look within, dismantle, and rebuild. Those most in a position to plan for and implement change include the planners responsible for crafting a coherent vision for the future, including the university presidents, deans, department heads, and program/track leaders, as well as the individual faculty themselves [11,13,57].

We believe the fun in graduate education is in its refinement, growth, and the way it can flex to meet contemporary and future challenges. Undergoing an evaluation of a graduate education program is not easy, but the alternative is a march towards irrelevance and resting upon the reputations of faculty that could retire at a moment’s notice. Although we note that change often requires the support of multiple levels of the institution, we see discomfort as key to moving from what is toward what could be. At the very foundation of learning, we know that new information often leads to discomfort, or as Piaget [58] calls it, a state of disequilibrium. Faculty have the choice to seek out and move through the discomfort of disequilibrium and courageously learn what they have protected themselves from knowing. The next step is to suspend judgement; of themselves and their colleagues, of students, of communities, and of the evaluative work that must precede the new development work. We are capable of creating new knowledge by critiquing what came before, but critique is different from judgement or criticism. The purpose, if we can suspend judgement, is to interrogate the systems, policies, and practices in place to find where they do not support students; and to look ever more bravely to ask what practices, policies, and systems often harm students, especially institutionally disadvantaged students. The key questions become:

• What can we strip away in policy and practice to approach this place?
• What are we willing to individually and collectively sacrifice for students to have the graduate school experience they need and deserve?
• What are faculty willing to sacrifice and change so they can actually provide what they—and their programs—claim to offer?

There are many places where students, faculty, and the literature find deficiency in graduate education, and though we are concerned with all students, we want to underscore that the constellation of deficiencies in institutional offerings are most destructive for our institutionally disadvantaged students. We need to, as Jennifer Freyd’s work on educational organizations articulates, move beyond simply identifying harm and move toward institutional courage [59]. Being courageous means having enough faith in the promise of a program and institution to examine them critically and work towards making them better. Courage means we stop being frightened by internal criticism and intransigence. Courage means doing what is necessary, even though it is threatening. Courage means doing the hard work, even though it may leave scars and bruise egos. A courageous institution proactively and regularly engages in evaluation, listens with, and responds to students as a good friend or other supportive person would—by listening well.

In this article, we described some of the challenges we have experienced and contributed to in the academy, from admissions to student support to curriculum to advising. We are both contributors to and beneficiaries from the current system, and we must make our individual and group amends through actions, not words alone. We will work to improve this system, but we cannot do it alone. It is possible we will fail, but this would be the result of courageous action, and we would rather fail courageously than surrender and further perpetuate inequities. We see attempting to change and failing—perhaps repeatedly—as different from sticking our heads in the sand, patting ourselves on the back, and assuring ourselves with saccharine lies that all is well in graduate education. All is not well, and each datum can tell us what can be performed better. Courage, even in this systematic evaluative approach, lies in our willingness to do more than we have perhaps been taught to do with data. As the Equity Score Card work articulates, we need to dig deep into the data, to interrogate them, see patterns most people do not see, ask questions rather than make assumptions, and then ask “what if’’ questions.

Evaluation is a way to strip away our ties to the status quo and remove the personal baggage we may be ignoring or have buried in our assumptions about ourselves, our students, and our work. Evaluation can support honestly revealing our actual practices and help us observe what organizational and institutional changes we have the autonomy to incur, and the courage to make right.