Applying Bibliometric Techniques: Studying Interdisciplinarity in Higher Education Curriculum

Abstract

Bibliometric methods are relevant for a range of applications and disciplines. The majority of existing scholarship investigating citation and reference patterns focuses on studying research impact. This article presents a new approach to studying the curriculum using bibliometric methods. Through a review of existing definitions and measures of interdisciplinary research and standardization procedures for comparing disciplinary citations, three measures were considered: variety, balance and dissimilarity. Bibliometric algorithms for assessing these measures were adopted and modified for a curriculum context, and three interdisciplinary programs were investigated that span undergraduate and graduate degrees. Data objects were course syllabi, and required references were coded for disciplinary affiliations. The results indicated that—despite purportedly pursuing a singular goal in the same academic unit—the programs employed distinct citation patterns. Variety was highest in the master’s program, and balance was highest in the doctoral program. Dissimilarity was highest in the doctoral program, yet a novel technique for disambiguating disciplinary composition was implemented to improve interpretation. The analysis yielded unexpected findings, which underscore the value of a systematic approach in advancing beyond discourse by harnessing bibliometric techniques to reveal underlying curricula structure. This study contributed a well-grounded bibliometric method that can be replicated in future studies.

1. Introduction

This paper contributes an applied bibliometric approach to measuring interdisciplinarity within higher education curriculum. The method is implemented in the assessment of intended curriculum for three academic programs within the same school: bachelor’s, master’s, and doctoral degree-granting programs. These programs share an intention to be interdisciplinary. However, the extent of interdisciplinarity has not previously been assessed, and thus this paper applies bibliometric techniques to study interdisciplinarity within the curriculum.

The paper begins by summarizing existing approaches to studying interdisciplinarity. A robust set of extant studies attends to measurement and operationalization of interdisciplinarity. This body of scholarship provides a summary of definitions for interdisciplinarity and description of its typical measurements. Scholars specify that multidisciplinarity is a necessary but not sufficient precondition of interdisciplinarity. Interdisciplinarity is a subsequent, more advanced step that requires the integration of multiple disciplines. Additionally, three measures are consistently engaged across studies: variety, balance, and dissimilarity. Approaches to coding and analyzing these measures are summarized in the literature review section of this paper.

Notably, most of the existing attention to measurement of interdisciplinarity focuses on research, in which interdisciplinary research (IDR) is defined as integrating multiple disciplines toward addressing a shared puzzle or research question. Attention to interdisciplinary curriculum (IDC) is lacking. Thus, the goal of the current study is to build from bibliometric approaches to studying interdisciplinarity within research toward assessing curriculum (IDR→IDC). The paper proposes that many of the bibliometric measurement and analytical techniques can be replicated in the study of curriculum. Yet, important adaptations also need to be made to shift from research data sources toward data relevant for curriculum. This translation work is not an automatic or self-given process and instead requires careful and thoughtful attention.

The purpose of this study is to apply bibliometric techniques in the study of interdisciplinarity within a curriculum context. The aim of this paper is to contribute to an evidence-based approach to computationally investigating a purported value: interdisciplinarity. As the existing studies within a research context show, interdisciplinarity is an important value, yet the intention to be interdisciplinary is not necessarily sufficient for implementing and achieving this value. Logically, a range of empirical manifestations could be achieved in practice, and these vary by subject area, disciplines engaged, construction of collaborator teams, and other measures of interest.

In summary, the approach of this study is to engage in a bibliometric approach to curriculum assessment, in which the first assumption is that intentions do not necessarily equate to empirical reality. To measure the degree of interdisciplinarity within the curriculum, this paper focuses on studying a set of degree programs that were created to bring together multiple disciplines from Social Sciences, Humanities, and more around a profession: philanthropy.

2. Literature Review

2.1. Interdisciplinarity

To begin, this section summarizes existing bibliometric techniques used to measure interdisciplinarity. A consistent finding from this scholarship is that interdisciplinarity has evidenced benefits on research impact, as summarized below. Within this context, it is logical to consider moving beyond traditional discipline-specific degree programs toward fostering interdisciplinary degrees. Yet, this scholarship underscores that the mere presence of multiple disciplines does not naturally or unequivocally result in the integration required to achieve interdisciplinarity. Thus, a central aim of this section is to summarize how interdisciplinarity is defined and measured.

2.1.1. Theories

A robust set of scholarship exists which attends theoretically to interdisciplinarity [1,2,3,4,5,6,7,8,9,10]. However, the majority of this scholarship attends to defining the problem to which interdisciplinarity is a response rather than defining what interdisciplinarity is or how it can be measured [11]. For example, researchers posit that interdisciplinarity can contribute to a greater degree of accountability through connecting scientists with the public and thus raising awareness of different disciplinary approaches among relevant stakeholders, as well as to knowledge advancement at the intersection of disparate theories and approaches. In this sense, interdisciplinarity can offer a triangulation process for how knowledge is generated and applied [12].

2.1.2. Definitions

The National Academy of Sciences delineates a distinction between multidisciplinary research (MDR) and interdisciplinary research (IDR), quoted below [13].

• Multidisciplinary research (MDR) is research that involves more than a single discipline in which each discipline makes a separate contribution. Investigators may share facilities and research approaches while working separately on distinct aspects of a problem. For example, an archaeological program might require the participation of a geologist in a role that is primarily supportive.
• Interdisciplinary research (IDR) is a mode of research by teams or individuals that integrates information, data, techniques, tools, perspectives, concepts, and/or theories from two or more disciplines or bodies of specialized knowledge to advance fundamental understanding or to solve problems whose solutions are beyond the scope of a single discipline or area of research practice.
• IDR is pluralistic in method and focus. It may be conducted by individuals or groups and may be driven by scientific curiosity or practical needs. Interdisciplinary thinking is rapidly becoming an integral feature of research as a result of four powerful “drivers”: the inherent complexity of nature and society, the desire to explore problems and questions that are not confined to a single discipline, the need to solve societal problems, and the power of new technologies. Successful interdisciplinary researchers have found ways to integrate and synthesize disciplinary depth with breadth of interests, visions, and skills. Students, especially undergraduates, are strongly attracted to interdisciplinary courses, especially those of societal relevance. The success of IDR groups depends on institutional commitment and research leadership. Leaders with clear vision and effective communication and team-building skills can catalyze the integration of disciplines [13].

Additionally, the National Science Foundation defines interdisciplinary research [14] by quoting verbatim the National Academy of Science definition (above second bullet). Those definitions pertain to disciplinary plurality within a research context.

Regarding curriculum, scholars state that curriculum is crucial for structuring in the overall design as to how different disciplines are intended to be combined [15]. They state that interdisciplinary learning also needs to focus on a theme or problem, rather than students acquiring discipline-specific knowledge. Focusing on a theme or problem is intended to form context-dependent knowledge that focuses on integrating multiple perspectives into complex understandings of a topic. Most importantly, beyond exposure to multiple disciplines alone, the crucial ingredient for interdisciplinarity is fostering integration. In addition, these scholars offer another definition, which refers to transdisciplinary approaches:

• A transdisciplinary approach involves nonacademic practitioners working with academics to identify, research, and develop solutions to real-world problems [15].

To synthesize, the above existing definitions combine to identify three approaches to pursuing disciplinary plurality: (1) Multidisciplinarity, (2) Interdisciplinarity, or (3) Transdisciplinarity. Multidisciplinarity refers to knowledge garnered from multiple distinct disciplines to inform a topic. Interdisciplinarity refers to knowledge garnered from integrating multiple disciplines into a new understanding of a topic. Transdisciplinarity refers to knowledge garnered from transcending multiple disciplinary boundaries by integrating disparate information into a new understanding that informs applications within a field of practice.

Additionally, these definitions identify three bibliometric contexts in which approaches to disciplinary plurality can be studied, within: (1) Research contexts (e.g., measuring impact by journal articles or funded grants), (2) Practice contexts (e.g., measuring impact by patients served, or programs implemented), or (3) Curriculum contexts (e.g., measuring impact by courses taught, or students graduated). Multidisciplinarity and Interdisciplinarity can both be implemented within a research context, with the distinction being that Multidisciplinarity has each discipline remaining distinct in its specialized knowledge contributions whereas Interdisciplinarity includes integration across disciplines. Based on extant studies, transdisciplinarity appears to be most relevant for practice impacts. This is because transdisciplinarity involves the application of research to practice, sometimes referred to as praxis, and is thus not primarily focused on the production of research as its objective but rather advances the application of research in attempting to solve or lessen social and human problems. Therefore, unlike measures of basic research impact, such as citation counts, the impact of applied research is measured by the extent to which it causes real-world changes.

Combining extant definitions, this section contributes a summative list of six items. Since educational programs can be designed to impact research and practice, all three approaches to pursuing disciplinary plurality are relevant for curriculum contexts. Thus, this paper combines these two dimensions (a) plurality approach with (b) impact context to offer six definitions (MDR, IDR, TDP, MDC, IDC, TDC), below and in

Table 1. Approaches to Disciplinary Pluralism in Research, Practice, Curriculum.

Discipline Plurality	Multi-	Inter-	Trans-
Research Context	Discipline-basedspecialists collaborate to investigate ashared researchquestion or puzzle → Acronym: MDR	Topic-basedspecialists integrate disciplines to studya shared research question or puzzle → Acronym: IDR	[Not applicable]
Practice Context	[Not applicable]	[Not applicable]	Field-based practitioners apply disciplines in addressing a real-world problem or issue → Acronym: TDP
Curriculum Context	Approach specialists are exposed tomultiple disciplines toward specializingin a particularapproach to a sharedquestion or puzzle → Acronym: MDC	Topic generalists integrate materialsfrom multipledisciplines towardunderstandingintersecting contextsof a shared topic → Acronym: IDC	Change practitioners transcend disciplinary boundaries in applying materials toward addressing real-world problems or issues through change efforts → Acronym: TDC

:

• Multidisciplinary Research (MDR): Discipline-based specialists collaborate to learn from distinct approaches and methods toward investigating a shared research question or puzzle.
• Interdisciplinary Research (IDR): Topic-based specialists integrate diverse disciplines, approaches, theories, and methods toward investigating a shared research question or puzzle.
• Transdisciplinary Practice (TDP): Field-based practitioners apply a range of approaches and methods towards transcending disciplinary boundaries in addressing a shared topic or puzzle.
• Multidisciplinary Curriculum (MDC): Approach specialists are exposed to multiple disciplinary resources to gain an understanding of distinct approaches and methods on a shared topic and be equipped to collaborate with other discipline-based specialists in a shared approach to that topic.
• Interdisciplinary Curriculum (IDC): Topic generalists are exposed to multiple disciplinary resources to integrate diverse approaches and methods toward understanding the intersecting contexts surrounding a problem or issue.
• Transdisciplinary Curriculum (TDC): Change practitioners are exposed to multiple disciplinary resources to transcend disciplinary boundaries in integrating and applying diverse approaches and methods toward informing change efforts and addressing a real-world problem or issue.

In summary, research endeavors are structured around bringing multiple disciplines together around a shared research topic, puzzle, or question (Table 1: Row 1). The topic itself causes a degree of unity to the endeavor. If the research endeavor failed, then the research project would cease to exist. Stated differently, the study of interdisciplinary and multidisciplinary research necessarily samples on successful outcomes: funded grants, published articles. Within this successful outcome context, a sufficient level of cohesion existed among the disciplines to foster a shared endeavor, or else the research would not have been funded or published. Similarly, application to practice coheres (Table 1: Row 2)

In contrast, it is plausible for curricular endeavors to exist without coalescing around a shared topic, puzzle, or question. The ideals described in the above table specify the intended curricular objectives, not the achieved impacts of the intended design. Thus, studying manifestations of different forms of disciplinarity is also important in assessing curriculum (Table 1: Row 3). Engaging more than one discipline within curriculum could manifest as multidisciplinarity (Table 1: Row 3, Column 1) or interdisciplinarity (Table 1: Row 3, Column 2), yet without necessarily sampling on success such as research grants only funding projects with integration achieved. Thus, impacts could instead be measured by enrollment or placement rates.

Additionally, a third possible curriculum manifestation exists: transdisciplinarity (Table 1: Row 3, Column 3). Since curriculum is the bridge between research and practice, and transdisciplinarity is relevant for practice, it is a third possible approach for curriculum endeavors, with practice impacts. Thus, a key aspect of empirically investigating manifestations of plural disciplines within curriculum is recognition of at least three ways that disciplines are intended to be combined: (1) multidisciplinary curriculum, (2) interdisciplinary curriculum, and (3) transdisciplinary curriculum.

2.1.3. Measures

Many existing studies operationalize and measure interdisciplinarity. These approaches mostly focus on research-related data. The primary methodological technique in these endeavors is bibliometrics. Scholars utilize bibliometric coding and analysis techniques to investigate how disciplines are grouped within research manifestations. Bibliometrics is a scientific method of measuring interdisciplinarity that analyzes the internal dynamics of bringing different disciplines together and assesses the results of those techniques [16]. The goal is to move beyond discursive attention to, and tacit understandings of, interdisciplinarity toward attempts to measure its empirical manifestations [17].

Bibliometric approaches share a focus on references as the basis of the data, and the bibliometric analysis reveals the structure of interdisciplinarity manifested within the reference outputs. For example, discrete disciplines were coded within research funded by the National Science Foundation (NSF) [18], research published within journal articles included in the Web of Science (WoS) database [19], research published within Essential Science Indicators (ESI) database, Clarivate Analytics, and the National Institute of Science and Technology Policy (NISTEP) of Japan [20]. Coded disciplines are grouped within subject areas, such as Social Sciences, Humanities, and Natural Sciences. These broad subject area groupings of disciplines are then analyzed alongside the structure of interdisciplinarity.

In bibliometrics, broad consensus exists on the importance of three measures of interdisciplinarity: (1) variety, (2) balance, and (3) dissimilarity [15,18,19,20,21]. First, variety refers to the number of disciplines involved. Second, balance refers to the distribution or evenness across disciplines, in terms of the degree of depth to which each discipline is included. Third, dissimilarity (also referred to as disparity, difference, or diversity) refers to the degree of difference between included disciplines. The computations of variety and balance are straightforward to compute and thus utilized consistently. Dissimilarity is the ‘fuzziest’ characteristic of these three measures, yet a consensus on its estimation has emerged. The dissimilarity score is sometimes also labeled as the Rao-Stirling index, Integration score, or Diversity Index [19,20,22,23,24,25].

2.2. Curriculum Assessment

The focus in the current study is on curriculum assessment for six reasons. First, curriculum assessment can be utilized to improve higher education [26]. However, the key to achieving improvement is designing assessment processes in ways that are meaningful, intentional, purposive, and consequential. Second, without an intentional process, assessments can be viewed skeptically as externally driven activities that are completed simply for the bureaucratic purpose of fulfilling state and federal requirements [27]. In this context, a common barrier to successful curriculum assessments is institutional resistance by faculty and administrators who doubt the general utility of assessments, perceive generic assessment tools to be low quality due to not being aligned with intended curriculum, and lack consensus regarding the most relevant and important domains and data to assess in studying curriculum.

Nevertheless, a third reason for focusing on curriculum assessment is that evidence-based storytelling can aid clarification of intended learning purposes [28]. Research findings can be utilized to “force education to explain itself” by facilitating shared meaning-making which supports faculty and staff in collaborating to advance student learning. Fourth, successful implementation of a collaborative assessment process is predicated on the existence of a professional learning community (PLC). PLCs are “collaborative teams who work independently to achieve common goals” [29]. This collaborative culture can promote: (a) clarity of purpose: shared mission, vision, and goals; (b) collective inquiry: an accurate sense of the current reality and best practices; (c) action orientations: unwillingness to tolerate inaction, (d) continuous improvements: a persistent disquiet with the status quo and relentless searching for a better way; (e) focus on results: basis in results rather than intentions; and (f) strong principles: committed to facing and overcoming adversity, conflict, and anxiety.

Fifth, as with other empirical endeavors, it is not wise to assume that theoretical intentions fully manifest. For instance, it is not prudent to merely assume that collaboration is a function of good intentions alone, instead empirical analyses can reveal a possible disjuncture between good intentions and the reality of actual manifestations [30]. In this context, DuFour and colleagues identify three different levels of curriculum: (1) intended curriculum (what was designed to be taught); (2) implemented curriculum (what was actually taught); and (3) attained curriculum (what students actually learned) [29]. Sixth, in order to build toward an assessment process that measures the second and third levels, it is necessary to first measure the intended curriculum by operationalizing what was intended to be taught. For the purposes of this project, the focus is on measuring interdisciplinarity within the intended curriculum.

In summary, curriculum assessment is important for multiple reasons. However, assessing curriculum endeavors is distinct in important ways from studying research activities. Despite key differences between the context of studying disciplinary plurality within curriculum versus research, it is important to build from how existing studies have operationalized key features of interdisciplinarity within a research context. This analysis replicates existing approaches to studying interdisciplinarity within research contexts, with needed modifications for curriculum.

Replications and Modifications

In the current study, the assertion is that the three well-studied measures of interdisciplinarity can be replicated within the context of curriculum assessment. In particular, this analysis replicates attention to variety, balance, and dissimilarity. Yet, important modifications are needed in the study of curriculum for two reasons. First, existing approaches to studying interdisciplinarity are conducted within the context of large and representative research-related databases, such as journal-article and funded-grant databases with hundreds of thousands of data points that represent thousands of researchers and research projects. In this context, the logic of the subject category approach is based in cluster analyses, such as principal component analysis, in which the structure of the category is revealed through the disciplines included within it [19]. The reliability of such an approach is dependent on having a sample that is large and representative [31]. There is no obvious parallel database for curriculum. For example, there is not a widely accessible database that includes thousands of course syllabi representing hundreds of interdisciplinary programs and instructors and with a shared coding scheme. Thus, to be utilized within the context of curriculum, the bibliometric approach needs significant modifications to adapt to available data sources.

Second, subject categories are more standardized within research databases than curriculum endeavors. This is due to the fact that journals and their publishers typically require researchers to categorize their articles within pre-set subject categories [20]. Similarly, it is a common practice for funders to require principal investigators to categorize research projects within subject categories [18], such as the National Science Foundation (NSF) Big Ideas, which are meant to support interdisciplinary research that draws upon the expertise of multiple NSF programs and divisions. There is not an obvious parallel centralizing categorization scheme in the study of curriculum. Thus, to be utilized within the context of curriculum, the bibliometric approach needs significant modifications to adapt to available subject categories.

Despite these differences, replication with modification is possible for two reasons. First, there are curriculum-related data sources that are relevant for modifications of bibliometric techniques. In particular, scholars implemented a bottom-up measurement approach in which they analyzed individual researchers and their affiliations to investigate the degree of interdisciplinarity represented within their research outputs [17]. For example, an article author was identified (e.g., last name of Binsted), and then the current departmental affiliation of this author was identified as their discipline (e.g., Computer Science). Thus, translating from research affiliations of Principal Investigators (PIs) to disciplinary affiliations of faculty and instructors provides a modified curriculum-related replication of the bibliometric coding processes for identifying disciplines within the people responsible for design decisions (PIs/researchers → Faculty/instructors).

Additionally, scholars examined disciplinary diversity within the departmental affiliations included in reference lists [21]. Specifically, the reference list of published articles was coded by discipline according to the current disciplinary affiliations of its cited authors. Thus, translating from journal article reference lists to course syllabi reference lists provides a modified curriculum-related replication of the bibliometric coding processes for identifying disciplines within the degree of exposure to interdisciplinary content (journal/grant reference list for cited content → course syllabi reference list for required content). In summary, to replicate a bibliometric approach to the study of interdisciplinarity, this study implements two sets of modifications to translate from research-relevant to curriculum-relevant data sources.

Second, a curriculum-related database exists that can be drawn upon for an externally validated taxonomy of disciplinary programs. Namely, the National Center for Education Statistics (NCES) developed a coding scheme called the Classification of Instruction Programs (CIP) which codes reported fields of study by disciplinary-like codes [32]. Disciplines are grouped within larger subjects according to a Dewey-decimal-like system, such that seven-digit numbers roll up within two-digit numbers (for example: the seven-digit CIP codes of 45.0601: Economics and 45.1101: Sociology roll up within the broader two-digit CIP code of 45: Social Sciences). These CIP codes are a federal standard for curriculum subject categories, which are engaged by state-level commissions of higher education for policy making decisions (e.g., [33]), and which in turn inform institutional curriculum design and assessment (e.g., [34]). The next sections describe how it was used.

Moreover, the CIP codes are utilized by scholars in investigating topics ranging from STEM to leadership development [35,36]. Thus, translation from journal article and grant database subject categories to degree program subject categories provides a modified curriculum-related replication of the bibliometric analysis process for assessing the structure of relationships between disciplines and subject categories (disciplines within journal/grant subject categories→ disciplines within NCES subject categories). In summary, to replicate a bibliometric approach to studying interdisciplinarity within curriculum assessment, this analysis utilizes the NCES standardized system of subject categories (CIP). Thus, to translate the empirical study of disciplinary plurality manifestations from a research context to a curriculum context, several replications are possible, with necessary modifications.

Table 2. Research to Curriculum Context Replications and Modifications.

Study Design	Existing: Research Context	This Study: Curriculum Context
Method	Bibliometric techniques	Bibliometric techniques
Data Sources	Publication and grant databases	Course syllabi
Data Points	Reference list: works cited PIs/researchers	Reference list: required content Faculty/instructors
Disciplinary Categories	SC: How disciplines are grouped within journal/grant-databases	CIP: How disciplines are grouped within NCES databases

summarizes the replications and modifications from research to curriculum data sources.

3. Methods

3.1. Data

The first step in advancing the study of interdisciplinarity within curriculum assessment is to operationalize interdisciplinarity within the context of academic degree programs. This study implements a novel process for how to apply bibliometric techniques to investigate interdisciplinarity in academic curriculum. Since the majority of existing studies on interdisciplinarity focus on empirical measurement within the context of research, the goal of the current project is to offer a new approach for adapting studies of interdisciplinarity into curriculum-related activities. This study asserts that many of the bibliometric techniques for studying interdisciplinarity within the context of research can be replicated within the context of curriculum. Specifically, this approach replicates a focus on the three most common operationalizations of interdisciplinarity: variety (number of distinct disciplines), balance (distribution of disciplines), and disparity or dissimilarity (difference or distance between disciplines).

While many aspects of IDR can be replicated, there are also notable differences in studying curriculum that require careful adaptations from studying research activities. Importantly, the source of the data is distinct. In studies of interdisciplinary research, data is drawn primarily from two sources: journal article and grant award databases, both of which are typically pre-categorized within existing subject areas. Alternatively, curriculum-related activities are not readily available within existing data sources. To create a comparable data source in curriculum assessment, a selection of course syllabi represent each of three programs.

3.1.1. Field Site

The field site is an interdisciplinary school—the Lilly Family School of Philanthropy—which is devoted to the interdisciplinary pursuit of “improving philanthropy to improve the world” [37]. The school currently offers three degree programs (BA, MA, PhD) in philanthropic studies. Philanthropic studies is defined as “an integrated field of study that develops positive and lasting change in the world” [37]. One of the diverse scholars within the school defined the intended curriculum as follows: “Philanthropic Studies takes as its area all aspects that surround the voluntary giving, voluntary associations, and voluntary actions that have been part of most societies since their beginnings and central to Western, especially civil, societies” [38]. In this context, the field site provides a rich context for studying interdisciplinarity within the curriculum, since the degree programs in philanthropic studies purport to bring many disciplines together (MDC) in the pursuit of integration (IDC) that impacts practices (TDC).

3.1.2. Intended Curriculum

To operationalize the intended curriculum [15,29], the focus is on a sample of course syllabi as the source of data for this analysis. This is for three reasons. First, as was stated: “every program is unique but it is still possible to have a common process for quality assessment” [39]. Coding course syllabi offers a common process across programs. Second, there are two approaches to establishing learning expectations. One is top-down in assessing external standards that are intended to regulate faculty approaches (such as federal or state standards), while the second is bottom-up in designing assessments based upon faculty or program expectations, for example as specified in program learning outcomes or syllabi requirements [39]. Course syllabi offer a way to implement a bottom-up approach to assessment that prioritizes faculty expectations as the basis for what students are expected to know, and syllabi can be examined for explicitly stated expectations. Third, there are two levels of assessment: program level and course level. Coding syllabi offers an avenue for understanding intended curriculum both within courses as well as across courses by aggregating courses to program-level characteristics.

3.1.3. Identification Process

The sample of course syllabi selected for this coding process was identified through the following procedures. First, input on which courses to select was sought from educational professionals who attend to and work within all three programs included in this analysis: bachelors, masters, and doctoral. Second, input on which courses to select was sought from a diverse variety of academicians: faculty, staff, students, and alumni.

Both forms of input were achieved through the same avenue: an academic programs and assessment committee was consulted. The committee was composed of fifteen academicians who represented three academic program administrators, three faculty members, one student-services staff member, six students, and two alumni. Two student representatives represented each program (two from bachelor’s, two from master’s, and two from doctoral). Additionally, one of the faculty members teaches in the masters and doctoral programs, and the other two faculty members teach in both the masters and undergraduate programs. One of the academic program administrators was responsible for managing the undergraduate program, a second was responsible for managing the masters and doctoral programs, and a third was the executive administrator responsible for leading all three programs. The student-services staff member was responsible for supporting students and alumni in all three programs.

The members of the academic programs committee represent diverse higher education roles, and each contributed input into the course identification process. The primary goal of the identification process was to select courses that best represent the foundational content that students in each program are required to know. To achieve this, the committee identified only required courses in each program, excluding elective courses. Second, the committee focused on the required courses that are relatively early in the program course sequence, identifying first and second-year courses rather than third year or more. Third, all members of the committee were asked to nominate courses for consideration based on the above criteria. Fourth, after initial discussions on the merits of considering one course versus another, and the lack of initial feasibility in coding all, a consensus was achieved that multiple courses from each program should be included within the sample. In particular, three to four courses were selected from each program based on expert identification of a sufficient initial sample of required program courses.

3.1.4. Sample

The sample of course syllabi consisted of three bachelor’s courses, three master’s courses, and four doctoral courses. In subsequent sections, these courses are labeled as BA_Course1, BA_Course2, and so forth. This corresponds to the following list of courses:

• BA_Course1—PHST 201: Introduction to Philanthropic Studies
• BA_Course2—PHST 210: Philanthropy and the Social Sciences
• BA_Course3—PHST 211: Philanthropy and the Humanities
• MA_Course1—PHST 521 Nonprofit and Voluntary Sector
• MA_Course2a—PHST 524a Civil Society in Comparative Perspective
• MA_Course2b—PHST 524b Civil Society in Comparative Perspective
• PhD_Course1—PHST 660: Ethical, Moral, Religious Aspects of Philanthropy
• PhD_Course2—PHST 662: Historical & Cultural Perspectives of Philanthropy
• PhD_Course3—PHST 664: Philanthropy and Nonprofits in Society I
• PhD_Course4—PHST 665: Philanthropy and Nonprofits in Society II

These courses are understood to be the foundational required courses for each program.

3.1.5. Required References

This section provides descriptive statistics regarding the course sample and required reference lists. These data form the objects for the coding procedures described in a subsequent Section 3.2, and the results of that analysis are presented subsequently (4). In total, a sample of ten course syllabi were pooled into a shared syllabi bank. The syllabi requirements were abstracted from the syllabus document, namely identifying the course required readings and resources. For example, if a course required readings from six book chapters and four journal articles, then a combined list of those ten references was abstracted as the basis of required learning in the intended curriculum. Likewise, if a course required ten readings and five organizational site visits, then a combined list of reference information for the ten readings and the five organizational representatives was abstracted. The focus was on bibliometric data listed in the syllabi as required content.

There was a wide range in the number of required readings and resources for each course, and across each program. As described above, there were 3–4 courses composing the sample for each program. In

Table 3. Number of Required Resources by Program.

	BA Program	MA Program	PhD Program
min	17	11	8
max	41	36	183
mean	25	25	84

, N denotes the number of courses: 3 bachelor’s courses, 3 master’s courses, and 4 doctoral courses. The other numbers reported in Table 3 represent the number of required resources within each course. The minimum number of syllabi required references was 17 for the undergraduate level (for BA_Course3), 11 for the master’s level (for MA_Course1), and 8 for the doctoral level (for PhD_Course2). The maximum number of syllabi required references was 41 for the undergraduate program (for BA_Course2), 36 for the master’s program (for MA_Course2), 183 for the doctoral program (for PhD_Course3). The average number of required references was 25 for the undergraduate courses, 27 for the master’s courses, and 84 for the doctoral courses. Table 3 summarizes these descriptive statistics for the number of required references.

3.1.6. Measures

The curriculum-related data for this study included three measures: variety, balance, and dissimilarity. The data for these measures were developed through seven steps. First, the intended curriculum was represented from a set of required courses for each program. Second, the syllabi for each of these courses were collected and shared within a syllabi bank. Third, the required resources were extracted from the list of course requirements within each course syllabus. Fourth, reference entries were cleaned to extrapolate to researcher full names. Fifth, disciplines were identified via a combination of publication source and author affiliations. Sixth, disciplines were coded and grouped within subject categories utilizing the NCES Classification of Instruction Programs (CIP) taxonomy [32,40,41,42,43]. Seventh, calculations of variety, balance, and dissimilarity were computed using replicated formulas. The procedures for coding and analyzing these measures are described in further detail within the following sections.

3.2. Coding

To operationalize interdisciplinarity within curriculum, course readings and resources were coded using the following procedures. The coding process focused on identifying the disciplines included within the required resources for each course. The five coders for this project were: (1) a non-administrative faculty member who teaches three of the sampled courses, (2) a doctoral student who completed the four doctoral courses, (3) an undergraduate student who completed the three bachelor’s courses, and (4–5) two alumni who graduated from the master’s program the previous year and completed the three master’s courses. These coders were also members of the academic programs committee described above, and three also compose the co-authorship team for this paper. Coders first identified disciplinary affiliations of the publication source, and then for non-identified publications proceeded to identify the authors.

3.2.1. Coding Structure

The coding scheme used to study levels of curriculum interdisciplinarity was the taxonomic Classification of Instructional Programs (CIP). The CIP coding system, “is the accepted federal government statistical standard on instructional program classifications” and “is used by state agencies, national associations, academic institutions, and employment counseling services for collecting, reporting, and analyzing instructional program data” [32]. The CIP taxonomy was created in 1980 by the National Center for Education Statistics (NCES), the primary entity within the U.S. Department of Education that gathers and reports on education data [41]. The most recent 2010 edition of CIP codes is its fourth revision and takes into account the vast proliferation of both academic and career-focused degree programs over the last several decades [32,42].

Additionally, the CIP system is a list of fields of study of recognized programs across higher education institutions, and the codes are utilized in degree creation policies and approval processes. For example, only one degree can be classified within the same CIP code at the same university. Within the taxonomy, the broadest grouping of degree programs is represented by a series of two-digit numbers. For the next steps down, a series of four-digit numbers represents mid-level groupings and a six-digit series represent certain programs [32]. The field site for this study, a Philanthropic Studies program, is coded most narrowly as CIP 44.0201) Community Organization and Advocacy, which is nested under 44.02) Community Organization and Advocacy, and most broadly, 44) Public Administration and Social Service Professions.

In this study analyzing interdisciplinarity in curriculum, utilizing CIP codes proved to be an effective form of measurement. While the CIP taxonomy is not perfect, it does offer a federal standard for coding disciplines that is backed by an extensive research process, and which is engaged by commissions of higher education when making policy decisions. These policy decisions in turn inform curriculum decisions within academic institutions. In the Philanthropic Studies field site outlined in this study, despite widespread use of CIP codes in initial programming decisions, the codes remain under-utilized in curricular assessment practices. The goal here is to engage CIP codes in measuring whether interdisciplinarity exists in the philanthropic studies curriculum.

3.2.2. Publication Source

For required readings, a sequential prioritization was implemented based on the publication source as the priority and co-author team as the second priority. First, the reading was coded by publication source: (a) single-discipline or multi-discipline journal, (b) interdisciplinary journal, (c) book, or (d) book chapter. A: The reading was coded as published in a single-discipline journal if the name of the journal specified a single discipline, and if so then coded as that discipline. The rationale is that the journal subject experts, such as the editor and the peer reviewers, awarded publication for this article as a work representing the discipline specified in the journal title. For example, an article published in the American Economic Review was coded as CIP 45.0601: Economics. Additionally, the reading was coded as published in a multi-disciplinary journal if the name of the journal specified more than a single discipline. For example, an article published in Journal of Economics and Management Strategy was coded as CIP 45.0601: Economics and CIP 52.0201: Business Administration and Management.

B: The reading was coded as published in an interdisciplinary journal if the name of the journal did not specify any disciplines, and the mission statements for these journals confirmed their purpose was to be interdisciplinary. For example, a reading of a journal article published in the Nonprofit and Voluntary Sector Quarterly was determined to be interdisciplinary. C+D: The reading was coded as a book or book chapter if the reference listed the publisher as a book press, such as University of Chicago Press. When a reference was coded as a B-D publication source, the following co-author team process was implemented to identify the included disciplines.

3.2.3. Disciplinary Affiliations

To identify author disciplines, full reference details were inputted, including spelling out in full the first name and last name of each author, as well as spelling out in full any abbreviated journal or book publisher names. The most labor-intensive aspect of this process was attending to APA-formatted references, which abbreviate author first names to a single letter. In these instances, search engines were utilized to find the publication parameters and locate the full author names. Author first and last names were then utilized to identify each contributing author to locate a professional biography and/or Curriculum Vitae listing their disciplinary affiliations. Author disciplinary affiliations were then coded by CIPs. For example, Daniel P. Aldrich is the author of an article published in the Journal of Civil Society [1]. His professional biography at Northeastern University identifies him as a Professor of Political Science, Public Policy, and Urban Affairs, PhD and MA in Government, MA and BA in Asian Studies, as well as a member of the American Political Science Association, Association for Asian Studies, and Disasters and Recovery. As a result, his reference was coded with the following disciplinary affiliations: CIP 45.1001: Political Science and Government, 44.0501: Public Policy Analysis, 45.1201: Urban Studies/Affairs, 43.0302: Crisis/Emergency/Disaster Management, 05.0104: East Asian Studies. Each reference could have multiple disciplines, and the disciplines of all authors were coded. Thus, one major advancement of this study was to assess disciplines through current affiliations along with educational backgrounds.

3.2.4. Interrater Reliability

To enhance interrater reliability, two procedures were implemented. First, each author name was hyperlinked with the source information to enable the other coders to review the data utilized to assign a disciplinary code. For example, a doctoral student assessed the relevant discipline, and then the faculty member reviewed the disciplinary codes for accuracy. Second, as an additional alignment check, two coders attended to the same set of references and applied codes independently. The codes were then reviewed by the faculty member who deemed one of the coders to be implementing a more accurate process than the other coder. This process was collectively reviewed and described to implement the same coding procedures across each syllabus. Subsequently, two members of the academic programs committee who did not code or author reviewed the paper to further check the reliability, and results were presented to additional faculty for comment.

3.2.5. Additional Resources

On occasion a course required additional resources beyond readings. For example, one course integrated organizational site visits that were specifically identified in the syllabi, and around which several required assignments were based. In this example, the students were explicitly introduced to the disciplinary educational backgrounds of each of the representatives at the site, and the students were prompted to write about these within their own career reflections after the site visit. Since the required course resources presented the opportunity for students to gain explicit exposure to the disciplinary affiliations in the educational backgrounds of each of the organizational representatives, these were also coded (see Section 3.2.3 Disciplinary Affiliations above).

3.2.6. Course and Faculty Affiliations

When listed, courses were coded according to the primary disciplinary set in the title. For example, BA_Course2: Philanthropy and the Social Sciences was coded as a social sciences course, whereas BA_Course3: Philanthropy and the Humanities was coded as a Humanities course. Since each program and particular courses also include opportunities for students to gain exposure to the disciplinary affiliations of the faculty members in the school, a similar coding process for author affiliations was implemented for faculty affiliations (see Section 3.2.3 Disciplinary Affiliations above). Faculty affiliations were also utilized to designate courses within a specific disciplinary set when one was not specified in the course title. For example, the history and ethics courses in the doctoral program (PhD_Course1, PhD_Course2) are regularly taught by historian faculty who identify as Humanities, and thus those were coded as Humanities courses. Similarly, the ‘in Society’ courses in the doctoral program (PhD_Course3, PhD_Course4) are regularly taught by economics and international relations faculty who identify as Social Sciences, and thus those were coded as Social Science courses.

3.3. Analysis

The analysis of interdisciplinary curriculum consisted of three primary measures that were designed to replicate existing measures within interdisciplinary research: (1) variety, (2) balance, and (3) dissimilarity [15,18,19,20,21]. As described in the previous sections, important modifications were made to adjust these measures to a curriculum context.

3.3.1. Variety

Variety was measured as a count across all the required readings and resources to indicate the number of unique disciplines to which students were exposed within each course. Each course has variety computed (Figure 1 Variety: course-level), and each course also has a variety score based on the average number of disciplines within each required reference (Figure 1 Variety Score: course-level). To facilitate comparisons across programs, variety scores were also averaged across courses to compute a variety score for each degree program (Figures 4 and 5 Variety Score: program-level: BA, MA, and PhD).

3.3.2. Balance

Balance was measured as the average evenness of disciplines across required references to indicate the average depth of attention to each discipline within each course. Each course has a balance score (Figure 2 Balance: course-level), and courses are identified by disciplinary emphasis. Scores were averaged across representative required courses to compute a balance score for each program (Figure 3 Balance: program-level). Thus, the degree programs can be compared for interdisciplinary balance.

3.3.3. Dissimilarity

Dissimilarity was measured as the degree of difference between disciplinary subject categories. Since the field site was developed out of a liberal arts tradition steeped in the Humanities, Humanities was set as the baseline (1*a), where a represents the number of resources that exposed students to disciplines within the Humanities subject category. Social Sciences was set as one degree of difference (-1*b), Professions as two (-2*c), and Natural/Computational Sciences as three (-3*d). The dissimilarity formula was computed as = (1*a) + (-1*b) + (-2*c) + (-3*d). Each course received a dissimilarity score (Figure 4 Dissimilarity: course-level), and scores were averaged across representative required courses to compute a dissimilarity score for each program (Figure 5 Dissimilarity: program-level). Due to the imperfections with this measure, raw counts for subject categories were also computed (Figures 6–10). Thus, the degree programs can be compared for interdisciplinary subject category dissimilarity and exposure.

4. Results

4.1. Course-Level Curriculum

In terms of variety (number of disciplines), the results indicate that there is a wide range. For BA-level, the variety ranges from a low of 7 disciplines in a Humanities course to a high of 63 disciplines in a Social Science course. A range from 27 to 39 is found from MA-level courses. For the PhD-level, courses range from a low of 7 disciplines in a Humanities course to a high of 99 in a Social Science course. The results for variety scores (average number of disciplines in each reference) shows a relatively flat pattern. Variety score ranges from 1.13 to 3.34 across BA-level courses, from 3.27 to 4.33 across MA-level courses, and from 1.61 to 4.12 across PhD-Level courses. The relatively flat pattern within references (variety score) with tremendous variation across courses (variety) indicates that the variety of disciplinary exposure appears to be due to the structure of the course references rather than disciplinary structure within references. Results shown in Figure 1.

Regarding the results of balance, referring to the relative depth of attention, there is also a range across courses. In the BA courses, disciplines are cited an average of about 2 times in each resource. The MA-level courses varied from an average low of about 2 disciplinary citations in a Humanities course, to a high average depth of nearly 4 in a Social Sciences course. The PhD-level courses ranged from a low average depth of 1.6 in a Humanities course to a high average depth of nearly 5 in a Social Science course. The results are summarized in Figure 2.

Ranges are also found in the dissimilarity measure, referring to the degree of difference. In interpreting the results, it is necessary to understand that negative scores represent a higher degree of dissimilarity than positive numbers. In the BA courses, the dissimilarity score ranges from not highly dissimilar (positive 14) for a Humanities course to highly dissimilar (negative 83) for a Social Science course. The MA level courses ranged from a dissimilarity score of negative 36 in a Humanities course, to a high of negative 166 in a Social Sciences course. The PhD-level courses ranged from low dissimilarity, 6 and 21 in two Humanities courses, to the highest degree of dissimilarity across all programs, negative 527 and negative 360 in 2 Social Science courses. The dissimilarity scores are summarized in Figure 3.

However, analyzing the dissimilarity score, as replicated from previous bibliometric studies, obscures a clear understanding of the actual degree of difference. This is because multiple sets of disciplines are collapsed into a single measure. Thus, a subsequent section will disambiguate disciplinary composition utilizing a novel technique.

4.2. Program-Level Curriculum

In summary, Figure 4 visualizes the program-level results, comparing variety and balance scores across programs. The MA degree has the highest discipline variety score as 3.86. Balance increases in a stepwise fashion across programs, from lowest in BA (2.22), middle in MA (3.26), highest in PhD (3.5).

Figure 5 aggregates the program-level results for variety and dissimilarity scores. Both the BA and MA courses have a variety of 34, whereas the PhD courses have a higher variety score of 58. The BA courses are found with the smallest dissimilarity score (negative 43), and PhD with the largest dissimilarity score (negative 294).

4.3. Disciplinary Composition

Therefore, to gain a clearer sense of the composition of each program, the following data visualizations disambiguate each of the four sets: (1) Humanities, (2) Social Sciences, (3) Professions, and (4) Natural and Computational Sciences. Figure 6 shows that in the bachelor’s program, the compositional structure, in descending order from highest to least coverage, is 0.5 Social Science, 0.23 Professions, 0.21 Humanities, and 0.05 Natural & Computational Sciences.

Figure 7 shows that the master’s program is composed of 0.76 Social Science, 0.64 Professions, 0.45 Natural & Computational Sciences, and 0.07 Humanities.

Figure 8 shows that the doctoral program is composed of 0.64 Social Science, 0.24 Professions, 0.07 Humanities, and 0.05 Natural & Computational Sciences.

Additional analyses examined the disciplinary composition of faculty members, which is displayed in Figure 9 as composed of 0.45 Social Science, 0.3 Professions, 0.18 Humanities, and 0.06 Natural & Computational Sciences.

Combining the compositional results (as shown in Figure 10) with the earlier findings regarding variety and balance, it appears that the PhD program most embodies Multidisciplinary Curriculum (MDC), with relatively low variety with highest balance score. Distinctly, the BA program appears to have the greatest degree of integration, having a high variety with lowest balance, and the most representation of Humanities. In this regard, the BA program appears to most embody Interdisciplinary Curriculum (IDC).

5. Discussion

In summary, this bibliometric analysis of curriculum revealed three primary findings regarding studying interdisciplinarity. First, multidisciplinarity, defined as the presence of multiple disciplines that are brought together for specialized contributions to a shared topic, appears to be manifested in the doctoral program. Second, interdisciplinarity, defined as the integration of diverse disciplines, appears to be manifested in the bachelor’s program. Third, transdisciplinarity, defined as the application to practice that transcends disciplinary boundaries in addressing real-world problems, appears to be manifested in the master’s program.

While interdisciplinarity is often the default term utilized, this data-based bibliometric approach to analyzing curriculum indicates that there are important distinctions between approaches to discipline plurality. Notably, Trinh and colleagues state [44]:

• Unlike multidisciplinarity, which additively contributes insights and methods from multiple fields of knowledge but does not challenge discipline boundaries, and unlike transdisciplinarity, which subordinates disciplinary knowledge to systems-level approaches, interdisciplinarity integrates insights and methods from various fields to form a coherent body of knowledge [45].

One implication of this in curricula practice is that programs can be more transparently and authentically described by engaging precision in distinguishing between the three approaches.

5.1. Studying Interdisciplinarity

More generally, this application of bibliometric techniques was effective in studying interdisciplinarity within a curriculum context. Considering the preconceived subjective perceptions that these academic programs manifest interdisciplinarity, it was unexpected to find that the three programs manifest disciplinary plurality in distinct ways. The programs are typically described as situated within a liberal arts tradition, which is represented most strongly by Humanities disciplines. In this context, it was also unexpected to find that the compositional structure across all three programs was tipped toward the Social Sciences. Likewise, it was notable that the degree of representation of each disciplinary set varied across the three programs. These insights underscore the benefits of applying an evidence-based approach to studying curriculum. Bibliometric data can advance professional learning communities beyond subjective perception of what could be toward logical, analytical, and evidence-based accounts of what is. The results were presented to all members of the academic committee, several additional faculty members, and the broader university system during a conference about online degree programs. The results were well-received, and the audience within this field site confirmed that the findings were unexpected given the perceived unity in approaches.

As Stobierski stated [46]: “Society has imbued the concept of ‘intuition’—of simply knowing when something is right or wrong—with a tremendous amount of prestige, importance, and influence...it’s through data that you verify, understand, and quantify.” Additionally, MacLaughlin [47] describes data as a tool for increasing organizational effectiveness, a way to improve programs and services, stating: “Without data, decisions are left to tribal knowledge or worse, the whims of the Highest Paid Person’s Opinion (HiPPO).” Data are important reflection tools within what Weick described as the ‘sensemaking process’ in organizational structure and behavior [48]. When organizations make retrospective sense of the situations in which they exist, the clarification process itself creates beneficial changes. This process of learning is key for designing organizations to be resilient in addressing complexity [49]. In this context, the act of operationalizing and analyzing curriculum can aid university academic units in overcoming organizational ‘immunity to change’ and becoming adaptive, responsive, learning organizations [50,51].

5.2. Intended Curriculum

A central purpose of this study was to apply bibliometric techniques, which were designed to study research impact, within the study of curriculum. When assessing variety, balance, and dissimilarity in the curriculum of the bachelor’s, master’s, and doctoral programs at the field site, the results reveal that the programs have different purposes and as a result, embody differing modes of disciplinarity. The doctoral program appears to manifest a multidisciplinary curriculum. Here, students are exposed to multiple disciplinary resources to gain an understanding of distinct approaches, so they can then make specialized contributions to a shared topic. This enables doctoral students to collaborate with other discipline-based specialists on their topics. In contrast, the bachelor’s program utilizes an interdisciplinary curriculum. In this curriculum, students are exposed to multiple disciplinary resources, learn to integrate diverse approaches, and intersect contexts around specific issues. Third, the master’s program has a transdisciplinary curriculum, showing an application to practice that transcends disciplinary boundaries. This curriculum moves students from exposure to multiple disciplines to a broader method of thinking that allows them to apply diverse methods to addressing real-word problems.

Additionally, the degree of representation of each disciplinary set varied in interesting ways across the three programs. From this study of the various programs, it appears that the curriculum can be classified in at least three different models. First, the intended curriculum can be thought of as promoting a field of study, where its goal is multidisciplinarity. Here, it relies on the legitimacy of multiple existing disciplines to utilize their tools to study the topic [38,52]. For example, Public Economics could be considered a field of study, which is in some respects a subfield of Economics and in other respects is broader than the discipline of Economics in intentionally engaging in multidisciplinary endeavors with the discipline of Public Policy, among others. For a field of study curriculum, the goal of a doctoral program would be to place students in specific disciplinary departments. Success of this intended curriculum, relative to uni-disciplinary endeavors would be empirical verification of placements within multiple disciplinary departments, as placements within a single discipline would not empirically validate that the intended multidisciplinary curriculum was manifested. Within this study, our assessment revealed that this model is most relevant for the doctoral program.

Second, the intended curriculum can be thought of as contributing to a professional field, where transdisciplinary curriculum integrates existing curriculum models in service to the profession [53]. Existing models for this approach include medical and law schools, and in the field site of this study is most relevant for the master’s level program. In this model, the goal would be to have students hired across a specific set of supported subfields. For example, within the medical field, one school of medicine could be known for its expertise in oncology, whereas another could specialize in pediatrics. Likewise, a single law school could offer specializations within both constitutional law and health care law. In the context of this study, the academic field site focus on the professional field of philanthropy could include specializations within specific subfields, such as health philanthropy and education philanthropy. Success of this intended curriculum could be measured by the number of placements and typical salary ranges of students into each of these subfields.

Third, the intended curriculum could alternatively be thought of as its own discipline and aim to develop a new CIP code, where the purpose is truly interdisciplinary, to integrate traditional disciplines to form a new discipline with its own theories; this would utilize an interdisciplinary curriculum [53]. Existing models of this type of approach include International Relations, American Studies, and Biochemistry. Within the field site of this study, this approach is most relevant to the bachelor’s degree program. One goal of this intended curriculum could be to facilitate the development of other programs that utilize this same CIP code, for example as occurred when social work programs were created and propagated. Another goal of this intended curriculum could be to place doctoral students in programs that also utilize this newly developed CIP code. Perhaps a third goal could be to form an accrediting body that organizes and assesses curricula for all programs within this CIP code.

In the field site of this study, the unresolved set of directions for the approach to how multiple disciplines are incorporated leads to curricular confusions regarding purposes, goals, and approaches of the bachelor’s, master’s, and doctoral programs. Despite the stated value of interdisciplinarity, little was known empirically about its curricular practice. This study helped to rectify this problem by addressing a data-based gap in knowledge.

5.3. Diversity, Equity, and Inclusion

Beyond the intrinsic value of better understanding disciplinarity and its manifestations, there are also broader extrinsic goods that could result from these pursuits. Specifically, many existing studies have identified a relationship between demographic characteristics and disciplines. Jones [54] found that clusters of academic disciplines resulted in distinct socialization styles that were characterized by different teaching beliefs and practices, departmental functioning and leadership, and employed research practices. Moreover, de Brey and colleagues [55] analyzed NCES data and found that disciplines correlated with race and ethnicity. For example, at the undergraduate level, natural sciences have higher concentrations of students who identify as Asian and lower concentrations of students who identify as black. Comparatively, the Social Sciences and history have more diverse concentrations across all racial and ethnic categories, with the highest representation among students who identify as Hispanic and as multiracial. Health professions and business have the highest concentrations of students who identify as Pacific Islander and American Indian.

Additionally, Welde [56] analyzed NSF and NCSES data and found that over two decades of graduation rates, underrepresented minorities made significant gains in bachelor’s (about 12 to about 19 percent) and master’s (about 7 to about 16 percent) degrees. However, the trend for doctoral degrees differed by disciplinary set, such that underrepresented minority degree gains for natural and computational sciences was stagnant (about 4 to about 6 percent), relative to marked increases for all other doctoral dissertation degrees (about 8 to about 16 percent).

Plus, disciplines also correlate with gender. Stepan-Norris and Kerrissey [57] found that the proportion of women with doctoral degrees being hired into new tenure-track appointments was highest in Education and Humanities/Social Sciences (about 62 and 52 percent, respectively) and lowest in natural and computational sciences (about 19 percent). Similarly, Tran and colleagues (2019) found that disciplines varied by gender ratios in faculty composition. Of natural and computational science departments, about 48 percent were predominantly men. Alternatively, about 15 percent of Social Science and Humanities departments were predominantly men, and about 5 percent of communication and librarian disciplines were predominantly men.

Perhaps one of the mechanisms of this gender, racial, and ethnic sorting process is the embedded skill sets in disciplines. For example, Owens and Lilly [58] found that technology skills correlated with discipline, such that pharmacy and law students consistently ranked higher in technology capacities than social work and nursing students. Another potential set of mechanisms for disciplinary self-selection are social psychological factors, such as math self-concept, aspirations for science, and self-perception as analytical versus passionate. Yet, research shows that interest gaps are shaped within cultural milieus, family disadvantages, and other interpersonal experiences which accumulate across pre-college paths and experiences. In this context, fostering discipline plurality within a single academic unit could be a way to address deep-rooted diversity, equity, and inclusion issues. The problematic aspects of self-selection processes embedded within disciplinary sorting could be transcended through academic approaches that intentionally design disciplinary plurality into curriculum, and as a result bring together the demographic diversity of students channeled by discipline.

5.4. Limitations and Future Studies

Despite several contributions of this study, it is worth noting that there are at least three limitations. First, replicating the dissimilarity score of prior studies resulted in interpretation difficulties, at least within the context of curriculum. Variety and balance measures are more straightforward to calculate and interpret within the context of curriculum. As prior studies have noted [20], dissimilarity is considerably more challenging to measure and standardize across data sources. In reviewing prior studies, it seems that the computation of the measure has typically been developed inductively from the data source to a calculation theory. This makes it a challenging measure to replicate in a study of distinct context: curriculum. To improve upon prior models for operationalizing this construct, this paper contributed an approach to analyzing the compositional structure of the required content and categorized this content within four major disciplinary sets: Humanities, Social Sciences, professions, and natural/computational sciences. This disambiguation of the measure can improve its relevance for instructors. Future studies can replicate and further test this approach.

Second, the corpus of sampled course syllabi is relatively small, yet the courses represent the foundational required knowledge for each program. These required courses are not necessarily representative of all the curriculum in each program, nor of curriculum in other academic units. Questions remain about the extent to which multidisciplinarity, interdisciplinarity, and transdisciplinarity are taught in the curriculum outside of the core foundational courses that were coded as part of this study. Thus, future studies can replicate the method offered in this study to code additional syllabi, increase sample size, and facilitate comparisons across academic units and universities. For example, a nonprofit curriculum library has recently been developed to partner in sharing course resources [59]. At the time of this writing, the library contains 20 syllabi. Though the size of this sample is modest, the establishment of a curriculum library signals the future potential of the method contributed in this study advancing to a larger sample.

Third, the method proposed and implemented in this study focused on the first step of the assessment process: the intended curriculum. This was an important starting point to engage faculty instructors in this process. Future studies need to study outcomes (such as student learning, graduation, and placement) to understand the impact of this content. For example, it is interesting to note that of the three programs, the online MA program is the one that has experienced the greatest enrollment growth in recent years.

6. Conclusions

In summary, this study offers five important contributions that advance the relevance of bibliometrics in the study of interdisciplinarity. First, this study synthesizes and replicates existing approaches to studying citation and reference patterns. Second, the methods innovated in this study propose and test a set of procedures designed to translate existing bibliometric techniques from studying interdisciplinarity within a research context to a curriculum context. Third, the design includes a novel focus on the intended curriculum as manifested through required reference content. The systematic implementation of a bibliometric analysis and visualization facilitates an objective evaluation of the curriculum structure. The result was a counter-intuitive finding that there are three approaches to manifesting disciplinary plurality embedded within the same academic unit. Fourth, a novel process for utilizing bibliometric techniques to operationalize and assess the intended curriculum was developed). Fifth, this process could facilitate efforts to increase diversity, equity, and inclusion within curriculum design by integrating the uni-disciplinary disparities by gender, racial, and ethnic statuses. For example, through disciplinary plurality the high male composition of computational and engineering programs could be balanced with a higher proportion of females within social sciences.