Understanding the Effects of a Math Placement Exam on Calculus 1 Enrollment and Engineering Persistence

Abstract

Educational institutions are grappling with declining enrollments and low mathematical achievements. This study investigates how a math placement exam (ALEKS) influences enrollment in Calculus 1 and student persistence, taking into account academic preparation and demographic factors. It also evaluates the effects of remedial math courses for students near the placement cutoff. Using Astin’s input–environment–outcome model, this study analyzed data from 3380 students employing a Kitagawa-Oaxaca-Blinder decomposition and fuzzy regression discontinuity. These methods were used to identify unexplained differences across demographic groups and capture outcomes near the math placement cutoff. Based on the findings, a cutoff of 80% for the ALEKS exam is appropriate. This study underscores the role of math placement exams in shaping engineering enrollment and student success. These findings prompt reevaluating placement strategies and support mechanisms, particularly for URM, first-generation, and female students, to enhance equity and retention in engineering.

1. Introduction

Higher education faces a looming enrollment cliff due to declining birth rates, resulting in fewer students pursuing higher education (Bauman, 2024). Some colleges will not survive the changes in student demand (Barshay, 2018). Therefore, there has been a call for institutions to use data to inform practices to respond to the uncertainty of future enrollment trends (Campion, 2020). The enrollment cliff is particularly concerning for engineering since continued concern exists about meeting the demand for engineering professionals in the future (National Academies of Sciences, Engineering, and Medicine, 2018), and the U.S. Bureau of Labor Statistics (2023) shows the engineering profession needs to grow significantly over the next 10 years (Occupational Projections and Worker Characteristics), so it is crucial to consider barriers that prevent capable, interested students from pursuing engineering. Moreover, to meet the complex challenges of the future and ensure that engineering solutions are effective, inclusive, and innovative, we need to increase the number of students studying engineering and diversify the workforce (National Academy of Engineering, 2018). Therefore, increasing the number of students from diverse backgrounds pursuing STEM degrees, particularly in engineering, is essential to addressing these challenges. Historically marginalized groups, including women, underrepresented minorities (URM), and first-generation students, remain underrepresented in engineering programs. By broadening participation and reducing barriers for these students, we can not only fill the growing demand for engineering professionals but also foster a more diverse workforce that drives innovation and produces more equitable and effective solutions.

One of the major barriers interested engineering students face in their pursuit of an engineering degree is mathematics. Mathematics placement is often used to predict student success in the major and curriculum trajectory (Gardner et al., 2007; Heinze et al., 2003). Further, Calculus 1 is often a course engineering students take in their first semester (Bressoud, 2015; Krause et al., 2015), showing math placement and readiness for Calculus are crucial for students interested in pursuing engineering. Many universities have newly instituted or continued to use math placement exams to determine eligibility for engineering programs, such as the ALEKS assessment. Though intended to gauge academic readiness, mathematics placement tests, and subsequent remedial courses are costly and disadvantageous to students’ progress through an engineering degree (Main & Griffith, 2022), thus showing the potential barriers math placement exams have on engineering students.

Math achievement for students in U.S. higher education reached a historic low in the aftermath of the COVID-19 pandemic and lockdown (Binkley, 2023), with similar impacts on mathematics achievement reported globally (Contini et al., 2022; Moliner & Alegre, 2022). Emergency transition to remote learning, extended time away from the classroom, and the social isolation students experienced contributed to changes in their academic readiness (Dorn et al., 2021). With varying restrictions across states and school districts (Bollyky et al., 2023), high school students had varied learning experiences during this time, and math achievement gaps were exacerbated across diverse groups of students (Kuhfeld et al., 2022), specifically students from underrepresented groups (Lewis & Kuhfeld, 2023) and with low socioeconomic status (SES; Jones-Alford, 2023). More engineering students have entered higher education as pre-math-ready or placed below the first-year math class required for the major. This is problematic because mathematics is a critical component of the engineering curriculum, and students often need to demonstrate proficiency in math by completing mathematics prerequisites required for discipline-specific engineering classes (Heileman et al., 2017).

We are interested in understanding the impacts of tools used for mathematics placement, such as the ALEKS assessment, and its impact on student trajectories in first-year engineering. The ALEKS assessment was intended to support student learning; however, it is now used by some universities for mathematics placement. Thus, this paper investigated current practices and the use of math placement for engineering students post-pandemic at a large university. By examining ALEKS placement outcomes and its ramifications, we aimed to contribute to a dialogue on how we support engineering students in mathematics, considering increasing achievement gaps and the looming enrollment cliff. Our study aimed to illuminate differences in Calculus 1 enrollment across demographic groups, impacts on persistence in engineering in the first term, and future performance in Calculus 1.

2. Literature Review

2.1. Theoretical Framework

This study used Astin’s input-environment-outcome (I-E-O) model (Astin, 1991). Astin’s I-E-O model emphasizes the importance of considering outcomes in the context of various student input characteristics and environments. In this model, inputs refer to personal characteristics students bring to the educational program, environment refers to student’s experiences in the educational program, and outcomes refer to skills being developed through the educational program. In the context of this study, input characteristics include gender, first-generation status, underrepresented minority status, zip code, and academic preparation. The environment of the educational program we considered was the ALEKS test or math placement mechanism for engineering students, as that is part of the educational program. Finally, outcomes relate to student’s progression in engineering through their enrollment and grades in Calculus 1. The ultimate goal of the I-E-O model was to learn how to structure educational environments to maximize talent development, which aligns with the goals of this investigation into the impact of the ALEKS assessment as a placement mechanism for engineering. In the following sections, we outline existing literature related to the inputs, environment, and outcome variables included in this study and how they relate to math readiness.

2.2. Inputs: School and Student Characteristics for Math Readiness

Disparities in high school resources are a function of inequities across the U.S. public school system. Students attending a low-resourced school likely have less qualified teachers, and limited course offerings. An exploration of this literature supports the consideration of these factors as inputs related to math readiness. For this reason, considering zip code, gender, first-generation status, and underrepresented minority status was appropriate for our analysis.

2.2.1. School Characteristics and Math Readiness

The U.S. public school system is not an equal playing field; differences in resources, teacher certifications, class offerings, and pedagogical approaches impact students’ academic achievement (Clotfelter et al., 2007; Committee on Developing Indicators of Educational Equity et al., 2019; Darling-Hammond et al., 2005; Smith et al., 2013). Teacher quality is a significant factor that impacts students’ readiness for engineering. For example, the most qualified science teachers primarily teach at suburban schools (National Center for Education Statistics, 2012). Furthermore, suburban schools tend to have less teacher turnover than urban schools (Ingersoll & May, 2012), which undoubtedly affects teacher quality. Moreover, a study investigating the National Educational Longitudinal Study and U.S. Census data on residential zip codes of students revealed disadvantaged neighborhoods and schools are directly associated with lower levels of math achievement (Catsambis & Beveridge, 2001). School characteristics significantly impact the instruction students receive and their math readiness. Educational opportunity gaps and school resources affect students beginning in preschool and persisting through high school (Barnett & Lamy, 2013).

2.2.2. Student Characteristics and Math Readiness

First-generation students, or students who are the first in their families to attend college, face significant barriers in their pursuit of a college degree. First-generation students are more likely to delay their entry to college and take remedial classes, which impacts their persistence (Engle, 2007). Further, first-generation students are less likely to take advanced high school mathematics courses (Horn & Nuñez, 2000), affecting their mathematics placement and readiness for college-level mathematics courses, which aligns with findings showing first-generation students are more likely to enroll in remedial classes upon entering college (Chen, 2016). The variation in math readiness for first-generation college students, as well as the potential additional challenges they may face in college, could deter interested, capable students from pursuing engineering.

In the United States, studies on mathematics readiness have shown that despite efforts to achieve gender equity, gender disparities still exist in mathematics achievement at the high school level (Houser & An, 2015). Despite varied findings, most literature shows female students underperform on standardized math tests. Further, recent research suggests the gender gap in engineering may be attributed to differences in self-efficacy and interest levels. For instance, a study by Buontempo et al. (2017) surveyed girls enrolled in a high school engineering course and found girls generally exhibit lower self-efficacy and interest in engineering than boys. From a global context, a meta-analysis showed that on average, the gendered difference in performance is minimal; however, more positive mathematics attitudes toward male students were found (Else-Quest et al., 2010). Therefore, these differences seem to be more related to cultural factors.

Significant intersections exist in the literature between student demographics, such as underrepresented minority student status, and school characteristics. For example, Black and Latinx students from low SES backgrounds are less likely to take advanced math courses in high school (Conger et al., 2009; Kelly, 2009). This difference is related to structural issues. Black, Latinx, and low SES students are more likely to attend schools with fewer rigorous courses offered compared to their White, Asian, and high SES peers (U.S. Department of Education Office for Civil Rights (OCR), 2012), which affects their ability to take rigorous math courses valuable for engineering. Differences in math achievement between URM and non-URM students can be attributed to the type of instruction students receive (Johnson & Kritsonis, 2006). Historical work found that White students experienced more instruction that aligned with the National Council of Teachers of Mathematics standards than Black students (Lubienski, 2001). Disparities in high school resources are a function of inequities across the U.S. public school system, which is rooted in systemic racism and have adverse effects on the opportunity gaps for minoritized students (Banaji et al., 2021; Darling-Hammond, 2013). This opportunity gap significantly impacts students’ access to experiences and courses.

2.3. Environment: Mathematics Placement Testing

The ACT and SAT exams have been a foundation for U.S. higher education admissions processes and have often been used by universities for mathematics placement. Historically, these tests were designed to offer a consistent measure for colleges and universities to assess applicants from different backgrounds (Clauser & Bunch, 2021). Grade point average and SAT scores have been successfully used to predict the retention of STEM and business students (Rohr, 2012). However, equity and bias concerns exist regarding standardized tests, like the ACT and SAT (Breland, 2007; Buchmann et al., 2010; Goodman et al., 2018; Nankervis, 2011). For example, a study that conducted a multiple regression analysis found the SAT math score marginally contributed to students’ achievement in their college-level math classes; however, researchers found different readiness levels (in regard to SAT math scores) between URM students and White students (Atuahene & Russell, 2016), which reveals equity concerns about the test. As schools are beginning to require standardized tests again, understanding potential equity concerns is more important than ever.

The ALEKS assessment is a tool that creates an individualized mapping of students’ knowledge. The test adapts to each student’s previous answer in an attempt to accurately measure their knowledge. To do this, ALEKS tailors the assessment based on the questions students get correct or incorrect (About ALEKS, n.d.). After students take the assessment for the first time, they are given a summary of their score, breaking down performance across multiple topic areas. Students can visit topical modules to master the content before retaking the assessment. The ALEKS assessment was not originally designed to be a placement exam and has been used in engineering as a supplemental tutoring tool (Carpenter & Hanna, 2006). At the university for this study, the ALEKS assessment was implemented in 2020 to help place engineering students into math classes because the traditional placement mechanisms that relied on the SAT and ACT were no longer an option. We are seeking to explore the impacts of this placement exam on engineering students’ Calculus 1 enrollment and persistence.

2.4. Outcomes: Effects of Mathematics Placement in Engineering

2.4.1. Impact of Remedial Math Courses

The current engineering curriculum and prerequisites are barriers for students who are not math-ready. If students are pre-math-ready, they likely need to enroll in a course below the desired first-year class, otherwise known as a remedial course. For engineering, the desired first-year course is often Calculus 1, so students who enroll in Pre-Calculus are in the remedial course for engineering. At most institutions, the curriculum does not support students who begin their degree in Pre-Calculus to complete their degree in 4 years. To persist, students need to do a significant amount of independent work to catch up and succeed in college (Van Dyken & Benson, 2019). Due to the rigid curriculum, these students often need to take additional classes to catch up or add time to their degree (Main & Griffith, 2022). Generally, engineering students who are placed in Pre-Calculus have lower levels of persistence compared to their peers who are placed in Calculus 1 (Rabb et al., 2016). Furthermore, a study comparing engineering students who were placed into Calculus 1 compared to Calculus 2 found students who were placed into Calculus 1 had lower GPAs and longer time-to-degrees (Inkelas et al., 2021). Math placement and remedial classes significantly impact students’ academic pathways.

2.4.2. Persistence in Engineering

A significant amount of the messaging around engineering focuses on the idea that engineers need to be good at math and science (National Academy of Engineering, 2008). However, many students see math classes as barriers they must overcome to pursue an engineering major (Meyer & Marx, 2014). Engineering culture is meant to “weed out” students, who often struggle to persist in barrier courses (Suresh, 2006). Engineering has high attrition rates, so it is important to understand engineering student persistence, and one of the factors that contributes to engineering students’ persistence is math and science confidence (Eris et al., 2010). Furthermore, some work has shown that math class placement and grades are particularly important to understanding engineering students’ persistence in the major (Middleton et al., 2014; Van Dyken et al., 2015). Mathematics plays a critical role in the engineering curriculum, and math readiness has historically been considered a predictor of engineering student success in first-year courses and the major (Heinze et al., 2003; Tyson, 2011). Therefore, persistence in engineering is often intertwined with math class placement. A significant amount of literature has shown that appropriate math placement can be beneficial for student success (Lougheed, 2015; Wilkins et al., 2021), but remedial classes have adverse effects on persistence due to the additional time and money that may be required.

2.5. Justification of Methods and Positionality

This study focuses on two early stages of students fulfilling the engineering major mathematics requirements to assess how sorting students based on mathematics readiness affects student progress in Calculus 1. The first stage is students’ performance on the placement exam, which sorts them into Calculus 1 or Pre-Calculus based on a score cutoff. The second is persistence in the major after completing the additional requirement of the Pre-Calculus remedial course for students who scored below the cutoff for Calculus 1 eligibility. Given underrepresented students may face additional barriers to achieving math readiness, this sorting and assessment process can create obstacles for students interested in engineering from pursuing their degree. However, as the literature has shown, these students may benefit from the ALEKS interactive environment and the remedial Pre-Calculus course designed to improve math readiness. Therefore, we aimed to contribute to how we can support students in mathematics and what the impact of math placement exams is in the current academic landscape. As a result, we used two main analyses to understand the differences in Calculus 1 enrollment between demographic groups and the impact students’ ALEKS scores have on their persistence in the major and future Calculus 1 performance: the Kitagawa-Oaxaca-Blinder decomposition and a fuzzy regression discontinuity analysis.

The Kitagawa-Oaxaca-Blinder decomposition is used to attribute group-level characteristic differences to outcome differences (Kröger & Hartmann, 2021). In this paper, the Kitagawa-Oaxaca-Blinder decomposition was conducted by performing separate regressions by group, such as male or female, and another regression with both groups pooled. The Kitagawa-Oaxaca-Blinder decomposition quantifies how much inequality of dependent variables can be explained by differences in mean covariate For instance we can quantify how much inequality in Calculus enrollment between URM and non-URM students can be explained by the difference in average GPA between both groups. The Kitagawa-Oaxaca-Blinder decomposition also quantifies how much inequality is unexplained due to differences in how covariates are treated by group, specifically, differences in the group-specific estimated coefficients from the ordinary least squares (OLS) method. We use the Kitagawa-Oaxaca-Blinder decomposition to assess what explains differences in students’ ability to enroll in Calculus 1 and score above the ALEKS cutoff.

The fuzzy regression discontinuity analysis allows us to understand how the additional imposition of the Pre-Calculus remedial course based on having an ALEKS test score below the cutoff impacts persistence in the engineering degree measured by dropout during the first semester, enrollment in Calculus 1 during the sample period, and grade performance in Calculus I conditional on taking the course. This method takes advantage of the fact that a cutoff score was used to sort students into Calculus 1 by comparing outcomes between students just above and below the cutoff of the ALEKS test score. This way, the method allowed us to estimate the local average treatment effect of the Pre-Calculus course on student outcomes for students at the margin (Imbens & Lemieux, 2007).

Our research team consists of economics and engineering education researchers, which influenced our analysis and framing of this study. Two members of the team are in engineering education and are interested in understanding institutional barriers in engineering related to math readiness and placement. Two other members of the team are economists interested in understanding what explains inequality in higher education and STEM outcomes between underrepresented students and students from backgrounds who are overly represented. We are particularly interested in broadening participation in the field of engineering and have concerns about current trends in the academic climate.

Through our review, we have shown major equity concerns surrounding math readiness, standardized testing approaches, and persistence in the engineering major. We aim to contribute to a dialogue on how we support students in mathematics and the impact of math placement exams through the following research questions:

• RQ1: To what extent might the unexplained enrollment in Calculus 1 between different groups (first-generation college students vs. second-generation and above, male vs. female, and URM vs. non-URM students) be attributed to academic preparation, zip code characteristics, demographics, and ALEKS testing behavior?
• RQ2: For students who scored near the placement cutoff, does the remedial Pre-Calculus course impact performance in Calculus 1 and persistence in the engineering major in the first year?

3. Methods

3.1. Data Set and Population Description

The data source used in this study is from a mid-Atlantic U.S. institution with a high engineering ranking. It is important to note that before 2020, the institution in this study did not offer a Pre-Calculus course. Students who needed to take Pre-Calculus participated in a self-paced course in the math center. The data set includes institutional data from incoming students who had an intended major of engineering entering in Fall 2021 and Fall 2022. Our data set included students’ high school GPA, demographic information, math course grades, declared major, and scores from the university-administered Math Placement Exam, ALEKS. The data used in this study were collected through a university analytics office and shared with us. Students self-reported their demographics to the university and test scores were taken directly from the testing platform. These data points were anonymized to maintain student confidentiality before being shared with the research team. The research process was reviewed by the IRB and aligned with the Family Educational Rights and Privacy Act (FERPA) to safeguard student privacy and confidentiality.

In this study, the data surrounding gender uses the binary of male and female. Many large data sets simplify gender to this binary, and we recognize gender is not binary, nor are the terms gender and sex interchangeable. Many of our results focus on female students to explain how this group is minoritized in engineering with respect to math placement. We acknowledge this binary use of gender as a limitation of our data set.

3.1.1. Study Population’s Enrollment in Calculus 1

Summary statistics of the demographics of students who enroll in Calculus 1 can be found in Appendix A. General trends show that 74% of the students enroll in Calculus 1 and only 20% of students enroll in the remedial Pre-Calculus course. The large number of students who enroll in Calculus 1 is evidenced by 71% of students ending up with a test score above the ALEKS score cutoff of 80, and the mean math score is 80.01. Additionally, female, first-generation, and underrepresented minority students, including students who identify as Black, Hispanic, or Native American, are underrepresented in Calculus among incoming students who intend to study engineering. Of the students who were enrolled in Calculus 1 and intended to study engineering, around 23% were female students, 22.7% were URM students, and 19.6% were first-generation students.

3.1.2. Factors Contributing to the Study Population’s Enrollment in Calculus 1

To further describe the data set and population, we provided a table showing the characteristics of students enrolling in Calculus 1, which includes the characteristics of underrepresented students, defined as female, Black, Hispanic, Native American, or first-generation students.

Table 1. Enrollment in calculus.

	(1)	(2)	(3)	(4)	(5)
Variables	Over-Rep	Under-Rep	Female	URM	1st Gen
Dropout First Term	0.022	0.02	0.019	0.023	0.04
Enroll Calc 1	0.80	0.686	0.749	0.614	0.614
Grade Calc 1	81.9	80.93	81.15	80.31	80.45
Transfer Calc 1	0.491	0.455	0.513	0.406	0.419
Enroll Pre-Calc	0.172	0.242	0.210	0.291	0.268
Above Cutoff	0.761	0.660	0.667	0.619	0.614
Math Score	81.75	78.35	79.66	76.37	76.16
Date 1st Attempt (Days before Fall)	61.68	62.66	64.40	62.08	58.52
First Score	51.91	49.25	50.29	48.26	48.71
Number of Attempts (if Below)	3.78	3.86	3.87	3.86	3.83
Avg Score Change (if Below)	11.88	11.79	11.34	12.18	11.9
Avg Time (Days) Between (if Below)	20.86	21.41	20.39	23.15	21.77
HS GPA	4.108	4.103	4.210	3.993	4.055
In-State Student	0.366	0.589	0.518	0.614	0.714
Zip Median Earnings Parent’s Education	78,420	65,080	72,530	68,460	39,540
Zip Unemployment Rate	3.837	3.991	3.837	4.223	4.216
Zip % Bachelor’s Degree	51.55	48.79	50.35	49.03	43.50
Zip % Bachelor’s in STEM	53.22	48.32	42.72	51.05	49.86
Sample Size	1646	1734	778	767	661

Note: Shows mean values of different covariates related to student demographics and Calculus 1/Pre-Calculus enrollment.

presents mean outcomes and characteristics by demographic group.

The Calculus 1 enrollment rate for over-represented students is 80%, which is significantly higher than that for female (74.9%), URM (61.4%), and first-generation students (61.4%). Under-represented students are much more likely than over-represented students to enroll in Pre-Calculus, where 17.2% of over-represented students took Pre-Calculus versus 29.1% of URM and 26.8% of first-generation students. It is important to mention there is a discrepancy between scoring above the cutoff and enrolling in Calculus 1. This is likely because some students are leaving the major before they even enter higher education.

When reviewing characteristics related to the placement test, we see that 76.1% of over-represented students score above the cutoff, whereas 66.7% of female, 61.9% of URM, and 61.4% of first-generation students score above the cutoff. This is evidenced by the higher mean math placement score of over-represented students, 81.75%, versus 79.66% for female, 76.37% for URM, and 76.16% for first-generation students, respectively.

We next will describe background characteristics across different groups and see that over-represented students have lower high school grade point averages (GPAs) than female students but slightly higher average high school GPAs than URM and first-generation students. Over-represented students are more likely to come from zip codes with higher median earnings for adults with their parent’s education level, a lower unemployment rate, and a higher fraction of bachelor’s degrees and bachelor’s degrees in STEM fields.

3.1.3. Proxies in the Analyses

Some of our analyses use proxies to better represent certain variables in the data. To better account for social resources and socioeconomic status, administrative data were merged with zip code-level data from the 2020 American Community Survey, including median earnings, employment, education rates, and the percentage of adults with a bachelor’s degree who majored in STEM or STEM-related fields in the respondent’s zip code.

Using zip code data, we constructed a median earnings variable for adults with education similar to that of the student’s parents. For first-generation students, we used a weighted average of median earnings for adults with a high school diploma, less than a high school diploma, and some college, where the weights reflect the distribution of each group among the zip code population with less than a bachelor’s degree. For second-generation and beyond students, we used a weighted average of median earnings for adults with a bachelor’s degree and adults with a graduate degree, where the weights reflect the distribution of each group among the zip code population with a bachelor’s or more.

Similarly, in the analysis that follows, we classified the student’s high school GPA and their first score on the ALEKS exam as measures of academic preparation. ALEKS variables include the number of attempts, date of the first attempt, average score change between attempts, and average time between attempts interacted with the first score to capture effort put into the ALEKS modules and exams. Finally, demographic information included under-represented minority status, gender, the first term of enrollment, and whether the student was an in-state or out-of-state student. When we focused on subgroups such as female, URM, or first-generation students, demographics included all variables except the category the subgroup belonged to.

3.2. Data Analysis

An important aspect of our analysis includes the ALEKS placement cutoff scores. In Spring 2020, the university began allocating students to Calculus 1 using a cutoff rule based on their performance on the math placement test. During the 2021 and 2022 academic years, the cutoff stipulated that students scoring 80 or higher on the math placement test were placed into Calculus 1. Students who scored below 80 were required to take Pre-Calculus before taking Calculus 1.

3.2.1. Kitagawa-Oaxaca-Blinder Decomposition

For this study, we used the Kitagawa-Oaxaca-Blinder decomposition to determine which groups of covariates explained mean differences in Calculus 1 enrollment and placing above the cutoff score between students from under-represented backgrounds and students from over-represented backgrounds. This type of analysis has been used in a similar study to understand racial gaps in achievement tests (Duncan & Sandy, 2013).

Specifically, we compared outcomes between first-generation and second-generation plus students; male-identifying students and female-identifying students; and White or Asian students and Black, Hispanic, or Native American students. We also divided the portion of the gap explained as attributable to covariates by the raw difference in outcomes between groups to interpret the results as the percentage of the total gap explained by mean differences of covariates between groups. In this paper, we will focus exclusively on how much inequality can be explained by groups of covariates (e.g., academic preparedness, ALEKS test-taking behavior, demographics, and zip code characteristics). For more information on how the Kitagawa-Oaxaca-Blinder estimates how much the overall gap in outcomes can be explained by mean differences in covariates, see Appendix B.

The Kitagawa-Oaxaca-Blinder decomposition allows us to assess how under-represented students’ inputs and environment influenced the first stage of sorting students into the engineering major, specifically performance on the ALEKS exam and enrollment in Calculus 1. In the results section, we explain how we estimated the impact of the remedial course on student progression in the major for students near the cutoff.

3.2.2. Fuzzy Regression Discontinuity

We also used a regression discontinuity design to evaluate the second stage of the sorting mechanism (i.e., placement into the remedial Pre-Calculus course for students who scored below the cutoff). The regression discontinuity is a quasi-experimental technique that leverages the use of discrete cutoffs to assign treatment to individuals. Comparing students just below and above the cutoff can serve as a good treatment and control group if scoring above the cutoff depends on some element of randomness, and if there is a discontinuous change in treatment. Specifically, we used a fuzzy regression discontinuity, which is similar to an instrumental variable approach and allows for non-perfect compliance with treatment. In this case we have non-perfect compliance since there were students who still took Pre-Calculus despite scoring above the cutoff and students who did not enroll in Pre-Calculus despite scoring below the cutoffs. We examined three measures of persistence: enrolling in Calculus 1 during the study period, dropping out during the first term of study, and grade received in Calculus 1. Fuzzy regression discontinuity has been used in a similar study, which evaluated the effectiveness of a remedial math and language course (De Paola & Scoppa, 2014).

An explanation of the fuzzy regression discontinuity and the validation of the assumptions can be found in Appendix B.

3.3. Data Limitations

We acknowledge several limitations with the data and study design. First, the data used in this study are from only one institution. The institution is a large R1 school with a large percentage of engineering majors. If more institutions were included, we could develop a more holistic picture of the effects of math placement exams on student outcomes. Second, the data we analyzed reflect students who experienced remote learning during the COVID-19 pandemic, which affected their academic readiness. Recent work has shown the pandemic had significant effects on student test scores. Kuhfeld et al. (2022) used test scores from 5.4 million students from grades 3–8 to measure achievement differences in math and reading prior to the pandemic to Fall 2020 and Fall 2021. They found students’ math scores dropped more than their reading scores in comparison to pre-pandemic Fall 2019. The gaps were significantly larger for students from low-SES backgrounds and under-represented minorities. Reports from the early pandemic showed the effects of school closures could exacerbate achievement gaps among students (Dorn et al., 2020). Additionally, research has shown that students had very different mathematics learning experiences during the pandemic (Ryan & Sajadi, 2024). This context is an important consideration but limits generalizability.

4. Results

The Kitagawa-Oaxaca-Blinder results address the first research question by describing differences in Calculus 1 enrollment, placement score above the cutoff, and first placement score. The second research question is addressed by the fuzzy regression discontinuity analysis, which describes the effect of the remedial Pre-Calculus course on persistence in engineering and performance in Calculus 1.

4.1. RQ1: Unexplained Enrollment in Calculus 1

In this section, we examine the first stage of sorting students into Calculus 1, which was a student’s performance on the ALEKS exam, whether they scored above the cutoff and eventually enrolled in Calculus 1. This includes determinants of students’ first and final scores on the ALEKS exam, whether they eventually scored above 80 on the exam, and their ultimate enrollment in Calculus 1. We did this by estimating the relationship between the four dependent variables and various independent variables via ordinary least squares (OLS), which allowed us to determine how changes in individual covariates related to outcome likelihood while holding other covariates constant. To further quantify what is responsible for gaps, we performed a Kitagawa-Oaxaca-Blinder decomposition on Calculus 1 enrollment and scoring above the cutoff.

4.1.1. Factors Contributing to Calculus 1 Scoring

Table 2. Factors contributing to Calculus 1 sorting.

	(1)		(2)		(3)		(4)
Variables	Enroll Calc1	(SD)	Above Cutoff	(SD)	Final Score	(SD)	First Score	(SD)
Female	0.0102	(0.0161)	−0.0511 **	(0.0195)	−0.4569	(0.6275)	−1.9604 **	(0.9345)
URM	−0.0685 ***	(0.0150)	−0.0660 ***	(0.0181)	−2.6865 ***	(0.5824)	−0.9600	(0.8660)
First Gen	−0.0943 ***	(0.0219)	−0.0166	(0.0266)	−2.0162 **	(0.8552)	0.1480	(1.2752)
In-State Student	0.0315 **	(0.0132)	0.0213	(0.0160)	0.4798	(0.5142)	−0.3341	(0.7662)
First Term Enrolled 2022	−0.1420 ***	(0.0124)	−0.0962 ***	(0.0218)	−4.4492 ***	(0.6988)	−0.4390	(0.9698)
HS GPA	0.2127 ***	(0.0172)	0.1685 ***	(0.0208)	6.7954 ***	(0.6686)	7.4085 ***	(0.9850)
Zip Median Earnings Par Edu ($10 k)	−0.0020	(0.0045)	0.0161 ***	(0.0054)	0.2854 *	(0.1732)	0.4355 *	(0.2581)
Zip Unemployment Rate	−0.0015	(0.0036)	0.0020	(0.0043)	0.1037	(0.1385)	0.2459	(0.2065)
Zip % Bachelors	0.009 *	(0.0005)	0.0015 **	(0.0006)	0.0914 ***	(0.0204)	0.0971 ***	(0.0303)
Zip % Bachelors in STEM	0.0001	(0.0007)	0.0014 *	(0.0008)	0.0487 *	(0.0264)	0.0386	(0.0394)
Above Cutoff	0.4557 ***	(0.0138)
First Score			0.0100 ***	(0.0013)	0.4314 ***	(0.0415)
# Attempts			0.0932 ***	(0.0268)	5.3102 ***	(0.8592)
Date First Attempt			0.0011	(0.0008)	0.0391	(0.0251)	0.0436 ***	(0.0153)
Avg Score Change			0.0079 ***	(0.0011)	0.2782 ***	(0.0369)
Avg Time (Days) Between Attempt			−0.0033 ***	(0.0008)	−0.0654 **	(0.0267)
First Score × Number of Attempts			−0.0014 ***	(0.0004)	−0.0630 ***	(0.0137)
First Score × Date 1st Attempt			−0.0000 ***	(0.0000)	−0.0013 ***	(0.0004)
First Score × Avg Score Change			−0.0000	(0.0000)	−0.0005	(0.0007)
First Score × Avg Days between Attempt			0.0001 ***	(0.0000)	0.0015 ***	(0.0004)
Constant	−0.3847 ***	(0.0828)	−0.8596 ***	(0.1336)	14.3670 ***	(4.2882)	19.5044 ***	(4.8278)
Observations	3380		3380		3380		3380
R-squared	0.3668		0.1394		0.1885		0.0407

Standard errors in parentheses. *** p < 0.01, ** p < 0.05, and * p < 0.1. Note: Shows OLS coefficient results for enrolling in Calculus 1, final score above the cutoff, final math placement score, and initial math placement score.

presents OLS results for different ALEKS test outcomes based on student demographics, high school GPA, home zip code, and math score above the cutoff. Overall, our results show academic readiness measured by high school GPA is strongly related to enrolling in Calculus 1, math placement scores, and scoring above the cutoff. Holding all else constant, a one-unit increase in high school GPA is associated with a 21.27 percentage point and 16.85 percentage point gain in the likelihood of enrolling in Calculus 1 and scoring above the cutoff, and a 6.8 and 7.4 point increase in final and initial math placement scores, respectively. Similarly, the final math score and scoring above the cutoff are strongly correlated with the first score, where an additional point on the first score is associated with a 0.43 percentage point increase in the final score and a 1 percentage point increase in the probability of scoring above the cutoff.

We also found that test-taking behavior is also strongly correlated with test performance. Additional attempts, score changes between attempts, and taking less time between attempts are positively correlated with the final score and scoring above the cutoff. Coefficients from these variables interacted with the first score, suggesting the association between these test-taking behaviors and the final score is weaker for students who were closer to the cutoff on their first score. For students further away from the cutoff, our findings captured aspects of students’ math readiness, such as organization, effort, and grit, that could be positively correlated with math performance.

Findings also show URM and first-generation students are 6.85 percentage point and 9.43 percentage point less likely to enroll in Calculus 1, respectively, compared to their non-URM or non-first-generation peers with similar high school GPAs, zip code characteristics, and math scores with respect to the cutoff. Similarly, female and URM students are 5.11 percentage point and 6.6 percentage point less likely to score above the cutoff than students who differed in these demographics but who are otherwise similar with respect to the other covariates. Additionally, we found female students are predicted to have an initial math placement score 1.96 points lower than male students who are otherwise similar. We also found URM and first-generation students have average final placement scores that are 2.69 and 2.02 points lower than otherwise similar non-URM and non-first generation students.

We examined behavior related to scoring above the cutoff. Compared to otherwise similar students, students who scored above the cutoff are 45.5 percentage points more likely to enroll in Calculus 1. This is significant because scoring below the cutoff does not preclude one from enrolling in Calculus 1 later, as students have to enroll in the remedial Pre-Calculus course before beginning Calculus 1. This suggests the extra requirement of enrolling in Pre-Calculus may act as a significant enough obstacle to prevent students from continuing in the engineering major.

We also found zip code is positively correlated with placement scores. Holding all other covariates constant, a 1 percentage point increase in the percentage of adults with a bachelor’s degree in a student’s home zip code is associated with an additional 0.0914 points on the final math score and an additional 0.0971 points on the initial math placement score. Similarly, holding all other covariates constant, a 1 percentage point increase in the percentage of adults with a bachelor’s degree and the percentage of bachelor’s degree holders with a STEM degree are also associated with a 0.15 and 0.14 percentage point increase in the likelihood of having a final score above the cutoff, respectively.

Finally, median earnings for adults with similar education to students parents are strongly positively correlated with test performance. A $10,000 increase in median earnings is associated with a 1.61 percentage point increase in the probability of scoring above the cutoff, and an additional 0.44 and 0.29 points on the first and final scores, respectively. This relationship between median earnings and test scores could be explained by this variable acting as a proxy for parental income. This variable could also reflect any zip code-level effects through schooling and social capital due to higher resources spent on education in a student’s local area.

4.1.2. Kitagawa-Oaxaca-Blinder—Calculus 1 Enrollment

In this section, we examine inequality in Calculus 1 enrollment. We compare first-generation to second-generation and above, female to male, and URM to non-URM students. We split the explained portion into a portion explained by differences in academic preparedness, including high school GPA and first score, a portion explained by differences in demographics composed of enrollment term, gender, first-generation status, and URM status, and a portion explained by zip code characteristics including the percentage of adults with a bachelor’s degree, the percentage of bachelor’s holders with a STEM degree, and median earnings for adults with similar education to the parents.

Table 3. Explanations for gaps in placement score above cutoff.

	1 Gen vs. 2+ Gen	Female vs. Male	URM vs. Non-URM
Variables	% Explained	% Explained	% Explained
Academic Preparation	36.27 ***	−20.44	40.12 ***
ALEKS Behavior	−17.50 **	0.0	−3.02
Demographic	1.19	2.77	−0.26
Zip Code Characteristics	65.93 ***	23.39	6.30 **
Total % Explained	85.81 ***	5.89	43.05 ***
Total % Unexplained	14.18 ***	94.11 ***	56.95 ***
Mean Over-rep Group	0.7319 ***	0.7214 ***	0.7352 ***
Mean Under-rep Group	0.6142 ***	0.6671 ***	0.6193 ***
Achievement Gap	0.1177 ***	0.0543 ***	0.1159 ***
Total Under-rep Group	661	778	767
Total Over-rep Group	2719	2602	2613

Standard errors in parentheses. *** p < 0.01, ** p < 0.05, and * p < 0.1. Note: Presents results from the Kitagawa-Oaxaca Blinder Decomposition, which calculates how much of the achievement gap was due to group mean differences in covariates and how much was unexplained. Results are aggregated into groups of covariates and divided by the total achievement gap to obtain the percentage of the gap explained by differences in mean covariates in each group. p-values tested the null hypothesis of whether the component explained by each group of covariates is equal to zero, meaning differences in these mean groups did not explain any of the achievement gaps.

presents our first set of results for inequality in Calculus 1 enrollment gaps. In this analysis, academic preparation includes the time of first attempt to measure skills such as timeliness, and measures like first score and high school GPA. This set of results also included an explained portion due to differences in rates of final scores being above the cutoff between groups. It is important to note female enrollment rate in Calculus 1 was higher than the male enrollment rate in Calculus 1. These results were included as they suggest interesting findings related to gender inequality.

When we compared first-generation to second-generation and above students, differences in mean covariate levels can explain 40% of the enrollment gap. We saw most of this difference is explained by differential rates of obtaining a final math placement score above the cutoff, which explained 33.82% of the enrollment gap between first-generation and second-generation and above students. In addition to the cutoff, 8.33% can be explained by differences in academic preparation, such as high school GPA, timeliness, and first score.

Although the enrollment rate of female students is higher than male students, there are some interesting findings to note. Specifically, the positive value on the above cutoff portion suggests lower rates of scoring above the cutoff are associated with a lower likelihood of enrollment in Calculus 1 for female students, holding all else constant. The negative value on the academic preparation portion suggests higher measures of academic preparation among female students should mean that female enrollment rates in Calculus 1 would be much higher than male students’ enrollment in Calculus 1 if these characteristics were treated the same. In other words, if female students had the same rate of scoring above the cutoff as male students, there would be more enrollment in Calculus 1, and despite female students having higher measures of academic readiness, they had a lower likelihood of scoring above the cutoff.

When comparing URM to non-URM students, 58.51% of the enrollment gap in Calculus 1 was due to differences in mean characteristics. For instance, lower URM rates of scoring above the cutoff explain 31.98% of the URM Calculus 1 enrollment gap, and lower measures of academic preparation, such as high school GPA and first score, explain 18.66% of the URM Calculus 1 enrollment gap. Additionally, differences in demographics, such as a higher rate of URM students being first-generation students compared to non-URM students, explain 7.27% of the URM Calculus 1 enrollment gap. Finally, for both URM and first-generation students, a large amount of the gap in Calculus 1 enrollment is left unexplained.

4.1.3. Kitagawa-Oaxaca-Blinder—Placement Score Above the Cutoff

In this section, we examine the inequality in obtaining a placement score above the cutoff. For gaps in placement scores above the cutoff, we have a portion explained by differences in math placement behavior, such as the number of attempts, score changes, and timing of attempts interacted with the first score. For this analysis, academic preparation includes only high school GPA.

When comparing first-generation to second-generation and above students, 85.81% of inequality can be explained by differences in mean covariates. The negative value for math placement behavior actually suggests that if first-generation students exhibited the same behavior as second-generation and above, then they would have lower rates of scoring above the cutoff and hence inequality would be predicted to grow by 17.5%. Additionally, we see zip code characteristics explain 65.93% of inequality between first-generation and second-generation and above students. This is because these students are from zip code locations with less income for adults with similar education to their parents, and with lower rates of bachelor’s attainment. This suggests social capital and access to resources could play an important role in the early stages of students’ STEM education.

When comparing female to male students, there is no statistically significant evidence that differences in characteristics explain the higher rate of males scoring above the cutoff. The comparison between the groups shows that 94% of the cutoff score differences are unexplained. ALEKS behavior, academic preparation, demographics, and zip code do not explain very much of the gender gap.

When comparing URM to non-URM students, 43.05% of the inequality in scoring above the cutoff can be explained by differences in mean covariates. Most of these differences are due to academic preparation, which explains 40.12% of the URM gap in scoring above the cutoff score. Similar to the first-generation and gender comparisons, math placement behavior does not explain very much of the URM gap, and differences in zip code explain 6.3% of the gap in scoring above the cutoff.

For first-generation versus second-generation and above, most of the inequality can be explained by differences in covariates, and for female versus male and URM versus non-URM, a large amount of the inequality in scoring above the gap is unexplained— 94.11% and 56.95%, respectively. As math placement behavior does not explain these differences, there are likely other factors contributing to the difference in scoring above the cutoff between different groups.

4.2. RQ2: Effect of Pre-Calculus on Engineering Persistence

This section explores the second stage of sorting students into the major. We examined whether the remedial Pre-Calculus course impacts engineering students’ success, specifically their eventual enrollment into Calculus 1, their grades in Calculus 1, and whether these students continue their studies beyond the first semester.

Table 4. Effect of remedial Pre-Calculus on Calculus 1 enrollment during sample period.

	(1)	(2)	(3)	(4)	(5)
Variables	Pooled	UR Pooled	URM	1 Gen	Female
Enroll Pre-Calculus	−0.5186 ***	−0.5361 ***	−0.3369	−0.8549 ***	−0.3894 *
	(0.1462)	(0.1721)	(0.2596)	(0.3319)	(0.2085)
Score Distance to Cutoff	0.0156 ***	0.0152 ***	0.0191 **	0.0092	0.0170 ***
	(0.0039)	(0.0048)	(0.0084)	(0.0101)	(0.0051)
Above Cutoff × Score Distance	−0.0125 ***	−0.0104	−0.0071	−0.0132	−0.0101
	(0.0046)	(0.0063)	(0.0105)	(0.0135)	(0.0069)
Unconditional Mean	0.7941	0.7735	0.7079	0.7273	0.8282
Took Pre-Calculus Mean	0.5171	0.4873	0.4390	0.5303	0.5397
No Pre-Calculus Mean	0.8851	0.8717	0.8270	0.8061	0.9079
Observations	1180	618	267	231	291
R-squared	0.2509	0.2603	0.3543	0.0	0.3354

Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, and * p < 0.1. Note: Fuzzy regression discontinuity results for all students enrolling in Calculus during the sample period. Local average treatment effect of remedial Pre-Calculus is given by the coefficient Enroll Pre-Calculus. All specifications were controlled for in-state student status, first-generation student status, gender, the term of enrollment, zip code socioeconomic status, high school GPA, first ALEKS score, and first ALEKS score interacted with average score increase between attempts, the date of first attempt, and the average time between attempts.

shows results for the effect of enrolling in Pre-Calculus on eventually taking Calculus 1 during the sample period. For students who enrolled during Fall 2021, this would be if they enrolled in or transferred Calculus 1 credit between Fall 2021 and Spring 2023. For students who enrolled during Fall 2022, this would be if they enrolled in or transferred Calculus 1 credit between Fall 2022 and Spring 2023.

For most specifications, taking Pre-Calculus reduces the probability students will enroll in Calculus 1. For instance, for the pooled sample, enrolling in Pre-Calculus led to a 51.86 percentage point decline in the probability of taking Calculus 1. For the sample, pooling together all under-represented students, taking Pre-Calculus reduces the probability of ever enrolling in Calculus 1 by 53.61 percentage points. The effect size is smaller for female students at −38.94 and largest for first-generation students at −85.49. The effect is also negative but not statistically significantly different for URM students. Although this effect size seems large, the difference in the unconditional mean between those who took Pre-Calculus is close to a 40 percentage point change for all samples.

Table 5. Effect of remedial Pre-Calculus on dropout during first term.

	(1)	(2)	(3)	(4)	(5)
Variables	Pooled	UR Pooled	URM	1 Gen	Female
Enroll Pre-Calculus	0.1651 **	0.2443 ***	0.2985 **	0.4373 **	0.1109
	(0.0815)	(0.0929)	(0.1435)	(0.2206)	(0.0834)
Score Distance to Cutoff	0.0028	0.0023	0.0076 **	0.0068	0.0010
	(0.002)	(0.0021)	(0.0037)	(0.005)	(0.0023)
Above Cutoff × Score Distance	−0.0037 *	−0.0023	−0.0091 *	−0.0076	−0.0027
	(0.0022)	(0.0026)	(0.0047)	(0.0067)	(0.0029)
Unconditional Mean	0.0212	0.0210	0.0225	0.0433	0.0172
Took Pre-Calculus Mean	0.0274	0.0443	0.0488	0.0758	0.0317
No Pre-Calculus Mean	0.0191	0.0130	0.0108	0.0303	0.0132
Observations	1180	618	267	231	291
R-squared	0.0	0.0	0.0	0.0	0.0333

Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, and * p < 0.1. Note: Fuzzy regression discontinuity results for dropout during the first term. Local average treatment effect of remedial Pre-Calculus given by coefficient on Enroll Pre-Calculus. All specifications control for in-state student status, first-generation student status, gender, the term of enrollment, zip code socioeconomic status, high school GPA, first ALEKS score, and first ALEKS score interacted with average score increase between attempts, the date of first attempt, and the average time between attempts.

shows that for most specifications, taking Pre-Calculus increased the probability students would drop out during the first term. For instance, for the pooled sample, enrolling in Pre-Calculus led to a 16.51 percentage point increase in the probability of dropping out. For the sample, pooling together all under-represented students, taking Pre-Calculus increased the probability of dropping out by 24.43 percentage points. For URM and first-generation students, the effect size was 29.85 and 43.73, respectively. For female students, the effect is not statistically significant but still positive.

The results for dropout suggest a very strong negative impact of taking Pre-Calculus on continuing enrollment at the university. However, it is important to note that these results are primarily driven by a huge spike in dropout near the cutoff region, as shown in Appendix B, Figure A4. We also performed a sensitivity analysis, dropping students in two points of the cutoff, and found results consistent with those in

Table 6. Effect of remedial Pre-Calculus on Calculus 1 grade if enrolled in Calculus 1.

	(1)	(2)	(3)	(4)	(5)
Variables	Pooled	UR Pooled	URM	1 Gen	Female
Enroll Pre-Calculus	−2.8209	0.0321	−0.7107	−3.3080	1.5424
	(2.6894)	(3.8062)	(4.5336)	(12.9954)	(3.3690)
Score Distance to Cutoff	−0.0992 *	−0.0380	0.0845	−0.0138	−0.1001
	(0.0533)	(0.0922)	(0.0976)	(0.2286)	(0.0918)
Above Cutoff × Score Distance	0.1522 ***	0.0678	−0.0399	0.0191	0.1271
	(0.0567)	(0.0965)	(0.1133)	(0.2548)	(0.1082)
Unconditional Mean	8.46	8.42	7.91	8.08	8.89
Took Pre-Calculus Mean	7.60	7.95	8.25	7.085	9.32
No Pre-Calculus Mean	8.62	8.51	7.83	8.34	8.82
Observations	643	324	134	118	152
R-squared	0.0756	0.148	0.166	0.0	0.713

Robust standard errors in parentheses. *** p < 0.01, ** p < 0.05, and * p < 0.1. Note: Fuzzy regression discontinuity results for grade in Calculus 1 for sample who enrolled in Calculus 1. Local average treatment effect of remedial Pre-Calculus given by coefficient on Enroll Pre-Calculus. All specifications control for in-state student status, first-generation student status, gender, the term of enrollment, zip code socioeconomic status, high school GPA, first ALEKS score, and first ALEKS score interacted with average score increase between attempts, the date of first attempt, and the average time between attempts.

, where students who took Pre-Calculus were more likely to drop out.

Finally, Table 6 shows that, in all specifications, taking Pre-Calculus has no statistically significant impact on grades in Calculus 1. This means there is no evidence that taking Pre-Calculus before Calculus 1 hampers student learning, and it could still help student learning as suggested with the positive coefficient pooling together under-represented groups.

It is important to note that fuzzy regression discontinuity best estimates the local average treatment effect, which means the treatment effect is specifically for those who scored close to the cutoff of 80. As a result, our estimate compared students just below 80 who had to take Pre-Calculus to students just above 80 who were eligible to directly enroll in Calculus 1. The fact the effects were zero is consistent with the cutoff being set at an appropriate level/This means that it is consistent with the Pre-Calculus course providing adequate preparation to students who scored below the cutoff so that they could perform just as well as students above the cutoff. If the effect was positive, this would suggest students just above 80 would do better if they enrolled in Pre-Calculus, which would suggest the university should set the score cutoff higher. If the effect was negative, then this would suggest the course did not do a good job of preparing students just below the cutoff, as they would perform worse than their classmates with similar scores who were not required to take Pre-Calculus.

5. Discussion

5.1. Inputs

5.1.1. Female Students

The Kitagawa-Oaxaca-Blinder results show female students are enrolling in Calculus 1 at higher rates compared to their male counterparts (see Table 3). However, there is a significant number of unexplained differences in female students scoring above the cutoff compared to their male peers (see Table 4). Ninety-four percent of the difference in scoring above the cutoff is not related to academic preparation, zip code, or other demographic characteristics investigated in this study. Differences in the score gap could be related to the placement test itself; literature has shown competitive pressures of test-taking environments can explain the gender gap in mathematics performance. Women may not perform as well in competitive test-taking environments compared to men (Niederle & Vesterlund, 2010). Some of the unexplained differences could be related to elements not captured in our analysis but shown as possible contributors in the literature (e.g., testing anxiety (Ashcraft & Ridley, 2005), stereotype threat (Spencer et al., 1999), self-efficacy (Parker et al., 2016), or hidden curriculum in engineering.

There is significant nuance in the results across gender, which show a score gap between female and male students but not an enrollment gap. This result speaks to female students choosing not to pursue engineering, even if they pass the exam, which aligns with prior work that found male students with mediocre math scores are more likely to pursue engineering fields than females with high math scores (Cimpian et al., 2020). One explanation may be that high-achieving female students have more major options since they often score highly in multiple subjects compared to their male counterparts, who only score high in math (Valla & Ceci, 2014).

The gender difference in engineering pursuits may also be related to the chilly climate in STEM and engineering for students from the non-dominant group. For example, studies have highlighted the hidden curriculum of STEM course syllabi that promote masculine thinking and the idea women are incompetent (Bejerano & Bartosh, 2015). Additionally, to gain acceptance in engineering, many women need to undo their gender to help maintain the accepted, masculine environment (Powell et al., 2009). The STEM education that women have been exposed to before entering the university may already deter them from pursuing an engineering degree or believing they can be engineers.

5.1.2. First-Generation Student Status

The Kitagawa-Oaxaca-Blinder results show that for first-generation students, the difference in enrolling in Calculus 1 can partially be explained by scoring above the cutoff. However, differences in scoring above the cutoff are explained by academic preparation and zip code characteristics but are negatively impacted by ALEKS test-taking behavior. The relationship between zip code, academic preparation, and scoring above the cutoff may be due to differences in the quality of the high school. Math readiness is better understood as an opportunity gap rather than an achievement gap (A. Flores, 2007). Factors such as high school resources, access to advanced placement (AP) courses, and the quality of prior math instruction—factors largely beyond a student’s control—shape their math preparedness (National Academy of Engineering, 2018; A. Rodriguez, 2018). These inequities are deeply rooted in structural disparities, such as differences in property tax revenue, which directly affect school funding and resources. As a result, the U.S. public school system perpetuates an unequal playing field, where students’ academic opportunities and outcomes are strongly influenced by their socioeconomic and geographic contexts (Clotfelter et al., 2007; Committee on Developing Indicators of Educational Equity et al., 2019; Darling-Hammond et al., 2005; Smith et al., 2013).

For first-generation students, a significant amount of the explained difference can be attributed to zip code characteristics. Comparing the percentage of the population of the student’s zip code that holds a bachelor’s degree shows that students who come from a community where more people hold bachelor’s degrees may score better on the ALEKS exam. This finding connects to the idea of social capital, suggesting students from communities with a higher percentage of bachelor’s degree holders may outperform their peers on assessments such as the ALEKS exam. It is likely that students who come from a particular zip code, with a certain percentage of STEM degree holders in their neighborhood, hold a particular social, familial, or navigational capital that influences their interest in STEM. A literature review analyzed research in STEM focusing on the various types of capital students possess (Denton et al., 2020). These findings illuminate the complex web of socio-cultural and educational factors that contribute to performance disparities in mathematics, advocating for a nuanced approach to addressing equity in educational outcomes.

5.1.3. Under-Represented Minority Student Status

The Kitagawa-Oaxaca-Blinder results show that URM students’ enrollment in Calculus 1 can mostly be explained by scoring above the cutoff and academic preparation. The explained difference in scoring above the cutoff is mostly due to academic preparation; however, a little over half of the difference is unexplained. For URM students, structural inequities that exist in STEM can contribute to their lower performance on the placement exam and enrollment in Calculus 1 (Whitcomb et al., 2021). Structural inequities also include limited access to Pre-Calculus and Calculus 1 courses, which is a function of high school characteristics (National Center for Education Statistics, 2023a). Literature has also shown that differences in mathematics achievement between students who are under-represented minorities and those who are not can be attributed to the type of instruction URM students receive (Johnson & Kritsonis, 2006). Also, White students were experiencing more instruction that aligned with the National Council of Teachers of Mathematics standards than Black students (Lubienski, 2001). Further, literature that investigated the Black–White test score gap found that after controlling for all variables, two-thirds of the score gap still remained (Fryer & Levitt, 2006), which could be explained by Black students being treated differently in the classroom. The type of mathematics instruction students receive affects their academic preparation for college-level mathematics.

Cultural factors may also play a significant role in shaping URM students’ experiences and performance in mathematics. For example, the unwelcoming or exclusionary culture of STEM fields can affect students’ sense of belonging in these disciplines (S. L. Rodriguez & Blaney, 2021; G. M. Flores et al., 2024). Addressing these cultural barriers requires intentional efforts to foster inclusive environments and challenge the deficit-oriented narratives often imposed on URM students in STEM.

5.2. Environment: The ALEKS Placement Exam

The fuzzy regression discontinuity analysis allowed us to investigate the effects of enrolling in Pre-Calculus for students who scored near the cutoff. Our results show that for students who do persist in the major, taking Pre-Calculus has no negative impact on grades, which suggests the cutoff is set at an appropriate level that allows students with less math preparation to catch up to similar peers who scored just above the cutoff. This indicates the cutoff score of 80% on the exam is set appropriately. We were not able to find publicly available information from ALEKS about the appropriate cutoff score for Calculus 1, but we have found many schools use a cutoff score near or slightly below 80%. For example, through an internet search, we found multiple schools that use a cutoff score of 76%, and other schools that use a cutoff score of 80%. Research that was completed before the pandemic examined the effectiveness of the ALEKS placement exam and found that a cutoff score of 70% on ALEKS for Calculus 1 was too low and suggested increasing the score to 73% (Woods, 2017). These findings, combined with the alignment of our cutoff score with those used by other institutions and prior research, provide evidence that the current cutoff score of 80% is both effective and appropriate for supporting student success in engineering pathways.

Further, findings from the OLS analysis show that although testing behaviors such as additional attempts, score changes between attempts, and taking less time between attempts were positively correlated with the final score and scoring above the cutoff, they were not explanatory for students near the cutoff score. Therefore, interventions related to test-taking strategies may be less useful for these students. Additionally, the Kitagawa-Oaxaca-Blinder decomposition showed that test-taking behavior explains very little of the gap in scoring above the cutoff between over-represented and under-represented students.

5.3. Outcomes: Impact on Student Persistence and Calculus 1 Performance

After analyzing scoring on the ALEKS placement exam, we analyzed the impact of Pre-Calculus on engineering students’ academic achievement and trajectory. If students do not score above the cutoff, they are required to enroll in Pre-Calculus. Our results indicate the effects of enrolling in Pre-Calculus increases the likelihood of dropping out during the first term. For all groups except females, the effect was statistically significant. At most schools, the curriculum does not support students who begin their degree in Pre-Calculus. Due to the long prerequisite chain of courses, students need to begin their degree in Calculus 1 to take engineering-specific classes in the correct sequence (Ellis et al., 2021). Since the engineering curriculum does not support students who begin their degree in Pre-Calculus, they need to do a significant amount of work to catch up in their pursuit of an engineering degree (Main & Griffith, 2022; Van Dyken & Benson, 2019), which may not be feasible or worth it for all students. The cutoff rule and requirement for taking Pre-Calculus is enough of a barrier to students enrolling in Calculus 1 and persisting in the major. The extra requirement of enrolling in Pre-Calculus may act as a significant enough obstacle to prevent students from continuing with their engineering major or at least completing the degree in a timely manner.

Our results also indicate that enrolling in Pre-Calculus does not have a significant effect on students’ Calculus 1 grades. This shows that if engineering students are placed into Pre-Calculus and persist, they may perform just as well in Calculus 1 as their peers who were originally placed in Calculus 1. Enrolling in Pre-Calculus does not necessarily help or hamper students learning, which aligns with literature that shows students taking Pre-Calculus in college do not necessarily do better once they take Calculus 1 (Sonnert & Sadler, 2014).

Further, our results suggest the policy of using the ALEKS test to sort students into Calculus 1 and Pre-Calculus has some benefits along with costs. Enrolling in Pre-Calculus does appear to prevent students from persisting in the engineering major in two aspects. The first is that students who take Pre-Calculus are less likely to ever enroll in Calculus 1, a core requirement for the engineering major. Second, students who take Pre-Calculus are more likely to drop out, which may be due more to the discouragement of missing the cutoff score than the effect of the course itself. However, on the benefit side, it does appear that for students who do persist, taking Pre-Calculus has no negative impact on grades, which suggests the cutoff is set at an appropriate level that allows students with less math preparation to catch up to similar peers who scored just above the cutoff. This aligns with the literature highlighting the importance of students starting in the math class they are ready for (Wilkins et al., 2021). These results show the real downside to implementing a math placement exam to sort students has more to do with imposing additional requirements for students with less math preparation and less to do with remedial courses providing inadequate preparation for advanced engineering courses.

5.4. Implications

This work has many implications for engineering programs in the current academic climate, with significant variation in mathematical readiness among students. National math scores are continuing to decline (National Center for Education Statistics, 2023b), and more students are entering college pre-math-ready. Institutions need to consider how they support engineering students entering pre-math-ready and the implications of high-stakes exams. Further, if math placement at these institutions greatly impacts time-to-degree and there is a lack of support and infrastructure for Pre-Calculus, schools may begin to see decreased enrollment in engineering, particularly for marginalized populations.

Findings from this analysis support interventions related to academic preparation, as a significant amount of the difference in scores could be attributed to academic preparation for first-generation and URM students, pointing to the importance of bridge programs (Ashley et al., 2017; Galbraith et al., 2021), culturally responsive teaching (Vavrus, 2008), structured mentoring programs (Markle et al., 2022), and learning communities (Ricks et al., 2014). These interventions aim to reduce inequities in academic preparation and foster success for first-generation and URM students in engineering. However, these types of programs may not necessarily be as helpful for female students, where academic preparation was not an explanation for differences in scores. For female and URM students, a significant amount of the difference was left unexplained. Future work could seek to understand some of the potential factors contributing to the unexplained differences in scoring above the cutoff and enrolling in Calculus 1. Notably, our findings illustrate that each group has different needs, showing that engineering student support systems should be strategically focused based on student needs. A one-size-fits-all approach is likely to leave some inequities in place.

This paper presents evidence for the implications of the ALEKS exam and its potential impacts on persistence. These results should encourage schools to consider if they want a high-stakes exam to determine whether students can enter engineering and evaluate the engineering pathways as they relate to mathematics. Recent work has shown that performing well in lower-level first-year math classes indicates persistence and success in the major compared to students who struggle in higher-level first-year math classes (Wilkins et al., 2021). Institutions have tried individual approaches to supporting engineering students who need to take remedial math courses (Ellis et al., 2021; Fick & Bauer, 2020; Ohland et al., 2004), but there have been no universal changes. Engineering students who are placed in Pre-Calculus may have limited engineering major options due to the complex curriculum in engineering (Heileman et al., 2019) that relies heavily on mathematics, which is an area of future research. One approach is to reconsider the engineering curriculum by examining the prerequisite and corequisite math classes for engineering courses and the amount of math actually needed for those classes. A recent study evaluated the amount of calculus used in fundamental engineering courses: statics and circuits. They found these classes use minimal calculus and require only basic calculus skills (Faulkner et al., 2020), showing that engineering programs have room to make changes in the curriculum to better support students.

6. Conclusions

Based on the findings from the study, we illuminated differences in Calculus 1 enrollment across demographic groups and impacts on persistence in engineering in the first term of college. We found unexplained differences contributing to Calculus 1 enrollment and placement scores for under-represented student groups, highlighting the complexities of sociocultural and educational factors contributing to disparities in mathematics performance. Moreover, we found the effects of enrolling in Pre-Calculus increased the likelihood of dropping out during the first term for students near the cutoff. Our research indicates that taking Pre-Calculus leads to higher dropout rates, but students who persist into Calculus 1 experience no detrimental effects on their grades.

These findings suggest the additional barrier posed by math placement exams primarily disadvantages students with less prior math preparation rather than indicating a deficiency in the ability of remedial coursework to prepare students for advanced engineering courses. This has critical implications for engineering programs, underscoring the need to re-evaluate the design and implementation of placement mechanisms and mathematics course sequences. Some recommendations engineering programs should consider include the following:

• Reconsider high-stakes placement exams: These results should encourage institutions to reconsider their use of high-stakes exams related to mathematics in engineering because of the unintentional barriers they create for under-represented students. We also encourage schools to thoughtfully consider the placement cutoff score; we found that 80% is an appropriate score for Calculus 1 placement. We believe placement mechanisms are important to ensure students are placed for success. However, the mechanism must be thoughtfully designed and consider the varying backgrounds of entering engineering students.
• Support structures for pre-calculus students: Given the higher dropout rate among students placed in Pre-Calculus, engineering programs should implement target support structures, such as bridge programs, learning communities, and tutoring to help students succeed and persist in engineering.
• Flexible course sequencing: To reduce barriers in engineering, we recommend that institutions reevaluate prerequisite and corequisite mathematics course requirements in engineering, as prior work has shown that there is room to make changes regarding math sequencing in engineering. Moreover, our findings highlighted that taking Pre-Calculus doesn’t mean students will receive a lower grade in Calculus 1, so by creating pathways that allow students to begin their degree in Pre-Calculus, institutions can broaden access to engineering programs without compromising student success in subsequent mathematics courses.

These recommendations provide some guidance for engineering programs to better support under-represented students in engineering and foster more equitable pathways for all.