Next Article in Journal
Blockchain and IoT-Driven Optimized Consensus Mechanism for Electric Vehicle Scheduling at Charging Stations
Next Article in Special Issue
Digital Health in Schools: A Systematic Review
Previous Article in Journal
Antiviral and Antifungal of Ulva fasciata Extract: HPLC Analysis of Polyphenolic Compounds
Previous Article in Special Issue
The Influence of Content Presentation on Users’ Intention to Adopt mHealth Applications: Based on the S-O-R Theoretical Model
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 14
Issue 19
10.3390/su141912801
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Pilot Study on the Impact of the BumptUp® Mobile App on Physical Activity during and after Pregnancy

by
Rachel A. Tinius
1,*,
Maire M. Blankenship
2,
Alison M. Colao
1,
Gregory S. Hawk
3,
Madhawa Perera
1 and
Nancy E. Schoenberg
4
1
Exercise Science, Western Kentucky University, Bowling Green, KY 42101, USA
2
Nursing and Allied Health, Western Kentucky University, Bowling Green, KY 42101, USA
3
Department of Statistics, University of Kentucky, Lexington, KY 40506, USA
4
Gender and Women’s Studies, College of Arts and Sciences, University of Kentucky, Lexington, KY 40506, USA
*
Author to whom correspondence should be addressed.
Sustainability 2022, 14(19), 12801; https://doi.org/10.3390/su141912801
Submission received: 7 September 2022 / Revised: 28 September 2022 / Accepted: 5 October 2022 / Published: 7 October 2022
(This article belongs to the Special Issue The Dawn of mHealth Innovation)
Browse Figures
Review Reports Versions Notes

Abstract

:
To combat maternal morbidity and mortality, interventions designed to increase physical activity levels during and after pregnancy are needed. Mobile phone-based interventions show considerable promise, and BumptUp® has been carefully developed to address the lack of exercise among pregnant and postpartum women. The primary goal of this pilot study was to test the potential efficacy of BumptUp® for improving physical activity among pregnant and postpartum women. A randomized controlled clinical trial was performed (N = 35) with women either receiving access to the mhealth app or an educational brochure. Physical activity and self-efficacy for exercise data were collected at baseline (in mid-pregnancy) and at three additional timepoints (late pregnancy, 6 and 12 weeks postpartum). For moderate-to-vigorous physical activity, a clear trend is observed as the mean estimated difference between groups increases from −0.35 (SE: 1.75) in mid-pregnancy to −0.81 (SE: 1.75) in late pregnancy. For self-efficacy for exercise, the estimated difference of means (control–intervention) changed from 0.96 (SE: 6.53) at baseline to −7.64 (SE: 6.66) in late pregnancy and remained at −6.41 (SE: 6.79) and −6.70 (SE: 6.96) at 6 and 12 weeks postpartum, respectively. When assessing the change in self-efficacy from mid-to -ate pregnancy only, there was a statistically significant difference between groups (p = 0.044). BumptUp® (version 1.0 (3)) shows potential for efficacy. Pilot data suggest key refinements to be made and a larger clinical trial is warranted.

1. Introduction

Maternal mortality and morbidity rates are strikingly high in the U.S. compared to other industrialized countries [1]. A contributing factor is the large number of women presenting for prenatal care with chronic health conditions including obesity, hypertension, and diabetes, all of which lead to complicated pregnancies [2], and women in certain marginalized groups may be even more vulnerable to chronic health conditions and have less access to prenatal care than women in other settings [3,4].
Physical activity during and after pregnancy contributes to better/optimal maternal health outcomes including reduced weight gain, improved glucose control, lower blood pressure, and better mental health [5]. Despite these well-documented findings, there are few sustainable, scalable, and accessible interventions that can successfully increase physical activity and reduce gestational weight gain, particularly among underserved women.
Mobile phone-based interventions show considerable promise because they can be tailored to the target population, can be delivered at any place and at any time, are interactive, and are accessible to the majority of the population irrespective of socioeconomic status (96% of U.S. women aged 18–49 have smartphones) [6,7]. Previous research demonstrates the ability of mobile apps to positively impact physical activity behaviors [8]; thus, a unique and timely opportunity exists to increase physical activity among pregnant and postpartum women through mobile health apps.
Mobile apps have emerged as a primary mode of health information for women during pregnancy [9]. In fact, pregnancy is the medical condition with the highest number of apps available [9]. However, to our knowledge, mobile apps designed specifically to increase physical activity during pregnancy are sparse [8,10,11]. Mobile apps designed to alter lifestyle during pregnancy have been unable to elicit statistically significant differences in physical activity among intervention vs. control groups of pregnant women [12,13,14]. In addition, many focus more on weight status and diet than physical activity [12,13,15]. Ainscough et al. focused on the impact of a mobile health program for overweight and obese pregnant women and found some improvements among physical activity variables in the intervention arm [15]; however, the intervention contained limited physical activity guidance (i.e., no personalized programming) and physical activity was measured via self-report, which is a major limitation. Several recent reviews concluded that exercise apps designed for pregnancy fail to consider: (1) current evidence-based physical activity guidelines, (2) screening for contraindications to physical activity, (3) appropriate personalization features to account for an individual’s characteristics, and (4) the involvement of qualified experts during the development of the app [10,11].
Akin to the postpartum focus on infant care rather than maternal health [16], a limitation to nearly all of the aforementioned mobile apps is the lack of continuation of physical activity support during the postpartum period [8]. Over 50% of maternal deaths occur between 7 and 365 days postpartum, many of which are related to factors modifiable with physical activity (i.e., obesity, diabetes, hypertension, and mental health) [4]. Given that pregnancy is an important opportunity for making lasting lifestyle changes [17], and keeping women engaged through postpartum can help with long-term adherence, the development of an mHealth app that also assists women through the postpartum period is critical.
To address these gaps, our team has engaged in extensive preliminary studies including focus groups and interviews with pregnant women, postpartum women, and obstetric health care providers [18] in order to develop an acceptable and promising approach to increase physical activities during and after pregnancy. Key features identified from these sessions and integrated into the development of a mobile health app (BumptUp®) include progress tracking, social support, evidence-based and safe exercise programming, videos, and symptom tracking [18], and these features set apart BumptUp® from existing mobile health interventions for pregnant and postpartum women. The primary goal of this pilot study was to test the efficacy of BumptUp® for improving physical activity and self-efficacy for physical activity among pregnant and postpartum women.

2. Materials and Methods

Participants: Study information was provided via social media, the physicians at the designated health clinic, and word of mouth. Sixty-nine women reached out regarding participation between November 2020 and August 2021. Inclusion criteria included: (1) Age 18–44; (2) Confirmed singleton viable pregnancy; (3) English-speaking (the app is currently only available in English); (4) Physician release to participate in exercise; (5) Ownership and willingness to use a smartphone; and (6) Plans to deliver at The Medical Center in Bowling Green, KY. Exclusion criteria included: (1) Multiple gestation pregnancy; (2) Inability to provide voluntary informed consent; and (3) Any medical condition (pregnancy-related or not) that would preclude exercise. All 69 were screened; 38 were deemed eligible and enrolled. Three women dropped out (one experienced a spontaneous abortion (control group) and two were lost to follow up (intervention)); thus, a total of 35 were included in the final analyses.
Study procedures: All procedures were approved by the Western Kentucky University Institutional Review Board (#20-257) and registered with clinicaltrials.gov (NCT04480931). An overview of the study design is provided in Figure 1.
Data were collected at the following timepoints: baseline/mid-pregnancy (23–25 weeks), late pregnancy (35–37 weeks), and 6 and 12 weeks postpartum. Of note, 12-week interventions during pregnancy are customary and have been shown to elicit clinically meaningful changes in outcomes [19,20]. There were two telephone check-ins (one during pregnancy and one during postpartum upon medical clearance at ~6 weeks) to aid with retention, as well as allow participants to ask any questions about the app or the educational brochure. The late-pregnancy timepoint was selected as women tend to be least active during late pregnancy [21]. Six weeks postpartum was chosen, as this is when most women have their only postpartum clinical appointment (and thus presented an opportunity for objective clinical data extraction). The 12-week time point was selected for assessing behavior as women transition back into activity after medical clearance during the early postpartum days, which is important for long-term sustained activity.
After the baseline assessment (~23–25 weeks gestation), participants were randomized into the intervention or control group. The intervention group received free access to the mobile app (BumptUp®). BumptUp® users were started with an introductory video on the app’s features which reiterated the goal of the app: to reach the recommended 150 min per week of physical activity as per recommendations from the American College of Obstetricians and Gynecologists [5]. The app contains evidence-based workout information and education as well as several resources aimed at increasing physical activity and overcoming barriers for women during and after pregnancy. Key additional features of the app include physical activity tracking, customized calorie tracking, customized gestational weight gain tracking with visuals, pregnancy-specific exercise videos with modifications and difficulty ranges, a build-your-own-workout system, symptom-tracking with a flagging feature for when to contact a provider, social support and communication, weekly educational articles, and careful screening mechanisms built into the flow of the app.
The control group underwent the exact same study protocols and received the same amount of attention from the study team (an attention control); however, control participants received an evidence-based educational brochure about physical activity during pregnancy instead of access to BumptUp®. The evidence-based brochure has been shown to increase knowledge and influence beliefs about physical activity during pregnancy [22], and an intervention utilizing the educational brochure was shown to reduce sedentary time during pregnancy [23].
Data collection: To assess the primary outcome of physical activity, participants wore an Actigraph wGT3X-BT Accelerometer (ActiGraph, LLC, Pensacola, FL, USA) on their wrist for seven consecutive days at each time point (Pregnancy: 23–25 weeks, 35–37 weeks, Postpartum: 12 weeks). Monitoring physical activity for one week at multiple time points is standard in physical activity research [24]. Wrist-worn tri-axel accelerometers are a valid measure of physical activity in pregnant women [25]. Devices were initialized by the study team in the Exercise Physiology Laboratory and delivered to each participant at a location of their choice. At the end of the 7 days, the study team picked up the device from the location desired by the participant. Data were collected for seven consecutive days at 30 Hz. The accelerometer output was sampled by a 12-bit analog-to-digital converter. Categories of activity were determined using the following cut points: Sedentary (0–99 counts/minute), light (100–1951 counts/min), moderate (1952–5724 counts/minute), and vigorous (≥5725 counts/min) [26]. The percentage of time spent sedentary as well as the amount of time spent participating in different categories of physical activity ranging from light to vigorous was calculated. Non-wear times were excluded from the analyses. Women were asked to wear the devices for 7 consecutive days without removal. If they did remove it, they were asked to document it so the study team could be sure it did not interfere with the analyses. The study coordinated with participants to select a week when they did not anticipate a reason for removal. As such, compliance was very high. All women wore the device for all 7 days with few or no removals.
Physical activity was also assessed subjectively using the Pregnancy Physical Activity Questionnaire (PPAQ) [27]. Self-efficacy was assessed via the Self-Efficacy for Physical Activity Survey [28] and delivered to the patient electronically via Research Electronic Data Capture (REDCap)(version 11.2.2) [29]. Self-efficacy was chosen as an important outcome to assess as self-efficacy is intricately linked to engagement in physical activity [30], and this is especially true among pregnant women [31,32].
Electronic surveys were distributed at each time point via REDCap [29]. Additional surveys which will serve as additional future outcomes and/or potential covariates in the larger trial included: General Demographics Survey, The Edinberg Postpartum Depression Questionnaire [33,34], Center for Epidemiological Studies Depression Scale [35], Social Support and Exercise Survey [36,37], CDC Barriers to being Active Quiz [38], Pregnancy Symptoms Inventory [39], Pelvic Floor Health (PDFI-I) [40], and the National Cancer Institute Multifactor (Diet)Screener [41]. Some surveys will be used to determine the potential impact of the mHealth intervention on a particular outcome related to maternal health. For example, mental health surveys (Edinberg Postpartum Depression Questionnaire and the Center for Epidemiological Studies Depression Scale) were included as depression plays a significant role in maternal morbidity and mortality [42,43], and physical activity improves depressive symptoms during and after pregnancy [44]. Therefore, it is useful to understand if the mHealth app improves mental health.
At the end of the protocol, participants assigned to the intervention group were given an app satisfaction survey (using the Mobile Application Rating Scale (MARS)) [45] in addition to one-on-one exit interviews. The MARS survey is a validated instrument and has excellent reliability (Omega 0.79 to 0.93) [46].
Medical Data from Participant Charts: Key medical data were collected directly from patient electronic medical record systems. These data included height, weight, and blood pressure values at prenatal appointments that correspond to data collection time points (i.e., 23–27 weeks-wider time range due to appointments still being monthly for many women during early-to-mid pregnancy, 35–37 weeks, and 6-weeks postpartum). All data at 12 weeks postpartum were patient/participant self-report as there are no routine postpartum visits at 12 weeks as per the standard-of-care. These extracted data were used to determine the impact of the intervention on weight gain, weight retention, blood pressure, and glucose tolerance.
Statistical Analysis: Pilot data are used to provide an estimate of the standard deviation and effect size, which will then be used to determine sample size and what will be observed in the main trial (forthcoming). Therefore, statistically significant differences between groups were not expected. However, appropriate statistical tests were still performed in order to examine trends and estimate effect sizes to use in the planning of a larger trial in the future.
The baseline characteristics of the intervention and control groups were compared using t-tests and chi-square tests, as appropriate. To investigate the effect of the intervention on study outcomes over time, a full-factorial repeated-measures ANCOVA model was fit for each outcome, analyzing overall differences across the two groups at each timepoint while adjusting for body mass index (BMI). Likelihood ratio testing and Akaike Information Criterion (AIC) were used to select appropriate covariance structures in each case. Due to the pilot nature of the study, group-level least-square means and their pairwise differences were calculated at each timepoint and adjusted for multiple comparisons, as appropriate. A Kenward–Roger adjustment was used to correct for negative bias in the standard errors and degrees of freedom calculations induced by the small sample size. All analyses were completed in SAS 9.4 (SAS Institute Inc.; Cary, NC, USA) or SPSS (version 28). All data were entered, stored, and maintained in the REDcap data management system [29].

3. Results

There were no baseline differences between the control and intervention groups for any of the demographic variables assessed including BMI, marital status, income level, race/ethnicity, educational attainment, employment status, self-reported health status, and physical activity levels. Table 1 contains demographic information for control and intervention participants.
The mean pre-pregnancy BMI of the study cohort was 29.0 ± 7.5 kg/m2 (range 21.4–59.4 kg/m2). In the study sample, 29% of participants had pre-pregnancy BMI within normal limits, while 38% were classified as overweight and 33% were classified as obese. According to the Centers for Disease Control, ~37% of women of childbearing age in Kentucky are classified as obese [47], suggesting a sample of women representative of women in the community.
Efficacy: There were no statistically significant differences between groups over time in physical activity levels (or other obstetric health outcomes such as weight gain/retention, glucose control, and blood pressure), which is expected given the small sample size. However, all data demonstrate promise. For example, Figure 2 shows moderate-to-vigorous physical activity levels (MVPA) for both groups based on objective accelerometer data. Table 2 contains all accelerometry data. While not statistically significant, a clear trend is observed as the mean estimated difference between groups (when adjusted for BMI) changes from −0.35 (SE: 1.75) in mid-pregnancy to −0.81 (SE:1.75) in late pregnancy. Analyses were adjusted for BMI because BMI is associated with MVPA levels in the present study (mid- pregnancy: r = −0.410, p = 0.016) and in the existing literature [48]). There are clear and directionally consistent changes in MVPA; however, the large standard errors induced by our small sample size did now allow for statistical significance.
Figure 3 shows PPAQ data, demonstrating that the control group saw a 15.1% decrease in total activity (sum of light, moderate, and vigorous), while the intervention group still managed to increase their activity levels by 6.7% from mid to late pregnancy (p = 0.11) (Figure 4).
Self-efficacy (via the SEES) was assessed across all four timepoints and controlling for BMI, and the estimated difference of means (control-intervention) changed from 0.96 (SE: 6.53) at baseline to −7.64 (SE: 6.66) in late pregnancy and remained at −6.41 (SE: 6.79) and −6.70 (SE: 6.96) at 6 and 12 weeks postpartum, respectively. Of note, BMI was controlled for because BMI is correlated with self-efficacy for exercise both in the literature [49] and in the current study (late pregnancy and SEES: r = −0.412, p = 0.021). When assessing the change in SEES from basline to late pregnacy only, there was a statistically significant difference in self-efficacy between groups (p = 0.044) (Figure 4). Furthermore, self-efficacy scores (both groups) were correlated with physical activity (MVPA) (r = 0.421, p = 0.012).
Based on an app satisfaction survey (using the MARS, Table 3), participants found the app to have trustworthy (4.0 out of 5) and reliable (3.9 out of 5) information, and they identified the workouts as safe (4.2 out of 5). Analyses for additional clinical outcomes were assessed, but statistical significance was not obtained and these data are not reported.

4. Discussion

Compared to an attention control group, pregnant and postpartum women using BumptUp® showed improved self-efficacy for physical activity. Thus, while the modest sample size precluded statistical significance for MVPA, we conclude that BumptUp® has potential to increase physical activity behaviors among pregnant and postpartum women. Study findings suggest key refinements to be made per pilot project feedback and a larger, adequately powered clinical trial is warranted.
The increase in self-efficacy for exercise among intervention participants is a critically important first step as public health theories demonstrate self-efficacy is intricately linked to engagement in physical activity [30], and this is especially true among pregnant women [31,32]. This change in self-efficacy demonstrates the potential of well-designed mobile apps to positively influence physical activity-related self-efficacy and subsequent behaviors (i.e., physical activity levels). This result is further demonstrated by the relationship noted between physical activity and self-efficacy among women in the study. Given that most women experience a decline in physical activity as pregnancy progresses [50] due to many new barriers that exist [51], and that self-efficacy for activity late in pregnancy influences exercise levels [52], this significant result is important and suggests that BumptUp® may favorably influence activity levels among pregnant women.
Important theoretical implications for this work exist. Because the features of the app were designed within the framework of the Health Belief Model [18], future intervention strategies should consider using the constructs of the Health Belief Model to guide the development of the intervention tools employed. Self-efficacy was added to the Health Belief Model because it can better explain individual differences in health behaviors [53,54]. Given the known relationship between self-efficacy for exercise and exercise behavior [55], targeting pregnant and postpartum women’s ability to feel confident in their ability to be active is critically important and is oftentimes ignored in exercise interventions. It is important to design interventions that not only seek to provide physical support for exercise, but mental support as well. Interestingly, self-efficacy for exercise may be even more important for determining long-term health outcomes than self-assessed physical activity levels [51]. Women with higher levels of self-efficacy for exercise may have higher levels of general confidence, positive well-being, as well as reduced fatigue and psychological distress during and after exercise compared to those with low self-efficacy [56].
While post-test data did not demonstrate a statistically significant change in physical activity levels, these pilot data show promise given improvement in self-efficacy and other promising trends in the physical activity data. Given that physical activity levels typically drop as pregnancy progresses [50], a resource that can prevent or minimize this drop in activity levels has strong potential to favorably impact clinical outcomes, which is precisely what the data demonstrate in this pilot project.
At the end of the study, intervention participants were all contacted for an optional exit interview. Of the 19 contacted, 10 were interviewed. Women were given the opportunity to provide open-ended feedback regarding the app, which was overall very positive. For example, a representative quote from one pilot study participant at 14 weeks postpartum said:
“The BumptUp® App acted as a great guide both during and after pregnancy. The activity tracker kept me accountable and motivated especially during the last few weeks before giving birth. The app provided excellent tips and educational articles that helped me make my way through the changes I was experiencing each week as well. After giving birth, the app didn’t stop! It was a great tool for a time when I needed support the most. The guided exercises may have been my favorite part because I knew that I would not be overexerting myself as I eased back into moving on a daily basis! Each workout was feasible yet challenging all at the same time, and I would recommend this type of guidance to any mom recovering from birth!”
The study team also asked each participant about the acceptability of the intervention itself, and all 10 reported that the surveys were not overly time-consuming and were well worth the small financial incentive. A common theme that emerged during the interviews was the appreciation of the electronic delivery of surveys so they could complete them anywhere/anytime as well as the porch delivery/pick-up of activity monitors. Feasibility and acceptability were demonstrated by the fact that only three women dropped out of the study (8% attrition rate), which is a strength of the study; interventions typically average a much higher rate of drop out [57]. Feasibility was also demonstrated by the fact that the sample was recruited in less than 9 months despite passive recruitment tactics (i.e., flyers hung at local clinics and on social media platforms from which potential participants reached out regarding participation).
In addition, the app satisfaction survey (the MARS tool [45]) provided feedback on areas to continue to develop and improve, which is a future direction of work. These include enhancing the capacity for customizations, improving usability and ease of navigation, and increasing the choices for workout programming within the app. Collectively, the information gathered from all of these data collections support the potential for impact of BumptUp® in future trials.
While data from the BumptUp ® pilot study did not demonstrate statistically significant improvements in physical activity levels (which was expected in a small pilot study), the app shows considerable promise for favorably impacting self-efficacy for exercise via the Self-Efficacy for Exercise Survey (SEES), and the importance of maintaining self-efficacy for exercise during a time point where physiologic changes can make exercise especially challenging for women [51] (i.e., pregnancy and recovering for pregnancy) should not be undersold. From a practical standpoint, the study is an important first step in showing that digital health technology can become an important and timely way to reach pregnant and postpartum women with much-needed physical activity education and support.
Limitations of the study include a small sample size; thus, statistical significance was unlikely to be obtained for many outcomes. In addition, the study population was not very diverse, thus limiting generalizability. Another limitation is the inability to directly determine adherence; the study team was unable to determine (based on app data) who achieved the goal of 150 minutes per week and logged this on the app. This is an important concept for future app improvement—an administrative feature that allows the team to determine who logged 150 min/week into the app and who did not. Despite the limitations, the study has many notable strengths. One strength of the study is its inclusion of an intervention designed based on public health theory and directly by key stakeholders. To our knowledge, BumptUp® is the only app created that fits these criteria. Another strength is the objective quality of the data collections, particularly the physical activity data (Actigraph) and the medical outcome data directly from patient charts (not self-reported). Another notable strength is the low rate of attrition; most interventions have a higher rate of drop-out [57]. Another key strength is that the intervention proposed has potential for scalability. Many times, physical activity interventions are successful at changing outcomes, but the ability to scale them up and/or sustain them is impossible (i.e., an exercise intervention with one-on-one coaching sessions). BumptUp® offers a potential solution that could easily be used by millions of women without placing burden on the health care team.
Given the maternal mortality and morbidity crisis in the United States, evidence-based and potentially sustainable intervention strategies are needed. The mobile app tested in this pilot project has potential to be further refined and scaled-up to serve women across the nation. Future directions include app refinement, a larger clinical trial, and additional testing among health disparity populations in order to make sure the app is effective, accessible and culturally appropriate for all women. Another pilot study trial is planned in a lower-middle-income country to test the acceptability and effectiveness of the app in another country. From a broader perspective, the opportunity that exists for BumptUp® (and other digital health interventions) is unique and timely. Obstetricians are seeking resources to help patients improve health outcomes such as blood pressure, insulin resistance, complicated deliveries, and depression/anxiety; physical activity can improve all of these things. A simple resource that providers can give to patients, without increasing clinical demands or expecting them to prescribe exercise (for which they are not qualified [51]) is critical in our nation and world. With further testing and refinement, BumptUp® could become that resource for pregnant women, postpartum women, and their healthcare team.

5. Patents

BumptUp® is registered for US Trademark (Reg. No. 6,837,038; Serial No. 90-812,606, Registered 6 September 2022).

Author Contributions

Conceptualization, R.A.T., M.M.B., A.M.C. and N.E.S.; methodology, R.A.T. and M.M.B.; software, R.A.T. and G.S.H.; formal analysis, G.S.H. and R.A.T.; investigation, R.A.T., A.M.C., M.M.B. and N.E.S.; writing—original draft preparation, R.A.T., N.E.S., G.S.H. and M.P.; writing—review and editing, R.A.T., A.M.C., M.M.B., N.E.S., G.S.H. and M.P.; funding acquisition, R.A.T., N.E.S. and M.M.B. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by NIGMS-funded IDeA Networks of Biomedical Research Excellence in Kentucky (3P20GM103436-20S1, ULRF_18-0975S2-01).

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of Western Kentucky University (#20-257, date of approval: 28 April 2020).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Trends in Maternal Mortality: 1990 to 2013. Estimates by WHO, UNICEF, UNFPA, The World Bank and the United Nations Population Division. World Health Organization: Geneva, Switzerland, 2014. Available online: http://www.who.int/reproductivehealth/publications/monitoring/maternal-mortality-2013/en/ (accessed on 18 July 2014).
  2. Agrawal, P. Maternal mortality and morbidity in the United States of America. Bull. World Health Organ. 2015, 93, 135. [Google Scholar] [CrossRef] [PubMed]
  3. Elixhauser, A.; Wier, L.M. Complicating conditions of pregnancy and childbirth, 2008: Statistical brief #113. In Healthcare Cost and Utilization Project (HCUP) Statistical Briefs; Agency for Healthcare Research and Quality: Rockville, MD, USA, 2006. [Google Scholar]
  4. Committee on Health Care for Underserved Women. ACOG committee opinion no. 586: Health disparities in rural women. Obstet. Gynecol. 2014, 123 Pt 1, 384–388. [Google Scholar] [CrossRef] [PubMed]
  5. Syed, H.; Slayman, T.; Thoma, K.D. Physical Activity and Exercise During Pregnancy and the Postpartum Period: ACOG Committee Opinion, Number 804. Obstet. Gynecol. 2020, 135, e178–e188. [Google Scholar] [CrossRef]
  6. Henriksson, P.; Sandborg, J.; Blomberg, M.; Alexandrou, C.; Maddison, R.; Silfvernagel, K.; Henriksson, H.; Leppanen, M.H.; Migueles, J.H.; Widman, L.; et al. A smartphone app to promote healthy weight gain, diet, and physical activity during pregnancy (HealthyMoms): Protocol for a randomized controlled trial. JMIR Res. Protoc. 2019, 8, e13011. [Google Scholar] [CrossRef] [Green Version]
  7. O’Brien, O.A.; McCarthy, M.; Gibney, E.R.; McAuliffe, F.M. Technology-supported dietary and lifestyle interventions in healthy pregnant women: A systematic review. Eur. J. Clin. Nutr. 2014, 68, 760–766. [Google Scholar] [CrossRef] [Green Version]
  8. Tinius, R.A.; Polston, M.; Bradshaw, H.; Ashley, P.; Greene, A.; Parker, A.N. An assessment of mobile applications designed to address physical activity during pregnancy and postpartum. Int. J. Exerc. Sci. 2021, 14, 382–399. [Google Scholar] [PubMed]
  9. Brown, H.M.; Bucher, T.; Collins, C.E.; Rollo, M.E. A review of pregnancy apps freely available in the Google Play Store. Health Promot. J. Aust. 2020, 31, 340–342. [Google Scholar] [CrossRef]
  10. Hayman, M.; Alfrey, K.L.; Cannon, S.; Alley, S.; Rebar, A.L.; Williams, S.; Short, C.E.; Altazan, A.; Comardelle, N.; Currie, S.; et al. Quality, features, and presence of behavior change techniques in mobile apps designed to improve physical activity in pregnant women: Systematic search and content analysis. JMIR Mhealth Uhealth 2021, 9, e23649. [Google Scholar] [CrossRef] [PubMed]
  11. Hayman, M.J.; Alfrey, K.L.; Waters, K.; Cannon, S.; Mielke, G.I.; Keating, S.E.; Mena, G.P.; Mottola, M.F.; Evenson, K.R.; Davenport, M.H.; et al. Evaluating evidence-based content, features of exercise instruction, and expert involvement in physical activity apps for pregnant women: Systematic search and content analysis. JMIR Mhealth Uhealth 2022, 10, e31607. [Google Scholar] [CrossRef] [PubMed]
  12. Sandborg, J.; Söderström, E.; Henriksson, P.; Bendtsen, M.; Henström, M.; Leppänen, M.H.; Maddison, R.; Migueles, J.H.; Blomberg, M.; Löf, M. Effectiveness of a smartphone app to promote healthy weight gain, diet, and physical activity during pregnancy (HealthyMoms): Randomized controlled trial. JMIR Mhealth Uhealth 2021, 9, e26091. [Google Scholar] [CrossRef] [PubMed]
  13. Dodd, J.M.; Louise, J.; Cramp, C.; Grivell, R.M.; Moran, L.J.; Deussen, A.R. Evaluation of a smartphone nutrition and physical activity application to provide lifestyle advice to pregnant women: The SNAPP randomised trial. Matern. Child Nutr. 2018, 14, e12502. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Choi, J.; Lee, J.H.; Vittinghoff, E.; Fukuoka, Y. mHealth physical activity intervention: A randomized pilot study in physically inactive pregnant women. Matern. Child Health J. 2016, 20, 1091–1101. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  15. Ainscough, K.M.; O’Brien, E.C.; Lindsay, K.L.; Kennelly, M.A.; O’Sullivan, E.J.; O’Brien, O.A.; McCarthy, M.; De Vito, G.; McAuliffe, F.M. Nutrition, behavior change and physical activity outcomes from the PEARS RCT-an mHealth-supported, lifestyle intervention among pregnant women with overweight and obesity. Front. Endocrinol. 2019, 10, 938. [Google Scholar] [CrossRef]
  16. Wendland, C.L. The vanishing mother: Cesarean section and “evidence-based obstetrics”. Med. Anthropol. Q. 2007, 21, 218–233. [Google Scholar] [CrossRef] [PubMed]
  17. Joy, E.A.; Van Hala, S.; Marshall, E. Pregnancy as an opportunity for behavior change. Curr. Sports Med. Rep. 2004, 3, 327–329. [Google Scholar] [CrossRef] [PubMed]
  18. Tinius, R.; Duchette, C.; Beasley, S.; Blankenship, M.; Schoenberg, N. Obstetric patients and healthcare providers perspectives to inform mobile app design for physical activity and weight control during pregnancy and postpartum in a rural setting. Int. J. Womens Health 2021, 13, 405–432. [Google Scholar] [CrossRef] [PubMed]
  19. Hawkins, M.; Chasan-Taber, L.; Marcus, B.; Stanek, E.; Braun, B.; Ciccolo, J.; Markenson, G. Impact of an exercise intervention on physical activity during pregnancy: The behaviors affecting baby and you study. Am. J. Public Health 2014, 104, e74–e81. [Google Scholar] [CrossRef]
  20. Bisson, M.; Almeras, N.; Dufresne, S.S.; Robitaille, J.; Rheaume, C.; Bujold, E.; Frenette, J.; Tremblay, A.; Marc, I. A 12-week exercise program for pregnant women with obesity to improve physical activity levels: An open randomised preliminary study. PLoS ONE 2015, 10, e0137742. [Google Scholar] [CrossRef]
  21. Nascimento, S.L.; Surita, F.G.; Godoy, A.C.; Kasawara, K.T.; Morais, S.S. Physical activity patterns and factors related to exercise during pregnancy: A cross sectional study. PLoS ONE 2015, 10, e0128953. [Google Scholar] [CrossRef] [Green Version]
  22. Alvis, M.L.; Morris, C.E.; Garrard, T.L.; Hughes, A.G.; Hunt, L.; Koester, M.M.; Yocum, I.C.; Tinius, R.A. Educational brochures influence beliefs and knowledge regarding exercise during pregnancy: A pilot study. Int. J. Exerc. Sci. 2019, 12, 581–589. [Google Scholar] [PubMed]
  23. Tinius, R.; Edens, K.; Link, K.; Jones, M.S.; Lyons, S.; Rebelle, T.; Pearson, K.J.; Maples, J. Effect of evidence-based materials and access to local resources on physical activity levels, beliefs, and motivation during pregnancy in a rural setting. J. Phys. Act. Health 2020, 17, 947–957. [Google Scholar] [CrossRef] [PubMed]
  24. Dunn, A.L.; Marcus, B.H.; Kampert, J.B.; Garcia, M.E.; Kohl, H.W., 3rd; Blair, S.N. Comparison of lifestyle and structured interventions to increase physical activity and cardiorespiratory fitness: A randomized trial. JAMA 1999, 281, 327–334. [Google Scholar] [CrossRef]
  25. Van Hees, V.T.; Renstrom, F.; Wright, A.; Gradmark, A.; Catt, M.; Chen, K.Y.; Lof, M.; Bluck, L.; Pomeroy, J.; Wareham, N.J.; et al. Estimation of daily energy expenditure in pregnant and non-pregnant women using a wrist-worn tri-axial accelerometer. PLoS ONE 2011, 6, e22922. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Freedson, P.S.; Melanson, E.; Sirard, J. Calibration of the computer science and applications, inc. accelerometer. Med. Sci. Sports Exerc. 1998, 30, 777–781. [Google Scholar] [CrossRef] [PubMed]
  27. Chasan-Taber, L.; Schmidt, M.D.; Roberts, D.E.; Hosmer, D.; Markenson, G.; Freedson, P.S. Development and validation of a pregnancy physical activity questionnaire. Med. Sci. Sports Exerc. 2004, 36, 1750–1760. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Resnick, B.; Jenkins, L.S. Testing the reliability and validity of the self-efficacy for exercise scale. Nurs. Res. 2000, 49, 154–159. [Google Scholar] [CrossRef]
  29. Harris, P.A.; Taylor, R.; Thielke, R.; Payne, J.; Gonzalez, N.; Conde, J.G. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J. Biomed. Inform. 2009, 42, 377–381. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  30. Holloway, A.; Watson, H.E. Role of self-efficacy and behaviour change. Int. J. Nurs. Pract. 2002, 8, 106–115. [Google Scholar] [CrossRef] [PubMed]
  31. Melton, B.; Marshall, E.; Bland, H.; Schmidt, M.; Guion, W.K. American rural women’s exercise self-efficacy and awareness of exercise benefits and safety during pregnancy. Nurs. Health Sci. 2013, 15, 468–473. [Google Scholar] [CrossRef] [PubMed]
  32. Shafieian, M.; Kazemi, A. A randomized trial to promote physical activity during pregnancy based on health belief model. J. Educ. Health Promot. 2017, 6, 40. [Google Scholar] [CrossRef] [PubMed]
  33. Cox, J.L.; Holden, J.M.; Sagovsky, R. Detection of postnatal depression: Development of the 10-item Edinburgh Postnatal Depression Scale. Br. J. Psychiatry 1987, 150, 782–786. [Google Scholar] [CrossRef] [Green Version]
  34. Schaper, A.M.; Rooney, B.L.; Kay, N.R.; Silva, P.D. Use of the Edinburgh Postnatal Depression Scale to identify postpartum depression in a clinical setting. J. Reprod. Med. 1994, 39, 620–624. [Google Scholar] [PubMed]
  35. Radloff, L.S. The use of the Center for Epidemiologic Studies Depression Scale in adolescents and young adults. J. Youth Adolesc. 1991, 20, 149–166. [Google Scholar] [CrossRef]
  36. Sallis, J.F.; Grossman, R.M.; Pinski, R.B.; Patterson, T.L.; Nader, P.R. The development of scales to measure social support for diet and exercise behaviors. Prev. Med. 1987, 16, 825–836. [Google Scholar] [CrossRef]
  37. Noroozi, A.; Ghofranipour, F.; Heydarnia, A.R.; Nabipour, I.; Shokravi, F.A. Validity and reliability of the social support scale for exercise behavior in diabetic women. Asia Pac. J. Public Health 2011, 23, 730–741. [Google Scholar] [CrossRef]
  38. Centers for Disease Control and Prevention (CDC) Barrier to Being Active. Available online: https://www.cdc.gov/diabetes/ndep/pdfs/8-road-to-health-barriers-quiz-508.pdf (accessed on 29 January 2022).
  39. Foxcroft, K.F.; Callaway, L.K.; Byrne, N.M.; Webster, J. Development and validation of a pregnancy symptoms inventory. BMC Pregnancy Childbirth 2013, 13, 3. [Google Scholar] [CrossRef] [Green Version]
  40. Zuchelo, L.T.S.; Bezerra, I.M.P.; Da Silva, A.T.M.; Gomes, J.M.; Soares Junior, J.M.; Chada Baracat, E.; de Abreu, L.C.; Sorpreso, I.C.E. Questionnaires to evaluate pelvic floor dysfunction in the postpartum period: A systematic review. Int. J. Womens Health 2018, 10, 409–424. [Google Scholar] [CrossRef] [Green Version]
  41. Thompson, F.E.; Midthune, D.; Subar, A.F.; Kahle, L.L.; Schatzkin, A.; Kipnis, V. Performance of a short tool to assess dietary intakes of fruits and vegetables, percentage energy from fat and fibre. Public Health Nutr. 2004, 7, 1097–1105. [Google Scholar] [CrossRef]
  42. Palladino, C.L.; Singh, V.; Campbell, J.; Flynn, H.; Gold, K.J. Homicide and suicide during the perinatal period: Findings from the National Violent Death Reporting System. Obstet. Gynecol. 2011, 118, 1056–1063. [Google Scholar] [CrossRef] [Green Version]
  43. Slomian, J.; Honvo, G.; Emonts, P.; Reginster, J.Y.; Bruyere, O. Consequences of maternal postpartum depression: A systematic review of maternal and infant outcomes. Womens Health 2019, 15, 1745506519844044. [Google Scholar] [CrossRef]
  44. Nakamura, A.; van der Waerden, J.; Melchior, M.; Bolze, C.; El-Khoury, F.; Pryor, L. Physical activity during pregnancy and postpartum depression: Systematic review and meta-analysis. J. Affect. Disord. 2019, 246, 29–41. [Google Scholar] [CrossRef] [PubMed]
  45. Stoyanov, S.R.; Hides, L.; Kavanagh, D.J.; Zelenko, O.; Tjondronegoro, D.; Mani, M. Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015, 3, e27. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  46. Terhorst, Y.; Philippi, P.; Sander, L.B.; Schultchen, D.; Paganini, S.; Bardus, M.; Santo, K.; Knitza, J.; Machado, G.C.; Schoeppe, S.; et al. Validation of the mobile application rating scale (MARS). PLoS ONE 2020, 15, e0241480. [Google Scholar] [CrossRef]
  47. Centers for Disease Control and Prevention (CDC). Behavioral Risk Factor Surveillance System Survey Questionnaire; Department of Health and Human Services, Centers for Disease Control and Prevention: Atlanta, GA, USA, 2022. [Google Scholar]
  48. Andersson-Hall, U.; de Maré, H.; Askeli, F.; Börjesson, M.; Holmäng, A. Physical activity during pregnancy and association with changes in fat mass and adipokines in women of normal-weight or with obesity. Sci. Rep. 2021, 11, 12549. [Google Scholar] [CrossRef] [PubMed]
  49. Brooks, J.M.; Deiches, J.; Xiang, X.; Batsis, J.A.; Chan, F.; DiMilia, P.; Chiu, C.; Thompson, K.; Bartels, S. Differences in self-reported physical activity, exercise self-efficacy and outcome expectancies, and health status by body mass index groups in people with chronic pain. J. Rehabil. 2018, 84, 46–52. [Google Scholar] [PubMed]
  50. Borodulin, K.M.; Evenson, K.R.; Wen, F.; Herring, A.H.; Benson, A.M. Physical activity patterns during pregnancy. Med. Sci. Sports Exerc. 2008, 40, 1901–1908. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  51. Bergström, G.; Börjesson, M.; Schmidt, C. Self-efficacy regarding physical activity is superior to self-assessed activity level, in long-term prediction of cardiovascular events in middle-aged men. BMC Public Health 2015, 15, 820. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  52. Cramp, A.G.; Bray, S.R. A prospective examination of exercise and barrier self-efficacy to engage in leisure-time physical activity during pregnancy. Ann. Behav. Med. 2009, 37, 325–334. [Google Scholar] [CrossRef]
  53. Orji, R.; Vassileva, J.; Mandryk, R. Towards an effective health interventions design: An extension of the health belief model. Online J. Public Health Inform. 2012, 4. [Google Scholar] [CrossRef] [Green Version]
  54. Rosenstock, I.M.; Strecher, V.J.; Becker, M.H. Social learning theory and the health belief model. Health Educ. Q. 1988, 15, 175–183. [Google Scholar] [CrossRef]
  55. McAuley, E.; Blissmer, B. Self-efficacy determinants and consequences of physical activity. Exerc. Sport Sci. Rev. 2000, 28, 85–88. [Google Scholar] [PubMed]
  56. McAuley, E.; Talbot, H.M.; Martinez, S. Manipulating self-efficacy in the exercise environment in women: Influences on affective responses. Health Psychol. 1999, 18, 288–294. [Google Scholar] [CrossRef] [PubMed]
  57. Hui, D.; Glitza, I.; Chisholm, G.; Yennu, S.; Bruera, E. Attrition rates, reasons, and predictive factors in supportive care and palliative oncology clinical trials. Cancer 2013, 119, 1098–1105. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Sustainability 14 12801 g001 550
Figure 1. Study flowchart and procedures.
Figure 1. Study flowchart and procedures.
Sustainability 14 12801 g001
Sustainability 14 12801 g002 550
Figure 2. Moderate-to-vigorous physical activity levels between control and intervention participants (mean ± SEM).
Figure 2. Moderate-to-vigorous physical activity levels between control and intervention participants (mean ± SEM).
Sustainability 14 12801 g002
Sustainability 14 12801 g003 550
Figure 3. Change in total activity from mid-to-late pregnancy in control and intervention partic-pants (mean ± SEM).
Figure 3. Change in total activity from mid-to-late pregnancy in control and intervention partic-pants (mean ± SEM).
Sustainability 14 12801 g003
Sustainability 14 12801 g004 550
Figure 4. Self-efficacy foe exercise scores in control and intervention participants (mean ± SEM). * p < 0.05.
Figure 4. Self-efficacy foe exercise scores in control and intervention participants (mean ± SEM). * p < 0.05.
Sustainability 14 12801 g004
Table
Table 1. Demographic Information for Control and Intervention Participants.
Table 1. Demographic Information for Control and Intervention Participants.
Control (n = 18)Intervention (n = 17) p-Value
Pre-pregnancy BMI (kg/m2)30.2 ± 8.727.8 ± 6.00.348
Married 18 (100%)16 (94%)0.486
Annual income 0.285
>$20 k0 (0.0%)2 (11.8%)
$20–40 k3 (16.7%)3 (17.6%)
$60–80 k3 (16.7%)5 (29.4%)
>$80 k12 (66.7%)7 (41.2%)
Race 0.472
Caucasian18(100%)17 (100%)
Educational Attainment 0.537
Highschool/GED1 (5.6%)0 (0.0%)
Trade/Technical School0 (0.0%)2 (11.8%)
Associates degree1 (5.6%)0 (0.0%)
Bachelor’s degree9 (50.0%)9 (52.9%)
Master’s degree5 (27.8%)4 (23.5%)
Doctoral degree or higher2 (11.1%)2 (11.8%)
Employed 16 (88.9%)14 (83.4%)
Self-reported health status 0.386
Excellent4 (22.2%)8 (47.1%)
Very Good10 (55.6%)7 (41.2%)
Good3 (16.7%)2 (11.8%)
Fair1 (5.6%)0 (0.0%)
Poor0 (0.0%)0 (0.0%)
Baseline Physical Activity Levels (%)
Sedentary52.1 ± 14.751.8 ± 8.80.946
Light35.3 ± 10.535.6 ± 6.50.932
Moderate12.5 ± 5.913.5 ± 5.10.612
Table
Table 2. Accelerometry data for all cut points across all participants at each time point of the study.
Table 2. Accelerometry data for all cut points across all participants at each time point of the study.
% of Time over 7 Consecutive Days (mean ± SD)
SedentaryLightModerate
TimepointIGCGAllIGCGAllIGCGAll
Mid-Pregnancy51.8 ± 8.852.1 ± 14.752.0 ± 12.135.6 ± 6.535.3 ± 10.535.4 ± 8.613.5 ± 5.112.5 ± 5.913.0 ± 5.5
Late Pregnancy51.9 ± 10.953 ± 11.352.5 ± 10.934.3 ± 7.034.7 ± 7.434.5 ± 7.113.8 ± 5.712.3 ± 4.813.0 ± 5.2
12 weeks postpartum48.8 ± 7.649.0 ± 8.048.9 ± 7.736.3 ± 5.437.3 ± 4.736.8 ± 5.014.6 ± 5.513.8 ± 5.014.3 ± 5.2
IG: Intervention Group; CG: Control Group.
Table
Table 3. MARS survey data.
Table 3. MARS survey data.
QuestionLikert Scale ChoicesScore (0–5)
How trustworthy is the information on the app?Not trustworthy at all to very trustworthy 4.0 ± 0.9
How reliable was the information provided on the app?Not at all to extremely reliable3.9 ± 2.0
Was there enough information on the app?Not enough to as much as I could want3.5 ± 1.3
Was there enough education topics on the app?Not enough to as much as I could want3.5 ± 1.1
How would you rate the usability of the app?Impossible to use to very easy to use2.9 ± 1.1
How easy to use was the app?Too confusing to use to very easy to use3.0 ± 1.1
Did the app work well?Impossible to use to functioned perfectly2.9 ± 1.0
Was it easy to navigate the app?Not at all to very easy3.3 ± 1.1
Were there any problems with the app’s performance?Too many to use to none at all3.1 ± 1.3
How would you rate the quality of the workouts?Very poor to excellent3.4 ± 1.1
Did the workouts feel unsafe to you?All of them to none of them4.2 ± 0.9
Were the workouts fun?Not at all to very fun2.8 ± 1.1
Were the workouts an appropriate length for you?Too long/short to perfect length3.2 ± 1.0
Were you able to personalize the workouts as much as you wanted to?Not at all to as much as I wanted to2.6 ± 0.8
Was the app fun to use?Not at all to very fun2.9 ± 0.8
Were you able to tailor the app to you as much as you wanted?Not at all to all I wanted3.4 ± 1.0
Would you recommend this app to friends?Would discourage use to would highly recommend3.0 ± 1.0
What would your star rating of this app be?1 through 53.3 ± 1.0
How likely are you to download this app and use it again?Absolutely will not to absolutely will2.8 ± 1.2
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Tinius, R.A.; Blankenship, M.M.; Colao, A.M.; Hawk, G.S.; Perera, M.; Schoenberg, N.E. A Pilot Study on the Impact of the BumptUp® Mobile App on Physical Activity during and after Pregnancy. Sustainability 2022, 14, 12801. https://doi.org/10.3390/su141912801

AMA Style

Tinius RA, Blankenship MM, Colao AM, Hawk GS, Perera M, Schoenberg NE. A Pilot Study on the Impact of the BumptUp® Mobile App on Physical Activity during and after Pregnancy. Sustainability. 2022; 14(19):12801. https://doi.org/10.3390/su141912801

Chicago/Turabian Style

Tinius, Rachel A., Maire M. Blankenship, Alison M. Colao, Gregory S. Hawk, Madhawa Perera, and Nancy E. Schoenberg. 2022. "A Pilot Study on the Impact of the BumptUp® Mobile App on Physical Activity during and after Pregnancy" Sustainability 14, no. 19: 12801. https://doi.org/10.3390/su141912801

APA Style

Tinius, R. A., Blankenship, M. M., Colao, A. M., Hawk, G. S., Perera, M., & Schoenberg, N. E. (2022). A Pilot Study on the Impact of the BumptUp® Mobile App on Physical Activity during and after Pregnancy. Sustainability, 14(19), 12801. https://doi.org/10.3390/su141912801

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop