Next Article in Journal
Digital Social Responsibility towards Corporate Social Responsibility and Strategic Performance of Hi-Tech SMEs: Customer Engagement as a Mediator
Previous Article in Journal
Can System Log Data Enhance the Performance of Credit Scoring?—Evidence from an Internet Bank in Korea
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Development and Optimization of a Topical Formulation with Castanea sativa Shells Extract Based on the Concept “Quality by Design”

by
Nágilla Oliveira
1,
Maria de la Luz Cádiz-Gurrea
2,
Ana Margarida Silva
2,
Catarina Macedo
2,
Francisca Rodrigues
2,* and
Paulo Costa
1,3
1
UCIBIO—Applied Molecular Biosciences Unit, MedTech-Laboratory of Pharmaceutical Technology, Faculty of Pharmacy, University of Porto, Rua de Jorge Viterbo Ferreira 228, 4050-313 Porto, Portugal
2
REQUIMTE/LAQV, Polytechnic of Porto—School of Engineering, Rua Dr. António Bernardino de Almeida, 4249-015 Porto, Portugal
3
Associate Laboratory i4HB—Institute for Health and Bioeconomy, Faculty of Pharmacy, University of Porto, 4050-313 Porto, Portugal
*
Author to whom correspondence should be addressed.
Sustainability 2022, 14(1), 129; https://doi.org/10.3390/su14010129
Submission received: 30 November 2021 / Revised: 20 December 2021 / Accepted: 21 December 2021 / Published: 23 December 2021

Abstract

:
The proposed study aims to develop and optimize a topical formulation with Castanea sativa shells extract considering the concept of Quality by Design, focusing on a planned development that consider the vulnerabilities of the entire process through risk analysis tools and design of experiments (DoE). A Box–Behnken design with three factors and three levels was used as a statistical tool for the execution of the DoE and the analysis of the response surface methodology responses. The independent variables studied were the quantity of sodium lauryl sulfate (%) (X1), beeswax (%) (X2) and macadamia oil (%) (X3); the dependent variables were pH (Y1), viscosity (Y2) and adhesiveness (Y3). According to the mathematical model, the optimal formulation contains 0.93% of sodium lauryl sulfate, 5.00% of beeswax and 10.00% of macadamia oil. The optimal formulation with the extract was prepared and characterized over the time, regarding organoleptic and technological characteristics, allowing conclusions to be reached regarding its stability. The formulation presented a pleasant odor and was light brown in color, it also demonstrated pseudoplastic-thixotropic behavior and a small reduction in the formulation consistency after 30 days of storage. This study demonstrated the efficiency of the Quality by Design methodology to understand the product variability, supporting that this approach favors a better understanding of the whole process and enables to design a robust development stage, reducing costs and generating high-quality products.

1. Introduction

Skin aging is a multifactorial process that has gained attention by consumers in recent years due to new concerns of contemporary society related to beauty and aging [1,2]. Several factors contribute to skin aging, being divided into intrinsic (related to chronological changes) and extrinsic factors (associated with environmental influences) [3]. Nevertheless, skin aging is a complex process and a misunderstood concept that may lead to modifications in skin structure and functions [4]. Oxidative stress plays a key role in the aging process, being normally generated by the excess skin of free radicals due to the accumulation of reactive oxygen species (ROS) or a deficiency of antioxidant defenses, which lead to the damage of biological macromolecules such as lipids, proteins and nucleic acids [5,6]. Currently, the cosmetic market has a growing interest in replacing synthetic products with natural ones, in particular phytocompounds, and in view of this the bioactive properties of plants have been highlighted. Phenolics, the principal plant bioactive compounds, are well known for their antioxidant properties that are responsible for inhibiting the formation and/or preventing the action of free radicals or non-radical species, reducing the risk of diseases related with oxidative stress [7,8]. Therefore, plant extracts are an excellent option to incorporate into semi-solid cutaneous preparations as active ingredients.
Castanea sativa Mill. is a species of the Fagaceae family present in the south of Europe, Asia and Africa. During chestnut processing large amounts of byproducts are generated, mainly shells, inner and outer skins and burs [9,10]. The environmental concerns associated with the increasing demand for ecological products highlights the potentialities of using food byproduct ingredients against skin aging. Different authors discussed the antioxidant properties of chestnut shells extracts, reporting their richness in bioactive compounds, particularly in phenolic acids and tannins (condensed and hydrolysable) [11,12,13,14]. Squillaci et al. [14] observed that chestnut shells extracts, in addition to antioxidant action, also present anti-inflammatory activity, leading to a reduction of the nitric oxide production, as well as an induction of nitric oxide synthase (responsible for catalyzing the production of nitric oxide). Furthermore, the authors reported the skin’s hydration capacity and the collagen protection of the extracts. In another study, Rodrigues et al. [9] confirmed the antioxidant properties of a conventional chestnut extract prepared with water, as well as the presence of some amino acids (in particular, arginine and leucine) and vitamin E.
Topical semisolid products are one of the fastest growing products on the market and their requirements are constantly changing. To ensure the quality, safety and efficacy of a product, a constant challenge must be faced, requiring well-structured projects and a deep understanding. The application of the Quality by Design concept is an excellent strategy which favors the development of in-depth product information as well as the study of the formulation and process variables that influence a product’s quality, thus favoring a deep knowledge of the design process and the possible improvements involved [15,16]. Quality by Design is encouraged by global regulatory agencies and symbolizes an innovative approach and an advance for industries and regulatory authorities. This concept is a systematic approach that allows the establishment of the product’s quality based on proven data, recognizing knowledge and risk management [15]. In addition, this approach indicates a transformation that diverges from conventional research (Quality by Testing, QbT) which is based on the concept that the process defines the product, and the specifications are based on the performance of the process. Since it is an empirical approach, problems related to the manufacturing process are often difficult to identify. Furthermore, in this type of approach there is an increased difficulty in identifying the variability of the whole process [15,17]. Thus, Quality by Design favors the cosmetic industry, and although the regulations in this sector are less stringent, quality and safety are increasingly important for the formulations produced.
A development following this approach can be compromised of the following detailed elements: the definition of the quality target product profile (QTPP) and the critical quality attributes (CQAs), conducting risk assessments to identify critical quality attributes (CMAs) and the critical process parameters (CPPs), the design space (DS) based on the design of experiments (DoEs), the creation of a control strategy and the continuous improvement during the entire product life cycle [15,17].
The design of experiments (DoE) is an important tool in the Quality by Design approach, allowing the use of structured and organized methods to evaluate the impact and understand the possible interactions between the factors that affect the process (inputs) and the response variable (output) [15]. The DoE has often been employed to understand the product and the manufacturing process and to determine the optimal conditions that should be implemented. Currently, many applications of DoE can be found in scientific literature on screening and optimization, such as the application of the screening project that allows the detection of CMA and CPP which affect CQAs and, therefore, the QTPP [18,19,20,21,22].The project optimization has been employed in the definition of design space, product optimization, process and analytical methods [23,24,25,26]. In this work, an optimization procedure involving the response surface methodology was performed. Despite the advantages of Quality by Design approaches in product development, few studies are currently available regarding its applications in the development of cosmetic products. The aim of this work was to develop and optimize a semisolid topical formulation with C. sativa shells extract according to the concept of Quality by Design, through more planned development, considering the vulnerabilities of the entire process through risk analysis tools and DoE and ensuring the successful development of a high-quality formulation. It is expected to develop an antioxidant formulation demonstrating an antiwrinkle effect, which can nearly be tested in vivo on human volunteers.

2. Materials and Methods

2.1. Materials

Chestnut shells were kindly provided by Sortegel (Sortes, Bragança, Portugal). Cetyl alcohol, sodium lauryl sulfate and coconut oil were purchased from Guinama SL (Valencia, Spain). Beeswax, glycerin, methylparaben and propylparaben were provided by Acofarma distribucion AS (Madrid, Spain). Macadamia oil was acquired in Galeno SRL (Carmiganano, Italy). All ingredients were of pharmaceutical and cosmetic grade. Water was purified by ion-exchange using synthetic resins.

2.2. Methods

2.2.1. Extract Preparation

C. sativa shells extract was prepared according to Pinto et al. [27]. Firstly, C. sativa shells were dried and grounded. Then, samples (10 g) were mixed with deionized and degassed water (100 mL) and extracted at 220 °C during 30 min in a 400 mL Parr Reactor (Series 4560 high-pressure mini-reactors, Parr Instrument Company, Moline, IL, USA) equipped with a Parr Reactor Controller (Series 4848, Parr Instrument Company, Moline, IL, USA). A pressure of 40 bar was maintained. During extraction, the sample was agitated with a four-blade impeller at 200 rpm. After extraction, the extract was filtered through Whatman n° 1 paper, centrifuged and incorporated into the formulation.

2.2.2. Preparation of the Cream Formulations

A hydrophilic cream (o/w) was prepared by a single-phase methodology. Initially, the raw materials were weighed (Macadamia oil: 10.00 g; Cetyl alcohol: 9.0 g; Glycerin: 8.00 g; Beeswax: 5.00; Coconut oil: 3.00 g; Sodium lauryl sulfate: 0.93 g; Propylparaben: 0.20 g; Methylparaben: 0.10 g) and manually mixed. Then, the mixture was heated in a thermostatic bath (Nahita, 601/5, Germany) at 70–80 °C until complete fusion of all constituents. Subsequently, the mixture was stirred in a heating plate (IKA, C-MAG HS 7, Germany) for 5 min at 700–1000 rpm, using the propeller agitator, Heidolph (RZR 2041, Germany). After removing the heating plate, cooling was promoted with stirring for 25 min, and at 40 °C the extract (63.77 g; substituting the aqueous phase) was added to the mixture. Finally, the prepared mixture was completely cooled to room temperature, conditioned and stored at 25 ± 1 °C. Formulations were prepared in triplicate.

2.2.3. Quality by Design

Definition of QTPP and CQAs

The quality target product profile (QTPP) is the first step in the development of a product based on the Quality by Design concept, which includes the definition of the product profile development, the dosage, the route of administration, the aspect, the consistency, the release profile and the viscosity, among others. For the QTPP elaboration, the quality of the final product produced was considered.
After the procedure described above, the second step took place, comprising the identification of the critical quality attributes (CQAs) that derived from the description of the QTPP. Physical, chemical, biological, or microbiological properties or characteristics were defined, within an appropriate limit, in order to ensure the predefined product’s quality. These attributes were investigated, controlled, and guaranteed during the development and production process and based on prior knowledge and scientific literature.

Risk Assessment

According to the ICH Q9 guidance [28], risk assessment consists of the “identification of hazards and the analysis and evaluation of risks associated with exposure to those hazards”. This aspect is extremely important at the development stage due to the possibility of identifying the potential risks in the formulation, as well as in the process, that may affect the final product quality. Therefore, risk identification allows the definition of studies and the determination of the critical variables, facilitating the control strategy and expanding the knowledge of the entire process.
To carry out the risk assessment, the Ishikawa diagram was employed. A risk estimation matrix (REM) was elaborated using the main factors observed in the diagram to estimate the level of criticality of the material’s attributes and process parameters that could influence the identified CQAs of the product. Based on the REM data, a Pareto chart was built to demonstrate which material attributes and process parameters are critical to ensure the product’s QTTP.

Design of Experiments

After identifying the process variables that may have a significant impact on the product properties, the experiments were designed by a mathematical model, a response surface methodology (RSM), using the Box–Behnken model. This design allows one to obtain the first- and second-order polynomial response surface models [29,30]. A total of 17 experiences were designed by the computer program Design Expert (version 7, Stat-Ease Inc., Minneapolis, MN, USA). Three independent variables of the formulation were specified, namely the quantity of sodium lauryl sulfate (%) (X1), beeswax (%) (X2) and macadamia oil (%) (X3). These three formulation variables varied at three levels: low (−1), medium (0) and high (+1). Three dependent variables were studied, namely pH (Y1), viscosity (Y2) and adhesiveness (Y3). The design variables mentioned for the development of the product formulation of this work are summarized in Table 1.
A second-order polynomial equation effectively expressed the responses with respect to the selected independent variables. Three-dimensional contour plots of the adjusted polynomial regression equations were provided to visualize the interaction effect of the independent variables on responses. The response surface model was demonstrated by the following equation:
Y = β 0 + β 1 X 1 + β 2 X 2 + + β n X n
where Y is the answer, Xi is the independent variable, ß0 is the constant and ß1 to ßn are the coefficients of the response values.
An appropriate polynomial model was chosen, based on the statistical significance of the model. The quality of fit of the model was assessed using the coefficient of determination (R2) and the analysis of variance (ANOVA).

2.2.4. Characterization of the Formulations

Appearance

The centrifugation test was carried out by placing 2 g of formulation in a 15 mL centrifuge tube which was then centrifuged for two runs of 10 min at 5000 rpm in centrifugation machine. At the end of each cycle, tubes were investigated macroscopically for the presence of any possible phase separation.

Determination of pH

The pH was determined at room temperature, using pH meter equipment (Basic 20 pH—Crison, Spain). The pH meter was calibrated using standard buffer solutions of pH 4.01, 7.01 and 9.00. The analysis was performed by diluting 1.0 g of the formulation in 9.0 g of neutral water. The sample was vigorously shaken, and the pH value was immediately read on the equipment. The determination was carried out in triplicate 48 h after manufacture.

Rheological Tests

To measure viscosity, a rotational rheometer with plate-plate geometry was used (Malvern Kinexus Lab+, Germany) at a temperature of 25 °C and with a 1 mm gap between the plates. A small amount of the sample was inserted onto the bottom plate of the equipment, and torque was applied to the upper plate, promoting a shear stress on the sample. The rSpace Kinexus Lab+ (Version 1.75) software was employed with an initial frequency of 10.00 Hz and a final frequency of 0.100 Hz, using 10 samples per decade, and a 0.25% strain.

Texture Analysis

The spreadability test was carried out on a Texturometer (Stable Micro Systems TA-XT2i, Godalming, UK), using the Exponent software (version 6.1.12.0). Samples were inserted into the female cone (bottom probe) and kept in a stove at 25 ± 1 °C for 30 min before testing. Then, the appropriate probe was chosen and connected to the equipment at an initial position of 25 mm. For measurement, the upper probe (male cone) penetrated the sample in the female cone at a speed of 3 mm/s over a distance of 23 mm and then the probe returned to the starting position, providing the result. Due to the conical shape of the probe, compression and tangential forces were involved.

Size of the Internal Phase

For the evaluation of the droplet size, a Mastersizer 3000 laser diffractometer was used, with a liquid dispersion unit for samples with flexible volume Hydro EV (Malvern, UK). The sample was prepared by diluting a small amount with water in a test tube. Afterwards, the sample was inserted into a graduated beaker containing the chosen dispersing medium (500 mL of water), until a level of obscuration between 5 and 10% was reached. The Mie Model was used to determine the particle size distribution and five readings were taken by the equipment. The obtained result revealed the density in volume (%) in function of the size of the particles, in three parameters: Dv10, Dv50 and Dv 90 (percentile 10, percentile 50 and percentile 90).

2.3. Statistical Analysis

For statistical analysis of the results, the Design Expert (version 7, Stat-Ease Inc., Minneapolis, MN, USA) software was employed. An analysis of variance test (ANOVA) evaluated the statistical significance between the formulations. The differences were considered significant if p < 0.05. All tests were performed in triplicate and the results were expressed as mean ± standard deviation.

3. Results

3.1. Preliminary Study

The objective of the present study was to develop and optimize a semisolid topical formulation with C. sativa shells extract as aqueous phase. To achieve this goal, a systematic approach outlined in Quality by Design principles was adopted, to develop a product in a more reproducible and faster way. The formulation composition as well as the preparation process were selected after extensive bibliographic research. Following this, test batches were prepared to estimate the associated risks, the process parameters and the ideal formulation conditions. These data helped in the elaboration of the risk assessment. The preliminary study was essential to prepare for the next stages of the formulation development and optimization.

3.2. Definition of QTPP, CQAs, CMAs and CPPs

Initially, compliance with regulatory aspects, product quality expectations, consumer desire and scientific knowledge was studied, forming the product’s QTPP. The QTPP determination is summarized in Table 2.
The physico-chemical attributes designed for the product, categorized as critical (CQAs) were identified and flagged in Table 3, along with a theoretical justification regarding the criticality of the attribute. The attributes identified as most critical for the product quality were subsequently controlled. After determining the QTTP and identifying the CQAs, the selection of critical material attributes (CMAs) and critical process parameters (CPPs) was performed.

3.3. Initial Risk Assessement

After grouping the variables that may influence the product’s CQAs and, consequently, may lead to failure in the product’s quality, the Ishikawa diagram (Figure 1) and REM (Figure 2) were performed.
The Ishikawa diagram depicts the relations between the cause and effect of the variables that may influence the CQAs of the formulation and the process. The structure of this diagram has a horizontal line, with the end representing what the product seeks to obtain. Diagonal lines represent the main influencing factors. Finally, the sub-line on the diagonal lines is categorized by the influence of critical process parameters and critical material attributes. The diagram provided a clear picture of the entire preparation process and systematically distinguished the reasons that could generate a product “non-conformity”, in addition it also predicted the working conditions that should be performed to control and prevent the occurrence of errors.
Then, the main factors mentioned in Figure 1 were classified in an REM (Figure 2).
Evaluating this risk matrix, it is possible to visualize information about a certain set of risks, in addition to the possibility of concluding which risks should be prioritized. For the elaboration of the risk matrix, the correlation between the CQAs and the material attributes and process parameters was built, with the risks being assessed and classified individually by considering the probability of occurrence and the impact. A combination of values was used on a scale from one to five, with five representing a very high risk, four a high risk, three a medium risk, two a low risk and one a very low risk. Values were calculated and ordered in a risk classification system with high (red), medium (yellow) and low (green) priority. In general, through the initial risk analysis, it is possible to conclude that the product’s aspects, such as viscosity, droplet size, adhesiveness, and compressibility, highly influence the CQAs.
Based on the REM results, two Pareto charts were generated (Figure 3a,b). The Pareto chart illustrates the severity scores and demonstrates the CQAs and CPPs that had values higher than 100. Based on the results of the risk assessment, viscosity and aspect were classified as highly critical factors for CQAs. In contrast, odor and color are the least critical issues. In addition, according to Figure 3b, homogenization speed and homogenization time are highly critical factors for CQAs. Pareto charts also prioritize the critical attributes that should be more controlled and investigated in this study.

3.4. Design of Experiments

The construction of the experimental Box–Behnken project with three input factors related to product formulation (amount of sodium lauryl sulfate (%) (X1), beeswax (%) (X2) and macadamia oil (%) (X3)) was used to investigate the effects on the output’s responses (pH (Y1), viscosity (Y2) and adhesiveness (Y3)). The experimental planning matrix created by the statistical program is shown in Table 3. Low (−1), medium (0) and high (+1) values for each factor studied were defined based on the risk analysis, previous knowledge and scientific principles, whose variations were considered a risk for the QTTP (Table 2).
A total of 17 formulations were suggested by the Design Expert software and were successfully prepared without any visual signs of separation or coalescence after preparation. The formulations were carefully characterized according to their main quality attributes, and the effects of the variables on the responses were studied in detail and summarized below.
The experimental matrix of random executions for the independent and dependent variables are summarized in Table 3. All responses were fitted to a quadratic model (second-order); the adequacy of this model was validated by an ANOVA analysis, lack of fit test and multiple correlation coefficient tests.
The statistical analysis (Table 4) indicated that models were significant for all responses (p < 0.05). Table 4 presents the results of the analysis of variance, as well as the F-value and the p-value. For each answer, a mathematical equation (Equations (2)–(4)) was obtained, as determined by a multiple regression analysis. A detailed factor to factor analysis is provided below.

3.4.1. Effect of Independent Variables on Dependent Variables

Effects on pH

The results obtained from the pH evaluation are shown in Table 3. The formulation showed pH values between 7.11 (C1) and 7.86 (C13). Although pH was not considered a threat to QTTP, it was included in the experimental design as a qualitative variable to have a better understanding of its influence on the formulation. The semisolid topical formulation’s pH is a quality parameter that may interfere with the physical stability of the product and must be compatible with the pH of the skin [31]. The extract presented a pH of 4.78. The formulation should have a pH in line with this value in order to maximize the skin effects of the extract, particularly the antioxidant and antiradical activities.
The determined effects of factors, the p-value and the F-value correlated of ANOVA on pH (Y1) are shown in Table 4. As it is possible to observe, a p = 0.832 was calculated for the lack of fit, meaning that the model is adequate for the data observed at a 95.0% confidence level. Regarding the R2, the value obtained explains 83% of the pH variability. The adjusted R2, which is better suited to compare models with different numbers of independent variables, is 0.61.
As shown in Table 4, two factors demonstrated a significant antagonistic effect on pH, namely variables X2 (p = 0.0489) and the interaction between factors X1 (p = 0.0295).
Figure 4a shows the graph obtained for the Y1 (pH) response, illustrating the relationship between the pH value, the concentration of two factors (amount of beeswax (X2) and macadamia oil (X3)) and the amount of surfactant (X1) fixed at the medium level (1.0). According to the obtained result, at the lowest X2 level, the pH value decreased when the amount of X3 increased from 6 to 10. On the other hand, at the highest level of X2, the pH value did not change significantly.
The influence of the different factors and their interactions on pH values can be represented by the following equation:
pH (Y1) = 6.917 + 1.889 × X1 − 0.1 × X2 − 0.944 × X22

Effects on Viscosity

The viscosity values (Y2) for the formulations tested are shown in Table 3. The results revealed that viscosity ranged between 700 (C9) and 2832 (C5) Pa.s. The p-value and the correlated F-value of ANOVA on viscosity (Y2) are show in Table 4. Three significant effects for Y2 response were observed, namely X1 (p = 0.019), X2 (p = 0.023) and the interaction X3 (p = 0.031).
The lack of fit (p = 0.365) demonstrated that the model is adequate for the data observed at a significant confidence level of 95.0%. The R2 indicated that the model explains 82% of the viscosity variability. The adjusted R2 showed a value of 0.596.
The response surface graph (Figure 4b) shows the effects of the different factors (two input factors (amounts of beeswax (X2), macadamia oil (X3)) and the amount of surfactant (X1) fixed at 1.35) for viscosity (Y2). It is possible to observe a non-linear interaction. Figure 4b indicates that the viscosity and concentration of oily components had a notable impact on the formulation.
The influence of the different factors and their interactions on viscosity can be represented by the following equation:
Viscosity (Y2) = −6017.55 − 806.2 × X1 + 254.142 × X2 + 1910.56 × X3 − 119.64 × X32

Effects on Adhesiveness

According to the results obtained for the measurements of adhesiveness for the preparations, the values ranged from 22.86 (C7) to 58.82 (C11) N.mm (Table 3). The p-value and F-value of this parameter are shown in Table 4. The ANOVA table demonstrated that the effects X1 (p = 0.0271) and the interaction X1 (p = 0.0101) were significant.
The lack of fit (p = 0.5147) demonstrated that the model is adequate for the data at a confidence level of 95.0%. The R2 indicates that the model explains 86% of the adhesiveness variability. The adjusted R2 is 0.6717.
Figure 4c shows the response surface analysis for adhesiveness in relation to input factors X2 and X3 (with X1 set at 0.8). As it is possible to observe, the concentration of oily components had an impact on product adhesiveness, indicating that with an increase in factors X2 and X3, a greater force is used to remove the product from a surface and a higher retention of the cream on skin occurs.
The follow equation represents the influence of different factors and their interactions on adhesiveness:
Adhesiveness (Y3) = −27.262 + 134.567 × X1 − 59.105 × X12

Optimization and Validation by Box–Behnken Response Surface Methodology

A multiple response optimization process was carried out by combining the experimental factors to optimize the three responses simultaneously. Thus, responses Y1, Y2 and Y3 were transformed into the desirability scale being maximized. For the individual desirability function of each answer, maximum and minimum values were taken as the highest and lowest values of the responses, respectively (Table 3). The desirability value was calculated using the Design Expert software and the optimal conditions are shown in the desirability graph (Figure 4d).
The general desirability of an optimized formulation proposed by the software was 0.594. The optimal values for the factors were X1 = 0.93, X2 = 5.00 and X3 = 10.00. To confirm the adequacy of the model and validate the optimization process, the suggested optimized formulation was prepared. The results are shown in Table 5.

3.5. Characterization of the Optimized Formulation over Time

The formulation containing the ideal conditions established by the DoE was prepared and characterized to assess the accuracy of the model in optimal conditions regarding organoleptic and technological characteristics. The study was carried out over time (30 days) and the sample was stored at a controlled temperature (25 ± 1 °C).

3.5.1. Appearance

Organoleptic analysis demonstrated that the formulation presented a homogeneous aspect, without phase separation, and the consistency was considered adequate, as shown in Figure 5. A light brown and fruity odor like caramel was also observed.

3.5.2. Determination of pH

The pH values ranged between 4.9 and 4.76, without changes over time. These values are compatible with the skin pH (4.0–6.0) [31].

3.5.3. Rheological Tests

The rheological tests were performed on the optimal formulation with the aim of analyzing the behavior when subjected to a certain tension, evaluating the force applied when the product is spread on the skin. Furthermore, the viscosity was screened. The assessment of viscosity over time is a significant parameter for estimating the product’s stability [32]. The values obtained were between 2618 Pa.s and 2205 Pa.s. As shown in Figure 6, a small viscosity reduction was observed until a certain period, which was probably related to internal structure modifications. Nevertheless, this decrease is not significant, as it stabalized after 5 days.
The formulation has a pseudoplastic-thixotropic behavior since a sharp decrease in viscosity (43%) in the third stage was observed, demonstrating that the restoration of the structure will occur more slowly than the initial destruction.
To determine the stress yield, an external strain was applied under the material. A shear stress superior to 82.79 Pa was required to produce a flow.

3.5.4. Texture Analysis

Regarding texture analysis, adhesiveness and compressibility were evaluated. A small reduction in the formulation consistency was observed over time, which may be related to modifications in the internal structure of the material, presenting values for compressibility from 26.51 N.mm to 21.42 N.mm, for adhesiveness between 32.19 N.mm and 24.74 N.mm and for firmness between 9.58 N and 8.38.

3.5.5. Size of the Internal Phase

The formulation presented diameters on a micrometric scale (μm), revealing that values below 80 μm remained stable over time. The values obtained for Dv10 were between 8.91 and 8.55, for Dv were between 40.20 and 31.30 and for Dv 90 were between 79.20 and 73.20.
In fact, the results of the investigations described above all presented satisfactory results, in which it can be observed that there have been no significant changes over time. Therefore, it is possible to confirm the stability of the semisolid topical formulation with Castanea sativa shells extract. In addition, the results demonstrated fulfill the requirements outlined in the initial QTPP of the product.

4. Conclusions

The present study allowed us to demonstrate the applicability and relevance of using the Quality by Design approach during the development of a topical semisolid product, as this approach assists in understanding the formulation and manufacturing process, reduces product variability, and provides the desired quality of the final product. The design of experiments along with risk analysis allowed us to predict the parameters that could have a high impact on the quality of the product and establish safe ranges for its variations. Optimization was used through response surface analysis using the Box–Behnken model with three factors at three levels. A formulation containing 0.93% sodium lauryl sulfate, 5.00% beeswax and 10.00% macadamia oil was considered the most promising. After validation of the optimal formulation, its organoleptic and technological characteristics were evaluated, and it was verified that the formulation is stable and fulfils the requirements outlined in the product’s QTPP. The present study allowed us to demonstrate the practical gain of the Quality by Design approach in the development of cosmetic products, resulting in a shorter development time, lower costs, avoiding the specification products and decreasing the need of human resources. Although the results were satisfactory, other studies related to safety, efficacy and extended stability need to be done to fully understand the formulation process.

Author Contributions

Methodology; software; formal analysis; investigation; writing—original draft preparation, N.O.; methodology; software, M.d.l.L.C.-G.; formal analysis; investigation, A.M.S.; formal analysis; investigation, C.M.; conceptualization; methodology; software; validation; formal analysis; investigation; resources; data curation; writing—review & editing; supervision; project administration; funding acquisition, F.R.; methodology; software; validation; formal analysis; investigation; resources; data curation; writing—review & editing; supervision; funding acquisition, P.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by project PTDC/ASP-AGR/29277/2017—Castanea sativa shells as a new source of active ingredients for functional food and cosmetic applications: a sustainable approach, financially supported by national funds by FCT/MCTES and co-supported by Fundo Europeu de Desenvolvimento Regional (FEDER) throughout COMPETE 2020—Programa Operacional Competitividade e Internacionalização (POCI-01-0145-FEDER-029277).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data available on request due to restrictions e.g., privacy or ethical.

Acknowledgments

The authors are thankful to Sortegel for the samples. Ana Margarida Silva is thankful for her Ph.D. grant (SFRH/BD/144994/2019) financed by POPH-QREN and subsidized by the European Science Foundation and Ministério da Ciência, Tecnologia e Ensino Superior. Francisca Rodrigues (CEECIND/01886/2020) is thankful for her contract financed by FCT/MCTES—CEEC Individual Program Contract. This work was also financed by national funds from FCT - Fundação para a Ciência e a Tecnologia, I.P., in the scope of the project UIDB/50006/2020. This work was also supported by national funds from FCT - Fundação para a Ciência e a Tecnologia, I.P., in the scope of the projects UIDP/04378/2020 and UIDB/04378/2020 of the Research Unit on Applied Molecular Biosciences – UCIBIO and project LA/P/0140/2020 of the Associate Laboratory Institute for Health and Bioeconomy - i4HB.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Russell-Goldman, E.; Murphy, G.F. The pathobiology of skin aging: New insights into an old dilemma. Am. J. Clin. Pathol. 2020, 190, 1356–1369. [Google Scholar] [CrossRef]
  2. Gruber, F.; Kremslehner, C.; Eckhart, L.; Tschachler, E. Cell aging and cellular senescence in skin aging—Recent advances in fibroblast and keratinocyte biology. Exp. Gerontol. 2020, 130, 110780. [Google Scholar] [CrossRef]
  3. Sunder, S. Relevant Topical Skin Care Products for Prevention and Treatment of Aging Skin. Facial Plast. Surg. Clin. 2019, 27, 413–418. [Google Scholar] [CrossRef] [PubMed]
  4. Gu, Y.; Han, J.; Jiang, C.; Zhang, Y. Biomarkers, oxidative stress and autophagy in skin aging. Ageing Res. Rev. 2020, 59, 101036. [Google Scholar] [CrossRef]
  5. Mizushima, N.; Komatsu, M. Autophagy: Renovation of cells and tissues. Cell 2011, 147, 728–741. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Birch-Machin, M.; Bowman, A. Oxidative stress and ageing. Br. J. Dermatol. 2016, 175, 26–29. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Vázquez, G.; Fontenla, E.; Santos, J.; Freire, M.; González-Álvarez, J.; Antorrena, G. Antioxidant activity and phenolic content of chestnut (Castanea sativa) shell and eucalyptus (Eucalyptus globulus) bark extracts. Ind. Crops Prod. 2008, 28, 279–285. [Google Scholar] [CrossRef]
  8. Almeida, I.F.; Pinto, A.; Monteiro, C.; Monteiro, H.; Belo, L.; Fernandes, J.; Bento, A.; Duarte, T.; Garrido, J.; Bahia, M.F. Protective effect of C. sativa leaf extract against UV mediated-DNA damage in a human keratinocyte cell line. J. Photochem. Photobiol. B Biol. 2015, 144, 28–34. [Google Scholar] [CrossRef]
  9. Rodrigues, F.; Santos, J.; Pimentel, F.B.; Braga, N.; Palmeira-de-Oliveira, A.; Oliveira, M.B.P. Promising new applications of Castanea sativa shell: Nutritional composition, antioxidant activity, amino acids and vitamin E profile. Food Funct. 2015, 6, 2854–2860. [Google Scholar] [CrossRef]
  10. Pinto, D.; Braga, N.; Silva, A.M.; Costa, P.; Delerue-Matos, C.; Rodrigues, F. Chestnut. In Valorization of Fruit Processing By-Products; Galanakis, C.M., Ed.; Elsevier: London, UK, 2020; pp. 127–144. [Google Scholar]
  11. De Vasconcelos, M.D.C.B.M.; Bennett, R.N.; Rosa, E.A.; Cardoso, J.V.F. Primary and secondary metabolite composition of kernels from three cultivars of Portuguese chestnut (Castanea sativa Mill.) at different stages of industrial transformation. J. Agric. Food Chem. 2007, 55, 3508–3516. [Google Scholar] [CrossRef]
  12. Vázquez, G.; González-Alvarez, J.; Santos, J.; Freire, M.; Antorrena, G. Evaluation of potential applications for chestnut (Castanea sativa) shell and eucalyptus (Eucalyptus globulus) bark extracts. Ind. Crops Prod. 2009, 29, 364–370. [Google Scholar] [CrossRef]
  13. Gonçalves, B.; Borges, O.; Costa, H.S.; Bennett, R.; Santos, M.; Silva, A.P. Metabolite composition of chestnut (Castanea sativa Mill.) upon cooking: Proximate analysis, fibre, organic acids and phenolics. Food Chem. 2010, 122, 154–160. [Google Scholar] [CrossRef]
  14. Squillaci, G.; Apone, F.; Sena, L.M.; Carola, A.; Tito, A.; Bimonte, M.; De Lucia, A.; Colucci, G.; Cara, F.L.; Morana, A. Chestnut (Castanea sativa Mill.) industrial wastes as a valued bioresource for the production of active ingredients. Process Biochem. 2018, 64, 228–236. [Google Scholar] [CrossRef]
  15. ICH Harmonised Tripartite Guideline. Pharmaceutical Development; Q8 (R2); ICH: Amsterdam, The Netherlands, 2009. [Google Scholar]
  16. Lawrence, X.Y.; Amidon, G.; Khan, M.A.; Hoag, S.W.; Polli, J.; Raju, G.; Woodcock, J. Understanding pharmaceutical quality by design. AAPS J. 2014, 16, 771–783. [Google Scholar]
  17. Lolas, A.G.; Rathore, A.S. Regulatory Challenges in the QbD Paradigm. BioPharm Int. 2012, 25, 44. Available online: https://www.biopharminternational.com/view/regulatory-challenges-qbd-paradigm (accessed on 25 October 2021).
  18. Marto, J.; Gouveia, L.; Gonçalves, L.; Gaspar, D.; Pinto, P.; Carvalho, F.; Oliveira, E.; Ribeiro, H.; Almeida, A. A quality by design (QbD) approach on starch-based nanocapsules: A promising platform for topical drug delivery. Colloids Surf. B. 2016, 143, 177–185. [Google Scholar] [CrossRef] [PubMed]
  19. Su, R.; Yang, L.; Wang, Y.; Yu, S.; Guo, Y.; Deng, J.; Zhao, Q.; Jin, X. Formulation, development, and optimization of a novel octyldodecanol-based nanoemulsion for transdermal delivery of ceramide IIIB. Int. J. Nanomed. 2017, 12, 5203. [Google Scholar] [CrossRef] [Green Version]
  20. Lauterbach, A.; Ekelund, K. Rheological temperature sweeping in a quality by design approach for formulation development and optimization. Int. J. Pharm. 2019, 568, 118533. [Google Scholar] [CrossRef]
  21. Simões, A.; Veiga, F.; Vitorino, C. Developing cream formulations: Renewed interest in an old problem. J. Pharm. Sci. 2019, 108, 3240–3251. [Google Scholar] [CrossRef] [PubMed]
  22. Thakur, K.; Mahajan, A.; Sharma, G.; Singh, B.; Raza, K.; Chhibber, S.; Katare, O.P. Implementation of Quality by Design (QbD) approach in development of silver sulphadiazine loaded egg oil organogel: An improved dermatokinetic profile and therapeutic efficacy in burn wounds. Int. J. Pharm. 2020, 576, 118977. [Google Scholar] [CrossRef]
  23. Mitchell, M. Determining criticality-process parameters and quality attributes part II; design of experiments and data-driver criticality. BioPharm 2014, 1, 9. [Google Scholar]
  24. Lourenço, F.R.; Francisco, F.L.; Ferreira, M.R.S.; de Jesus Andreoli, T.; Löbenberg, R.; Bou-Chacra, N. Design space approach for preservative system optimization of an anti-aging eye fluid emulsion. J. Pharm. Pharm. Sci. 2015, 18, 551–561. [Google Scholar] [CrossRef] [PubMed]
  25. Saviano, A.M.; Francisco, F.L.; Ostronoff, C.S.; Lourenço, F.R. Development, optimization, and validation of a microplate bioassay for relative potency determination of linezolid using a design space concept, and its measurement uncertainty. J. AOAC Int. 2015, 98, 1267–1275. [Google Scholar] [CrossRef] [PubMed]
  26. Politis, S.N.; Colombo, P.; Colombo, G.; Rekkas, D.M. Design of experiments (DoE) in pharmaceutical development. Drug Dev. Ind. Pharm. 2017, 43, 889–901. [Google Scholar] [CrossRef] [PubMed]
  27. Pinto, D.; Vieira, E.F.; Peixoto, A.F.; Freire, C.; Freitas, V.; Costa, P.; Delerue-Matos, C.; Rodrigues, F. Optimizing the extraction of phenolic antioxidants from chestnut shells by subcritical water extraction using response surface methodology. Food Chem. 2021, 334, 127521. [Google Scholar] [CrossRef]
  28. ICH Harmonised Tripartite Guideline. Quality Risk Management; Q9. Current Step 4; ICH: Amsterdam, The Netherlands, 2005; Volume 4, p. 408. [Google Scholar]
  29. Aslan, N.; Cebeci, Y. Application of Box–Behnken design and response surface methodology for modeling of some Turkish coals. Fuel 2007, 86, 90–97. [Google Scholar] [CrossRef]
  30. Candioti, L.V.; De Zan, M.M.; Cámara, M.S.; Goicoechea, H.C. Experimental design and multiple response optimization. Using the desirability function in analytical methods development. Talanta 2014, 124, 123–138. [Google Scholar] [CrossRef] [PubMed]
  31. Mahdi, E.S.; Noor, A.M.; Sakeena, M.H.; Abdullah, G.Z.; Abdulkarim, M.F.; Sattar, M.A. Formulation and in vitro release evaluation of newly synthesized palm kernel oil esters-based nanoemulsion delivery system for 30% ethanolic dried extract derived from local Phyllanthus urinaria for skin antiaging. Int. J. Nanomed. 2011, 6, 2499. [Google Scholar] [CrossRef] [Green Version]
  32. Maia Filho, D.C.; Ramalho, J.B.; Spinelli, L.S.; Lucas, E.F. Aging of water-in-crude oil emulsions: Effect on water content, droplet size distribution, dynamic viscosity and stability. Colloids Surf. A Physicochem. Eng. Asp. 2012, 396, 208–212. [Google Scholar] [CrossRef]
Figure 1. Ishikawa diagram summarizing the cause-and-effect relationships of the variables that affect CQAs in the development of the formulation with C. sativa shells extract.
Figure 1. Ishikawa diagram summarizing the cause-and-effect relationships of the variables that affect CQAs in the development of the formulation with C. sativa shells extract.
Sustainability 14 00129 g001
Figure 2. Initial risk analysis of the individual product formulation and process parameters: Low = low risk parameter (green); Medium = medium risk parameter (yellow); High = high risk parameter (red).
Figure 2. Initial risk analysis of the individual product formulation and process parameters: Low = low risk parameter (green); Medium = medium risk parameter (yellow); High = high risk parameter (red).
Sustainability 14 00129 g002
Figure 3. Pareto chart for (a) CQAs and (b) CPPs screening.
Figure 3. Pareto chart for (a) CQAs and (b) CPPs screening.
Sustainability 14 00129 g003
Figure 4. Response surface graph representation (3D) main effects of input factors X1, X2 and X3 in relation to the response: (a) pH; (b) viscosity; (c) adhesiveness and (d) response surface graph of the combination of factors that achieves desirability.
Figure 4. Response surface graph representation (3D) main effects of input factors X1, X2 and X3 in relation to the response: (a) pH; (b) viscosity; (c) adhesiveness and (d) response surface graph of the combination of factors that achieves desirability.
Sustainability 14 00129 g004
Figure 5. Optimal formulation with C. sativa shells extract.
Figure 5. Optimal formulation with C. sativa shells extract.
Sustainability 14 00129 g005
Figure 6. Result of the viscosity of the formulation optimized over time.
Figure 6. Result of the viscosity of the formulation optimized over time.
Sustainability 14 00129 g006
Table 1. Variables used in the Box–Behnken plan.
Table 1. Variables used in the Box–Behnken plan.
Independent Variables (Factors)Levels
Low (−1)Medium (0)High (+1)
X1: Sodium lauryl sulfate (%)0.511.5
X2: Beeswax (%)23.55
X3: Macadamia oil (%)6810
Table 2. Determination of the QTTP and CQA of the formulation with extract of C. sativa shells.
Table 2. Determination of the QTTP and CQA of the formulation with extract of C. sativa shells.
QTPPTargetCQAsJustification
Product typeCosmetic--
Dosage formCream-Multiphase preparations consisting of a lipophilic phase and an aqueous phase.
Route of administrationTopic-Product administration location.
DosageNA--
Aspect/AppearanceHomogeneous smooth cream with Castanea sativa Mill. Shell extractYesChanges in the aspect/appearance of the product may affect quality, safety and effectiveness.
Changes in aspect/appearance are related to possible changes in the physical-chemical and/or microbiological stability of the product.
The aspect/appearance has an influence on the acceptance and adherence.
OdorCharacteristicYesChanges in the odor of the product may indicate physical, chemical and/or microbiological changes in the product.
Odor has an influence on consumer acceptance and adherence.
ColorLight brownYesThe color of the product indicates the presence of C. sativa shells which have a brown color, in addition to indicating product stability.
Color has an influence on consumer acceptance and adherence.
Viscosity700–2500 Pa.sYesThe determination of viscosity is eventually used to assess the quality of a product.
Changes in the product’s viscosity demonstrate a possible change in the physical structure of the product, which may interfere with the time the product will remain in the application site.
Viscosity is a critical factor in assessing product stability.
IdentificationEur. Ph/USP-It is a critical factor for product safety and effectiveness.
pH4.0–8.0YesThe pH has an influence on the physical-chemical stability of the product.
The pH of the product must be compatible with the pH of the skin in order to avoid irritation and damage to the integrity of the skin after application of the product.
CrystallizationEur. Ph/USPYesCrystallization has an impact on the uniformity and stability of the formulation.
Particle size100 nm–100 μmYesParticle size has an impact on effectiveness and stability.
Compressibility20.0–60.0 N.mmYesThe compressibility is related to product consistency. Changes in the consistency of the product may modify the characteristics of the product, impacting its spreadability and adherence.
Compressibility has an influence on acceptance and adherence.
Adhesiveness20.0–60.0 N.mmYesAdhesiveness is related to cream retention on the skin. It has an impact on the preservation of cream in situ.
Preservative testingEur. Ph/USPYesThe testing of preservatives present in the product will ensure the safety and stability of the formulation.
Microbial limitsEur. Ph/USPYesChange in microbiological limits may impact product safety.
StabilityICH Q1A/Eur. Ph/USPYesChanges in product stability may impact the quality of the product during the storage period.
It is a quality requirement.
Container closure systemAppropriate for the dosage form-The packing material must be appropriate to contain the physical form of the product.
Material integrityNo failure-The integrity of the material must be maintained in order to guarantee the quality, effectiveness and safety.
NA, note applicable; Eur. Ph., European Pharmacopeia; USP, United states Pharmacopeia.
Table 3. Matrix and result of planning the experiences and the responses observed in the Box–Behnken.
Table 3. Matrix and result of planning the experiences and the responses observed in the Box–Behnken.
Independent VariablesDependent Variables
RunX1: Sodium Lauryl SulfateX2: BeeswaxX3:Macadamia OilY1: pHY2: ViscosityY3: Adhesiveness
(%)(%)(%) (Pa s)(N.mm)
C11.5587.11 ± 0.091520.6 ± 90.4540.39 ± 5.91
C21.53.5107.30 ± 0.13931.3 ± 187.3130.15 ± 1.73
C313.587.43 ± 0.131820.8 ± 276.2135.83 ± 6.58
C413.587.33 ± 0.062086.2 ± 229.1241.30 ± 3.62
C50.5587.14 ± 0.032831.9 ± 1951.9826.09 ± 14.41
C61.53.567.43 ± 0.091029.0 ± 813.4949.31 ± 8.54
C70.53.567.32 ± 0.041415.9 ± 1074.7422.86 ± 4.71
C80.53.5107.22 ± 0.031353.5 ± 1171.7028.85 ± 9.91
C91.5287.33 ± 0.01700.3 ± 585.0246.56 ± 9.63
C1013.587.30 ± 0.061517.1 ± 1341.2444.62 ± 0.33
C1115107.50 ± 0.011667.5 ± 1465.2158.82 ± 18.07
C121567.43 ± 0.031428.0 ± 1205.4457.80 ± 3.67
C131267.86 ± 0.031015.6 ± 896.5347.35 ± 1.52
C1413.587.56 ± 0.071670.7 ± 1344.0852.13 ± 4.33
C150.5287.50 ± 0.031804.7 ± 1433.4523.18 ± 3.38
C1613.587.66 ± 0.051288.0 ± 1117.7350.98 ± 3.38
C1712107.60 ± 0.01877.7 ± 709.4144.97 ± 7.80
Table 4. Summary of the model and analysis of variance (ANOVA) of the dependent variables.
Table 4. Summary of the model and analysis of variance (ANOVA) of the dependent variables.
Sum of SquaresMean of SquaresF-Valuep-Value
Y1Y2Y3Y1Y2Y3Y1Y2Y3Y1Y2Y3
X1: Sodium lauryl sulfate0.01.2999 × 106535.1360.01.2999 × 106535.1360.014.2411.601.00000.01950.0271
X2:Beeswax0.1801.1626 × 10655.33520.181.1626 × 10655.33527.8312.741.200.04890.02340.3349
X3: Macadamia oil0.02427.78126.39010.02427.78126.39010.870.000.570.40390.94870.4915
X1.X10.252737765.096970.210.252737765.096970.2110.990.0121.030.02950.93140.0101
X1.X20.0110,701.920.61160.0110,701.920.61160.430.120.450.54570.74930.5404
X1.X30.0311.522158.1310.0311.522158.1310.0000.003.431.00000.95620.1378
X2.X20.012736811,078.676.50960.012736811,078.676.50960.550.121.660.49810.74510.2672
X2.X30.0435,607.72.890.0435,607.72.891.740.390.060.25770.56610.8147
X3.X30.0464211972,755.037.90110.0464211972,755.037.90112.0210.660.820.22840.03090.416
Lack of adjustment0.02383,258.0124.7040.00666667127,753.041.56810.291.400.900.83160.36530.5147
Pure error0.092365,051.0184.5170.02391,262.746.1292
Total0.664.23937 × 1062152.85
R2 (Y1) = 0.8303; R2 aj. (Y1) = 0.6121; R2 (Y2) = 0.8234; R2 aj. (Y2) = 0.5965; R2 (Y3) = 0.8563; R2 aj. (Y3) = 0.6716.
Table 5. Comparison of experimental and intended values prepared under optimal conditions.
Table 5. Comparison of experimental and intended values prepared under optimal conditions.
Y1Y2Y3
Experimental value7.43 ± 0.092214.44 ± 211.7445.19 ± 2.42
Intended value7.491779.0853.66
p-value 0.610.270.09
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Oliveira, N.; Cádiz-Gurrea, M.d.l.L.; Silva, A.M.; Macedo, C.; Rodrigues, F.; Costa, P. Development and Optimization of a Topical Formulation with Castanea sativa Shells Extract Based on the Concept “Quality by Design”. Sustainability 2022, 14, 129. https://doi.org/10.3390/su14010129

AMA Style

Oliveira N, Cádiz-Gurrea MdlL, Silva AM, Macedo C, Rodrigues F, Costa P. Development and Optimization of a Topical Formulation with Castanea sativa Shells Extract Based on the Concept “Quality by Design”. Sustainability. 2022; 14(1):129. https://doi.org/10.3390/su14010129

Chicago/Turabian Style

Oliveira, Nágilla, Maria de la Luz Cádiz-Gurrea, Ana Margarida Silva, Catarina Macedo, Francisca Rodrigues, and Paulo Costa. 2022. "Development and Optimization of a Topical Formulation with Castanea sativa Shells Extract Based on the Concept “Quality by Design”" Sustainability 14, no. 1: 129. https://doi.org/10.3390/su14010129

APA Style

Oliveira, N., Cádiz-Gurrea, M. d. l. L., Silva, A. M., Macedo, C., Rodrigues, F., & Costa, P. (2022). Development and Optimization of a Topical Formulation with Castanea sativa Shells Extract Based on the Concept “Quality by Design”. Sustainability, 14(1), 129. https://doi.org/10.3390/su14010129

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop