Fire Behavior, Fuel Consumption, and Turbulence and Energy Exchange during Prescribed Fires in Pitch Pine Forests
Round 1
Reviewer 1 Report
Manuscript Review
Relationships between fire behavior, fuel consumption, and turbulence and energy exchange during prescribed fires in pine-dominated forests
This study evaluated relationships between fuel consumption, turbulence and energy fluxes in a series of prescribed fires in the New Jersey pinelands. Measurements contribute to a growing body of literature on prescribed burns in pine forests of the eastern United States. As currently written, this manuscript is formatted as if it were to be a US Forest Service General Technical Report or Research Paper with a central objective of presenting measurements taken for this series of prescribed burns. However, the discussion provides better context for the study including the implications of low-intensity prescribed burns on smoke impacts to local communities and transportation corridors and informing ignition patterns for prescribed burn managers to achieve fuel reduction/ecosystem objectives while minimizing smoke impacts.
I recommend a rewrite of the Introduction to provide context for the study followed by objectives and key research questions. This format will then help to organize the results and discussion. The discussion should be reorganized for clarity and to highlight and discuss the findings that are of interest to fire and smoke management. See specific comments within the discussion section.
Specific comments:
Title – I’d recommend a refinement of the title to be less wordy and more specific to the New Jersey pinelands.
Abstract –
The abstract needs a stronger topic sentence with context from the revised introduction and guiding research questions. For example, how is fuel consumption and fire intensity associated with turbulence, and what are the implications of this research for smoke management and potential impacts to communities? How can research findings from this study apply to other locations across the world?
Line 18: Need a guiding question about why you are studying relationships between ignition patterns, fire behavior and turbulence and why these have implications for smoke management
Line 19: Eastern USA should be lowercase e
Line 24: “Followed the order fine fuels > understory vegetation…” This sentence needs to be rewritten for clarity – replace the > with words.
Line 41: delete comma after canopy fuels
Introduction –
Line 49: “because they are the most cost-effective method” isn’t always the case and isn’t really pertinent to the study.
Line 57: Eastern USA needs to be lower case
Line 65-66: Reword this sentence --- these variables are especially important for what reasons?
Line 75: reword to fuel loading and consumption
Lines 74-91: Replace this topic sentence with a central objective and research questions that guided this study. Replace “stands” with “sites”
Line 102 – Need strong topic sentence for this paragraph and a clear description of the study design. I’d recommend a table that summarizes the 6 stands and the prescribed burns.
Line 117: need a citation or description of the actual prescription window.
Figure 1: Great photos – nice addition to the paper. Remove extraneous “a” from “a low-intensity backing fire” etc.
Line 152: L horizon needs to be defined at first use.
Line 159: How were these plot locations selected?
Line 166: hyphen needed when units are adjectives before a noun: 110-m to 15-km radius.
Line 203: what are fuel components? Need to be defined with examples at first use.
Line 219-20: using SigmaPlot is not needed.
Results –
Line 238: Use pre-burn instead of initial.
Figure 2: Fine litter needs a different pattern to distinguish from understory in B&W prints of manuscript. Reword caption: “Pre- and post-bur fuel loading for eight prescribed fires, partitioned into understory, 1-hr wood, 10-hr wood and fine litter. Fuel loadings are presented for a) low-intensity burns and b) high-intensity burns. Table B1 presents source fuel loading values.”
Lines 256-258: Awkward and confusing wording. Jettison the > symbols and rewrite in plain language.
Figure 3: Fitted lines need to be defined either in the figure legend or caption.
Table 2: Replace “Statistics for the relationship between…” with Linear models of preburn loading…
Figure 4: I grappled with this figure. Normally, combined figures are helpful, but I think that the latent and sensible heat are confusing as depicted here and should be presented in a table or additional figure. Also, the caption needs to explain why there are no error bars for the low-intensity/low fuels category.
Line 323: Suggested reword: When integrated over each fire event…
Figure 5: With high scatter, low coefficients of determinations, and trends in the residuals, I’m not sure these figures are all that informative. Could they be included in supplementary material instead?
Table 4 should be moved to supplementary material.
Line 364: Save the surprisingly for the discussion. Start with a strong topic sentence about the overall comparisons within this paragraph.
Discussion –
Line 378: With a revised introduction and a focus on clear objectives and research questions, I’m hoping that this paper will turn into less of an exploration and more of an evaluation. The topic sentence here should lead with that context. Details about the prescribed burns should only be included as context for the results. I’d just retain the sentence about how the 8 Rx burns encompass much of the range of the NJ pinelands program.
Line 389. Delete “consumption of all fuel types.” Also – the discussion shouldn’t be a recap of the results but again a statement of the guiding questions and findings in the context of fire/smoke management and other studies.
Line 405: Start here with the discussion detailing the importance of better understanding drivers of heat flux and smoke production in low intensity fires in the eastern US.
Line 408-422: Bring this toward the front of the discussion.
Lines 423-427: Reword for clarity. Modifiers make this confusing to read.
Line 433: These details need to be first described in the methods section and then referred to here as another source of uncertainty.
Line 436: This is very long paragraph. I’d suggest breaking at “Despite these potential measurement errors…”
Lines 439-446: This section would benefit from having a companion table that lists these comparison results.
Line 466: flaming front?
Lines 470-475: I realize that relationships were weak for many of the variables that you evaluated, but where you have results that have on differences in heat flux between high-intensity crown fires and low-intensity fires should be emphasized. They are currently buried in the sources of uncertainty paragraph. I’d recommend a reorganization of the discussion with 1) results that are pertinent to fire and smoke management, 2) sources of uncertainty, and 3) recommendations for future studies and integration with CFD and smoke emission and dispersion models. You have a lot great information on methodology and instruments that can be harnessed to guide subsequent studies.
Lines 540-563: Nice conclusions section. I’d recommending using some of this distillation to construct the section #1 of the discussion.
Author Response
Please find attached a revised version of our manuscript, now entitled “Fire Behavior, Fuel Consumption, and Turbulence and Energy Exchange during Prescribed Fires in Pitch Pine Forests”. We have addressed all the comments provided by the two reviewers, and this has improved the focus and clarity of our manuscript considerably. We first describe the major comments and document how we revised the manuscript. We then detail how we addressed specific comments from each reviewer and have highlighted our responses in italic font.
Reviewer #1 requested that we reorganize the Introduction and Discussion sections to focus on the overall theme of fire behavior, turbulence and heat fluxes, and factors affecting smoke dispersion during prescribed fires. For the Introduction, Reviewer #1 recommended that we provide a more complete context for our study, followed by clearly stated objectives and key research questions.
We agree with this comment, because we had our main research questions “buried” in the objectives at the end of the Introduction section, and this made it more difficult for readers to understand what our major objective and research questions are. We have rewritten the Introduction to focus on the importance of fire behavior and fire intensity during prescribed burns on smoke impacts to local communities and transportation corridors. We have removed much of the text summarizing our approach, which was essentially a summary of our objectives and methods employed, and replaced this with a clear statement of our overall objectives followed by two specific research questions.
For the Discussion section, Reviewer #1 suggested that we focus the beginning of the discussion on the importance of better understanding drivers of turbulence, heat flux and smoke production during the prescribed fires that were measured. He or she also recommended that the discussion shouldn’t be a restating of the overall results but rather a statement of the guiding questions and findings in the context of fire/smoke management and other studies.
We have reorganized the first part of the Discussion section for clarity and to highlight and discuss the findings that are of interest to fire and smoke management, which ties in more effectively with the revised Introduction section. These changes included removing the summary of non-essential results that did not pertain directly to fire behavior and smoke management that had been included in the first paragraph of the Discussion in the previous version of the manuscript.
We then address the specific comments point by point, again using italics to highlight our responses:
Title – I’d recommend a refinement of the title to be less wordy and more specific to the New Jersey pinelands.
We have revised the title by removing “Relationships between…” and to include that our study was conducted in pitch pine-dominated forest.
Abstract
The abstract needs a stronger topic sentence with context from the revised introduction and guiding research questions. For example, how is fuel consumption and fire intensity associated with turbulence, and what are the implications of this research for smoke management and potential impacts to communities?
Line 18: Need a guiding question about why you are studying relationships between ignition patterns, fire behavior and turbulence and why these have implications for smoke management
These two are related comments. To address these, we have rewritten the first two sentences of the Abstract so that our research question is presented and linked the importance of mitigating the impacts of smoke and risk of escapes.
Line 19: Eastern USA should be lowercase e
We have changed “Eastern” to “eastern” throughout the manuscript.
Line 24: “Followed the order fine fuels > understory vegetation…” This sentence needs to be rewritten for clarity – replace the > with words.
We have rewritten this sentence and omitted the “>” symbols throughout.
Line 41: delete comma after canopy fuels
We have made this change to the Abstract text
Introduction
Line 49: “because they are the most cost-effective method” isn’t always the case and isn’t really pertinent to the study.
We have revised this slightly and have added “typically” because in some cases mechanical thinning can be less expensive. Mechanical thinning does involve lower smoke emissions, however, in the NJ pinelands and many other pine-dominated forests in the eastern US, it is the most cost-effective method to reduce hazardous fuels.
Line 57: Eastern USA needs to be lower case
We have changed “Eastern” to “eastern” throughout the manuscript.
Line 65-66: Reword this sentence --- these variables are especially important for what reasons?
We have added an additional phrase at the end of this sentence as a rationale, as these are the most important meteorological variables controlling fire behavior and fire line control.
Line 75: reword to fuel loading and consumption
This phrase is now reworded in the revision.
Lines 74-91: Replace this topic sentence with a central objective and research questions that guided this study. Replace “stands” with “sites”
Thank you, this is a very helpful comment. This is one of the major suggestions, requesting a reorienting of the Introduction from a set of objectives to an explicit research objective and a specific set of research questions. In the original version of our manuscript, we had partially buried our research question in the list of objectives, and this is not ideal.
We have rewritten much of the Introduction to define our primary objective and research questions, and to link them to key problems, primarily smoke management, and risks involved in conducting prescribed fires in populated areas of the eastern US.
Line 102 – Need strong topic sentence for this paragraph and a clear description of the study design. I’d recommend a table that summarizes the 6 stands and the prescribed burns.
We have inserted a stronger topic sentence at the beginning of this paragraph and revised the following sentences to explain our experimental design. We have also inserted an addition table (now Table 1) to clarify the experimental design we employed. Reviewer #2 made a similar comment about the clarity of our presentation of the experimental design.
Line 117: need a citation or description of the actual prescription window.
We have added citation from the New Jersey Forest Fire Service.
Figure 1: Great photos – nice addition to the paper. Remove extraneous “a” from “a low-intensity backing fire” etc.
Thank you. We have removed the “a” from each panel description in the legend for Figure 1.
Line 152: L horizon needs to be defined at first use.
We have now defined the L horizon as the litter layer
Line 159: How were these plot locations selected?
We located 1-m2 circular plots at random directions and distances from fixed points, typically at the edges or perimeters of tree census plots. Thus, we have inserted the term “randomly” in this sentence to clarify how 1-m2 plots were selected.
Line 166: hyphen needed when units are adjectives before a noun: 110-m to 15-km radius.
We have inserted hyphens in the appropriate locations in the text.
Line 203: what are fuel components? Need to be defined with examples at first use.
We have changed the term “fuel components” to “fuel types”, denoting the fine litter, 1 and 10-hour woody fuels, and understory vegetation.
Line 219-20: using SigmaPlot is not needed.
We have deleted “using SigmaPlot” from the end of this sentence.
Results
Line 238: Use pre-burn instead of initial.
We have adopted the term “pre-burn” throughout the manuscript, unless it is used at the beginning of a sentence.
Figure 2: Fine litter needs a different pattern to distinguish from understory in B&W prints of manuscript.
We have modified the color scheme in Figure 2 and Figure 3 so that bars for fine litter and understory are different. Reviewer #2 made a similar comment.
Reword caption: “Pre- and post-bur fuel loading for eight prescribed fires, partitioned into understory, 1-hr wood, 10-hr wood and fine litter. Fuel loadings are presented for a) low-intensity burns and b) high-intensity burns. Table B1 presents source fuel loading values.”
Thank you, we have reworded this caption as per this suggestion.
Lines 256-258: Awkward and confusing wording. Jettison the > symbols and rewrite in plain language.
We have rewritten this sentence to exclude symbols for the ranking of amounts of each fuel type consumed. We revised a similar sentence in the Abstract.
Figure 3: Fitted lines need to be defined either in the figure legend or caption.
We have indicated that the lines are plots of the linear relationships between pre-burn loading and consumption for each fuel type, presented in Table 3.
Table 2: Replace “Statistics for the relationship between…” with Linear models of pre-burn loading…
We changed the description of this Table (now Table 3) to the suggested wording. For consistency, we also changed the description of Tables 5 and 7 which also present linear model equations and statistics.
Figure 4: I grappled with this figure. Normally, combined figures are helpful, but I think that the latent and sensible heat are confusing as depicted here and should be presented in a table or additional figure. Also, the caption needs to explain why there are no error bars for the low-intensity/low fuels category.
We agree that our attempt to present the comparison of energy fluxes calculated from fuel consumption and measured using eddy covariance were not as clear as it could have been. To address this issue, we have separated this figure into two parts in the revised manuscript; Figure 4a. energy balance calculated from fuel consumption estimates, and Figure 4b. sensible and latent heat fluxes estimated from eddy covariance measurements. We have also clarified the lack of error bars (because there were only two low intensity fires so that a SE could not be calculated).
Line 323: Suggested reword: When integrated over each fire event…
We have reworded this sentence as suggested.
Figure 5: With high scatter, low coefficients of determinations, and trends in the residuals, I’m not sure these figures are all that informative. Could they be included in supplementary material instead?
This is a good suggestion. The table for the linear models in this figure is already in Appendix B. We have moved Figure 5 to Appendix B, and it is now Figure B1.
Table 4 should be moved to supplementary material.
Reviewer #1 suggested moving this table to supplementary material. However, we feel that this is important information, and explains the linear models in Figure 3, which does appear in the main body of the manuscript.
Line 364: Save the surprisingly for the discussion. Start with a strong topic sentence about the overall comparisons within this paragraph.
This is a helpful comment. We have removed “surprisingly” from the first sentence in this paragraph, and have written the first sentence of this paragraph to highlight the overall comparisons between fuel consumption and turbulence and heat flux statistics.
Discussion
Line 378: With a revised introduction and a focus on clear objectives and research questions, I’m hoping that this paper will turn into less of an exploration and more of an evaluation. The topic sentence here should lead with that context. Details about the prescribed burns should only be included as context for the results. I’d just retain the sentence about how the 8 Rx burns encompass much of the range of the NJ pinelands program.
Line 389. Delete “consumption of all fuel types.” Also – the discussion shouldn’t be a recap of the results but again a statement of the guiding questions and findings in the context of fire/smoke management and other studies.
Line 405: Start here with the discussion detailing the importance of better understanding drivers of heat flux and smoke production in low intensity fires in the eastern US.
Line 408-422: Bring this toward the front of the discussion.
These are helpful comments and incorporating them has led to a more focused Discussion section. We have generally followed these suggestions when we rewrote the first two paragraphs of the Discussion section. We have omitted much of the recap information, and now focus on key points that were “buried” in the first two paragraphs.
Lines 423-427: Reword for clarity. Modifiers make this confusing to read.
This sentence is now rewritten in the revised Discussion section.
Line 433: These details need to be first described in the methods section and then referred to here as another source of uncertainty.
We describe how we used estimated fuel consumption and fuel moisture measurements to calculate total heat of combustion and the partitioning of energy fluxes in Appendix A, and we thought it unnecessary to repeat this information in the Methods section. We believe that our description of sampling protocols for pre-burn fuel mass and fuel moisture contents are sufficient for readers to understand. Further, we discuss how spatial variation in fuel types contributes to uncertainty in their mass estimates lower in this paragraph.
Line 436: This is very long paragraph. I’d suggest breaking at “Despite these potential measurement errors…”
We now start a new paragraph at this sentence in the revised version of the manuscript.
Lines 439-446: This section would benefit from having a companion table that lists these comparison results.
We have added a table comparing pre- and post-burn fuel loading and percent consumption between this study and previously-sampled values for the Pinelands in Clark et al. This is now Table C1 in the Appendix.
Lines 470-475: I realize that relationships were weak for many of the variables that you evaluated, but where you have results that have on differences in heat flux between high-intensity crown fires and low-intensity fires should be emphasized. They are currently buried in the sources of uncertainty paragraph.
We have now highlighted the differences between low- and high-intensity fires that we measured in the first paragraph of the Discussion section. However, one of the differences that Reviewer #1 mentioned is appropriate to treat in the paragraph on error analyses, because we discuss how a number of factors, including potential mismatches between fuel consumption estimates and the footprint sampled by the sonic anemometers during fire front passage.
I’d recommend a reorganization of the discussion with
1) results that are pertinent to fire and smoke management,
2) sources of uncertainty
3) recommendations for future studies and integration with CFD and smoke emission and dispersion models. You have a lot great information on methodology and instruments that can be harnessed to guide subsequent studies.
Thank you for this helpful set of comments. We have generally followed this suggested outline in the revised discussion section. A number of ongoing studies and planned studies using similar field sampling protocols are focused on integrating specific models employing CFD (computational fluid dynamics) frameworks on these experiments, including WFDS, QUIC-Fire and ARPS Canopy smoke dispersion model.
Lines 540-563: Nice conclusions section. I’d recommending using some of this distillation to construct the section #1 of the discussion.
Thank you. We have retained the Conclusion section.
Reviewer 2 Report
A review of “Relationships between Fire Behavior, Fuel Consumption, and Turbulence and Energy Exchange 3during Prescribed Fires in Pine-dominated Forests”.
The manuscript reports more depth on the relationships between prescribed burning, fire behavior, and fluxes in combustion and energy than typical of field studies. These linkages are intriguing and the results interesting. The manuscript is well structured, comprehensive, and appropriate for publication. I do think the manuscript is dense and a bit long for the reader to get through, but any changes would come at the expensive of being comprehensive and reporting on the linkages. Therefore, I only have a few minor comments for consideration before the manuscript is ready for publication.
Comments
Line 24: I had a hard time with this sentence here and elsewhere. The sentence is a bit odd if you will. I am not suggesting it is grammatically incorrect, rather just odd to read. Are you referring to the temporal order? The percent of total fuels combusted? The absolute amount combusted?
Line 54-57: This sentence reads like prescribed fire is being used as a smoke management tool. I am not sure that is the intent because I am not certain how you are using the term “wildland fire”. Does the term “wildland fire” include prescribed fire and wildfire? Is it synonymous with wildfire? I generally agree prescribed fire can be a smoke management tool when compared to wildfires, but it is not clear to me how you are using it here.
Line 65: Drawing the relation to ignition pattern here is redundant with your opening sentence for this paragraph and is therefore unnecessary to include.
Line 82: Begins a long sentence that could be broken up.
Line 121 – 126: I became confused between the number of Rxburns and the number of stands. I had to think it through and write it out to understand it was eight fires in six stands. This of course is not the first time you are describing the number of stands and burns. Can you put a sentence in here that simply states there are six stands and eight burns, followed by the individual descriptions?
Line 146: Were the harvested fuels replaced, or did you sample different plots pre and post-fire?
Line 151: Fine litter is not defined as explicitly as the other fuel layers. Please do so.
Line 209: Does the non-combusted material play a role in heat fluxes? That is, are large stems still a sink here?
Line 530: remove the word “by”
Line 539: Is this section required? The manuscript is long, and dense, so cutting this section might be useful.
Thanks for including all the appendices!
Figures
Some readers still print papers in black and white. The bar charts are not conducive to interpretation when printed (e.g. Figure 2, understory and fine litter are not distinguishable when printed).
Figure 5: Some of these relationships look non-linear, particularly as one approaches zero on the x-axis. Did you try to test for that relationship?
Author Response
Please find attached a revised version of our manuscript, now entitled “Fire Behavior, Fuel Consumption, and Turbulence and Energy Exchange during Prescribed Fires in Pitch Pine Forests”. We have addressed all the comments provided by the two reviewers, and this has improved the focus and clarity of our manuscript considerably. We first describe the major comments and document how we revised the manuscript. We then detail how we addressed specific comments from each reviewer and have highlighted our responses in italic font.
The manuscript reports more depth on the relationships between prescribed burning, fire behavior, and fluxes in combustion and energy than typical of field studies. These linkages are intriguing and the results interesting. The manuscript is well structured, comprehensive, and appropriate for publication. I do think the manuscript is dense and a bit long for the reader to get through, but any changes would come at the expensive of being comprehensive and reporting on the linkages. Therefore, I only have a few minor comments for consideration before the manuscript is ready for publication.
Thank you for the positive feedback on our manuscript.
Comments
Line 24: I had a hard time with this sentence here and elsewhere. The sentence is a bit odd if you will. I am not suggesting it is grammatically incorrect, rather just odd to read. Are you referring to the temporal order? The percent of total fuels combusted? The absolute amount combusted?
We agree that the use of symbols in this sentence could result in unintended interpretations. We have revised this sentence and a similar sentence in the Results section that used “greater than” symbols.
Line 54-57: This sentence reads like prescribed fire is being used as a smoke management tool. I am not sure that is the intent because I am not certain how you are using the term “wildland fire”. Does the term “wildland fire” include prescribed fire and wildfire? Is it synonymous with wildfire? I generally agree prescribed fire can be a smoke management tool when compared to wildfires, but it is not clear to me how you are using it here.
We intended “wildland fire” here to include both wildfires and prescribed fires. However, we agree with Reviewer #2 that this is ambiguous. We have changed the term “wildland fire” in this sentence to “prescribed fire” to clarify. In addition, we do state the intended goal of prescribed fires above in the Introduction.
Line 65: Drawing the relation to ignition pattern here is redundant with your opening sentence for this paragraph and is therefore unnecessary to include.
We have rewritten this sentence in the revised Introduction section
Line 82: Begins a long sentence that could be broken up.
We have rewritten much of the Introduction section, and this sentence has been changed
Line 121 – 126: I became confused between the number of Rxburns and the number of stands. I had to think it through and write it out to understand it was eight fires in six stands. This of course is not the first time you are describing the number of stands and burns.
Can you put a sentence in here that simply states there are six stands and eight burns, followed by the individual descriptions?
Both reviewers suggested that we could improve the presentation of our experimental design here. We have split Table 1 into two tables that more clearly describes the experimental design with number of burn area and control towers. In addition, we added a revised topic sentence and a clearer description of the number of forest sites and prescribed fires in the second paragraph of the Site Description.
Line 146: Were the harvested fuels replaced, or did you sample different plots pre and post-fire?
Pre- and post-burn fuels were destructively harvested from independent 1 m2 plots, and we did not attempt to replace fuels and resample the same location. Rather, we selected a second set of randomly located 1 m2 plots following each burn, and sampled these in an identical manner as the pre-burn plots. We then calculated consumption by subtracting average post-burn loading from average pre-burn loading for each fuel type.
Line 151: Fine litter is not defined as explicitly as the other fuel layers. Please do so.
We have added the following phrase to the description of fine litter in the Methods section: “Fine litter, consisting of primarily needles, leaves, and miscellaneous material, and 1–hour and 10–hour wood in the litter layer (L horizon)…”
Line 209: Does the non-combusted material play a role in heat fluxes? That is, are large stems still a sink here?
This is a good question. Unfortunately, we did not measure stem temperatures during these fires, and can only approximate overall storage terms in the canopy air space using the thermocouples. This is a good suggestion to guide future research, because heat flux to stems is a major cause of tree and sapling mortality during prescribed fires.
Line 539: Is this section required? The manuscript is long, and dense, so cutting this section might be useful.
We and Reviewer #1 felt that this section was important because it describes how our research contributes to ongoing efforts to synthesize and simulate fire behavior and smoke emissions and dispersion during prescribed fires. These are ongoing efforts, with a number of multi-scale projects currently being conducted and planned for the future. Therefore, we have retained (and updated) this section.
Thanks for including all the appendices!
Thank you for this comment. We considered including some of this as supplemental information, but it is more easily accessed by readers as appendices.
Figures
Some readers still print papers in black and white. The bar charts are not conducive to interpretation when printed (e.g. Figure 2, understory and fine litter are not distinguishable when printed).
We have altered the color scheme in Figures 2 and 3 so that they can be distinguished. Reviewer #1 made a similar comment about Figure 2.
Figure 5: Some of these relationships look non-linear, particularly as one approaches zero on the x-axis. Did you try to test for that relationship?
We realize that a better fit occurs when using non-linear functions when all data are included in some of these relationships. However, we were most interested in the relationship between air temperature and vertical windspeed in buoyant plumes when temperatures were greater, and in this case > 5 °C greater than ambient temperature. At these enhanced temperatures, functions were generally best modelled as a linear function.
Round 2
Reviewer 1 Report
The manuscript is much improved. I'd recommend it for publication. Please note that I did not carefully check references.