Towards a System Dynamics Framework for Human–Machine Learning Decisions: A Case Study of New York Citi Bike
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe paper presents a comprehensive approach to integrating machine learning with system dynamics for decision-making in the context of bike-sharing systems. The case study provides a practical application and demonstrates the potential of the proposed framework to address complex inventory balancing challenges. The authors have clearly articulated the need for such a framework and have provided a detailed methodology for its development and testing.
1. How does the proposed framework differ significantly from existing models that combine machine learning with system dynamics?
2. Could the authors elaborate on the selection criteria for the case study and why Citi Bike was chosen?
3. What are the limitations of the current model in terms of scalability and applicability to other contexts beyond bike-sharing systems?
4. How does the model handle unpredictable events or sudden changes in user behavior?
5. The Bike-Share Problem (Section 3)
6. Can the authors provide more details on the data collection process and how it might affect the accuracy of their model?
7. In Section 3.2, Figure 2 shows evidence of poor incentivisation strategy. Could the authors explain the reasoning behind the conditional coloring formula used?
8. How confident can we be about the causal relationships presented in Figure 4, and what is the level of uncertainty associated with these relationships?
9. The problem statement and literature review are expertly crafted and presented in the Introduction section. Resilience is one of the most interesting topics, and the study will help to raise the bar for the resilience of the paper. The following article ought to be included in the Reference list: doi.org/10.1016/j.tust.2024.105640
10. What are the potential biases in the data that could affect the causal mapping?
11. In Section 3.5.1, the paper mentions using LSTM for demand forecasting. How does the performance of LSTM compare to other ML algorithms not considered in this study?
12. Regarding the causal inference model in Section 3.5.2, could the authors discuss the potential limitations of this approach and any assumptions it makes?
13. In Figure 10, how were the stocks and flows schematic designed to capture the complexity of the system, and what were the challenges encountered?
14. How does the model account for the potential variability in user responses to incentives over time?
15. What specific process metrics were used to evaluate the partial model, and how were these metrics selected?
16. Could the authors provide more insight into the unexpected worse performance of several stations under ML mentioned in Section 3.6?
17. How sensitive is the model to changes in the starting inventory policy, and what are the implications for different bike-sharing systems?
18. In Section 3.7.2, how were the decision variables selected for the simulation of performance across the fitness landscape?
19. The paper discusses the ruggedness of the decision-making landscape. Could the authors elaborate on how this ruggedness affects the robustness of the proposed model?
20. How do the findings from this study relate to or differ from other research on human-AI collaboration in decision-making?
21. The authors mention the potential for ML to explore a broader decision space. How can organizations leverage this while mitigating the associated risks?
22. What are the next steps for this research, and how might the framework be further developed or tested?
Comments on the Quality of English LanguageIt is Ok.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsThe paper Towards a System Dynamics Framework for Human-ML Decisions: A Case Study on New York Citi Bike is a study used to create a simulation-based model to quantify algorithmic interventions within organizational contexts, combining causal modelling and data science algorithms. The authors used specific data from New York Citi Bike sharing to create a model and to demonstrate potential joint human AI decision-making process.
The length of the paper is inappropriate, in my opinion. Most of the chapters are somewhat long and chaotic as well as the description of the model itself. The description of the model is chaotically separated into several sections and aspects and their relations are not very clear. Due to that the entire process is hard to follow and understand. Some parts of the model are obviously based on the statistical approach, while others use some AI and human decisioning. However, the entire modelling process and the workflow are unclear. The presented diagrams are hard to read, and it is not clear, how do they follow each other. I recommend rearranging the text completely and the description of the approach. I also recommend creating a complete flow diagram and simplified flowchart to make the whole process easier to follow. Finally, the method is verified using only one set of specific data. There is also no comparison between the presented method and any similar existing technique.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
Comments and Suggestions for Authors
The manuscript titled: Towards a System Dynamics Framework for Human-ML Decisions: A Case Study on New York Citi Bike, includes a pretty good number of references. In particular, reference 33. Shaheen SA, Guzman S, Zhang H. Bikesharing in Europe, the Americas, and Asia: Past, Present, and Future. Transportation 966
Research Record [Internet]. 2010 Jan 1 [cited 2022 Nov 30];2143(1):159–67. Available from: https://doi.org/10.3141/2143-20… this paper includes cases in Europe, the Americas, and Asia… Can the authors discuss if the topic Citi Bike applies to other cities in USA and abroad?. The discussion can also add rental cases, as the ones shown in Figure 21. Perturbing demands on the 24th by lifting rental demands until 10:00 and distributing it 837 to the remaining periods.
The Keywords emphasized in this manuscript are quite interesting: Machine learning; system dynamics; simulation modelling; algorithmic decision-making; supply chain planning; ; NY Citi Bike… some recent Works on system dynamics includes studies and analysis about complexity, which may apply to this manuscript. Can the authors discuss if the Citi Bike is a complex problem?
Looking at references:
1. Raschka S. Build a Large Language Model from Scratch. Manning Publications; 2024. 400 p. 901
2. Malone TW. MIT Sloan Management Review. 2018 [cited 2021 Sep 22]. How Human-Computer "Superminds" Are Redefining 902
the Future of Work. Available from: https://sloanreview-mit-edu.plymouth.idm.oclc.org/article/how-human-computer-super- 903
minds-are-redefining-the-future-of-work/ 904
3. Agrawal A, Gans JS, Goldfarb A. MIT Sloan Management Review. 2017 [cited 2021 Sep 14]. What to Expect From Artificial 905
Intelligence. Available from: https://sloanreview-mit-edu.plymouth.idm.oclc.org/article/what-to-expect-from-artificial-intelli- 906
gence/… It seems fuzzy to understand why they are mentioned in the very first sentence in section 1. Introduction..
Despite the rapid advancement of Artificial Intelligence (AI), particularly with gen- 30
erative AI [1], human judgment remains critical in decision-making [2], [3]… A reader may be more interested to know the motivation on the problem: Towards a System Dynamics Framework for Human-ML Decisions,.. It may be more interesting to discuss the state on the art on the development on system dynamics frameworks, and afterwards, the case study can be discussed: A Case Study on New York Citi Bike
In line 87, authors finish the section 1 Introduction with a sentence that is not clear to understand: This gap highlights the proposed model's novelty, facilitating a more comprehensive evaluation of human-AI collaboration... Authors may include a very brief description of the following sections to get a picture of the contribution and how does the manuscript is organized by each section.
Authors must add representative references to section 2. Modelling Framework, which includes ZERO references, so that a reader cannot appreciate what is going on the research side for this topic, and what are the open problems, so that highlighting the cases study can help to appreciate the problem formulation
Section 3. The Bike-Share Problem, is motivated by three references:
31. Oliveira GN, Sotomayor JL, Torchelsen RP, Silva CT, Comba JLD. Visual analysis of bike-sharing systems. Computers & 960
Graphics [Internet]. 2016 Nov 1 [cited 2022 Nov 29];60:119–29. Available from: https://www.sciencedirect.com/science/arti- 961
cle/pii/S0097849316300991 962
32. Shen Y, Zhang X, Zhao J. Understanding the usage of dockless bike sharing in Singapore. International Journal of Sustainable 963
Transportation [Internet]. 2018 Oct 21 [cited 2022 Nov 29];12(9):686–700. Available from: 964
https://doi.org/10.1080/15568318.2018.1429696 965
33. Shaheen SA, Guzman S, Zhang H. Bikesharing in Europe, the Americas, and Asia: Past, Present, and Future. Transportation 966
Research Record [Internet]. 2010 Jan 1 [cited 2022 Nov 30];2143(1):159–67. Available from: https://doi.org/10.3141/2143-20… BUT these references are too old, but authors may consider including more global Works, like reference 33. Shaheen SA, Guzman S, Zhang H. Bikesharing in Europe, the Americas, and Asia: Past, Present, and Future. Transportation 966
Research Record [Internet]. 2010 Jan 1 [cited 2022 Nov 30];2143(1):159–67. Available from: https://doi.org/10.3141/2143-20
Also, reference 34. Chung H, Freund D, Shmoys DB. Bike Angels: An Analysis of Citi Bike's Incentive Program. In: Proceedings of the 1st ACM 968
SIGCAS Conference on Computing and Sustainable Societies [Internet]. New York, NY, USA: Association for Computing Ma- 969
chinery; 2018 [cited 2022 Oct 24]. p. 1–9. (COMPASS' 18). Available from: https://doi.org/10.1145/3209811.3209866… is old and must be replaced by recent ones to appreciate the problema formulation.
The contribution is not clear and should be reorganized.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
Comments and Suggestions for AuthorsThe paper can be accepted in the current form. Congrats to all the authors.
Reviewer 2 Report
Comments and Suggestions for AuthorsIn my opinion, the authors have provided the revisions and updates in the text according to the reviews and comments. All my questions and comments were sufficiently addressed. I recommend accepting the paper now.
Reviewer 3 Report
Comments and Suggestions for Authorsaccept
