How do Online Learning Networks Emerge? A Review Study of Self-Organizing Network Effects in the Field of Networked Learning

Round 1

Reviewer 1 Report

Thank you for your review which offers good depth of analysis and synthesis on the networked effects of preferential attachments, reciprocity and transitivity.  The Carvalho and Goodyear theoretical model that depicts a loose coupling of task and activity, etc. is doing so to honour the difficulty of designing for 'learning between the ears'. This is very different to work in the SNA field in which data about activity along links and nodes is taken as a proxy for learning. I think large parts of the review are written as if learning networks are made up of people, with the notable exception of lines 735-740. Learners are not 'participants', they are human beings. At line 705 the authors posit that preferential attachment could provide insights into why some are more active/popular than others but seems like a qualitative research project? The review claims to be 'systematic'. This immediately sets up some expectations of the nature of the work. For example, is the question to be one of effectiveness, the 'lived experience', or more of a 'realist' (what works and for whom) methodology. Since it is none of these, I think it would help if the review was more modest in claims about being a systematic review. It might be better to talk of using some systematic review techniques but you are far from the aim of retrieving and synthesising everything ever written on a topic. At line 652, the authors express a hope that meta-analysis could be employed but this seems optimistic given the level of funding available and heterogeneity of included studies.  Could limitations be addressed explicitly? For example, the geographically limited representation of the articles (lines 204-9).   Line 36 I agree that we are talking about the loose coupling between design and actuality of learners behaviour but then to talk about self organisation because of emergence is a bit of a stretch. That happens on lines 40 to 43, and then the article moves into social network analysis territory - I think those are very different things unless you're talking about very big networks to the extent that they approach 'scale free' proportions. Line 425 features a 'large-scale MOOC' and a formal VLE-based network of 20 students in the same sentence. There are very different things.  However we may not always be talking about self organisation because tutors also have a role in organising groups - how often does that happen in comparison with 'self-organisation'? Line 55 statement is too strong: preferential attachment does not fully explain the growth of the web following further research post Barbarasi Line 58 The Lusher et al. reference is based in network theory and not networked learning as such. The statements either side of it are about network learning but the Lusher source is not sufficient support for these claims, ie. that networks that have a high proportion of reciprocal ties result in the balanced and dense networks, this is quite generalized and speculative. Line 64-5 it is a fair point to claim that reciprocity is a feature of Culture than a behaviour, however this also emphasizes the complexities of what you are trying to address and the disjuncture between the ostensible research the phenomena you are trying to rhetorically construct. Line 70 what is meant by cohesive here and why is it proposed as a quality of networks either positive or negative, presumably positive implications but without analysis of the content of messages, this could be destructive or ethically dubious activity. Also, are cohesive networks essentially better learning networks? The author seems to be assuming that no learning takes place apart from in the network but these are human beings and they are learning regardless of any network Line 77 - connectivism is of dubious standing and has been squarely critiqued (e.g. Gourlay & Oliver 2018) , but why is gaining access to other learners thought to be important? Line 79 - do you think that a comprehensive understanding is at all possible. Is the lack of this comprehensive understanding partly due to the nature of these theoretical notions? Line 82 has a superfluous close bracket. Cohesion is a complex, ambiguous concept and you have not stopped to define this. Line 86 to line 88 you proposed a tension between cohesive network structures and the benefits of weak ties without properly exploring whether these properties could be present in either context, of face-to-face or scale-free networks. They are very different things and perhaps it is important to consider that 'network' is only a metaphor. You needed to have defined more closely before launching into questions such as on lines 89 to 97. For example the question beginning on line 90 needed to define the parameters of what is meant by online learning networks, for example how many (even a rough figure) learners are involved? Line 111. PRISMA is, as mentioned in the acronym's  component words, about reporting. It is not a general methodology for undertaking systematic reviews. These exist. It is usual to report/describe inclusion and exclusion criteria (these are not merely opposites) at the search strategy stage, rather than at the screening phase. It is rare to impose a date limit in systematic reviews - this needs to be defended, as does the choice of databases. What other databases could have been used and why were they excluded? Please explain your decisions about grey literature. For transparency, your search terms and database search history, with hit counts, should be reported in a table here or as an appendix. Line 125 - what is meant by advice from other authors? Line 128 - although you referred to the Post digital, it is not clear how the work is linked to that field - SNA etc. seems quite 'noughties' - i.e. something that emerged in the mid-late 2000's.  Line 129 - defines learning as 'value creation' but this is ambiguous and anyway quite a limited construct compared with 'learning' per se. Line 142 - study, not 'stud' Line 158 - Under 'eligibility', the box about exclusions should very briefly detail the reasons for exclusion, not state, 'with reasons'. Line 201 refers to 'appraisal' but was critical appraisal used to analyse the papers for quality? If so, how was this done? Was a standardised tool used to assure consistency? Line 257 - give the constituent words of an acronym at first use Line 266 - Threads not 'threats' Lines 308-319 show some good analytical scholarship with reference to counting first posts to a forum or not. But I find SNA assumptions rather strong in general.  Line 325 'de'? Line 326 poor grammar Line 345 - 'binary learning networks' and 'quality label' - these phrases are ambiguous, please reword or explicate.  Line 410 advanced not 'advances' Line 684 collaboratively not 'collaborative' Line 730 the not 'de' Line 758 I am not convinced that the 'weak ties' concept is so straightforwardly transferable into a theory of networked learning design.  Line 800 - instructions not 'instructs'.  Lines 800-812 The effort to provide some practical guidelines is admirable. You should try to link with established texts by the likes of Gilly Salmon or David McConnell where similar points have been made. 

Author Response

Dear reviewer,

We hope this letter finds you well. We thank you for the opportunity to offer you a revised version of our manuscript, titled How do online learning networks emerge? A review study of self-organizational network effects in the field of Networked Learning.

We thank the reviewers for their thoughtful and detailed feedback on our manuscript. We appreciate the fact that they recognize the importance of this topic and urge us to resubmit our paper. We have used their comments and revised our manuscript thoroughly accordingly. In the attached table we describe for the reviewers’ comments the way we have addressed them in our text. We believe that the quality of the paper has increased significantly and are looking forward to your response. 

Best wishes,

Also on behalf of my co-authors.

Author Response File: Author Response.pdf

Reviewer 2 Report

The author proposes a meta-analysis of evidences founded in online learning networks related to self-organizational network effects They identified three key effects of this type of networks: preferential attachment, reciprocity and transitivity, and selected 23 papers from an initial set of 247 publications. They analysed this list of papers according the three network properties, and following Goodyear and Carvalho’s architectural perspective.

In my opinion is a very relevant and well-written paper, although there are relevant issues to be addressed before its publication.  

There are very few evidence (only five out of 23) that report on possible consequences, and support that Self-organizational network effects impact into learning outcomes. Authors should considered to increment the number of papers that give evidence on this direction.    Authors highlight three self-organizational network effects (preferential attachment, reciprocity and transitivity), however there are other network properties that could be considered.  For instances, cohesion is a well-known network property that can be measured by other metric different from transitivity such as density and clique analysis. There are many papers in the literature using other network metrics for analysing self-organizational learning network.  At least, author should motivate why they didn’t choose other metrics.   Preferential Attachment process is highly influenced by network size, and time frame when the learning activity is performed. It is less likely that this effect occurs in networks with a small number of nodes. Likewise, activity duration may impact on the apparition of such effect.  It would be interesting to include a deeper discussion on this effect, and to reconsider analysing others model of network formation   Authors advice (Line 809) that learning architects should design learning networks to have the change to emerge. However, they don’t present evidence self-organizational learning network is a better alternative to a designed learning network regarding learning outcomes.

Due to the previous comments, one possibility is to expand the list of paper to be analysed in the paper. With this goal in mind, I propose a list of candidate papers. It could be possible that some of them appear in the original set of candidates (247), but I don’t have access to this list to check it.

Nurmela, K., Lehtinen, E., & Palonen, T. (1999, December). Evaluating CSCL log files by social network analysis. In Proceedings of the 1999 conference on Computer support for collaborative learning (p. 54). International Society of the Learning Sciences.

Lipponen, L., Rahikainen, M., Lallimo, J., & Hakkarainen, K. (2001). Analysing patterns of participation and discourse in elementary students’ online science. Euro-CSCL, Maastricht, the Netherlands.

Reffay, C., & Chanier, T. (2003). How social network analysis can help to measure cohesion in collaborative distance-learning. In Designing for change in networked learning environments (pp. 343-352). Springer, Dordrecht.

Aviv, R., Erlich, Z., Ravid, G., & Geva, A. (2003). Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks, 7(3), 1-23.

De Laat, M., Lally, V., Lipponen, L., & Simons, R. J. (2007). Investigating patterns of interaction in networked learning and computer-supported collaborative learning: A role for Social Network Analysis. International Journal of Computer-Supported Collaborative Learning, 2(1), 87-103.

Harrer, A., Zeini, S., & Pinkwart, N. (2005, May). The effects of electronic communication support on presence learning scenarios. In Proceedings of th 2005 conference on Computer support for collaborative learning: learning 2005: the next 10 years! (pp. 190-194). International Society of the Learning Sciences.

Kossinets, G., & Watts, D. J. (2006). Empirical analysis of an evolving social network. science, 311(5757), 88-90.

Bratitsis, T., & Dimitracopoulou, A. (2007, July). Interaction analysis in asynchronous discussions: lessons learned on the learners' perspective, using the DIAS system. In Proceedings of the 8th iternational conference on Computer supported collaborative learning (pp. 90-92). International Society of the Learning Sciences.

Heo, H., Lim, K. Y., & Kim, Y. (2010). Exploratory study on the patterns of online interaction and knowledge co-construction in project-based learning. Computers & Education, 55(3), 1383-1392.

Vaquero, L. M., & Cebrian, M. (2013). The rich club phenomenon in the classroom. Scientific reports, 3, 1174.

Haya, P. A., Daems, O., Malzahn, N., Castellanos, J., & Hoppe, H. U. (2015). Analysing content and patterns of interaction for improving the learning design of networked learning environments. British Journal of Educational Technology, 46(2), 300-316.

Chen, B., Chang, Y. H., Ouyang, F., & Zhou, W. (2018). Fostering student engagement in online discussion through social learning analytics. The Internet and Higher Education, 37, 21-30.

Vaquero, L. M., Rodero‐Merino, L., & Cuadrado, F. (2019). The anatomy of information cascades in the classroom: An observational study. British Journal of Educational Technology, 50(1), 371-384.

Chen, B., & Huang, T. (2019). It is about timing: Network prestige in asynchronous online discussions. Journal of Computer Assisted Learning.

 

Minor issues:

 

Line 43: Missing reference: “The process of self-organization in (learning) networks is well-described in social network theory”

 

Line 85: Wrong citation: Granovetter’s [17] seminal work => [51]

 

Line 142: stud => study

 

Line 248: de conclusions => the conclusions

 

Table 1: there are several issues:  a) wrong citation format in General information column; b) incompleted cells in row: Pham et al (2012) column: Type of technology used; c) Nat. column could be omitted as all cells have the same value. d) LIW acronym in Physical Settings/Context is not explained in the text.

 

Section 3.1.2 Methods to select online learning ties for research purposes: add a summary table.

 

Line 330: missing whitespace [28]was => [28] was

 

Line 353: missing whitespace [39]to => [39] to

 

Line 386: missing whitespace theresults => the results

 

Line 405: wrong reference format [32, 43]

 

Line 412: missing whitespace [25]or => [25] or

 

Line 412: missing whitespace [48]and => [48] and

 

Section 3.2.2 add a summary table, and report reciprocity measures

 

Section 3.2.3 add a summary table, and report transitivity measures

 

Line 453: missing comma [42] [27] => [42], [27]

 

Line 454: too many decimals (0.812) => (0.81)

 

Line 508: missing whitespace [1]architectural => [1] architectural

 

Line 556: missing whitespace [31]explicitly => [31] explicitly

 

Line 730: into de => into the 

 

Line 797: enumerate the triggers. Example a) Provide clear [...]; b) Make participation …

 

References

Review all the references since there are many inconsistencies. In particular in the page number format of each publication.

Author Response

Dear reviewer,

We hope this letter finds you well. We thank you for the opportunity to offer you a revised version of our manuscript, titled How do online learning networks emerge? A review study of self-organizational network effects in the field of Networked Learning.

We thank the reviewers for their thoughtful and detailed feedback on our manuscript. We appreciate the fact that they recognize the importance of this topic and urge us to resubmit our paper. We have used their comments and revised our manuscript thoroughly accordingly. In the attached table we describe for the reviewers’ comments the way we have addressed them in our text. We believe that the quality of the paper has increased significantly and are looking forward to your response. 

Best wishes,

Also on behalf of my co-authors.

Author Response File: Author Response.pdf

Reviewer 3 Report

Overall, this work makes an important and interesting point regarding the role of self-organizing social network processes in collaborative online learning discussions, based on a review of the literature. Self-organization is a crucial factor in realizing online discussions' potential: top-down or pre-determined organization of the conversation puts constraints on the flow of information and on the creation of interesting connections. On the other hand, no organization at all can lead to cluttered and fragmented discourse.

The measures chosen for identifying emergent social structures were convincing. Other more system-level measures that could be considered to that extent is modularity or community detection analyses (see for instance Aviv, Erlich, Ravid, & Geva,2003, Kent & Rechavi 2018).

While the message of this paper is clear and the practical implications are evident, the text body could be improved in terms of flow, organization and readability. For instance, the formal definitions for network measures such as preferential attachment and reciprocity are only given in paragraph 4.1. They should be provided much earlier in the text, where these terms are first introduced. The findings regarding the practices that improved learning (paragraph 3.3) are given as running text – providing a summary and a generalization of these would be informative and assist readability. These generalizations can be found in the discussion, but introducing them earlier and with relation to the actual findings could be helpful.

Also, more thought should be put into the visual aids used (tables and figures):

Table 1 is very difficult to read. Perhaps consider splitting it into two tables: a summary of the data (country, size etc.), and a summary of the findings (self-organizing network measures and their quality). In table 2, consider separating the procedures (e.g. descriptive SNA) and the technologies (e.g. UNICET, R) into two separate columns – these are different aspects. Figure 1 is confusing and doesn't help in understanding, perhaps consider changing it. More figures could be added to visually explain some of the network measures. For instance, transitivity is easily presentable in a graph, by highlighting triads.

To sum, this review introduces an interesting and well-justified viewpoint on social network processes in online learning environments, makes the case that "learning cannot be designed, but only be designed for", and highlights some important points for future research in the field. The readability and the flow of the text should be improved.

Author Response

Dear reviewer,

We hope this letter finds you well. We thank you for the opportunity to offer you a revised version of our manuscript, titled How do online learning networks emerge? A review study of self-organizational network effects in the field of Networked Learning.

We thank the reviewers for their thoughtful and detailed feedback on our manuscript. We appreciate the fact that they recognize the importance of this topic and urge us to resubmit our paper. We have used their comments and revised our manuscript thoroughly accordingly. In the attached table we describe for the reviewers’ comments the way we have addressed them in our text. We believe that the quality of the paper has increased significantly and are looking forward to your response. 

Best wishes,

Also on behalf of my co-authors.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Thank you for your diligent response to the initial feedback. I think the article is much improved: easier to digest for the summary tables, clearer/more transparent. The language has been moderated where appropriate. Some of the replies to individual points I made were indirect but not poor enough to justify holding out for another version and afterall this is your publication and you are ultimately answerable to the reader. Just one thing that needs attention is that while you have omitted 'systematic' from the title, I recommend searching the paper for this word to ensure that you are happy that it is being used properly. Much has been written about performing systematic reviews and yet little of this has been referenced. So, the question remains - are you doing a systematic review or are you borrowing systematic review techniques? I feel it is the latter and the work needs to make this clearer.

Author Response

Dear reviewer, thank you very much for your positive feedback on the revisions of our article. We have tried to strengthen our work based on your comments.

We have omitted the word systematic in the overall article and described PRISMA as a systematic review technique we have used. We have made minor textual changes. Based on the answer of another reviewer we have also changed the central term to self-organizing network effects instead of self-organizational network effects. We perceive this term as a better translation of the concept. This change did not affect our search results.

We thank you for the opportunity to offer you a revised version of our manuscript, titled How do online learning networks emerge? A review study of self-organizing network effects in the field of Networked Learning.

We believe that the quality of the paper has increased significantly and are looking forward to your response. 

Best wishes,

Also on behalf of my co-authors.

 

Reviewer 2 Report

I appreciate that authors had made significant changes on the first version that considerably improve the quality and readability of the paper. Besides,  they properly answered my comments, and gave robust arguments when they didn't agree with my point of view.

Currently I have only three minor suggestions to be considered after publication.

A) Authors acknowledged that there are two journal articles (see below) relevant to the review, but they didn't include them since after widening the search they still did not appear in their list. In my opinion, this is a clear evidence that there are room for improvement the study coverage. Any review  paper should consider coverage as one of the key metric for measuring its quality.

Heo, H., Lim, K. Y., & Kim, Y. (2010). Exploratory study on the patterns of online interaction and knowledge co-construction in project-based learning. Computers & Education, 55(3), 1383-1392.

Vaquero, L. M., & Cebrian, M. (2013). The rich club phenomenon in the classroom. Scientific reports, 3, 1174.

B) In line  282, authors states:

We want to emphasize that these quality issue only relate to answer our research questions. The quality issues are not about the overall quality of the studies selected in our review.

 This metric is report in table 1 as low, medium, high, very high. I would suggest to change the name of this column since it is misleading. If you are not considering the overall quality of the article, it is not fair to associate just one of the quality dimension with the whole metric.   

C) I would suggest a last check on grammar and spelling english as well as  reviewing references numbering and citation format. 

Author Response

Dear reviewer, thank you very much for your positive feedback on the revisions of our article. We have tried to strengthen our work based on your comments.

 

A: Concerning the articles we have missed: The article of Heo is not on topic. They only look at density and a centrality measures. They did not mention one of the self-organizing network effects in their study. That is what we mean by the explanation and the reason why they are also not in the big list. It is not an indication of the quality of the review. The article of Vaquero, which is on topic, did not appear in our list. We have included Web of Science in our search (because Scientific Reports is not related to the field of learning or education) and widened the search with “social network analysis” to find this article, but still it did not appear. But we do not think it is due to the quality of the review study. It is more because of the limited search possibilities of Web of Science, as it only searches titles, abstracts and key words. ScienceDirect f.e. and Google Scholar do full text searches which provide better results.

B: Concerning the quality issue. You are correct. We have changed the title of the table in "Causality" instead of Quality.

C: We have checked the references and made minor language corrections.In your first comments of the first round of revision, you used the term self-organizing network effects in your explanation. Based on this remark we discussed that this is a better translation and use of the concept. We have changed the central term self-organizational network effects to self-organizing network effects. This change did not affect our search results.  

We thank you for the opportunity to offer you a revised version of our manuscript, titled How do online learning networks emerge? A review study of self-organizing network effects in the field of Networked Learning.

We believe that the quality of the paper has increased significantly and are looking forward to your response. 

Best wishes,

Also on behalf of my co-authors.

Reviewer 3 Report

The main points as well as the figures are clearer now.

 

Author Response

Dear reviewer,

 

We thank you for the opportunity to offer you a revised version of our manuscript, titled How do online learning networks emerge? A review study of self-organizing network effects in the field of Networked Learning.

We checked all figure and table numbers again. Based on a reviewers remark we have changed the central term self-organizational network effects to self-organizing network effects. We think this is a better translation and use of the concept. This change did not affect our search results.  

We believe that the quality of the paper has increased significantly and are looking forward to your response. 

Best wishes,

Also on behalf of my co-authors.