A Multiverse Graph to Help Scientific Reasoning from Web Usage: Interpretable Patterns of Assessor Shifts in GRAPHYP

Abstract

The digital support for scientific reasoning presents contrasting results. Bibliometric services are improving, but not academic assessment; no service for scholars relies on logs of web usage to base query strategies for relevance judgments (or assessor shifts). Our Scientific Knowledge Graph GRAPHYP innovates with interpretable patterns of web usage, providing scientific reasoning with conceptual fingerprints and helping identify eligible hypotheses. In a previous article, we showed how usage log data, in the form of ‘documentary tracks’, help determine distinct cognitive communities (called adversarial cliques) within sub-graphs. A typology of these documentary tracks through a triplet of measurements from logs (intensity, variety and attention) describes the potential approaches to a (research) question. GRAPHYP assists interpretation as a classifier, with possibilistic graphical modeling. This paper shows what this approach can bring to scientific reasoning; it involves visualizing complete interpretable pathways, in a multi-hop assessor shift, which users can then explore toward the ‘best possible solution’—the one that is most consistent with their hypotheses. Applying the Leibnizian paradigm of scientific reasoning, GRAPHYP highlights infinitesimal learning pathways, as a ‘multiverse’ geometric graph in modeling possible search strategies answering research questions.

1. Introduction

In general, the scientific approach is based on reading and possibly on citing articles previously published on the same research question, including articles which may be defended or, on the contrary, criticized, in cases in which a subject is controversial. How can we make better use of the documentary tracks revealed by the traces left by previous users of scientific publication servers? How can we use these documentary tracks to assist the scientific reasoning of new users exploring the same research problem? These are questions that motivate the present work, in the continuation of a paper [1] dedicated to a new knowledge graph model, GRAPHYP, and its application to identify adversarial scientific communities. Our hypothesis is that a comprehensive modeling of web usage logs can make it possible to represent the main directions of answers that researchers have previously given to a research question.

Recent developments in Artificial Intelligence (AI) suggest that new answers to these questions are possible but require additional research on science analytics and data modeling. Although ubiquitous, AI abilities to contribute to human reasoning still resist comprehensive and stable scientific categorization. For example, common sense reasoning is still insufficiently assessed considering all related aspects, as noted by Davis [2], relying on a survey of no less than 139 benchmarks. Categorization is also challenging regarding AI’s abilities to support scientific reasoning, which remains a matter of debate. Either Artificial Intelligence is seen as a threat due to its “dark sides” [3], or, on the contrary, it is studied extensively across a wide range of methodologies as a “powerful paradigm in scientific research” [4].

However, the current digital support for scientific reasoning in the communication of scholarly knowledge, despite intensive research [5], leads to unsatisfactory results; the assistance for scientific reasoning is still deemed embryonic [6]. Such a shortcoming raises a paradox and a priority.

The paradox is between, on the one hand, a strong incentive for “continuous online learning” in science; legislation steering the web and data mining in research has multiplied over the last decade on a global scale, in the USA and Japan, followed by the UK in 2014, France in 2016, Germany in 2017 and an EC Directive in 2022. On the other hand, there are few initiatives for modeling web usage technologies for the benefit of scientific reasoning. Search log data recorded from scholarly research remain either concealed or hidden [7] and always underexploited. However, these are data from the “reverse side of the coin” of web technologies. On one side is the response to a request; on the other is the trace left on the origin and the nature of the asked question. Abundant new knowledge can be captured in this context from web usage, with the aim of helping scientific reasoning by representing the choices of predecessors in the framework of continuous online learning.

This article aims to be a contribution in that direction. We present hereunder the motivation, the problem and the results.

1.1. Motivation: Turning Assistance to Scientific Reasoning into ‘Continuous Online Learning’

Scientific reasoning currently benefits from bibliometric services bringing new logic and new tools in the analysis of results, categorized where necessary in Scientific Knowledge Graphs (SKGs). In addition, the ORKG community (Open Research Knowledge Graph https://orkg.org/) recommends a significant extension of the representation of alternative assessments of scientific results. However, we currently observe that no digital service is available to assist scientific reasoning in canonical hypothesis-based science.

In that direction, it appears fruitful to start by collecting and modeling the search experience of researchers. As mentioned previously, this was our approach when we proposed GRAPHYP, a conceptual framework informed and modeled from the search experience of scholars who have explored the same research question [1]. From this descriptive conceptual framework, we now approach the process of the documentary choice of users/researchers among the available representations of data and documents. For this, we propose the modeling, again from the analysis of logs, of ‘interpretable patterns’ of changes in document judgment (‘assessor shifts’) coupled with the corresponding documentary tracks. The objective of the present study is to propose solutions to represent such interpretable patterns of assessor shifts through the comprehensive modeling of adversarial query strategies on a knowledge graph.

We observe that, today, it is hardly possible to explore eligible theories on the basis of a cartography of the uses of interrogation and choices of predecessors obtained via data mining techniques. Satisfying this need is our first motivation, in a framework of ‘continuous online learning’, also taking into account the dynamics of the evolution of scientific categorizations.

As we know, scientific information is constantly changing due to the daily stockpiling of new contributions in all fields, and categorizations are impacted by the ever-changing reactions (citations, annotations, social media features, etc.) of cognitive communities. In addition, an assessor’s shift in document judgment is depicted as being “relatively relevant”, as assessors often disagree about relevance and can be inconsistent among themselves [8]. Moreover, the epistemological directions of assessment can reveal contrast, whereas studies experimenting with the large-scale use of the same data show a radical dispersion of results [9]. The ability of AI to produce new outcomes from graphs of web usage data—already a long story [10]—receives with our research an additional extension; “continuous online learning” aims to connect comprehensive dynamic adaptations in the vocabulary and syntax of scientific reasoning, and to analyze the incoming flow of data and contributions.

Despite the construction of continuous online learning, the risk for science prevails in that the web content may spread without timely updates and without relevant concepts, in a “Tower of Babel” effect, driven by growing interactions of plain scientific categorizations that are barely updated. However, even though web mining technology is extensively practiced in scientific applications, there is still hardly any Scientific Knowledge Graph providing users with comprehensive dynamic modeling of the “commonwealth” of web usage and proposing a language for its interpretation; GRAPHYP 2 is intended to be a contribution in this direction.

1.2. Modeling an ‘Assessor’s Shift’ in Assistance to Scientific Reasoning

We come later (Section 2, Background) to the corpus of the literature that underlies our “multiverse graph for scientific reasoning”. The main assumptions for the development of GRAPHYP 2 (GR2) are first sketched below to provide a perspective to our research question. Research methods are developed in Section 3.

We assume that the assistance to scientific reasoning should rely on data captured from web usage traces, left in usage logs or in a restricted form, from any dataset equipped with a log record. The challenge is then to model differing uses and changing choices (assessor shifts) that the user can encounter, simulate and interpret, by visualizing and modeling the ‘documentary tracks’ (i.e., the profile of the documents consulted by the users of similar queries on a research question).

In a previous article [1], we published a background of our methodology, GRAPHYP 1, hereafter called GR1, depicting documentary tracks of scientific literature in retrievable cliques of interpretable adversarial subgraphs. In what follows, with an extended methodology, hereafter called GR2, we seek to model the documentary choices made by previous users in order to detect assessor shifts between documentary tracks. Our primary goal is to show that web usage modeling can serve as an aid to scientific reasoning, when taking advantage of the “assessor shifts” recorded during previous search experiments. GRAPHYP thus functions as a Graph Adversarial Network, in which GR1 is a “generator” of assessments (here, modeled documentary tracks) and GR2 a “discriminator” of modeled patterns [11] between assessment shifts.

With these two successive functions, GRAPHYP offers an original type of aid to scientific reasoning; in fact, it is neither a model for interpreting scientific documents nor a “discovery tool” documenting alternative theories. The added value of GRAPHYP is its production of a complete model of the documentary practices of a scientific subject by representing the contradictory paths and the patterns of choice that can be modeled between them. It aims to stimulate the understanding of the evaluations and to facilitate a possible change in evaluation, or assessor shift, that the choice of the users can privilege, according to their research question. The specific originality of this type of browser is its operation on an inverted matrix. It does not list the answers to a question as a classic browser does, but it represents the possible practices of answering a question (hypergraph modeling) and their uses (recorded logs of responses positioned on the hypergraph). The concept of GRAPHYP is essentially a modeling of the adversarial use of the web during a search and an identification of the changes in the appreciation of the relevance of documents.

We observe that assessment shift digital representations are important units of data in the support of scientific reasoning, and that there is currently no modeling of those representations. In addition, we note the importance for scientific reasoning of displaying disputes and controversies. A recent bibliometric analysis, studying scientific disagreements on a large scale [12], highlights the importance of “relative differences” as well as the “heterogeneity” of observed practices of disagreements. We therefore speculate that the representation of differences in retrieval practices of predecessors in cliques, which is a result that AI can perform with neutrality, can help users verify and secure their own assessment shifts on methods or on results, with regard to the observed usage of the other members of a cognitive community, labeled by keywords.

Our research question is thus as follows: How can we assist scientific reasoning with comprehensive representations of adversarial documentary tracks as a generator (see GR1 [1]), which can both be meaningful and usable to enable understanding in a discriminator (GR2)?

The analysis of assessor shifts can allow identifying which documentary “world” the assessor’s search experience belongs to; secondly, what the other possible documentary choices are for the same keyword, regardless of the assessment they represent; and third, what the “best possible” modelized documentary track is, in the assessor’s opinion. The judge here is the assessor. The documentary tracks are modeled data in a discriminator, and the assessor is the human in the loop.

The research question thus becomes the following: Which bodies of literature can we mobilize, and to what extent, to build the modeling of GRAPHYP2? Two main bodies of literature shall be mobilized. One deals with web usage mining to collect and model data, and the other deals with the methodologies of AI for aiding scientific reasoning in that context in this context.

Web usage log analysis and the interpretation of log data in scientific activities cover already some important tasks in research practices, for instance, in the comparison of exploration strategies across ontologies from usage logs in the NCBO BioPortal [13].

In addition, our modeling sources benefit from other fields of research to define assistance for scientific reasoning. The first field is Explainable AI [14], with a wide range of applications in this direction, and a second growing field concerns “message passing” geometric graphs [15].

Scientific reasoning progresses with efficient solutions found from controversies; more generally, in GR2, even more systematically than in GR1, adversarial modeling is at work to represent assessor shift patterns and applications. A resulting problem is that of censoring; with GRAPHYP, we model it via the representation of partial information adjusted to sensitive attributes (classes of documentary tracks), given a utility constraint, in the censoring framework of a geometric hypergraph [16]. In our opinion, current research on possibilistic graphical modeling methodologies [17] confirms the ability of our model in reasoning about assessor shifts in an environment of uncertainty and imprecision.

The type of reasoning analytics that we adopt is causal inference reasoning [18] and a very useful transfer learning methodology [19].

Finally, we link web usage mining techniques and innovative reasoning methodologies to build and use the classifier that is GRAPHYP. We integrate it in an operative type of explainable graph that we call a ‘Multiverse’ graph because it transposes the Leibnizian paradigm of possibilistic logics into the digital space of hypergraph representations [17]. The corresponding research methods are described at the beginning of Section 3.

1.3. Mapping Explainable Dialogue Search/Research with a Multiverse Graph

In GR1, we observed that GRAPHYP “is a graph designed not to represent information, but to model information representation”. By applying our modeling to assessor shifts of researchers, we showed that SKG GRAPHYP is able to describe interpretable pathways among the set of underlying documentary tracks, in a multi-hop assessor shift. We also showed that this approach is helping researchers explore the “best possible” documentary tracks, i.e., that which is the most consistent with their eligible hypotheses. In this way, we found that assessor shifts, coupled with documentary tracks, are key units of data for building up interpretable user pathways. We present in Section 3 a first implementation of these concepts and the results of tests carried out on a large sample of users.

Implemented in GRAPHYP, as observed above, the Leibnizian paradigm of scientific reasoning highlights infinitesimal learning pathways, in the announced new type of ‘multiverse graph’ with additional impacts on search strategies and pattern discovery (see Section 2).

The unit of output in GRAPHYP, a shift in assessment, can be retrieved, compared, simulated, mapped in time and space and analyzed in its qualitative documentary composition (data and analysis). Both of these features leverage existing refinements of web usage log analysis (see Section 2 and Section 3), which offer a wide range of potential services.

Given our assumption, our modeling seeks to provide users with functionalities of assistance to scientific reasoning, insofar as the outcomes of GRAPHYP are as follows:

–

“indirect”: SKG GRAPHYP does not provide ‘results’, but a comprehensive representation of clickable maps of topologized results;

–

“intermediary”: It does not deliver any final scientific assessment, but a methodology to reach the documentary set that seems to a scientist as being best adjusted to the hypotheses that are under review, as well as those that the user expects to simulate;

–

“neutral”: Our typology of classification, not being referred to any assessment on the scientific content of captured documents, deals only with the information profile and content of user logs, formalized in a triplet of parameters measured from anonymized data (see [1] and Appendix A for more details): Intensity (How many readers?), Variety (How many documents?) and Attention (What degree of balance-ratio between the number of readers and number of documents?).

Scheme 1 summarizes our implementation of the conceptual approach of a multiverse graph with two interactive phases (generator/discriminator) in which documentary tracks, recorded from our three variables, can provide a background helping scientific reasoning.

The rest of the paper is organized as follows: Section 2 reviews the background and other works. Section 3 analyzes GRAPHYP’s search of the best possible fit between documentary tracks and eligible theories. Section 4 conducts a discussion. Section 5 presents future works, and Section 6 provides the conclusions.

2. Background and Other Works

2.1. Background

Undiscovered interactions have been searchable treasures since long before the web; web usage gives them incredible extension, provided that there are data identification and comprehensive modeling, as well as ways to query the ‘second world’ of the web that are unevenly explored. At the same time, the Declaration on Research Assessment (DORA) gives high priority to “Reimagining academic assessment” (https://sfdora.org/dora-case-studies/, accessed on 15 March 2023).

For the present work on a ‘Multiverse graph for scientific reasoning’, we propose a background with two components: data and their interpretation. First, we review the literature on data availability and the capture and representation of logs in web usage techniques, with reference to scientific documentation from the perspective of where assessor shifts can be identified. We then review the literature that is applicable to scientific reasoning that is linked to modeling documentary uses and their exploitation and interpretation in relevance judgments. Using Scheme 2 below, we show the background for interpretable patterns of assessor shift.

2.1.1. Data Availability and Representation

We first review the literature relating to the availability and capture of data and then the literature relating to data representations.

• EXTRACTION OF NEW KNOWLEDGE FROM LOGS

Any new scientific contribution must be retrieved from a dedicated “conceptual fingerprint” [20] that must be precisely defined, which is an uneasy task in any innovative context and depends on the network-based learning of cognitive communities, which allows for the identification of possible undiscovered interactions [21]. This raises difficult questions of adversarial recommendations, not only in the categorization of scientific contributions [22].

The extraction of new knowledge can also proceed from the structural components of meaning with related metrics (e.g., Altmetrics: a manifesto http://altmetrics.org/manifesto/ accessed on 11 March 2023) in systems in which the interactions of the users and the search engine make sense [23]. ‘Searching as learning’ thus becomes a tandem of interactions between the exploration of search behaviors and outcomes in learning-related tasks [24]. This trend raises questions on the fairness, security and trustworthiness of those models within this type of deep learning model [25] and opens avenues toward meta-heuristics.

• DATA CAPTURE AND EXPLOITATION

Availability

The methodology of usage patterns from the analysis of web-based library catalog logs has been developed for almost two decades [26]. Librarians and publishers of scientific materials, notably with Elastic Stack suite (https://www.elastic.co/elastic-stack/, accessed on 23 February 2023) and Kibana for the management of logs of big data contents, have long been exploiting (more than 10 years) logs of documentary tracks of research activities (https://www.ezpaarse.org/) on a global scale for multipurpose data management operations. Data availability is generally limited by rules of integrity and privacy. Citation counts and usage analytics from logs are two alternative ways to document the quality of the same ‘seed article’ in scholarly digital libraries; both methodologies are supposed to present a complementarity [27]. Tests of data availability and first-degree analytics are given on a large scale in WorldWideScience (http://worldwidescience.org/), including experimentations on reusing log data to analyze the semantics of queries in science, with a comparative approach [28] and a large number of participating databases worldwide.

Exploitations in large scientific infrastructure: A few examples

Data capture and re-use from logs to improve services to users/researchers is at work in a growing body of very large infrastructure; we select here a few significant practices in techniques and results.

The OakRidge Leadership Computing Facility, which exploits logs in petascale file systems, has been developing since 2017, profiling usage behavior trends and the sharing trends of users, and it invites other HPC centers to develop a similar approach [29] with new methodologies [30] for efficient web mining taxonomy.

We already mentioned Bioportal, at the NCBO, for similar log management in the workflow of scientific research, at which they identify and profile browsing behavior types and “compare exploration strategies across ontologies” [13].

• DATA REPRESENTATIONS

Turning web usage data into knowledge involves a variety of operations, techniques and architecture that we examine in accordance with the related phases in GRAPHYP’s results on assessor shifts (see Section 3).

Techniques of log analytics

The main operations of log capture and exploitation for new knowledge have been well established for over a decade [31]. An extensive survey of researchers’ practices and opinions showed a favorable welcome to data sharing in general, as well as interest in modeling for data discovery [32]. Log files usually contain an important amount of accurate information about the user, typically the IP address, time stamp, access request, number of bytes transferred, result status, referred URL and user agent, plus others derived from this first list of data, as well as additional parameters that can be used for path analysis that are most commonly managed with graphs [33] and for additional analytical tasks, such as classification rules, clustering, pattern discovery and association rules. We consider these parameters in the different phases of the conceptual framework of GRAPHYP.

Data Architecture for log representation

Most data architecture aims at producing a scientific result for a unique scientist (for instance, in the technique of personalized web search), for the management of a research community or institution or for a learning entity [10]. We note here that the architecture of GRAPHYP does not belong to those categories and is fully neutral in its destination. The original feature of our approach is that we do not aim to create scientific knowledge or results, but to provide comprehensive and explanatory documentary assistance that is likely to stimulate the scientific reasoning of all potential users by responding to a need that scientists have always expressed. It accesses adversarial expressions of research questions and gathers comprehensive documentation for their own conclusions.

We can say that, put simply, the GRAPHYP approach is web-friendly and includes serendipity. The heterogeneity of log parsers remains a significant restriction on the expected commonality of web usage analytics in science; fortunately, for given inputs, there is a wide range of solutions regarding expected analytical outputs and performance levels of existing categories of parsers [34].

2.1.2. Reasoning Analytics

GRAPHYP aims at bringing assistance to scientific reasoning in relevance judgments, applying the innovative rules of possibilistic graphical models to representations expressed in a geometric graph [35] with multiverse reasoning analytics. We rapidly introduce those two points with remarks on the aid to scientific reasoning in the context of web usage analytics.

• Scientific reasoning and web usage analytics

Scientific reasoning is approached in the literature via attempts to reach the frontiers of the scientific method [36]. The numeric transposition of scientific reasoning [37] and assessments of results (articles, data and annotations) by means of interpretable metrics are still an ongoing challenge [38]. Differing conclusions of methodologies while observing the same body of facts are not unfamiliar (as shown in [9], already mentioned) when studying scientific disagreements and, namely, peer disagreement, thus highlighting the importance of interpreting the pathways of assessor shifts [39]. Digital technologies, for the first time, are enabling the powerful delineation of scientific information included in scientific evaluations to connect any assessment to existing results and to achieve results, such as an information extraction pipeline for knowledge graphs [37], with neural algorithmic reasoning [40]. We found justifications for a relativistic approach to assessor shifts in GRAPHYP, measuring the differences between the differences, with possibilistic logic [41] applied to the explainable geometric graphs [42]. We consider this as the only mode of reasoning, allowing a neutral representation of the documented adversarial opinions, and delivering a neutral classification of contradictory documentary tracks to let any scientist form their own opinion, with an overview of the practices of peers.

• Modeling assessor shifts in a framework of possibilistic graphs

Web usage modeling techniques are familiar in multipurpose analyses [43]. Methodologies analyzing adversarial web usage, elaborated from former adversarial web searches [44], allow pattern mining in web search or identifying process scenario discovery from web log captures [45]. Possibilistic modeling makes it possible to represent information in contexts of uncertainty and imprecision, when “information is obtained from human observers or imprecise measuring instruments” [17].

It should be noted that, in GRAPHYP, we do not stop at enumeration without borders, but we propose interpretable possibilistic modeling. Interpretation is ‘possible’ from the specific geometry of GRAPHYP (for a complete description, see [1]), and the answer delivered to the user is not presented as a solution. It is neutral assistance to reasoning that takes the form of the ability to exhaustively represent all ‘possible’ adversarial documentary tracks, considering the measured parameters and their geometry of representation in a bi-partite symmetrical hypergraph. This hypergraph indeed connects all ‘possible’ cliques in the modeling, which represents all the feasible associations of users and items based on a predefined triplet [1] (pp. 271–272).

Referring to the theory of “decidable forms” of Edmund Husserl (notoriously introduced by Jacques Derrida [46]), we follow here the recent contributions to the geometry of graphs [1]. The geometry of graphs quickly became popular because of its ability to assist scientific reasoning with directly interpretable graphs. The superposition of geometric laws in the design of a graph and of the corresponding information network makes it possible to represent the activation functions of nodes and edges, thus facilitating direct interpretation. GRAPHYP has the originality of linking its graphs in a bipartite symmetrical hypergraph (adversarial cliques: see [1]), which gives its expressive power to the neutral modeling of assessor shifts, which are isomorphic in our representation, with the analytic advantages of three-layer knowledge hypergraphs [47].

• Reasoning on assessor shifts from data logs of documentary tracks

Decidable information is the foundation of our modeling of assessor shifts, which represents the human assessments expressed in documentary track choices. Relevance judgments can differ, as a document is relevant to a specified search request and an assessor’s conceptions of a request [8]. This translates into the fact that assessments are “relatively relevant”, which provides confirmation, through the assessor shift observation, to the geometry of the GRAPHYP model. The theory of evolutionary games on graphs that optimizes behavior in adversarial environments contains similar premises [48]. Finally, the research question to be solved is, “how can a network be moved from one state to another state” [49], in conciliating causality and complexity in assessor shifts. This is also the case for science as a whole and in all its documented contributions, insofar as undiscovered information is the goal of its approach and where it strives to place it in the context of already known information (theories and facts recorded by predecessors). Among the issues we seek to address is the need to achieve modeling for adversarial positions with subgraph analytics (GR1), included in a hypergraph modeling of assessor shifts (GR2).

• A multiverse graph structure of search for research

In direct agreement with the geometry of GRAPHYP, the idea of a multiverse graph corresponds to assessor shifts that can be said to comprehensively identify the “possible worlds”, their compossibility and self-reference [50], founded on a clear methodology and fueled but not overtaken by idealistic references [51]. We therefore propose that our conceptual framework be given the name “multiverse graphs”, with the design of ‘possible’ solutions [41], opening the way to a multi-hop path to select “a best possible” choice. Methodologies of the multiverse analysis in machine learning have received well-identified applications in science analytics in various domains [52]. To our knowledge, this is the first application of the paradigm of Leibnizian scientific reasoning on the famous “best possible results” in the area of knowledge graphs, issued from a set of ‘possible’ options of choice (e.g., [53,54,55]).

Scheme 2. How can a network be moved from one state to another state? Background for assessor shift interpretable patterns.

Scheme 2. How can a network be moved from one state to another state? Background for assessor shift interpretable patterns.

3. Results: Retrieval Modeling in Research

We already mentioned that the assessor shift in document judgments can be described as “relatively relevant” [8]. We propose hereafter a model of this relativity in the document presentation and retrieval, adapted to the practices of scientific documentation in research activities. Section 3 is organized as follows:

• GRAPHYP modeling: main steps (summarizing [1]);
• Research methods and range of applications;
• Additional retrieval strategies toward modeling web usage;
• Search exploration and pattern discovery;
• Qualitative vs. quantitative features of measured assessor shifts;
• New explainable patterns for the search profile.

3.1. GRAPHYP Modeling: Main Steps

In GR1 [1], we showed how usage log data, in the form of ‘documentary tracks’ in assessment building, make it possible to determine distinct cognitive communities (called adversarial cliques) within sub-graphs; we established a typology of these documentary tracks through triplets of measurements (intensity, variety and attention) to describe the potential approaches to a research question (see Scheme 3). Our geometric hypergraph GRAPHYP assists interpretation as a classifier, with possibilistic graphical modeling. GRAPHYP shows a complete representation of all the typical intermediary situations between two limits of triplets of nodes, which gives us a tool for the classification of observed search sessions in documentary tracks.

Table 1 provides a summary of GRAPHYP’s main steps, following [1].

Table 1. GRAPHYP’s methodology: key steps. More details are given in Appendix A.

A. Definition of documentary tracks: We note N, the number of users in a session log, and K, the number of documents read; we note α/β, the ratio of their average values over a series of different sessions. α/β is an “expression of stability/disruption of behaviors” 1. It expresses the degree of attention emerging in cliques of the cognitive community’s documentary practice and makes it possible to measure the ‘stability’ or ‘disruption’ of the behaviors of users over the considered search sessions. Therefore, the ‘attention’ parameter contributes to measured changes in the dynamic of the search of documents met on documentary tracks.   B. Web usage classification of documentary tracks: Documentary tracks are being defined and measured with min and max values, and web usage logs find their explainable integration in the proposed modeling. This implies the classification of all non-contradictory solutions of the triplet parameters (N or K must be min or max and cannot be logically combined) with reference to their mean values measured for the whole sample (see Figure 1 in [1]).   C. User’s documentary track positioning: According to A and B, users of GRAPHYP can localize identifiable types of classified documentary tracks on a research question and, with the help of the modeling, they can localize the interpreted selection of documents that the laboratory has realized, compared to other typologized practices on the same research question.

¹ The ratio α/β is “calculated from a value of normalization, expressed from the mean value of N/K. We can consider a fraction α/β where α is the numerator calculated from N mean value and β is the denominator derived from the mean value of K. This fraction α/β will vary, consequently, with any recorded group of reader and article values. (An alternative procedure could be to note α the coefficient of increase of N and β the coefficient of increase of K when Q varies by one unit when an additional query on the same search is recorded.)”

Table 1. GRAPHYP’s methodology: key steps. More details are given in Appendix A.

A. Definition of documentary tracks: We note N, the number of users in a session log, and K, the number of documents read; we note α/β, the ratio of their average values over a series of different sessions. α/β is an “expression of stability/disruption of behaviors” 1. It expresses the degree of attention emerging in cliques of the cognitive community’s documentary practice and makes it possible to measure the ‘stability’ or ‘disruption’ of the behaviors of users over the considered search sessions. Therefore, the ‘attention’ parameter contributes to measured changes in the dynamic of the search of documents met on documentary tracks.   B. Web usage classification of documentary tracks: Documentary tracks are being defined and measured with min and max values, and web usage logs find their explainable integration in the proposed modeling. This implies the classification of all non-contradictory solutions of the triplet parameters (N or K must be min or max and cannot be logically combined) with reference to their mean values measured for the whole sample (see Figure 1 in [1]).   C. User’s documentary track positioning: According to A and B, users of GRAPHYP can localize identifiable types of classified documentary tracks on a research question and, with the help of the modeling, they can localize the interpreted selection of documents that the laboratory has realized, compared to other typologized practices on the same research question.

¹ The ratio α/β is “calculated from a value of normalization, expressed from the mean value of N/K. We can consider a fraction α/β where α is the numerator calculated from N mean value and β is the denominator derived from the mean value of K. This fraction α/β will vary, consequently, with any recorded group of reader and article values. (An alternative procedure could be to note α the coefficient of increase of N and β the coefficient of increase of K when Q varies by one unit when an additional query on the same search is recorded.)”

Scheme 3. GRAPHYP parameters used for measuring the variations in assessor shifts.

Scheme 3. GRAPHYP parameters used for measuring the variations in assessor shifts.

3.2. Research Methods and Range of Applications

The research methods are applied in the GRAPHYP modeling stages of GR2 use, as in the first stages (GR1), in which data from user logs of scientific research documentation are used (see GitHub link in GR1).

The main objective is to approach quantitatively and qualitatively the representation of the documentary choices made by the users on the items (in general, scientific articles) proposed for their common choice. For any documentary track, GRAPHYP is able to transmit its comparative position and thus provides researchers with data that can be used to

• position their own assessor shifts on the same research question;
• appreciate the conditions under which a given documentary track is selected (for example, many users, few items and an unbalanced ratio of users to items, relative to the sample mean).

We first tested this method with data that were presented first in Paper 1 [1] (document and item dynamics). In the following, as a complement, we include a synthetic appreciation of the different types of assessor shifts at various levels of measurement.

3.3. Application Scope: Additional Retrieval Strategies toward Modeling Web Usage

Assessor shift modeling with GRAPHYP opens the way to the management of a wide range of retrieval strategies. We propose hereafter a first classification of these new possible domains.

The literature on search experiences often suggests the need to supplement a keyword, not only with additional keywords, but also with retrieval strategies on the elicitation of data discovery contexts and with the construction of a collection of tests for assessing search results. Task models have also been implemented for the exploitation of usability test logs, which contribute significantly to web analytics [56]. Modeling web usage in the search for scientific concepts becomes more attractive as the approaches diversify; a recent survey underlined the growing impact of new metrics, such as “atypicality” and “disruption”, which highlight the adversarial motivations possibly carried by similar keywords or, conversely, the contradictory modes of retrieval by which a similar concept can be revealed. “Science is built on a scholarly consensus that shifts with time” [57]; it is clear that innovative contributions emerge at borders between old categorizations, where discovery traces its paths.

Modeling web usage is a prerequisite for modeling retrieval in research. Items and corpuses are considered successively below.

• Items

The technologies of web usage mining, as well as examples of applications (Section 2.1.1), illustrate the large range of technical opportunities for capturing log data in research analytics. Items must be adapted to the dimension of the involved cognitive community, which can be configured on different scales (from small to very large research infrastructure; see examples in Section 3.4).

GRAPHYP models search experiences on various scales of web usage, among which we can list three indicative levels of search strategies that can benefit from the observation of changes in relevance judgments and of the retrieval of assessor shifts:

–

First level: search optimization (search off the beaten track, search with a better method, search with better vocabulary…);

–

Second level: graph completion of documentation (Is a theory fully documented? In what unknown direction can new knowledge be expanded, and what are the best pathways?);

–

Third level: adversarial search experiences and adversarial theories (What are the items of correspondence between documents and eligible theories? Are there overlapping boundaries between a documentary track and an eligible theory?).

• Corpuses

The corpus’s size and scope are an open question that depends on the research question and on the degree of its semantic evolution over time, as well as the composition of cognitive communities. We can only say that the requirements are those of the already classic methodologies of text and data mining in research activities. This question must be answered with the relevant collegiality of scientists.

3.4. Assessor Shift Modeling: A Grid for Usability Test Logs

Starting from the modeling features of GRAPHYP, we developed a first analytic functionality of the model, with assessor shift retrieval for classification and navigation tasks.

SKG GRAPHYP can be considered a “compass”, recording and indicating the direction of digital navigation in an environment of articles and other scientific resources, with a methodology referenced to exploring generative adversarial networks and adversarial training [58].

More generally, GRAPHYP can contribute positively to the management of scientific information by improving the usability of the web and by promoting the overall optimization of service uses. We observe that recommendations to mine usage logs have already been familiar for a long time for the composition of web services and are at the source of important methodological evolutions [59]. In this section, we show that GRAPHYP, by helping to model the design of web services, contributes to bringing more flexibility and accuracy to the benefit of users, according to the assessor shifts revealed by the analysis of usage logs in the context of documentary tracks.

Hereafter, we give illustrations of usability tests that can explain or plead in favor of an assessor’s shift, and then we present the mechanism of analysis proposed in GRAPHYP’s methodology.

• Usability test logs of documentary tracks

Recreating users’ experiences in log files is a familiar technology with well-known techniques for extending the usability of websites [60]. In addition to its many advantages (accuracy, pattern description, ubiquitous data, standards formats, timestamps, user clicks, etc.), usability test logs find new added value in the comparison of documentary tracks, as proposed by the global modeling of GRAPHYP. It adds a comparative structural understanding of the whole set of logs on a queried keyword. A few examples follow.

Usage-based and citation-based methods are often compared as differing methods for recommending scholarly research articles [27]; changes in relevance judgment and assessor shifts between documentary tracks can highlight the additional comparative features of the two methods. GRAPHYP’s comparative study of assessor shifts and corresponding documentary tracks can also help enable semantic analyses of user browsing patterns in a wide variety of contexts and circumstances for which a theory or methodology is under discussion [61].

• Usability test log analysis grids

Pairs of edges in the GRAPHYP knowledge graph always belong to a given node (here, a, b, c, d….). Each node designates a grid of proximities between partially neighboring nodes; we then exploit these partial proximities to identify paths that, by using links with next neighbor circulation paths, can

• help identify the optimal “route of preferences” of a user, according to relevant identified proximities of edges and nodes in a given documentary track;
• record any observed path of users that, during their past search sessions, have used a recorded method in their successive document assessments.

Figure 1 shows the genealogy that identifies search routes and documentary track characteristics, analyzing the different paths of discovery. More details about the key features of GRAPHYP are given in Appendix A.

In Figure 1, we represent possible circulation paths in the GRAPHYP grids.

The grid on the left part of Figure 1a, whose basic characteristics are described in [1], shows how a user can

–

characterize their position from available data, around one of the six typical nodes of GRAPHYP;

–

assess, depending on the triplet values of the summit corresponding to their documentary track, which elements that this situation creates differences with other choices that could be preferable.

From the grid on the right of Figure 1b, the user can

–

assess the steps by which a move can be executed from the current position, to any preferred one;

–

measure the distance between different positions and calculate the “length of the route” separating two possible courses.

By successively using the two graph grids, GRAPHYP makes it possible to navigate from the position of any documentary track to another, until reaching the optimal choice. Depending on the types of documentary tracks and routes of assessor shifts experienced by the users, it becomes possible to carry out a comparative analysis of the sets of logs, according to their content with reference to the parameters N, K and α/β.

3.5. Comparison of Assessor Shifts

In Figure 2, three documentary tracks (A, B, C) are represented, with, for each of them, the measured values of the three parameters of GRAPHYP’s triplet: N, $\alpha /\beta$ and K.

In this representation, the locations are proportional to the values and the respective positions of A, B and C for the three values of the triplet, which give a ‘real world’ representation of our methodology and which can support both qualitative and quantitative analyses.

3.5.1. A Tool for Qualitative Comparisons of Assessor Shifts

In the example illustrated by Figure 2, A, B and C clearly show exploitable differences that can be interpreted in context.

Assessment C is considered as a ‘convivial’ assessment with comfortable situations in N and K, and $\alpha /\beta$ is located on the median line of mean values.

Assessment A, conversely, represents a ‘minority’ position, with weak values for N and K and, with A2, a value for $\alpha /\beta$ that is out of the mean values.

Assessment B can be considered a ‘profitable’ one with N users that is measured in a slightly favorable position in the Nmax zone, combined with a great score on the quantity of items (value of K is not so far from the Kmax frontier). Compared to the mean line value of the sample, the $\alpha /\beta$ value of assessment B is located exactly on the line, with B2.

There are many possible declinations of this figure that remain to be implemented in further testing and applied research with typologies in the real world.

A few examples of log analytics extracted from Figure 2 are as follows:

–

Learning to rank query recommendations via semantic similarities [62] is in use for all situations in which query expansion and optimization are under discussion; a formal overview, as in Figure 2, illustrates the help in orienting incoming query strategies.

–

Users’ exploration of ontologies on the web: In the already mentioned study of NCBO BioPortal usage logs [13], the authors remarked that “very little is known about how exactly users search and explore ontologies” and “what kind of usage patterns or user group exist in the first place”. They concluded that deeper insight into user support are requested, and they proposed browsing behavior types (see Section 2).

An approach such as that shown in Figure 2 provides a basis for additional methodology.

3.5.2. Additional Tests on the Added Value of the “Attention” Parameter

Successful tests of the robustness and significance of node triplets integrated in the image of various documentary tracks have been realized from real world search history records on a test panel of 10 million search sessions, with the first prototype using access log files from OpenEdition.org platforms, filtered to eliminate requests for files others than papers, and with readers coming mostly directly from web search engines. Log files were collected by the web analytics platform Matomo (https://matomo.org/about/, accessed on 10 February 2023). The logs were filtered in order to remove bots and access to pages that were not papers or lists of papers. Users were identified from anonymous addresses (over time, it was possible that several identifiers corresponded to the same user), and sessions were estimated from the timestamps. Hence, a session is a sequence of content requests and content accessed in a limited time. About 3/4 of the sessions correspond to access to three articles at most, but some sessions are very long with access to more than 100 papers. For each session, we estimated the number of other sessions in which the same items were offered and, for each possible item, the number of times it was accessed. Detailed results and analyses are given in [1].

As a complement to Figure 2, we present in Figure 3 new results of the above-mentioned testing on the parameter of attention (α/β) measured in the model, and we show its ‘added value’ to assessor shifts metrics. We note that this third parameter expresses the stability/disruption of behaviors and allows addressing multiple factors bound to attention as an additional feature (mechanisms, types of co-attention, intra-attention and all documentary behaviors to measure a “real-valued hint” in information retrieval), measuring prominent features of assessor shift transition. We recall that the process is as follows: “First, we divide all the logs into blocks of sessions. Then, for each session, within each block, we estimate the values of K as the number of articles read by a reader. This makes it possible to estimate, in each block, the number of readers N who have read K articles and thus the mean values of the different N and K for this block. $\alpha$ and $\beta$ values as well as their ratio are calculated, for each block of sessions except the first one, from the mean values of N and K” [1].

Figure 3 shows the variations above and under the mean value of this parameter for a panel of 200 blocks of search sessions. For some sessions, the β value deviates significantly from the average. The same is true for the ratio α/β, which also deviates significantly from its average for about 10% of the sessions (values between −500 and +500). These measurements confirm that the majority of the studied sessions correspond to a consensual behavior, and a minority is interested, at least occasionally, either in very specific and rare requests or consults documents that are not considered often for similar queries. If we had access to the queries used as input, we could thus differentiate the uses and see behavioral or contextual breaks emerge.

The test shows the magnitude of the recorded variations and makes it possible to determine the type of each session, according to the associated values of N, K and $\alpha /\beta$. Thresholds are applied to identify the tendencies toward the min/max values of N and K and toward variations for $\frac{\alpha }{\beta }$ above and under the mean values of the parameters in the recorded sample.

Lastly, the sequence of types (a, b, c, d, e, f) gives us the “routes of search” between assessor shifts.

The variations and amplitudes of the $\alpha /\beta$ ratio around their mean allow for the detection of breaks in behavior and assessor shifts, as well as the type of search session. The link between these values and both the search objects and the possible (retrieved) documents could be the subject of specific studies. It could allow for the identification of innovative research objects (their disruptive character moves the exploration away from the usual research paths), subjects of strong controversy or, on the contrary, those that are consensual.

These results pave the way for a wide range of future services in the exploration of the documentary tracks actually followed by researchers. Further comparative developments of „bibliography and citation optimization” can be seen by examining the results of real-world researchers. This examination could reveal, on the nodes of GRAPHYP, “topologies of citations” varying according to the cases recorded on the same research question. The challenge could then be to observe how the citation of an author is favored, according to the different values of intensity, variety and attention.

3.5.3. Visualization of User Selections of URLs Consulted during the Search

Opening the way to further studies implies visualizing users’ selections of consulted URLs on any route, as illustrated by Figure 4. Each node (blue points) corresponds to a URL (a web page associated with a paper) for sessions in which multiple pages were viewed. An edge connects two nodes when the user has successively consulted the corresponding pages. Edges are represented either with blue lines (low frequency) or red lines (high frequency). A total of 4704 nodes and 4550 edges for 1532 sessions are represented here. Nodes are positioned using the Fruchterman–Reingold force-directed algorithm (Spring Layout of Network Python package). Some long routes appear, with most of them in the middle of the graph.

Figure 4 offers another representation of the GRAPHYP source data of Figure 3. It presents a more direct visualization of the selections made by the readers and their search strategies, initiating further interpretations. Some long roads (long blue lines with multiples nodes) can be seen close to the center of the graph. One interpretation is that they correspond to sessions in which users deviated from the average behavior, viewing fringe and otherwise unpopular papers.

This result is decisive for future documentary services. By zooming into the results of Figure 4, it becomes possible to explore ways of analyzing discovery, by comparing the strengths and weaknesses of each of the pathways. Depending on the respective values of our three parameters, we thus have a robust tool for deepening this domain at the heart of the representation of information on the discovery process, in its environment of competitive attempts. “Explainable Pathways” can be developed in a large range of services to the authors and reviewers, significantly augmenting the efficiency of documentation tasks and the readability of documents.

3.6. Possible New Patterns for Search Profile Retrieval

• A new conceptual venue of “modeling retrievability” (opportunities, behaviors)

Within the framework of “multimodal knowledge acquisition”, scholarly knowledge appears open to the innovations that we have discussed in the previous sections and that can find additional value in novel ways of modeling information retrieval. User modeling that combines access logs, page contents and semantics [63] may appear out of reach in the near future, but is nonetheless consistent with a mainstream attitude found in this research, namely to “represent each user by a set of features derived from the different data sources, where some feature values may be missing for some users”.

The need for new patterns is clearly identified in the Woods Hole Open Access Server [64] and is illustrated by text analyses aimed at “mapping research output” (e.g., [65]).

• Real world practices and elaboration of patterns

Practices of observation and the exploitation of usage logs benefit scientific workflows such as Embase and its parallel universe, Scopus [66], to evaluate the degree of overlap between the “offer” and “demand” of scientific documentation through “the search carried out for a systematic review investigating validated existing track and trigger scores”. It creates a new method for the ‘canonicalization’ of the dataset’s uses and modeling at W3C with ongoing discussion on new standards (https://www.w3.org/TR/rdf-canon/#intro-uses/, accessed on 14 February 2023).

• Routes of knowledge and rules of linkage

Hypertext and hyperlink analysis is an open highway for modeling scientific logs for identifying the “routes of knowledge” taken by any cognitive community. In this direction, we should mention that research topic identification, in relation to search activities [67] and new mutualization initiatives, such as COST, can be used to implement crowd-sourced complex search tasks. Thus, there is the need to configure dynamic search results in scientific topics that we propose to call “IMAGO”, with reference to the biological equivalent concept, to distinguish those search results from other meta-data existing around scientific contributions, such as Google Trends [68].

• Search patterns and the canonicalization of datasets

Search patterns in the literature in the context of scientific research are opportunities to develop original analyses of “non factoid” answers [69] that aim at “investigating research gaps and research objectives that are promising to obtain insight into future research directions” and for which GRAPHYP’s methodology provides a prototype. Multimodal search creates the need to improve observations of new interactions between research documentation and influence, such as from other sources [70].

Are there modellable behaviors in the search experience? How are those behaviors related to sense-making activities on web graphs (WebGraph: https://webgraph.di.unimi.it/, accessed on 4 March 2023)? Is scientific research characterized by specific regularities?

4. Discussion

To answer the research questions raised in the Introduction of this study, we showed, with GRAPHYP, how a novel scientific knowledge graph can assist scientific reasoning with complete representations of adversarial documentary tracks, serving as a generator (GR1) and a discriminator (GR2). We found assessor shift representation to be a key unit of data in this regard, as it signals critical changes in the path of reasoning, and we advocate for a system that provides users with indirect, intermediary and neutral information about the research topic they are exploring.

Readers are endemic producers of information about the texts they consult, through the traces they leave, including traces that constitute documentary routes. Computer science now has the ability to exploit this information to the benefit of new knowledge [71]; with the filtering of search records on any research question, we argue that it becomes possible to capture the characteristics of the acquisition of documented knowledge. For the first time in the history of scientific work, one can apprehend, on the scale of the global web, a new type of information about sense-making activities, the impacts of which are considerable on the research workflow. This information can be sketched out as the ‘route’ that all new knowledge follows, based on the data on the search experience and its connections between users and items. In addition, the directions and choices of predecessors—previous users of a given query—can be captured from user logs recording the items selected during search sessions. New methodologies involve tagging a lot of interpretable data resulting from search experiences; formerly hidden or concealed, those technologies are now more widely shared.

We outlined the rich commonality of logs for research, and, with GRAPHYP modeling, we proposed a prototype to model assessor shifts.

Web mining usage and its related analytics too often remain an unfertilized commonality of cognitive communities in research activities; on the web as well as in databases, modelized data on the use of the web remains a kind of “res nullius” of the Roman era [72]. Modeled uses of predecessors are too often ignored, hidden or hardly defined and ill-protected.

However, retrieving a relevant result on the web remains an ambiguous practice. On one hand, the progression toward the result is instantaneous and obvious, and on the other hand, the diversity of neighboring results is so rich that the search remains full of unanswered questions. As keywords provide access to a zoo of meanings within which we are compelled to express our preferences, we show in an inverted approach what are the adversarial features of the existing choices to allow everyone to benefit from the comprehensive mapping of possible results, either to find others features to reach the same place or other places that can express the same features.

This was the reason why we designed a neutral model with a full inverted matrix of choices from research, to search adversarial characteristics. We then discovered rich additional search strategies and tools implied by geometric graphs, which can be applied in web graph approaches.

New approaches toward the exploitation of web usage open a wide avenue toward opportunities that have not yet all been measured, but which could make it possible to open or extend research in many fields with existing technology, mature and fully adapted to the tasks of research. Moreover, compared to other domains, these technologies are developed with common natural language standards (dictionaries, definitions and categorizations).

However, these options leave many important questions open in fields such as graph alignment methodologies [73], and we are also aware of having designed a methodology that must be completed in further subgraph interpretations [74] that we plan to develop in further research.

5. Conclusions

We have shown, throughout our study, the urgency of resorting to the uses of the web to characterize the documentary routes and, thus, to assist the scientific reasoning of new readers wishing to compare the paths used by predecessors when exploring a research question. We have also shown that a new generation of knowledge graphs has the capacity to visualize, for the users of new services which remain to be developed, not only the paths of knowledge but also the changes and reversals of evaluations (assessor shifts) that generally characterize scientific controversies. We have introduced a new methodology for visualizing the URLs consulted during the search process. Finally, GRAPHYP plays its role as a ‘multiverse’ geometric graph in modeling possible search strategies answering research questions.

Moreover, the documentary tracks followed during the scientific process deserve comments on their status and their uses, as here arises a question of correlation between information and results, theoretical and practical, which is not addressed in this study and is limited to direct relationships between data and assessment.

We assume in this article that retrievable adversarial documentary tracks of scientific reasoning have the value of possible ‘knowledge contents’ about revised assessments. What the user of the SKG GRAPHYP can identify and consider as evidence is that it exists as a modeled resource regarding a corresponding yet unreached research question.

6. Further Works

Due to the strategic importance of integrity and privacy in those fields, which is a matter in itself and cannot take place in the present article, the authors are preparing an additional article devoted to this issue that opens new venues in ethical and legal aspects of scientific investigation under the conditions of web usage.

With the analysis of web usage logs, GRAPHYP can contribute to highlighting the integrity issues of digital categorizations in relation to the indexing of scientific contributions in large datasets; experiments are expected in this direction.

In another work in progress, we plan to explore how data management and visualization are used in organizations adopting Elastic Stack solutions to collect integrated data from different sources in one place, and then to visualize and analyze them in near real time.