A Framework for Online Public Health Debates: Some Design Elements for Visual Analytics Systems

Abstract

Nowadays, many people are deeply concerned about their physical well-being; as a result, they invest much time and effort investigating health-related topics. In response to this, many online websites and social media profiles have been created, resulting in a plethora of information on such topics. In a given topic, oftentimes, much of the information is conflicting, resulting in online camps that have different positions and arguments. We refer to the collection of all such positionings and entrenched camps on a topic as an online public health debate. The information people encounter regarding such debates can ultimately influence how they make decisions, what they believe, and how they act. Therefore, there is a need for public health stakeholders (i.e., people with a vested interest in public health issues) to be able to make sense of online debates quickly and accurately. In this paper, we present a framework-based approach for investigating online public health debates—a preliminary work that can be expanded upon. We first introduce the concept of online debate entities (ODEs), which is a generalization for those who participate in online debates (e.g., websites and Twitter profiles). We then present the framework ODIN (Online Debate entIty aNalyzer), in which we identify, define, and justify ODE attributes that we consider important for making sense of online debates. Next, we provide an overview of four online public health debates (vaccines, statins, cannabis, and dieting plans) using ODIN. Finally, we showcase four prototype visual analytics systems whose design elements are informed by the ODIN framework.

1. Introduction

The information that people encounter when examining online public health debates can influence how they come to make decisions, what they believe, and how they act [1,2]. The ability to quickly examine and make sense of online health debates can help the general public assess the balance of such debates. Beyond the general populace, other stakeholders, such as public health practitioners and policy makers, may also need to make sense of such online debates quickly and accurately. This ability is also necessary for those in positions of responsibility so as to safeguard the populace against misinformation and disinformation [3].

Currently, however, making sense of such online debates is not straightforward. This is the result of several factors. Here, we highlight four of these factors: (1) number of sources—there are many sources of information about each health issue; (2) distribution of sources—information sources are distributed across the Internet on different websites, including a multiplicity of social media platforms; (3) veracity of information—it is difficult to examine the veracity of information originating from different sources; and, (4) positioning of sources—it is not easy to immediately understand the sentiments expressed by different sources and how they position themselves.

Computational tools can help alleviate some of the difficulties encountered above [4,5,6]. A subset of computational tools that can help stakeholders make sense of complex information, such as the information encountered in public health debates, is visual analytics systems (VASes) [7,8]. By integrating data analytics, data visualizations, and human-data interaction, VASes can facilitate sense-making activities [7,9]. It is important for such systems to help stakeholders not only perform the necessary sense-making activities, but also be human-centered, fitting the needs of the users. Without a proper understanding of the structure and elements of online public health debates, however, designing and building of such systems would be a haphazard rather than systematic in approach. To create human-centered tools in healthcare, we must have appropriate frameworks [10].

To help inform the design elements of VASes that facilitate making sense of online public health debates, in this work, we present and discuss a framework that we have developed called ODIN (Online Debate entIty aNalyzer). It should be noted that this is a preliminary work that can be expanded upon. This framework is for generalizing online public health debates and is based on a construct, which we call Online Debate Entities (ODEs). ODEs are the various sources of information—that is, organizations and people with an online presence debating public health topics (e.g., websites, Twitter profiles, Facebook users, and/or Reddit users). ODIN helps with the analysis of various attributes of ODEs which are needed to permit stakeholders to quickly make sense of any online debate. In this framework, we identify and define seven attributes: presence, shared presence, geographic location, registrant, age, focus, and sentiments. Using four examples of online public health debates (vaccines, cannabis, statins, and dieting plans), we demonstrate how ODIN can be used not only for systematizing the analysis of ODEs, but also for helping inform the design elements of VASes that facilitate stakeholders’ investigations of other online public health debates.

The remainder of this paper is organized as follows. Section 2, Background, discusses information spaces, sense-making, VASes, and online public health debates. Section 3 presents ODIN and discusses how it helps with analyzing online debates. Section 4 describes the four online debate cases using ODIN. Section 5 demonstrates how ODIN-based VASes can be developed and how they facilitate making sense of online debates. Section 6 presents some conclusions and discusses the limitations of this approach and describes future work.

2. Background

This section discusses the background concepts and terminologies used in this paper. We begin with information spaces and sense-making. We follow this with a discussion on VASes, why they are important, what they are made of, and examples of how they can help users to complete sense-making activities. Finally, we go over online public health debates and examine 4 specific cases: vaccines, cannabis, statins, and dieting.

2.1. Information Spaces and Sense-Making

Information spaces are bodies of information that are thought of as having spatial characteristics [7]. Compared to the related concept of “data”, which refers to information that has already been discerned and recorded, an information space is a useful concept for visual analytics research because it allows the freedom to conceptualize unstructured or abstract information. For online debates, these concepts are important because they are unstructured information where data are not necessarily readily available [11,12,13,14]. Information spaces are made up of information items (e.g., entities, properties and relationships) that exist at various levels of granularity. Making sense of information spaces is an example of a cognitive activity.

Cognitive activities are part of everyday modern life. Cognitive activities that are information-intensive and involve intense human cognition can be further described as complex cognitive activities [15,16]. Complex cognitive activities have two distinct characteristics: (1) they require the use of complex psychological processes; and (2) they exist in the presence of complex conditions [17]. An example of a complex cognitive activity is making sense of online public health debates.

Sense-making is an activity in which people gradually develop mental models of an information space about which they have insufficient knowledge [18,19]. Sense-making often includes a set of tasks, some of which can include scanning the information space, selecting relevance of items, and examining items in more detail [9]. Sense-making can require people to take complex information and uncover meaning from it that otherwise could go unnoticed. Sense-making activities are often problems that are ill-structured and open ended [20]. This includes, for example, investigating online debates [11]. Sense-making involves users establishing goals, discovering an information space’s structure, and determining what questions to ask as well as how the answers to those questions should be organized [21]. A challenge when completing sense-making activities is that relevant information for completing required tasks is not always easy to access, stored in the proper format, or located in the correct locations [22]. Visual analytics systems have been developed to support users in sense-making activities in a variety of information spaces [23,24].

2.2. VASes

In today’s environment of big data, people can get lost in and overwhelmed by the voluminous data presented to them, and consequently struggle to decipher meaning [25]. With VASes that combine human insight with powerful data analytics, data visualizations and human–data interaction, users can be alleviated of some of these problems they face. VASes can enable stakeholders to make sense of data in ways that have never before been convenient or possible. “Just like the microscope, invented many centuries ago, allowed people to view and measure matter like never before, (visual) analytics is the modern equivalent to the microscope” [26]. VASes combine data analytics with interactive visualizations to synthesize, analyze, and facilitate high-level cognitive activities, like sense-making, that involve investigating data [27,28].

VASes are composed of three integrated components: (1) an analytics engine; (2) data visualizations; and (3) human–data interactions [7,29]. The analytics engine pre-processes and stores data (e.g., data cleaning and fusion), transforms it (e.g., normalization), and analyzes it (e.g., multi-dimensional scaling, emotion analysis) [30]. For analysis of the Internet, data can be collected using individually developed web scrapers or online resources such as such as Alexa (alexa.com), MOZ (moz.com), or Majestic (majestic.com) that do the scraping for you. Examples of how data visualizations in a VAS can be visual representations of the information derived from the analytics engine. Visualizations extend the capabilities of individuals to complete tasks by allowing them to analyze data in ways that would be difficult or impossible to do otherwise [29,31]. Human–data interaction is used in VASes to allow the user to control the data they see and the way the data are processed. Interaction in VASes supports users through distributing the workload between the user and the system during their exploration and analysis of the data [4,7,32].

Sense-making requires people to rapidly compare and contrast information items unhindered by information formats or locations [33], for which VASes can be particularly useful. The sense-making loop for VASes can help us understand the process that users go through using these systems (Figure 1) [19]. Data are analyzed and fed into the VAS. To make sense of the data, the user first perceives it through the visualization, and then gathers new insights from it by conducting interactions on the VAS. Based on the new knowledge they acquire, the user can then further analyze and explore the information in new ways by specifying the VAS to what they need to see, which vary depending on the tasks at hand. The loop repeats as the user completes sense-making tasks and generates further insights from the data.

2.3. Online Public Health Debates

The term “online debate” is used in this paper to refer to a concept or topic that is widely discussed on the Internet and has two or more differing points of view. While some online debates exist on a micro-level (i.e., instances of a topic being debated back and forth between two ODEs on a web forum) [34,35,36], in this paper we are focused on online debates that manifest themselves on a macro-level (i.e., prominent ODEs promoting information and viewpoints on various positions of a debate) [11,37,38].

Some well-documented examples of topics that are debated online include: gun control [39,40], vaccines [11,39,41,42], abortion [43,44], and climate change [45,46]. Online debates are important for researchers to investigate because they have been shown to influence the way people perceive an issue and have an impact on their real-world decisions [47,48,49]. People often rely on the Internet to gather the information they need to form their opinions; this practice extends to topics regarding what is best for their health.

Every online debate is an information space made up of information items, including ODEs and their attributes (discussed in Section 3). The ODEs that participate in a debate have contrasting views and opinions to one another. ODEs include websites or social media profiles (e.g., Twitter) of people, organizations, companies, and government agencies. Each of these ODEs have important characteristics that help position them within the information space. Visually representing this information is important for stakeholders to be able to make sense of it.

Next, we examine the background and applications of four example online public health debates. These debate topics include: vaccines [36,38,50], cannabis [51,52], statins [53,54], and dieting plans [55,56].

2.3.1. Vaccines

There is little debate within the medical community on the efficacy of vaccines. Some public health experts claim that there have been up to 103 million prevented cases of contagious diseases since 1924 as a result of vaccines [57]. Despite the documented successes of the practice, there are still some in the medical community who have concerns about the risks or side effects of vaccines, and question the practice of mandatory vaccination [58]. The most infamous example of this is in a 1998 article published in The Lancet (now redacted) by Andrew Wakefield that reported a connection between the measles, mumps, and rubella vaccine and developmental disorders in children [59]. The findings of this study still influence the discussion around vaccination today—Wakefield is a prominent figure in the anti-vaccine community. The debate about vaccines is lively within the general public and has become even more contentious over time. It is well documented that that has been a recent rise in the number of unvaccinated children and the subsequent re-emergence of previously near eradicated diseases, such as measles [60,61]. Because of this trend, the anti-vaccination movement has been considered an emerging public health problem by some experts [62,63]. The World Health Organization even listed the rise of the anti-vaccination campaign as a top ten health emergency in 2019 [64].

2.3.2. Cannabis

Cannabis is the subject of debate both within the medical community and the general public. The cannabis debate occurs with regard to its medical applications, the movement towards legalization for recreational purposes, and the risks associated with its use both medically and recreationally. Cannabis has been examined by researchers for its potential medical uses in helping patients with conditions such as chronic pain [65], epilepsy [66], and Parkinson’s disease [67]. The drug has shown some promise to help with these conditions, and potentially has uses for some elusive illnesses like fibromyalgia [68] and insomnia [69]. While these beneficial applications of cannabis have been studied, there is also a great deal of research detailing the harm that cannabis can pose to users. Some reasons for concern include: addiction [70,71], infertility [72], and adverse effects on cognitive and motor functions [73]. Moreover, there are concerns that cannabis can act as a gateway into other more dangerous drugs like heroin or cocaine [74,75]. There is disunity in the medical community about the correct position on cannabis and whether the risks of the drug outweigh its potential benefits. Current literature about the potential applications or risks of the drug discusses the limited research there has been thus far as a result of the legal constraints, and calls for further research on the topic urgently [76].

2.3.3. Statins

The debate about cholesterol management and the use of statins is a public health issue within the medical community [77,78]. Statins are debated with regard to two important concepts: (1) who should receive the drug; and (2) how common are its side effects [79]. There are some medical professionals that believe statins should only be used to help manage cholesterol for secondary prevention, meaning that after an event such as a heart attack, statins are administered to a patient to help control or prevent a future event [80]. Others believe that statins can play a wider role in managing cholesterol via primary prevention, meaning that patients with high cholesterol, who are at risk for an event, take statins regularly to stop the event from happening. The reasons for these two diverging positions are varied. There is a range of concern about the effectiveness of statins in preventing events compared to the potential risks involved with the side effects that can be caused by taking the medication. A 2019 study found that 10% of patients who were eligible for statins declined the medication, citing side effects of the drug as their primary concern [81]. Some of the most commonly reported side effects of statins are muscle aches and pains, muscle breakdown (rhabdomyolysis), diabetes, and liver failure [82]. Patients’ concern regarding these side effects is believed to also influence how they might react to the drug itself. The term “nocebo” was recently used by the American Heart Association to explain a growing phenomenon of patients experiencing side effects as a result of their expectations of a side effect from the drug and not necessarily from the drug itself [82].

2.3.4. Dieting Plans

What and how much people eat is a critical part of their health, and dieting plan strategies for weight loss and health management are an important public health debate. There is a rich history informing the debate on dieting. Origins of the practice date back over 2000 years [83]. Specific trends and dieting plans themselves have evolved over time. The practice of dieting is expansive, and, depending on the diet and context, the can be harmful, ineffective, or beneficial [84]. There are many dieting plans promoted as the right choice for a healthy diet. For example, a vegan diet encourages people to abstain from eating any animal products (e.g., milk, eggs, meat). Not only is this encouraged for personal health and weight loss [85,86], but it has also been promoted as a way to combat climate change [87,88]. Another example is the Keto Diet, a popular diet that encourages people to eat less carbohydrates (e.g., bread, rice, pasta) and more fats (e.g., eggs, avocado, cheese). The diet has been promoted as a way to help people manage weight loss [89,90,91] as well as epilepsy in some situations [92,93]. Examples of other popular, competing dieting plans include: South Beach Diet [94], Atkins Diet [95], and Paleo Diet [96].

3. ODIN

In this section, we present a framework to analyze ODEs called ODIN (Online Debate entIty aNalyzer). Online debates occur on the general web and on social media platforms (e.g., Twitter) as a result of contrasting or conflicting information about a topic. They manifest themselves in the lack of agreement between ODEs about in what views on a topic or issue are correct or should be accepted. In a general sense, online debates are polarized around two or more opposing positions (pro- or anti- “phenomenon”). However, contained within each position are nuanced sub-positions, which share similarities with one another but may also be at odds with one another based on a variety of other factors.

Every online debate is an information space. Each ODE and its attributes are information items that help position it within the space. In this paper, we identify, define, and justify some attributes that we consider to be important for making sense of online debates. It is important to note that any characterization of attributes for ODEs is difficult to accomplish and is subjective in nature. Because of this, we root our analysis in the literature. There is a degree of redundancy in these attributes which leads us to the question of how can we search for and describe these attributes (there is no agreement on what the attributes are because this is a fairly new area of research). Our approach to this is by going through the literature and finding attributes about ODEs that researchers discuss. Of ten attributes that we have found in the literature, we present seven of them in ODIN, and we acknowledge that this list can be expanded as research in this area continues to grow.

The ten attributes are presence, shared presence, geographic location, registrant, age, focus, sentiments, frequency, funding, and veracity. Next, we demonstrate, by citing the literature, how online debates can be analyzed and measured. We then provide an example of how each attribute supports making sense of online debates. At the end of the section we provide Table 1, which summarizes the attributes and can be used as a reference for ODIN. In Section 4, we use ODIN and show how it can be applied in analyzing four online public health debates (vaccines, cannabis, statins, and dieting plans).

All ODEs have a presence, that is, the attention received by the ODE. The more presence an ODE has, the more popularity and/or authority it holds in a debate. Presence can be quantified by various metrics. On the general web, inlinks [39,97,98], website traffic [99,100], and website rankings from online resources such as Alexa (alexa.com, accessed on 2 November 2021), MOZ (moz.com, accessed on 2 November 2021), or Majestic (majestic.com, accessed on 2 November 2021) are all powerful tools for measuring presence. On Twitter, metrics like followers [101,102] or follower/following ratios [103,104] have been used as indicators of presence. With these metrics, it is possible to quickly make sense of the popularity and authority of ODEs. If ODEs have a strong presence, the information they are sharing about the online debate has greater reach than ODEs with a weaker presence.

Presence can also be examined to compare the similarity of ODEs. The more shared presence ODEs have, the more likely it is that they are similar to one another in their views and position. Shared presence has been determined using co-link (general web) [99,105] or co-follower (Twitter) [106,107] analyses. Using these metrics can help stakeholders make sense of an online debate’s structure quickly. If multiple ODEs share a lot of presence, they are likely to be similar to each other [108,109]. Because of this, stakeholders can use shared presence data to quickly detect ODEs that are similar to one another and identify potential debate position clusters. On the contrary, stakeholders can also use shared presence to see ODEs that are not similar to one another and identify ODEs that may appear on the surface to have one position but really have different positions at a deeper level.

The registrant is the person or organization that owns an ODE. The registrant is not always easy to determine based on the name and surface level appearance of an ODE; this makes uncovering the background of an ODE’s registrant an important task. Content analyses, along with a registrant classification system, can be used to collect these data [39,110]. Moreover, for the general web, services like WHOIS (https://lookup.icann.org/, accessed on 2 November 2021) that provide information on the registration of a website have been used to record this information [111,112]. Knowing the identity of an ODE registrant makes it easier to make sense of an online debate. Not only does it allow stakeholders to quickly see the obvious biases or conflict of interests, but the relationships between multiple ODEs can also become more transparent with this information. For example, a debate may have ODEs that share a position and have similar sentiments towards a variety of issues, but the registrant is the same person or organization. This would reveal that this position may be overrepresented.

The geographic location of an ODE describes where it is located in the real world. This can be measured by conducting a content analysis of the ODEs [11,113,114]. For the general web, services like WHOIS (https://lookup.icann.org/, accessed on 2 November 2021) that provide registration information for websites have been used [11,115,116], while on Twitter, location information of tweets are provided by the website itself [117,118]. Geographic location information is important because, when combined with the locations of other ODEs, they make it easy to make sense of the real-world dispersion of an online debate and its positions. For example, specific positions of an online debate may be spread out throughout the world or have regionalization. There could be important reasons for geographic clusters, such as differences in local public health policies or traditions.

The age of an ODE is the length of time that it has been registered. For the general web, the Internet Archive’s Wayback Machine (http://web.archive.org/, accessed on 2 November 2021) makes it possible to see how long a website has been active. On Twitter, the age of the profile is visible on its homepage. The age of ODEs can be used as a way to assess the evolution of an online debate [119,120,121,122]. If ODEs of one position are collectively older than ODEs of another, it can be an indication that the debate has evolved over time and the new position is becoming increasingly prominent.

ODEs share content that is necessary for stakeholders to examine to make sense of a debate. The focus of this content can indicate an ODE’s position in a debate. Frequently occurring words and phrases related to a debate reveals the focus of an ODE [11,123,124,125]. Natural language processing methods such as topic modelling [126,127] or content analysis [128,129] have also been used to identify the focus of ODEs. The focus is important to quickly make sense of the content shared by an ODE or a group of ODEs. For example, if various websites appear to have a similar position on a debate, but differ in their focus, they may share an overall position (anti- or pro-) but diverge from one another in their sub-position. If multiple ODEs share a common focus but differ in their sentiments about them, this would also be an indication that they are on opposite ends of a debate.

ODEs share sentiments with regard to various topics. These sentiments can take many forms and give stakeholders an important context to help make sense of a debate. Sentiments can be shared through the text, images, audio, and video elements of a website or social media profile. These sentiments can describe the overall tone of an ODE or the specific topics they discuss. ODEs’ text can be analyzed using content analysis [130,131] as well as various natural language processing tools to assess metrics like polarity (i.e., positive and negative sentiment) [132,133,134] or emotion (e.g., joy, fear, anger, sadness) [135,136,137]. With this information easily accessible, stakeholders can determine the position of an ODE quickly. This information can also help stakeholders determine the polarizing issues within a debate. For example, if multiple ODEs mention an issue frequently and some of them share negative sentiments while others share positive ones, it would be an indication of a polarizing issue within the debate.

Finally, while we present seven attributes of ODEs in this work, there are additional attributes that can be considered. We provide three additional attributes here to demonstrate how this list could be expanded in the future. ODEs share content at a different frequency. For example, some may share content on a daily basis while others may share content hourly. This can be especially important to consider in relation to and in evaluation of other attributes of an ODE [138,139]. Additionally, ODEs can be funded by different people and organizations. The attribute of funding can be important to consider when evaluating the intent behind sharing the content [140,141,142]. While funding is related to the registrant attribute, it can be different as one organization may operate and maintain an ODE, but another may fund it. ODEs can also have different levels of veracity, another attribute that can be important when making sense of online debates. As this is a subjective attribute, it is a particularly challenging one to quantify; nonetheless, it is important to consider. With recent focus on fact-checking [143,144], misinformation, disinformation, and similar concepts [145,146,147], there is a great deal of ongoing research that would contribute to coherent metrics for measuring and presenting this attribute.

Table 1. Attributes of ODEs.

Attribute	Definition	Measurement-General Web	Measurement–Twitter
Presence	Attention received by an ODE, which indicates its popularity and authority.	- Inlinks - Website traffic - Website ranking	- Followers - Follower/Following Ratio
Shared Presence	Presence various ODEs share with one another	- Co-Link Analysis	- Co-Follower analysis
Registrant	Person and/or organization that registered an ODE	- Registration information - Content analysis	- Profile registration name - Content analysis
Geographic Location	Geographic location of an ODE’s registration	- Registration information - Content analysis	- Profile registration location - Content analysis
Age	Time since ODE was created	- Internet Archive	- Reported creation date
Focus	Frequently mentioned topics and concepts of an ODE	- Word/term frequency - Topic Modelling tools - Content analysis	- Word/term frequency - Topic Modelling tools - Content analysis
Sentiments	Feelings, attitudes, and emotions of an ODE’s text and multimedia content	- NLP tools - Content analysis	- NLP tools - Content analysis

Table 1. Attributes of ODEs.

Attribute	Definition	Measurement-General Web	Measurement–Twitter
Presence	Attention received by an ODE, which indicates its popularity and authority.	- Inlinks - Website traffic - Website ranking	- Followers - Follower/Following Ratio
Shared Presence	Presence various ODEs share with one another	- Co-Link Analysis	- Co-Follower analysis
Registrant	Person and/or organization that registered an ODE	- Registration information - Content analysis	- Profile registration name - Content analysis
Geographic Location	Geographic location of an ODE’s registration	- Registration information - Content analysis	- Profile registration location - Content analysis
Age	Time since ODE was created	- Internet Archive	- Reported creation date
Focus	Frequently mentioned topics and concepts of an ODE	- Word/term frequency - Topic Modelling tools - Content analysis	- Word/term frequency - Topic Modelling tools - Content analysis
Sentiments	Feelings, attitudes, and emotions of an ODE’s text and multimedia content	- NLP tools - Content analysis	- NLP tools - Content analysis

4. Online Public Health Debates—Four Case Studies

In this section, we conduct an overview of four online debates: vaccines, cannabis, statins, and dieting plans. For each online debate, we discuss how it manifests itself online. We do this using seven attributes presented in ODIN, described in Section 3. We show how these components of the framework can be used to make sense of these online debates. It should be noted that in this section, we are not attempting to suggest that this is a thorough investigation into the complex content of these debates, or to conclude what the “right” position in each debate is. Rather, we are attempting to demonstrate that these debates exist online and can be made sense of through an examination of the attributes described in ODIN.

4.1. Vaccines

The online debate about vaccines consists of pro- and anti-vaccine ODEs. Within each position there are sub-positions that approach the debate differently [105]. Some anti-vaccine sub-positions include parents against vaccines (e.g., The Informed Parent and ThinkTwice Global Vacccine Institute), autism concern (e.g., Age of Autism), or general anti-vaccine (e.g., National Vaccine Information Center and Vaxxter). Some of the pro-vaccine sub-positions include flu vaccine focus (e.g., Families Fighting the Flu), vaccines for children (e.g., Shot of Prevention and Voices for Vaccines) and general pro-vaccine (e.g., Immunization Action Coalition and GAVI: The Vaccine Alliance).

Vaccine ODEs vary in their presence. Anti-vaccine ODEs like the National Vaccine Information Center (NVIC) and Age of Autism as well as pro-vaccine ODEs like Immunization Action Coalition or GAVI: The Vaccine Alliance have a strong presence. On the other hand, ODEs such as The Informed Parent (anti-vaccine) or Shot of Prevention (pro-vaccine) have a weak presence [11]. Based on the presence of data, it is possible to quickly identify ODEs that are popular and hold authority in the vaccine debate. One example of this is NVIC, which has both a strong presence in the debate on the general web [148] and Twitter (@NVICLoeDown has over 12,000 followers).

Moreover, the shared presence of vaccine ODEs quantifies which ODEs share a similar online presence, and which share little to no presence at all. The difference in shared presence is clear in the vaccine debate: ODEs that share either a pro- or anti-vaccine position will also share a presence with one another [105,148]. Stakeholders can use this information to help determine which ODEs belong to which positions. Within the anti- and pro-vaccine positions, other sub-positions can be identified using a shared presence as well. For example, two ODEs that have a very similar presence include Shot of Prevention and Voices for Vaccines [11,105]. Upon further examination, clearly both these websites follow the sub-position of pro-child vaccination.

Vaccine ODEs are owned and operated by various registrants, each with their own biases and motivations. For example, ThinkTwice Global Vaccine Institute’s registrant (an anti-vaccine ODE) is New Atlantean Press, a holistic health publisher. New Atlantean Press uses this ODE to promote anti-vaccine information, as well as to promote the sale of their books. Additionally, some vaccine ODEs are operated under the same registrant. Shot of Prevention and Vaccinate Your Family are both registered under the same registrant and likely share the same motivations and views on the debate. The registrant of vaccine ODEs can also be examined using classification systems that place the registrant into various categories. A 2017 study showed there was a significant difference in the classification of ODE registrants (those that linked to a known vaccine ODE) between anti- and pro-vaccine. The ODEs from the pro-vaccine sample were more likely to have registrants such as medical entities, while registrants of ODEs from the anti-vaccine sample were more likely to be individuals [38].

The geographic locations of vaccine ODEs reveal the distribution of the debate globally. Overall, there is a concentration of English-language vaccine ODEs in North America and Europe [11]. For example, The Informed Parent is an anti-vaccine ODE that is located in England and Age of Autism, another anti-vaccine ODE, is located in Virginia. There are also clusters of ODEs that share both their vaccine position and location in various regions of the world. An example of this is in North Western North America, where there is a concentration of anti-vaccine ODEs [11].

There has been online debate about the efficacy and morality of vaccines since near the beginning of the Internet. A 2017 study [38] found that the age of vaccine ODEs sampled were from 18 years to 6 months, with an average age of 8 years. Noticeably, there was no significant difference found between the mean ages of ODEs that were anti- and pro-vaccine. This indicates that both positions of the debate have been represented online for about equally as long.

Vaccine ODEs tend to focus on different topics depending on their debate position. Collectively, anti-vaccine ODEs have a strong focus on side effects (e.g., “autism”, “death”, “injury”) and vaccine choice (e.g., “exemptions”, “choice”, “required”) while pro-vaccine ODEs have a strong focus on vaccine-preventable diseases (e.g., “influenza”, “measles”, “flu”) and vaccine organizations (e.g., “Center for Disease Control”, “National Foundation for Infectious Diseases”). On an individual basis, the focus of an ODE reveals the specific components of the debate that an individual might be interested in. Sabin Vaccine Institute focuses on topics like “development” and “uptake” [11]. Upon further investigation, this is an appropriate indication of the ODE’s mission to create and spread new vaccines throughout the world.

The sentiments and emotions that ODEs share about vaccine topics, both in a general sense and about related concepts (e.g., specific vaccines, side effects), vary. Anti-vaccine ODEs often share negative sentiments about vaccines that use emotions like fear or anger [11]. Pro-vaccine ODEs, on the other hand, often share more positive sentiments about vaccines that use emotions like joy and pleasure when discussing vaccines [11]. Two examples which highlight this difference between the sentiments of the two sides of the debate are seen in Figure 2 and Figure 3. In Figure 2, news items on the anti-vaccine website Vaxxter are displayed. Phrases like “embroiled in fight with state’s laws”, “ineffective vaccines” or “vaccinated 6-year-old dies” all highlight the negative sentiments shared about vaccines by this ODE. On the other side of the debate, in Figure 3, the homepage of Voices for Vaccines is displayed. Phrases like “pro-vaccine”, “importance of on-time vaccination”, and “we want to protect all children” all highlight the positive sentiments shared about vaccination by this ODE. On a larger scale, the sentiments shared regarding vaccine topics are important for evaluating the content and views of the different sides of the debate.

4.2. Cannabis

The online debate about cannabis consists of ODEs with pro- and anti-cannabis views. There appear to be far more ODEs that are pro-cannabis, and within the pro-cannabis group there are a range of sub-positions including those focused on: cannabis legalization (e.g., NORML and Marijuana Policy Project), medical cannabis use (e.g., American Alliance for Medical Cannabis (AAMC), CannabisMD, Society of Cannabis Clinicians), cannabis lifestyle (e.g., High Times), or government cannabis administration (e.g., Ontario Cannabis Store (OCS)). On the other hand, the anti-cannabis position have ODEs that all show concern for the various effects of cannabis use on people and society (e.g., Citizens Against Legalizing Marijuana (CALM) and Smart Approaches to Marijuana (SAM)).

Cannabis ODEs vary in their presence. There appear to be more pro-cannabis ODEs in count, many of which seem to have a stronger presence than their anti-cannabis counterparts. Pro-cannabis ODEs like NORML and High Times have approximately 292,000 and 724,000 Twitter followers. Anti-cannabis ODEs, on the other hand like CALM and SAM have far fewer, with approximately 450 and 6000 Twitter followers. The shared presence of cannabis ODEs varies as well. Many of the anti-cannabis ODEs link to each other’s websites and have specific sections dedicated to “other organizations”. Moreover, government cannabis administration ODEs in Canada appear to have a strong shared presence with each other, as each province runs its own cannabis distribution network.

Cannabis ODEs are owned and operated by various registrants, each with their own biases and motivations. For example, there are some ODEs that are operated by non-profit organizations such as NORML, SAM, or AAMC. Some registrants may have biases that influence the content they share on the debate. High Times lifestyle magazine is operated by High Times Holding Company, whose stocks are publicly traded and connected with other cannabis retail companies. The content they share supports a pro-cannabis lifestyle but may also promote views that encourage people to purchase the products that this company sells.

The geographic locations of cannabis ODEs reveal the distribution of the debate globally. Overall, there appears to be a concentration of English-language cannabis ODEs in North America and Europe, with even more in areas where cannabis has been legalized like Canada and areas of the United States. NORML is a cannabis legalization ODE that is located in Washington D.C. while SAM, an anti-cannabis ODE, is located nearby in Virginia. There are also clusters of ODEs that share both their vaccine position and location in various regions of the world. An example of this is in California, where there appears to be a concentration of medical cannabis ODEs (e.g., Society of Cannabis Clinicians).

The debate around cannabis has existed online for a long time and appears to have become more prevalent over time. Older cannabis ODEs have been operating since the late 1990s (e.g., High Times, NORML), while newer ones have been created as recently as 2018 (e.g., OCS). This debate has evolved over time, with shifts seen in the discussion with regard to the forms of cannabis consumption that are promoted as well as the legal implications of cannabis laws [149,150,151]. As the trend of legalization in various regions of the world continues, new ODEs will likely emerge that discuss regional issues related to the use of cannabis.

Cannabis ODEs focus on various topics related to the debate depending on their position. The anti-cannabis ODEs appear to focus heavily on topics related to side-effects such as “mental illness”, “addiction”, and “lung cancer”. Within the pro-cannabis ODEs, the focus appears to have some variance depending on the sub-position of the ODE. The cannabis legalization ODEs appear to focus on topics such as “legalization”, “law”, “policy”, and “testing”. The medical cannabis use position appears to be focused on topics such as “healing”, “compassion”, as well as a variety of ailments that cannabis can treat (e.g.,“sleep”, “pain”, “depression”). The cannabis lifestyle position appears to be focused on topics such as “strands”, “shop”, and “culture”. Finally, government cannabis administrators appear to be focused on topics such as product types offered (e.g., “extracts”, “flower”, “edibles”) and topics related to consumption (e.g., “methods”, “safety”, and “learn”).

The sentiments and emotions shared by cannabis ODEs also vary depending on their position in the debate. The anti-cannabis ODEs appear to evoke fear and sadness when promoting their views. SAM has a page in their website titled “The Victims of Marijuana” where they discuss real life cases of individuals who have died in relation to cannabis use. In Figure 4, an example of a reported victim of cannabis is shown. In this example, the image used is of the deceased person’s father sitting in front of a memorial with a description in the text of how they died from heroin addiction. They connect the person’s cannabis use to them taking this harder drug. Among the pro-cannabis positions, many emotions and sentiments are shared. One pro-cannabis ODE (cannabis legalization) appears to evoke a more neutral, calm, and relaxed tone. For example, in Figure 5, the homepage for NORML is displayed. Here we see the content shared is mostly of a neutral tone, such as “NORML Responds” and “marijuana laws and penalties”. There are also phrases that appear to share a more pleasant tone, such as “help legalize” and “legalization has been a success”.

4.3. Statins

The online debate about statins consists of ODEs with pro-statin, anti-statin, and neutral views. There appears to be a greater amount of ODEs that are pro-statin than anti-statin. Within the pro-statin group, there also appears to be a variety of sub-positions including those focused on: non-profit medical organizations (e.g., American Heart Association, British Heart Association), general advocacy organizations (e.g., Take Cholesterol to Heart), and pharmaceutical companies (e.g., Kowa Pharmaceuticals). The neutral ODEs are government organizations such as Public Health Agency of Canada or the Center for Disease Control. While they do not warn of the risks of statins, like the anti-statin ODEs do, they do not recommend them as the solution for high cholesterol either. These ODEs share the latest information regarding statins for healthcare professionals. The anti-statin ODEs share the common viewpoint that taking statins for primary prevention is unadvisable. No ODEs were found that warned against statins for secondary prevention. Examples of anti-statin ODEs include Dr. Joseph Mercola, Alliance for Natural Health, The Health Examiner, and Dr. Aseem Malhorta.

The presence of statin ODEs ranges in both extremes. Non-profit medical organizations ODEs like the American Heart Association (291,000 Twitter followers) and British Heart Association (323,500 Twitter followers) have a very strong presence. The American Heart Association also has many associated ODEs that focus on particular portions of the populations (e.g., Go Red For Women) or regions (e.g., Midwest and Eastern). Furthermore, the anti-statin ODE Dr. Joseph Mercola (290,000 Twitter followers) has a strong online presence. There are also statin ODEs with a very weak presence, such as the anti-statin ODE The Health Examiner (22 Twitter followers). With regard to the shared presence, there appears to be connections between ODEs of the same position. For example, on Dr. Joseph Mercola’s website, there are several references, interviews, and links to another like-minded individual (Dr. Aseem Malhorta) who also operates an anti-statin ODE. All of the associated ODEs to the American Heart Association are linked on Twitter and often interact with one another. This is an indication that there is likely be a strong shared presence between these ODEs.

Statin ODEs are owned and operated by a variety of people and organizations. There are some ODEs that are operated by individual people, such as the two anti-statin ODEs run by Dr. Joseph Mercola and Dr. Aseem Malhorta. Other ODEs have registrants that are less straightforward in relation to their title that could have implications on their biases. For example, the ODE “Take Cholesterol to Heart” appears to be an independent advocacy organization for statins that is run or operated by Howie Mandel, the spokesperson for the organization. A deeper investigation into this ODE reveals that it is, in fact, owned by Kowa Pharmaceutical, a company that creates statins. This could bias what this organization recommends to the public regarding appropriate statin use, and is important to consider.

Statin ODEs are located in a variety of places all over the world. Many countries have their own non-profit heart health organizations that promote the use of statins. Some of these countries include the United States (the American Heart Association), Britain (British Heart Association), Canada (Heart and Stroke Foundation), and Australia (Australian Heart Foundation). In these countries, there are often government-run health organizations that share information on statins for the public. For example, in Canada there is the Public Health Agency of Canada while in the United States there is the Centre for Disease Control.

The discussion around statins has been ongoing since the drug’s commercial release in the 1980s [152]. As a result, the age range of statin ODEs is large, starting at the beginning of the Internet and lasting until today. Some of the oldest ODEs date back to the late 1990s (British Heart Foundation in 1999, Dr. Joseph Mercola in 1997) and early 2000s (American Heart Foundation in 2003). On the other hand, some statin ODEs have been created very recently, such as the advocacy group Take Cholesterol to Heart in 2017 or The Health Examiner in January 2020. There appears to be no slowdown in the discussion of the appropriate role of statins in cholesterol management.

Statin ODEs focus on various topics related to the debate, which vary depending on their position. Anti-statin ODEs appear to focus heavily on side effects (e.g., “neurological”, “muscle pain”, “liver damage”) and alternate methods for managing cholesterol (e.g., “dietary approach”, “omega-3 fatty acids”, “garlic”). Neutral ODEs appear to mostly discuss resources that people can use to find out further information on statins (e.g., “healthcare professional”, “drug product database”, “FDA”). Pro-Statin ODEs appear to be focused on the benefits of taking statins (e.g., “reduce risk”, “fight inflammation”) and, associated with this, the risks of having a high cholesterol (e.g., “narrowed arteries”, “heart attack”, “stroke”).

The sentiments and emotions shared by statin ODEs vary depending on their position in the debate. At one end of the debate, the anti-statin ODEs appear to evoke fear when it comes to statins. For example, displayed in Figure 6, an article published on Dr. Joseph Mercola’s website discusses the risks of taking statins associated with mental health issues. In the article, they discuss risks such as “depression”, “anxiety”, and “suicide”. The article further invokes fear in that it claims that “the scientific community is marked by a significant lack of interest in investigating the effects on personality”. On the other end of the debate, the pro-statin ODEs tend to invoke positive emotions like joy when discussing statins and fear when discussing the implications of high cholesterol. For example, in Figure 7, on the homepage of the ODE Take Cholesterol to Heart, there is a very pleasant picture of comedian Howie Mandel with text detailing how he likes inspiring individuals to take statins and that it is better than getting a laugh. On another page, discussing the risks of high cholesterol, there is another image of Mandel, this time looking concerned, with text detailing how many people have high cholesterol, and how it can be an invisible problem until the consequences are too late to manage.

4.4. Dieting Plans

The online debate about dieting plans is made up of many different positions. While there are some anti-dieting plans out there, the bulk of the debate appears to be among pro-dieting plans, specifically about which plans are best. Within the pro-diet group there are a range of sub-positions that promote specific diets, such as ketogenic diet (e.g., The Charlie Foundation, Matthew’s Friends, Keto Resources and My Keto Kitchen), vegan diet (e.g., Vegan, The Vegan Society, Vegan Action, The Vegetarian Resource Group, and Veg Source), south beach diet (e.g., South Beach Diet, South Beach Diet 101, and South Beach Diet Club), and Atkins diet (e.g., Atkins and Atkins Nutritionals). On the other side of the debate, the anti-diet position is aligned in their view that the problem is not about getting people to eat healthier, but rather that society needs to accept people for whatever their body types and eating habits are (e.g., Association for Size Diversity and Health, National Association to Advance Fat Acceptance, No Lose, and Fat!So?). In promoting body acceptance, these ODEs suggest that the risks of dieting plans (physical and mental) is more harmful on society than accepting obesity and different body types.

With regard to presence, it appears that the pro-dieting plan ODEs have a stronger presence than the anti-dieting plan ODEs. Most of the anti-dieting plan ODEs have a weak presence when compared to the pro-dieting plan ODEs. For example, the Association for Size Diversity and Health is one of the most present anti-dieting plan ODEs, with 3345 Twitter followers. One of the most present pro-dieting plan ODEs, on the other hand, is Vegan with over 250,000 Twitter followers. Among the pro-dieting plan positions and their ODEs, there is quite a lot of variability in their presence. Continuing with the vegan diet as an example, it appears that there are numerous ODEs with a strong presence. Aside from the one just mentioned, the Vegan Society is another example (230,100 Twitter followers). However, there are also many ODEs that promote this position that have a weaker presence. One example of this is the ODE Vegan Action, with 2691 Twitter followers. With regard to shared presence, there appears to be connections between ODEs of the same position or between ODEs of similar diet plans. For example, in The Charlie Foundation website, there is a section where they list friends of the organization. In this section, other keto diet ODEs, including Matthew’s Friends, are listed. This would indicate a strong shared presence between these organizations. The keto diet and south beach diet are very similar to one another in terms of their dieting plans and therefore the ODEs promoting these positions likely share a strong online presence. On the ODE South Beach Diet, for example, there are numerous references and links to keto diet material highlighting this.

Dieting plan ODEs are owned and operated by a variety of people and organizations. ODEs with a pro-dieting plan position and strong presence, such as Atkins and South Beach Diet, are often owned by organizations that created the diet and sell the related publications, materials, and food. Most dieting plan ODEs that are not directly connected to a business, such as the Association for Size Diversity and Health and National Association to Advance Fat Acceptance, are operated by non-profit organizations of the same name.

Dieting plan ODEs are located throughout the world, but there appears to be a large concentration of the English language ones in North America. ODEs such as Atkins (Denver, Colorado), South Beach Diet (Washington, Pennsylvania), and Matthews Friends (Surrey, BC) are all located in this region. While there are examples of ODEs located outside of North America, such as The Vegan Society (Birmingham, England), it appears to be less likely that this is the case.

The age of dieting plan ODEs is wide ranging. There have been ODEs for various dieting plans since the late 1990s (e.g., Atkins, The Vegan Society). However, specific dieting plans have come into existence or become more popular at various periods of time. By examining the age of the ODEs associated, the evolution of the debate about the best dieting plans is made clearer. The South Beach Diet, which was created in 2003, highlights this. The official ODE of this dieting plan’s creator (South Beach Diet) launched in 2003. Another unassociated but also popular ODE of the same position was then launched in 2004 (South Beach Diet 101) and as the diet became more popular, more ODEs were developed. The diet is still popular to this day, with new ODEs of this position created as recently as 2018 (South Beach Diet Club).

The focus of dieting plan ODEs varies among the different positions. Anti-dieting plan ODEs tend to focus on topics such as “fat acceptance” or “healthy at every size”. Pro-dieting plan ODEs have different focuses depending on the dieting plan requirements. For example, Atkins diet, ketogenic diet, and south beach diet ODEs all appear to focus heavily on “carbohydrates”, “low carb”, or “sugar levels”. This is because these diets promote reducing the amount of carbohydrates one eats and replacing it with options that will promote weight loss. However, within these positions there appear to be some variability between the ODEs. For example, the ketogenic diet has some ODEs that are focused almost entirely on the topic of “weight loss” or “body shaping” (e.g., Keto Resources, My Keto Kitchen). Other ketogenic diet ODEs (e.g., The Charlie Foundation) promote the keto diet for its applications in helping individuals with epilepsy and focus on different topics like “metabolic disorders” or “neurological conditions”. Vegan diet ODEs tend to have a very different focus from the other dieting plans mentioned. While the health benefits of the dieting plan are often mentioned, there is a strong focus typically on justice for animals and to combat climate change.

The sentiment of dieting plan ODEs varies for each position. For example, the anti-dieting plan ODEs appear to contain a mixture of negative sentiments (regarding dieting plans and the culture surrounding them) and positive sentiments (accepting people as they are). Figure 8 shows the Healthy at Every Size webpage of the Association for Size Diversity and Health. On this page, positive sentiments are shared with focuses on things such as “weight inclusivity” and “eating for well-being”. At the same time, there appear to be negative sentiments shared when referring to society’s status for health of individuals such as “judge” and “oppress”. Pro-dieting plan ODEs appear to share much more positive sentiments in general. These ODEs try to give a sense of encouragement and often share personal stories of weight loss or body change that are meant to encourage other individuals to feel positive about the possibility of losing weight or becoming healthy as well. Vegan diet ODEs appear to share different sentiments to the other diets mentioned. For example, Figure 9a shows the lifestyle page of the ODE Vegan, which is very positive. There is a happy picture of an individual and a dog and in the text and it refers to choosing a vegan diet as a “compassionate choice” to protect animals. In another section of the website, however (Figure 9b), negative sentiments (e.g., anger and sadness) are shared on animal cruelty. These sentiments are generated from the images of chickens being mass-killed by foam and the cruelty of the meat industry is described in the text with words such as “wasting” birds or referring to the foam as “chicken killing foam”.

5. ODIN-Based Design of Visual Analytics Systems

Making sense of online debates requires stakeholders to complete tasks that involve investigating the attributes outlined in ODIN (see Section 3). In this section, we discuss how ODIN-based VASes can enable stakeholders to complete these tasks to make sense of ODE attributes. Using four scenarios, one for each of the online public health debates described in Section 4, we demonstrate how ODIN can help inform the design elements of VASes that facilitate making sense of these online public health debates. To demonstrate this, in this section, we present four ODIN-based VASes: an existing tool (VINCENT) and three mock-up designs.

The existing ODIN-based VAS, VINCENT, was developed to help stakeholders make sense of the online vaccine debate [11,148]. In this system, we used seven of the attributes to help make sense of the debate. The design and empirical evaluation of VINCENT suggest that such systems can be very useful. An empirical user study of VINCENT showed that participants who used the system to complete tasks were quicker, more confident in their abilities, and more accurate in their responses than participants who did not have the system. Therefore, we use the same framework-based approach to design the other VASes.

The other three ODIN-based VASes are mock-up designs of potential tools for making sense of the online statin, cannabis and diet debates. It is important to note that these three VASes and scenarios are not based on real data but are rather conceptualizations of what the data could be. We use these conceptualizations to demonstrate how the integration of these attributes in VASes could help stakeholders make sense of the debates.

5.1. Vaccines

Figure 10 shows VINCENT, a VAS developed to help stakeholders make sense of the online debate about vaccines [11,148]. In VINCENT, vaccine ODEs have been identified, and attribute data including presence, shared presence, focus, geographic location, and sentiment have been integrated. VINCENT is composed of four main components: (1) the Presence Map (top left); (2) the Word Cloud (top right); (3) the Map of Website Locations (middle right); and (4) the Emotion Bar Charts (bottom). The Presence Map is a representation of the hyperlink or follower data analyzed from each website. Shared presence is determined by analyzing co-occurrences (links or followers) between ODEs using Multi-Dimensional Scaling (MDS) [105,153,154]. With this method of analysis, ODEs’ inlinks or followers are analyzed with a co-occurrence analysis and are visualized using MDS. This technique plots each ODE on the map as a circle positioned in proximity to other circles based on the similarity of the ODEs’ presence (i.e., how much shared presence they have). If the circles are plotted closer together, then the ODEs they represent share more presence, and vice versa. Each ODE’s circle on this map has been sized to reflect its individual presence. The bigger the circle, the greater presence it has. The Word Cloud is a representation of the 25 most common unique words that are related to the vaccine debate from each website. Words are sized based on the frequency with which they appeared on the website or group of websites. The bigger a word in the Word Cloud is, the more frequently it is used on the website. The Map of Website Locations shows a representation of the locations of each website on a world map. Similar to the online Presence Map, the Map of Website Locations uses circles to encode each website, but the circles have all been sized equally to help the user see the location of each website clearer. The Emotion Bar Charts represent positive and negative emotions for a selection of each website’s text about a set of vaccines, selected on the right side of the bar charts. The Emotion Bar Charts represent the negative (red) and positive (green) emotions detected by IBM’s Natural Language Processing API [135]. Each bar is made up of all the smaller rectangles (ODEs). The bar represents the overall detected emotion in the text of the complete set of ODEs. The width of each rectangle within each bar chart represents the degree of detected emotion in that ODE’s content. The wider the rectangle, the more the emotion is detected.

Scenario: A public health stakeholder suspects that there is a lot of anti-vaccine sentiment coming from North Western North America. They want to investigate this issue by identifying which part of North America has the highest concentration of anti-vaccine ODEs. If the North West does have the highest concentration of anti-vaccine ODEs, the stakeholder wants to know which vaccines in particular have the strongest negative emotions associated with them.

The stakeholder uses the VAS by first navigating to the Map of Website Locations and selecting the ODEs in the region they want to investigate. After highlighting the ODEs for each of the various regions of interest (North West, North East, South East, Mid-West, South West) and checking the position of the ODEs, they confirm that there are the highest concentration of anti-vaccine ODEs in the north west. The stakeholder then selects each of the listed vaccines and compares the Emotion Bar Charts. They find that the ODEs had the strongest negative emotions associated with the measles, mumps, and rubella vaccine. As the stakeholder continues to investigate the ODEs using the Word Cloud, they quickly find that some of the anti-vaccine ODEs in the north west, like Vaccination News and Vaccine Choice Canada, have a strong focus on the issue of autism. Given the popular belief among anti-vaccine groups that the MMR vaccine is linked to autism [35], the prevalence of this sub-position is notable.

Empirical Study: In an empirical study (see [148]), we gave study participants ten tasks, requiring them to make sense of the online vaccine debate. Thirty-four participants performed these tasks by investigating data from 37 vaccine-focused websites. Half the participants were given access to VINCENT, while the other half were not and were the control group. The ten tasks required participants to make sense of various elements of the set of websites, including online presence, shared online presence, geographic location, focus, emotion towards specific vaccines or vaccines in general, and/or a mixture of these. Study participants from both groups were subsequently interviewed. Examples of questions and issues discussed with interviewees were: how they went about completing specific tasks, what they meant by some of the feedback they provided, and how they would have performed on the tasks if they had been placed in the other group. Overall, the results of the study showed that VINCENT was a highly valuable resource for users, helping them make sense of the online vaccine debate much more effectively and faster than those without the system (e.g., users were able to compare websites similarities, identify emotional tone of websites, and locate websites with a specific focus). For more detail on this study and considerations for developing VASes for online health debates, the reader is referred to [148].

5.2. Cannabis

Figure 11 shows a VAS envisioned to help stakeholders make sense of the online debate about cannabis. In this VAS, cannabis ODEs have been identified and attribute data including presence, shared presence, focus, sentiment, age and registrant have been integrated. The VAS consists of three components: (1) the Selection Panel (top); (2) the Presence Map (bottom right); and (3) the Focus and Information Panel (bottom left). The Selection Panel allows the user to control how presence is evaluated, including the type (shared or individual), the area (general web or twitter), and the measurement method. Moreover, users can choose to filter the ODEs in the VAS using this panel. In the Presence Map, based on the presence type selected, either a tree map of individual presence (Figure 11) or a (MDS) map of shared presence (Figure 12 in Section 5.3) is populated. The VAS is set to individual presence for the example in this section and the tree map is displayed. The bigger the cell of the tree map, the more presence the ODE it represents has. Finally, the Focus and Information Panel allows the user to drill into specific ODEs by selecting them from the Presence Map. When an ODE is selected, its name, registrant, age, and position become immediately available. Stakeholders can then investigate the ODE’s sentiment regarding a particular focus by choosing a word or phrase. The system then analyzes the related content from the ODE and returns the emotions detected, as well as the strength of the ODE’s focus on the word or phrase.

Stakeholders investigate the debate with this VAS by first selecting the level that they want to analyze presence (individual or shared). Next, they choose the area of the Internet they want to analyze (general web or Twitter) and then set the measurement technique they want the system to use. Furthermore, in this panel, they determine whether or not they want to filter the ODEs and if so, how. The resulting tree map shows the presence of the ODEs based on the specifications. The cells of the tree map are organized and color-coordinated based on their position. Stakeholders can investigate the debate by selecting specific cells to reveal ODE information and by repeating this process with all the relevant ODEs to their investigation.

Scenario: A public health stakeholder has come across claims on social media that suggest using cannabis can help individuals who struggle with sleep. The stakeholder has investigated the literature on the subject and has found some evidence to support this claim. They now need to investigate whether this claim is shared widely on Twitter, and specifically among the medical cannabis ODEs. They use the VAS to examine the online debate on Twitter. In Figure 11a, the VAS has been set for individual presence on Twitter using follower/following metrics without any filter of position or ODE selected. The stakeholder continues their investigation by filtering the VAS, so only ODEs with a medical position are shown (Figure 11b). They also investigate by selecting the individual cells to reveal further information about the ODE.

In Figure 11b, the ODE that has been selected is one with a moderately strong presence called “The Medical Cannabis Community”. The stakeholder can quickly see in the information panel on the left the ODE’s Twitter handle, presence, position, registrant, and age. They can also input “sleep” into the focus and see the sentiments they have on the topic. In this case, they have found that the ODE has a moderately strong focus on the topic. Furthermore, they can see that overall, the tweets from the ODE with this focus have an excited and pleasant sentiment. By repeating this process with other ODEs of interest, both with the same position or not, the stakeholder can quickly see how the discussion of sleep in the online cannabis debate manifests itself.

5.3. Statins

Figure 12 shows a VAS envisioned to help stakeholders make sense of the online debate about statins. In this VAS, statin ODEs from Twitter have been identified and attribute data including age, position, registrant, presence, focus, and sentiments have been integrated. The VAS consists of four components: (1) the Focus Panel (top left); (2) the ODE Information Box (top right); (3) the Sentiment Map (middle); and (4) the timeline (bottom). The Focus Panel is made up of the focus input, where a user can specify words or phrases that they want to know more about. Furthermore, in this panel, the stakeholders can filter the ODEs to display a specified position. The ODE information box reveals information on individual ODEs as they are selected by the stakeholders in the Sentiment Map. This information includes the ODE’s Twitter handle, position, number of tweets, registrant, years active, and presence (follower/following ratio). The Sentiment Map displays all the ODEs that match the criteria specified in the Focus Panel within the time frame selected on the timeline. Each ODE is represented on the map with a circle. The placement of the circle is based on a sentiment analysis of tweets containing the specified focus (this method of sentiment analysis has been adopted from [155]). The timeline shows the years and months since the beginning of Twitter (2006) to the present. The yellow highlighted sections display the selected time period by the stakeholders.

Figure 12. VAS for Statins.

Figure 12. VAS for Statins.

Stakeholders investigate the debate using this VAS by first specifying the period they are interested in investigating in the timeline. They then choose the focus of the debate they wish to know more about and the position(s) of the ODEs that they want to include in their search. With the specifications made, the ODEs are scanned and those that match the criteria are then plotted onto the Sentiment Map based on the aggregation of the sentiment of their focus-related tweets. Stakeholders identify specific ODEs on the Sentiment Map for which they want to learn more about. By choosing a circle of an ODE on the map, they reveal its content in the ODE information box. When new data points are chosen or the date range is changed, the data in the system updates accordingly and the user can begin examining the new data presented.

Scenario: A public health stakeholder wants to make sense of what happened in the summer of 2017, when an event occurred that the stakeholder hypothesizes may have led to an uptick in hesitancy of patients choosing to take statins based on the fear of side-effects. They use the VAS in Figure 12 to help them make sense of the debate as it occurred on Twitter. They first specify the period of time they want to investigate (24–26 July 2017) and the focus (“side-effects”). The VAS then processes the data from the ODEs active during that time and outputs a Sentiment Map based on the content they shared related to the focus. The stakeholder then selects the data points of interest on the Sentiment Map and reveals further information about each ODE.

The stakeholder has come across a particular ODE (Dr. Joseph Mercola) that they want to know more about. The tweets analyzed have a nervous and unpleasant sentiment towards the focus of “side-effects”. By selecting the ODE, the stakeholder can see that its position is anti-statin, that it has been active on Twitter since 2008 and that it has a strong presence, as indicated by the follower/following ratio. In the time period they have selected, they published three tweets that contained reference to “side-effects”. They can also scroll through these tweets in the information box and evaluate them. By repeating this process through selecting other ODEs on the Sentiment Map, or by specifying another time period they are interested in, the stakeholder can start to make sense of how the debate was shaped on Twitter during this event.

5.4. Dieting Plans

Figure 13 shows a VAS envisioned to help stakeholders make sense of the online debate about dieting plans. In this VAS, dieting plan ODEs have been identified and the attribute data described in Section 5.2 have been integrated in a similar fashion. The VAS has the three main components: (1) Selection Panel (top); (2) the Presence Map (bottom right); and (3) the information and Focus Panel (bottom left). In this example, the VAS has been set to analyze shared rather than individual presence. The shared presence is analyzed using co-link analysis and MDS, the same method described in Section 5.1. To make the information easier to interpret, the position of the ODEs has also been integrated into the MDS map by coloring the circles according to position, and then highlighting the areas on the map that contain ODEs with those positions.

Stakeholders investigate the debate using this VAS by first selecting (in the Selection Panel) the area of the Internet, measurement level of co-occurrence analysis (page or domain), and position they want to investigate. For co-occurrence analysis, depending on the level of measurement the stakeholders select [105], the ODEs’ inlinks are analyzed and plotted accordingly in the Presence Map. As in Figure 13, stakeholders can investigate the ODEs by selecting circles on the map to reveal details in the Focus and Information Panel.

Scenario: A public health stakeholder is comparing diet positions to one another. Specifically, they are interested in identifying (1) diet communities that have greater similarity to each other; and (2) how those diet communities view one another. They use the VAS to investigate the online debate on the general web. In Figure 13, the VAS has been set for shared presence on the general web using page-level co-link analysis without any filter of position. The stakeholder begins to select individual cells of interest to reveal further information about the ODE in the Focus and Information Panel. They notice that South Beach Diet and Keto Diet are close to one another on the map, which makes sense since both dieting plans promote reducing carbs as an important component of the diet.

In Figure 13, the stakeholder selects the ODE “Keto Resource”. The stakeholder can quickly identify, in the Focus and Information Panel, the ODE’s Twitter handle, presence, position, registrant, and age. In this case, the stakeholder is interested in the “South Beach”. The stakeholder finds that the selected ODE has a somewhat strong focus on the phrase, indicating the dieting plans are likely associated with one another. As well, they can see that, overall, the related content on the website has an excited and pleasant sentiment. By repeating this process with other ODEs of interest, both with the same position or not, the stakeholder can quickly see how the discussion of sleep in the online diet debate manifests itself.

6. Discussion

In this work, we have presented a framework for analyzing ODEs called ODIN. This framework consists of seven ODE attributes, discussed in detail in Section 3. For each attribute, we have described the types of data collection and analysis methods that can be used. We then demonstrated how the framework can be applied in the analysis of four online public health debates (vaccines, cannabis, statins, and dieting plans).

We used the analyses to guide the development of four framework-based VASes. These VASes were designed to demonstrate that one can use the framework presented to approach their design elements. Online debates are complex information spaces and making sense of them can be challenging. Without the assistance of VASes, it is difficult, or in some cases impossible, to quickly assess the various attributes of these debates. There are many ways in which ODIN-based VASes can implement attributes from the framework, as we have discussed in this paper. One of the VASes presented was VINCENT, a VAS that has already been developed and been shown to be very useful for stakeholders [148]. Moreover, we described three other mock-ups of ODIN-based VASes to show other potential applications of the framework for analyzing online public health debate.

The framework presented in this work is meant to be generalizable so that it can apply to other debates, either related to public health or in other areas (e.g., political debates). As long as the discussion of interest consists of multiple opposing positions, has ODEs that can be identified, and has data about the attributes that can be analyzed, the framework can be implemented.

Limitations and Future Research

While we have included seven ODE attributes in ODIN that we consider to be important for making sense of online debates, this list can likely be expanded with other ODE attributes. Examining other online debates (both about public health topics or other topical areas of debate) and including a large range of social media platforms such as Facebook or Reddit would likely result in the inclusion of other attributes.

We have focused our discussion of online debates on two specific manifestations of the Internet: the general web and Twitter. However, there are many other social media platforms (e.g., Facebook, Instagram, Reddit) in which these debates also occur. Each area of the web has unique qualities and uses different labels and structures for the various attributes described in ODIN. While we have not discussed each of these social media platforms in detail individually, the attributes described are generalizable so that they can be applied or adapted to these other social media websites. The application of such attributes to such alternate social media websites could be a fruitful area of investigation.

It is also important to note that the data that researchers can access is constantly changing. Considerations of what the best data sources are, as well as how data may be manipulated is important. For example, Twitter follower data can be manipulated through users buying followers [156]. While methods exist (e.g., follower/following ratio) that attempt to expose those that artificially boost their social media statistics, improved approaches for identification of profiles with inflated numbers are needed to gauge a more accurate view of presence.

In this work, we have demonstrated that the ODIN framework can help to inform the design elements of visual analytics systems in a more systematic way. However, this paper does not map the attributes to specific visual design features. In future research, clear descriptive guidelines can be developed to determine how to map from the ODIN framework to visual analytics systems to optimize such investigations.

Finally, there is also a need for the examination of alternate methods of data analytics, data visualizations, and human–data interactions for ODIN-based VASes. For example, in this paper we have discussed and implemented MDS for integrating shared online presence in VASes. However, other methods of social network analysis could have been utilized. As more VASes are created to meet the challenges of making sense of online public health debates, best practices for their design can be catalogued.