Is Citation Count a Legitimate Indicator of Scientific Impact? A Case Study of Upper (1974) “The Unsuccessful Self-Treatment of a Case of Writer’s Block” and Its Derivatives

Abstract

The work by Upper (1974) was a blank paper. Multiple replication studies were published. This work examined the number of citations received by these papers, and manually checked the citing papers to determine why they made the citations. The Dimensions literature database was queried with the search string: (unsuccessful treatment writer’s block). The search yielded 14 articles, two of which were irrelevant and excluded. The 12 papers remained after screening included the original study by Upper (1974), nine replication studies, one review, and one meta-analysis. The original work received 43 citations, but related works had fewer than 10 citations each. One fourth of citations of Upper (1974) were being satiric on “nothing” or “precise” from papers dealing with unrelated concepts, and five citations were deemed erroneous/digressed. One citation was made to acknowledge the reviewer’s comments to Upper (1974), which did not involve Upper’s own ideas. This work exposed a scenario where there were limitations of using citation count as the only metric to gauge scientific impact of journal articles.

1. Introduction

The number of citations a paper receives has long been considered a key metric for measuring its scientific impact [1,2,3]. However, it is important to remember that citation counts may not always accurately reflect the quality or impact of a paper. For example, a paper that has been retracted due to scientific misconduct or errors may still continue to receive citations long after its retraction, particularly if it had already gained a large number of citations prior to its retraction [4]. Similarly, it was found that papers criticized by a complementary piece of commentary or a letter to the editors were generally much more likely to be among the most cited papers of a journal and more cited than non-commented (i.e., not criticized) papers [5]. In addition, the Matthew effect can skew citation counts, as researchers tend to selectively cite references that have already been highly cited, regardless of their actual scientific merit [6]. For instance, a large scale citation analysis indicated that a small number of elite scientists has received increasing citation shares over time and the citation inequality is on the rise in natural sciences, medical sciences, and agricultural sciences [7]. Furthermore, citation practices may be influenced by factors such as author reputation, gender, and institutional affiliations, which may not necessarily reflect the actual quality of the specific research being cited [8]. Another potential issue with citation metrics is that they do not take into account the context of each citation. For instance, a paper may be cited for reasons other than its scientific merit, such as to criticize its flawed methodology, or to highlight a minor point that is not central to the main findings of the paper. Citations could be made by papers from so-called predatory journals, papers with false authors, or papers deemed having research integrity issues; in these scenarios, some researchers might even wish to refuse the citations to distance themselves from the citers [9]. These limitations suggest that, while citation counts can be a useful tool for gauging the impact of a paper, they should be interpreted with caution and should not be relied upon as the single measure of scientific quality.

Instructional papers such as reporting guidelines might be published concurrently in multiple journals to enable maximum readership. One example was the CONSORT (CONsolidated Standards of Reporting Trials) 2010 guideline, simultaneously published in 10 journals. Within these 10 duplicate papers, it could be observed that some duplicates gained nearly as many as 2000 citations, whereas some duplicates received as few as 1 citation [10]. In addition, each citation may carry its own context, with some citations being “wrong” in the sense that the cited reference did not support the stance of citing paper or contain the information quoted by the citing paper [11,12]. These examples highlighted the disadvantage of making a judgment on a paper’s scientific impact based on citation count alone. Other research qualities, such as originality and societal value, should also be considered to have a more comprehensive evaluation [13].

The current study focused on an unusual set of papers that have attracted significant attention in the scientific community. The origin of this collection of papers can be tracked back to Upper’s paper published in 1974 [14] titled The Unsuccessful Self-Treatment of a Case of “Writer’s Block”. This paper was a satirical take on a case study, consisting of a blank page with only a title and a brief footnote saying that portions of the paper were not presented at a particular conference and reprints might be obtained from the sole author, Dennis Upper. This work should have no originality or any societal value if viewed in a serious sense. Despite its lack of content, the paper was accepted for publication and even received comments from a “Reviewer A”, who praised it as “concise” and recommended that it should be published without revision. Since its publication, Upper’s paper has become a well-known example of scientific satire, and has even spawned numerous replication studies. These studies range from direct imitations of the original paper, to more elaborate parodies that are built upon Upper’s paper. The proliferation of these replication studies raises interesting questions about the nature of scientific citation and its role in shaping scientific discourse. The current study aimed to identify the replication studies that have been inspired by Upper’s paper. Specifically, the current study sought to investigate the context in which these papers have been cited, and whether they have received similar numbers of citations as the original paper. By examining the citation patterns of these replication studies, it was hoped to shed light on the ways in which scientific satire can have an impact on the research literature, and have a reflection on the meaning of scientific citations.

2. Materials and Methods

On 30 December 2022, the Dimensions literature database (https://www.dimensions.ai/ accessed on 30 December 2022) was queried with the search string: (unsuccessful treatment writer’s block). This search string was referenced from a prior meta-analysis [15]. This query searched for any articles that mentioned all these four words in their title and/or abstract. Articles were excluded if they were not purely satirical in nature. The search resulted in 14 articles. One of them contained all four words separately in its abstract, but was totally irrelevant, being an essay on a musician, and thus excluded [16]. Another paper had a very long abstract that expressed the views of the author on the non-dual, a psychological or philosophical concept [17]. That paper mentioned the Upper paper [14] in its long abstract, and had a blank main text. Since this paper indeed contained serious informative content through its abstract, it was also excluded. Finally, 12 articles remained after screening: one being the original study by Upper [14], nine being replication studies, one being a review on this topic, and one being a meta-analysis on this topic [15]. Both the review and meta-analysis papers were purely satirical in nature.

Table 1. Summary of 12 papers related to unsuccessful treatment of writer’s block.

Reference	Paper Type	Key Points	Citations	AAS Score
Upper (1974) [14]	Original	The original work. Abstract, main text, and reference were all blank.	43	3825
Molloy (1983) [19]	Original	Abstract and main text were blank. Reference cited [14].	8	82
Hermann (1984) [20]	Original	No abstract. Main text contained two sentences, one finished and the other unfinished. Cited [14,19].	4	63
Olson (1984) [21]	Review	No abstract. Main text contained two sentences. It concluded that “the consistent outcome of these studies is [blank]”. Cited [14,19,20]	1	28
Skinner et al. (1985) [22]	Original	Abstract and main text were blank. Reference cited [14,19]. Footnotes explained that the treatment procedures for weekly 1-h sessions over 2 years were ineffective for the group.	3	22
Skinner and Perlini (1996) [23]	Original	Abstract mentioned that the treatment procedures for weekly 1-h sessions over 10 years were still ineffective for the group. Main text was blank. Cited [14,19,22].	1	18
Didden et al. (2007) [24]	Original	Abstract and main text were blank. Reference cited [14].	3	217
McLean and Thomas (2014) [15]	Meta-analysis	Abstract and main text looked genuine. Concluded that group treatment tended to be more unsuccessful, due to the partially successful self-treatment by [20]. Cited [14,19,20,21,22,23,24].	4	26
Artino Jr (2016) [25]	Original	Abstract and main text were blank. Reference cited [14].	2	55
Brodhead et al. (2019) [26]	Original	Abstract and main text were blank. Reference cited [14].	1	33
van Compernolle and Leontjev (2020) [27]	Original	Abstract had 4 sentences, claiming that COVID-19 has made the authors suffer from writer’s block and cannot be cured by the treatment proposed by Upper in 1974. Main text was blank. Cited [14]	0	0
Ampatzidis (2021) [28]	Original	Abstract and main text were blank. Reference cited [14,19,20,25]	0	6

lists the key points of these 12 papers. The citation count and Altmetric Attention Score (AAS) of these 12 papers were recorded from Dimensions. Papers citing them were read to categorize the rationale of such citations into the following groups: (1) paper replicating the work by Upper in 1974; (2) paper seriously dealing with writer’s block or related psychological concepts; (3) being satiric on the concepts of “having nothing” or “writing with precision” from paper dealing with unrelated concepts; (4) erroneous/digressed citation; (5) wrong entry by Dimensions/publisher; or (6) no access and thus could not be determined. Similar to a recent citation analysis study [18], these citation groups were originally devised for this study, and subsequently amended and finalized after evaluating the collected data.

3. Results

As expected, the original work by Upper (1974) received the highest number of citations (n = 43) and AAS score (3825). Subsequent related works had fewer than 10 citations each and did not seem to have strong impact to the academia (Table 1). Next, the citations received by Upper (1974) were thoroughly examined, and it was revealed that one fourth of its citations were made by its replication studies and review/meta-analysis of this group of studies (

Table 2. Breakdown of citations received by the 12 papers related to unsuccessful treatment of writer’s block.

Reference	Citation Type
	Type 1 (Replication)	Type 2 (Serious)	Type 3 (Satiric)	Type 4 (Irrelevant)	Type 5 (Wrong Entry)	Type 6 (No Access)
Upper (1974) [14]	11	11	12	5	3	1
Molloy (1983) [19]	6	1	1	0	0	0
Hermann (1984) [20]	3	1	0	0	0	0
Olson (1984) [21]	1	0	0	0	0	0
Skinner et al. (1985) [22]	2	0	1	0	0	0
Skinner and Perlini (1996) [23]	1	0	0	0	0	0
Didden et al. (2007) [24]	1	1	1	0	0	0
McLean and Thomas (2014) [15]	0	3	1	0	0	0
Artino Jr (2016) [25]	1	0	1	0	0	0
Brodhead et al. (2019) [26]	0	0	1	0	0	0
van Compernolle and Leontjev (2020) [27]	0	0	0	0	0	0
Ampatzidis (2021) [28]	0	0	0	0	0	0

Type 1: Citations from papers replicating the work by Upper in 1974. Type 2: citations from papers dealing seriously with writer’s block or related psychological concepts. Type 3: citations being satiric about the concepts of “having nothing” or “writing with precision” from papers dealing with unrelated concepts. Type 4: citations deemed erroneous/irrelevant. Type 5: wrong entry by Dimensions/publisher (actually no citation). Type 6: no access to the citing paper.

). Another fourth of its citations came from papers seriously dealing with writer’s block or related psychological concepts (

Table 3. Citations to Upper (1974) from papers dealing seriously with writer’s block or related psychological concepts.

Citing Sentence	Citing Paper
“Previous researchers have attempted to develop interventions based on strategies that writers have described as useful in overcoming writer’s block [citation here, 4 other citations]”. “Most of these solutions have been taken from interviews with successful novelists about their routines and writing habits [citation here, 3 other citations]”. “Unlike the samples in previous studies, which comprised mainly students [citation here, 2 other citations], the current sample consisted of semiprofessional and professional writers”.	[29]
“A famous and funny series of articles around this phenomenon concerns the unsuccessful self-treatment of writer’s block [citation here, 2 other citations]”.	[30]
“Writer’s block is a well-known phenomenon recognized by faculty [citation here, 1 other citation]”.	[31]
“Frankly, it is easy to catch a bit of writer’s block. Luckily, this has been dealt with in a number of articles throughout the years. Perhaps the most important one is [citation here] “Unsuccessful self-treatment of a case of ‘writer’s block’”. This is a truly unique article”.	[32]
“In my twenty-five years of conversations with colleagues on this topic, the best-known of articles follow the same format: They are humorous, consisting of a title about blocking (for example, “My latest attempt to overcome my writing block”) followed by a blank page [citation here]”.	[33]
“Consider Bergler’s (1950) likening of writing blocks to a rejection of mother’s milk, Ross’s (1985) claim that stymied writers can write while they sleep, or Upper’s (1974) [citation here] widely-known joke (a blank page) about blocking”.	[34]
“Because people often study phenomena that have personal meaning and application to themselves, there may be anxious writers studying writing anxiety [citation here, 2 other citations]”.	[35]
“So it is that when we discuss writing at all we tend to a nervous kind of humor, the single best-known article on writing in psychology is Upper’s (1974) [citation here] irreverent report of unsuccessfully treating a writing block”.	[36]
“Finally, Upper (1974) [citation here] has presented a humorous example of the unsuccessful self-treatment of a case of writer’s block”.	[37]
“This technique seemed perfectly suited for me because, like many of my colleagues [citation here] I had completed a number of research projects, but had not prepared them for publication”.	[38]
“However. little attention has been given to adults who have the more specific problem of writer’s block, except, for example, in a humorous fashion with an unsuccessful case [citation here]”. “No doubt, writer’s block has also affected many researchers [citation here]”. “Thus, a rapid technique (five sessions, including information gathering and an unsuccessful session without contingency-fulfillment) for attenuating the common problem of writer’s block [citation here, 1 other citation] was demonstrated with no evidence of recidivism”.	[39]

): many of these citations recognized that the Upper (1974) paper was satirical, but some of them seemed to cite it as a legitimate study without explicitly telling readers about its satire. Equally, another fourth of citations were being satiric on “nothing” or “precise” from papers dealing with unrelated concepts (

Table 4. Representative example citations to Upper (1974) being satiric about the concepts of “having nothing” or “writing with precision” from papers dealing with unrelated concepts.

Citing Sentence	Citing Paper
On “writing with precision”: “While most reports may not be as accurate and succinct as an example from the remote past [citation here], the comments of “Reviewer A” should be our goal: “I have not detected a single flaw in either design or writing style”.	[40]
On “nothing”: “The possibility that the Universe is indeed electrically neutral is appealing, perhaps, as it aligns with the hypothesis that the Universe is literally an intricate embodiment of nothing [citation here, 2 other citations]”.	[41]

). Five citations were deemed erroneous/irrelevant, and they were listed in

Table 5. Citations to Upper (1974) that were deemed irrelevant.

Citing Sentence	Citing Paper
“As the value and reliability of analyzing such a highly altered pedigree is questionable, no segregation analysis was performed [citation]”.	[42]
“R-Biopharm: 33/45 (73.3%; 95% CI: 58.4%, 84.2%)].…(citation)”.	[43]
“With this they confirmed the concept of gaining insight into an author’s intentions by X-rays [citation]”. (“They” referred to investigators that used micro x-ray fluorescence to examine an old book)	[44]
“Very little is known about the transcriptional regulation of gene expression in diatoms (citation)”.	[45]
“Since its initial publication, no comprehensive description [citation] of the ViennaRNA Package has appeared”.	[46]

, several of which were related to gene analysis.

4. Discussion

Submitting a blank manuscript seemed to be a bold action by the authors. Since each submission may have its own contextual background, in certain circumstances it is true that journal editors will appreciate the satirical nature of the submission and even allow so-called replication studies of a blank paper. However, if some journal editors cannot appreciate the submission, there may be a chance for the submitting authors to be dismissed by the editorial office of the journal for future submissions [47]. Besides, it would be very difficult for readers to comprehend the original intentions or underlying thoughts of the authors with a blank paper, if it was not purely satirical. In fact, some authors published commentary to supplement their papers that were blank or only contained a few words. Some notable examples were discussed below.

The first example was from the philosophy field. Habgood-Coote et al. published a paper titled Can a Good Philosophical Contribution Be Made Just by Asking a Question? with no abstract, main text, references, or supplementary files [48]. They published a commentary to explain their reasoning and intentions of this act [49]. The second example was from the law field. Jensen published a paper titled The Shortest Article in Law Review History with only three words, “This is it”, and two footnotes [50]. Years later, he also published a separate commentary to explain the why he wrote it and how some readers responded [51].

Of course, there could be some satirical publications. From the papers citing [14], there was a paper titled Nothing: A Review [52]. It had no abstract and main text, but the reference list cited several replication studies of [14], as well as two other blank papers: [53] from the linguistic field and [54] from the chemistry field.

Though Upper (1974) [14] was a blank paper, readers usually intended to perceive it as a lively example of writer’s block, as its title suggested. However, one paper cited it at the end of a sentence that said some investigators evaluated an author’s underlying intentions by checking an old manuscript with X-rays [44]. The idea of checking a manuscript with X-rays did not originate from Upper, but it was within the reviewer’s comments given to, and printed with, Upper (1974), which said:

“I have studied this manuscript very carefully with lemon juice and X-rays and have not detected a single flaw in either design or writing style”.

This citation brought out the issue of proper recognition to the idea contributor. In this case, the authors actually wanted to acknowledge the reviewer of Upper (1974), not Upper himself. A similar scenario was when the authors cited a review paper instead of the original articles, then they unfairly gave credit to the review authors for other researchers’ ideas [55]. However, currently there is no proper mechanism to cite reviewer comments, though more and more journals are now publishing papers alongside with open peer review reports.

Several citations of Upper (1974) were related to gene analysis and deemed irrelevant. It remained to be elucidated if the researchers who made these citations were affecting each other or made the citation independently. At first glance, it does not make sense to cite a blank paper to support any concept or justify any methodology concerning genetics. Searching the literature, one could identify a much more prominent and famous example of irrelevant citations: a three-page commentary on moth pheromone by Vickers (2017) [56]. It has attracted over 2300 citations according to Google Scholar. According to Vickers and bibliometricians, over 99% of citations of Vickers (2017) were off-topic or invalid [57]. One potential reason for the presence of so many invalid citations could be the use of a wrong DOI (truncated DOI) to search for Vickers (2017) in Google Scholar during the creation of a citation file for citation management software [57]. Hence, there could be many reasons for making irrelevant or invalid citations, but they could not be revealed without the input from the citing authors. A less prominent issue with citations is wrong entry by databases, thus over- or under-counting the number of citations. This issue is equally difficult to check in an automated manner on a large scale.

One issue with citation count, not evaluated in the current work, is the different coverage by different literature databases. In particular, the Web of Science platform hosts several databases, with the most popular pre-set combinations called Core Collection. Even though the name of the database seems straightforward and intuitive, the number of databases actually included in the Core Collection and their corresponding coverage timespans vary according to the subscription plan [58]. Therefore, results may not be directly comparable across studies if different databases and coverage timespans are used to generate dataset for the analysis. Meanwhile, nowadays, bibliometric analysis has become a popular paper type in the scientific literature, ranging from the analysis of a pre-defined dataset to the identification of top 100 most cited papers for a particular topic [59]. Especially for the latter type of papers, they may identify and cite the top 100 most cited papers for their field in the format of a table, but may not actually discuss or elaborate on the content of each of the top 100 papers. These citations may inflate the citation count of the highly cited papers with the context of Matthew effect, which means the highly cited papers tend to accumulate more citations due to their established recognition [60].

Another phenomenon observed from the results was that the replication studies of Upper (1974) were much less cited than the original work. A prior study also reported that the majority of articles citing the original studies did not cite relevant replication results from later studies [61]. In fact, even some of the latest replication studies of Upper (1974) neglected to cite their earlier counterparts, see [25,26,27].

The above examples highlighted the issue of citation analysis with simple citation counts. Upper (1974) and its derivative works were theoretically identical in terms of the message delivered to the readers and to the academic community. Though some of them were cited more than the others, it should not mean that some of them had a larger scientific impact. Instead of simply counting the number of citations to determine the impact of the cited references, it was more informative to consider the citing behavior, or the context of the citations. Readers are directed to a comprehensive review by [62]. For example, a citation could be made by professional motivations (contextually need to be cited) versus connectional motivations (social connections or personal benefits) [63]. Such a kind of categorization/analysis was not possible without making direct queries to the authors. Another categorization method focused on the citation context from the text, such as historical background, description of relevant work, use of established methodology, and/or mathematical equations [64,65]. A more intuitive categorization method was to classify citations into supportive, negative, and others (neutral) [62]. This was easier to be performed manually or by artificial intelligence [66,67].

Since Upper (1974) was a blank paper, none of the above categorization methods should be applicable. Still, the citation sentences could be examined to determine the context of the citation and if erroneous citations could be found. It was found that 12% of the citations received by Upper (1974) came from irrelevant fields such as genetic studies. It was unsure if such citations were, indeed, made by the authors, or due to typesetting errors during article production. Notwithstanding, this finding seemed to be consistent with the estimation that only 20% of citers read the original paper [68], and that 13.3% of references listed at the end of published papers contained errors [69].

The current analysis of the citations of Upper (1974) demonstrates the limitations and potential pitfalls of using citation count as the only measure of scientific impact. One may argue that this is only an isolated incident, but it may actually serve as a valid reflection of a critical issue in the current academic landscape: the over-reliance on citation metrics can distort the perceived value of research and lead to unintended consequences. There were two reasons. First, citation metrics do not necessarily reflect the potential research or societal impact of the cited paper. In the case of Upper (1974), the humorous and satirical nature of the blank paper has garnered a significant number of citations. If researchers are not aware of this extreme case before, they may easily perceive it as a genuine case report based on its title. Second, even this satirical blank paper of Upper (1974) received unrelated or erroneous citations made to justify certain procedures or concepts in gene studies. Therefore, it is important to adopt more holistic and multi-dimensional evaluation metrics. These should consider the quality, reproducibility, and even relevance beyond academia such as impact on policy, customer service, and society, rather than solely relying on the quantity of citations [70].

In conclusion, the findings of this study have demonstrated that even a satirical blank paper could receive many citations. Even more, some of these citations were made to substantiate certain concepts or justify some procedures for gene studies. Hence, it is not always legitimate to rely on citation count as the single metric to evaluate the scientific impact of research works. Although this may be an extreme case or isolated incident, research performance will be better evaluated with a more holistic approach in the future. Apart from citation count, relevance beyond academia such as impact on policy, customer service, and society could be considered. Even for citation count itself, supplementary metrics in terms of citation context, such as ratio of citations that are supportive, negative, or neutral, could be exploited.