|
A blog from WHSLA (Wisconsin Health Sciences Library Association) featuring posts on medical and health science libraries, NLM, and learning opportunities for medical and health science librarians and library staff.
|
There has been an explosion of artificial intelligence (AI) tools over the past few years. A category of AI tool that has been getting some traction is tools that summarize individual articles. A few such tools include Elicit, SciSpace, Perplexity, and EndNote 2025's new Key Takeaway tool (however, there are many, many more out there!). Among other things, these generative AI tools provide brief, easily digestible summaries of an article.
While these article summarizers have been lauded for their efficiency, there have been some concerns relating to their accuracy. Additionally, with the black box nature of AI tools, it can be difficult to determine just how much of an article AI tools are "looking at" when generating high level summaries.
As someone who recently got access to EndNote 2025's Key Takeaway tool, I decided to play around (or, more aptly, clown around) with the tool. Using the text of an article I had published with JMLA, I systematically replaced different sections of the article with nonsense text to see if the Key Takeaway tool would pick up on the shenanigans. The "nonsense text" consisted of snippets of a fictitious study on identifying malicious clowns hiding within the general public, which I generated using Microsoft Copilot.
In terms of replacing individual parts of an article, one section at a time, with clown nonsense, I found each and every replaced section (i.e., title, abstract, introduction, methods, results, discussion, conclusion, and references), by itself, managed to fly under the radar in the Key Takeaway tool (i.e., no clown shenanigans detected).
I also tested a few section combinations. My most interesting finding was that I was able to fully replace the methods, results, and references sections of the article (resulting in 47% of the article being comprised of text about clowns) without EndNote 2025's Key Takeaway tool mentioning anything about clowns in its generated summary!
I tested this same PDF (i.e., with nonsense methods, results, and references) out in SciSpace, Perplexity, and Elicit, and the clown shenanigans remained undetected in their generated summaries, as well (note that I only tested the high level summaries, and not the summaries these tools generated for each individual section of the article).