Tina Vrieler

The library services at Western Norway University of Applied Sciences invited me to participate in a panel on AI research and trust (see picture above and below) on Wednesday, the 23rd of September. I was very excited to contribute to this very important issue, which also gave me good reasons to read and study more on this topic. To keep this post short, I will summarize some of the key messages I brought forward at the panel. Many of them are inspired by my own reflections and experiences, as well as by the excellent paper by Messeri, L. and Crockett, M. J.

1. I believe anyone doing cognitively intensive jobs, such as researchers, is particularly vulnerable to overuse or overtrusting GenAI because of the way the research process and practice are structured. We are keen to trust information that comes in neat packages, and that looks like expert opinions, something, as we know, GenAI is very good at. GenAI appeals to researchers because research often carries an implicit ideal of objective knowledge; human shortcomings are viewed as a “threat” to scientific knowledge. On top of that, research work comes with precarities such as heavy workloads, tight deadlines, the rush to publish, and difficulty securing long-term employment contracts. Together, these pressures can push researchers to take shortcuts and use GenAI in disadvantageous ways.
2. Messeri and Crockett [1] warn of the illusion of understanding and depth if we overrely on GenAI. This is more likely when we use GenAI for things we know little about or have little expertise in. This is not research-related, but not long ago, I asked Google for all the names of PhD students who are, or used to be, part of UpCERG (my former research group). Quickly and elegantly, Google generated a list of names, but somehow left out mine. Given that I have been a PhD student in the group for nine years (including three years of maternity leave and working in positions of trust), published five articles mentioning the IT department at Uppsala University, have a LinkedIn profile mentioning UpCERG, and maintain a personal website that also mentions UpCERG, I was surprised the algorithm didn’t find me. This left me feeling invisible, which was also a surprise, as I thought only humans could do that 😉 On top of that, the list of names was not correct, and I knew this because I know who has been and is currently part of the group (a sort of expert knowledge, you could say). Now imagine this happening every time someone asks AI to answer a question. It looks complete, but don’t let it fool you. More often than not, it provides only partial, highly selective, and erroneous knowledge.
3. By now, I think we have all heard that GenAI tends to produce average outputs because it’s based on statistical probability. This becomes a problem when you look at it from a larger picture: Imagine 100 000 scholars in the same field using GenAI to learn and produce knowledge — the epistemic risk here is that the field will all be comunicating and producing knowledge in increasingly similar way, contributing to the homogenization of the field. On top of that, Messeri and Crockett highlight how the appeal and efficiency of using GenAI will further nudge us to choose the methods and questions that lend themselves more easily to AI assistance – i.e., we might become more “machine-like”. What we then have is a recipe for a monoculture of knowing and knowers, making us, science, our institutions, and society as a whole more vulnerable and weaker.

So how do we overcome these problems with GenAI, research, and trust? There is, of course, as usual, no easy answer. One thing I know for certain is that we need to be more cautious and critical when using GenAI. We also need to become critical scholars and keep up to date with critical AI studies. I turn to the wise words of Ott and Mack [2, p. 16-17] here, who argue that to be critical when studying media/technology is to embody:

1. A skeptical attitude: “a deep distrust of surface appearances and ‘commonsense’ explanations” [p.16])
2. A humanistic approach: “Emphasizes self-reflection, critical citizenship, democratic principles, and humane education. [It] entails “thinking about freedom and responsibility and the contribution that intellectual pursuit can make to the welfare of society. […] Knowledge created is never complete, fixed, or finished.”
3. Political assessment: Critical scholars are interested in the practical and political implications of research findings with the ultimate aim of improving lives for the most marginalized and vulnerable, which leads us to the final point.
4. Social justice ambition: “A desire for a better world.” A commitment to identify injustices and contribute to change by confronting and challenging injustices.

These characteristics are what I try to live by as a researcher. What I hope for is that the system in which research practices, processes, and policy exist will change. I wish scholars could get funding to do research and the time necessary to communicate it. For my part, I would love to spend more time on research outreach instead of writing (unsuccessful) research applications. I wish we could focus on learning, development, curiosity, and doing research for good or even for its own sake rather than for the sake of competition, money, power, status, and the like.

References

[1] Messeri, Lisa, and M. J. Crockett. 2024. “Artificial Intelligence and Illusions of Understanding in Scientific Research.” Nature 627 (8002): 49–58. https://doi.org/10.1038/s41586-024-07146-0.

[2] Ott, Brian L., and Robert L. Mack. 2014. Critical Media Studies: An Introduction. 2nd ed. John Wiley & Sons, Inc.