RELIC: Investigating Large Language Model Responses using Self-Consistency
METADATA ONLY
Loading...
Author / Producer
Date
2023-11-28
Publication Type
Working Paper
ETH Bibliography
yes
Citations
Altmetric
METADATA ONLY
Data
Rights / License
Abstract
Large Language Models (LLMs) are notorious for blending fact with fiction and generating non-factual content, known as hallucinations. To tackle this challenge, we propose an interactive system that helps users obtain insights into the reliability of the generated text. Our approach is based on the idea that the self-consistency of multiple samples generated by the same LLM relates to its confidence in individual claims in the generated texts. Using this idea, we design RELIC, an interactive system that enables users to investigate and verify semantic-level variations in multiple long-form responses. This allows users to recognize potentially inaccurate information in the generated text and make necessary corrections. From a user study with ten participants, we demonstrate that our approach helps users better verify the reliability of the generated text. We further summarize the design implications and lessons learned from this research for inspiring future studies on reliable human-LLM interactions.
Permanent link
Publication status
published
Editor
Book title
Journal / series
Volume
Pages / Article No.
2311.16842
Publisher
Cornell University
Event
Edition / version
v1
Methods
Software
Geographic location
Date collected
Date created
Subject
Human-Computer Interaction (cs.HC); Computation and Language (cs.CL); FOS: Computer and information sciences
Organisational unit
09684 - Sachan, Mrinmaya / Sachan, Mrinmaya