This record is in review state, the data has not yet been validated.
In-context learning enhanced credibility transformer
Loading...
Author / Producer
Date
2025-01-01
Publication Type
Journal Article
ETH Bibliography
Citations
Web of Science:
Altmetric
Data
Rights / License
Abstract
The starting point of our network architecture is the Credibility Transformer which extends the classical Transformer architecture by a credibility mechanism to improve model learning and predictive performance. This Credibility Transformer learns credibilitised classification tokens that serve as learned representations of the original input features. In this paper we present a new paradigm that augments this architecture by an in-context learning mechanism, i.e., we increase the information set by a context batch consisting of similar instances. This allows the model to enhance the classification token representations of the instances by additional in-context information and fine-tuning. We empirically verify that this in-context learning enhances predictive accuracy by adapting to similar risk patterns. Moreover, this in-context learning also allows the model to generalise to new instances which, e.g., have feature levels in the categorical covariates that have not been present when the model was trained-for a relevant example, think of a new vehicle model which has just been developed by a car manufacturer.
Permanent link
Publication status
External links
Editor
Book title
Journal / series
SOUTH AFRICAN ACTUARIAL JOURNAL
Volume
25
Pages / Article No.
Publisher
Event
Edition / version
Methods
Software
Geographic location
Date collected
Date created
Subject
Transformer; credibility transformer; in-context learning; attention layer; foundation model; insurance pricing