Between ritual and relief – when the computer squints: Analysing AI-supported indexing in the Image Archive of the ETH Library

Abstract

Since 2021, the ETH-Library’s Image Archive has been using artificial intelligence to automatically classify its images, not as a replacement but as a supplement to intellectual keyword indexing. After the first run with the computer vision model "General" from the company Clarifai, over a million images were tagged with keywords. The time and computing power required was considerable. Initial samples showed that the quality of the results of the automated recognition varied greatly. For example, there are keywords with almost 100 % accuracy, such as "drummer". However, there are also translation errors, such as "Erleichterung" for "relief", incorrect assignments, such as mountains and glaciers as "Bechamel sauce" or even distorted assignments such as "ritual". In the absence of own ground truth data, and in order to improve the quality of the AI-based classification, the Image Archive’s data pool of over one million tagged images was subjected to qualitative analysis. To do this, all 4,600 keywords were visualised individually to evaluate the recognition quality of the AI and to validate the keywords. The analysis revealed that about half of the keywords were incorrect or problematic in terms of content and had to be deleted. The remaining keywords were then subjected to a semantic analysis. The following questions were examined: Are there any word types that are better suited for AI-based image classification? Which content works well and which leads to problematic classifications? The qualitative analysis of the ETH-Library’s Image Archive provides valuable insights for the development and application of AI-based image classification methods in practice. The results show that AI can be a valuable tool for digitising and indexing image data. However, it is important to be aware of the limitations of AI and to take these into account when using it.