More Efficient Manual Review of Automatically Transcribed Tabular Data

Author(s)

  • Bjørn-Richard Pedersen Norwegian Historical Data Centre, UiT The Arctic University of Norway
  • Rigmor Katrine Johansen Department of Health and Care Sciences, UiT The Arctic University of Norway
  • Einar Holsbø Department of Computer Science, UiT The Arctic University of Norway
  • Hilde Leikny Sommerseth Norwegian Historical Data Centre, UiT The Arctic University of Norway
  • Lars Ailo Bongo Department of Computer Science, UiT The Arctic University of Norway

DOI:

https://doi.org/10.51964/hlcs15456

Keywords:

Population census data, Machine Learning, Historical data, Manual review, Occupation codes, Norway 1950, Norwegian population data, Efficient manual review, Automatically transcribed, Tabular data, Manual review and correction, Interviews, Norwegian occupation data, Historical occupation data, Norwegian Historical Data Centre, UiT The Arctic University of Norway

Abstract

Any machine learning method for transcribing historical text requires manual verification and correction, which is often time-consuming and expensive. Our aim is to make it more efficient. Previously, we developed a machine learning model to transcribe 2.3 million handwritten occupation codes from the Norwegian 1950 census. Here, we manually review the 90,000 codes (3%) for which our model had the lowest confidence scores. We allocated these codes to human reviewers, who used our custom annotation tool to review them. The reviewers agreed with the model's labels 31.9% of the time. They corrected 62.8% of the labels, and 5.1% of the images were uncertain or assigned invalid labels. 9,000 images were reviewed by multiple reviewers, resulting in an agreement of 86.4% and a disagreement of 9%. The results suggest that one reviewer per image is sufficient. We recommend that reviewers indicate any uncertainty about the label they assign to an image by adding a flag to their label. Our interviews show that the reviewers performed internal quality control and found our custom tool to be useful and easy to operate. We provide guidelines for efficient and accurate transcription of historical text by combining machine learning and manual review. We have open-sourced our custom annotation tool and made the reviewed images open access.

Downloads

Download data is not yet available.

Downloads

Published

2024-04-04

Issue

Section

Articles

How to Cite

More Efficient Manual Review of Automatically Transcribed Tabular Data. (2024). Historical Life Course Studies, 14, 3-15. https://doi.org/10.51964/hlcs15456