TIL: Annotation Datasets

Let’s study annotators

Vincent Warmerdam koaning.io
2022-10-06

For a while, I’ve been interested in studying annotators and their labels more closely. Many datasets, however, don’t provide this information so I started looking for some.

Thankfully, I found an article titled “On Releasing Annotator-Level Labels and Information in Datasets” that talks about the need for sharing annotator information and they also list a few datasets that do provide this information.

I figured I’d keep track of a list. There are a lot of tricky issues related to annotator disagreement, and it’d be great to have more datasets to study.