About This Project
Data visualizations are important to readers' understanding but in many cases, are unavailable to visually impaired individuals due to poor or missing captions. In this project, we investigate if figure captioning can be automated using machine learning techniques in order to improve accessibility of data visualizations. To this end, we aim to create a dataset of scientific figures with carefully crafted, human-generated captions that describe the high-level trends and insights of the figure.
Ask the Scientists
Join The DiscussionWhat is the context of this research?
Data visualization is an important tool for conveying information and drawing attention to certain aspects of the data. Visualizations are commonly used in news articles and research papers to assist readers in their understanding but in most cases, are completely inaccessible to visually impaired users due to poor or missing captions.
With substantial advances in the field of video and image captioning, it is natural to start extending some of these ideas to automated figure captioning in order to help make data visualizations more accessible to the visually impaired.
What is the significance of this project?
To this day, research in the field of automated data visualization captioning is quite limited. Rule-based approaches to this task are not easily generalizable to more complicated data visualizations and the lack of appropriate datasets for this task is a barrier to kick-starting machine learning research in this field.
Through this project, we will be creating a dataset of line charts extracted from scientific papers with crowd-sourced captions that describe the high-level insights of these charts. We will help progress machine learning research in this field by providing public access to any collected data and developing machine learning techniques that can create detailed captions for data visualizations.
What are the goals of the project?
In this project we first aim to create a dataset of line charts and crowd-sourced captions that convey the high-level insights of the figures. We will then develop machine learning approaches to automated captioning of data visualizations while taking inspiration from previous ML research in the field of image captioning and visual question answering.
Budget
Successful machine learning approaches require large amount of high-quality data, the creation of which is typically expensive in terms of both time and money. To our knowledge, no suitable, public dataset exists for our project of automated figure captioning for improved accessibility. Thus, we aim to pave the way by introducing a premiere dataset for this task, which will benefit not only our work but an entire field of related research. Specifically, we will be creating this dataset by crowdsourcing figure captions through the Amazon's Mechanical Turk platform, where we will be paying Mechanical Turk workers for captioning figures.
The creation of this dataset is partly funded by the Harvard Data Science Initiative’s Faculty Special Projects Fund, and any money raised from this fundraiser will further go towards paying Mechanical Turk workers.
Endorsed by
Project Timeline
We aim to launch our Mechanical Turk experiment in January 2022 in order to crowdsource figure captions for our dataset. Following the preprocessing of gathered data and development of models, we will be providing public access to our dataset and results to help foster further research and development in this area.
Nov 30, 2021
Designing a Mechanical Turk experiment for labelled the figures
Nov 30, 2021
Scraping and pre-processing a diverse set of data visualizations
Dec 10, 2021
Project Launched
Dec 31, 2021
Preliminary study to evaluate the experimental design and make appropriate adjustments
Jan 31, 2022
Launch Mechanical Turk experiment to gather labels
Meet the Team
Affiliates
Team Bio
We are a multi-disciplinary team interested in machine learning, data visualization, computer vision, and natural language processing. Anita is a Master of Data Science student at Harvard who is being advised by Chris and Zona as she works towards completing her master's thesis.
Anita Mahinpei
I'm a Master of Data Science student at Harvard University. I'm passionate about researching the applications of machine learning to under-explored domains. Currently, I conduct research as a member of Harvard's Data to Actionable Knowledge Lab and as a MS in Data Science student at Harvard's Institute for Applied Computational Science. Previously, I was a student at the University of British Columbia where I studied Physics and Computer Science.
Zona Kostic
Zona Kostic is a lecturer and research fellow at Harvard University. She teaches courses on data visualization and AI-powered web applications. Her research focuses on immersive visual analytics and novel interaction modalities. Zona received her PhD in computer science from University of Belgrade in 2014. Zona was also a member of the Visual Computing Group (Harvard SEAS) in 2016 and the Innovation Labs (Harvard Business School) in 2018.
Chris Tanner
Chris is a lecturer at Harvard, where he conducts research and teaches courses that concern NLP, Data Science, and Machine Learning. Specifically, a persistent theme of his research focuses on discourse and trying to better understand what is discussed and who is who within any body of text. He received a PhD from Brown University in 2019, and before then, he was an Associate Staff researcher at MIT Lincoln Lab.
Lab Notes
Nothing posted yet.
Project Backers
- 10Backers
- 46%Funded
- $230Total Donations
- $23.00Average Donation