CMed: Crowd Analytics for Medical Imaging Data
Ji Hwan Park, Saad Nadeem, Saeed Boorboor, Joseph Marino, Arie Kaufman
The CMed system: (A) Timeline View displays a summary of annotations for each video, (B) Worker View shows workers' annotation and the corresponding event patterns, (C) Frame View presents details of selected frames, (D) Matrix View shows the correlation between users' event patterns and their accuracy, (E) Class View displays characteristics of worker classes based on event patterns, (F) Video View shows a selected video, and (G) Control Panel for selecting and reordering data. Views are linked, e.g., selected frames for the top video in the Timeline View (A), highlighted with a gray bounding box (pointed to by a red arrow), are shown in the Frame View (C), and the same selected frames are also highlighted in the Worker View (B).
Abstract
We present a visual analytics framework, CMed, for exploring medical image data annotations acquired from crowdsourcing. CMed can be used to visualize, classify, and filter crowdsourced clinical data based on a number of different metrics such as detection rate, logged events, and clustering of the annotations. CMed provides several interactive linked visualization components to analyze the crowd annotation results for a particular video and the associated workers. Additionally, all results of an individual worker can be inspected using multiple linked views in our CMed framework. We allow a crowdsourcing application analyst to observe patterns and gather insights into the crowdsourced medical data, helping him/her design future crowdsourcing applications for optimal output from the workers. We demonstrate the efficacy of our framework with two medical crowdsourcing studies: polyp detection in virtual colonoscopy videos and lung nodule detection in CT thin-slab maximum intensity projection videos. We also provide experts' feedback to show the effectiveness of our framework. Lastly, we share the lessons we learned from our framework with suggestions for integrating our framework into a clinical workflow.
Publications
  1. "CMed: Crowd Analytics for Medical Imaging Data"
    Ji Hwan Park, Saad Nadeem, Saeed Boorboor, Joseph Marino, Arie Kaufman
    IEEE Transactions on Visualization and Computer Graphics, 2021 PDF Video
  2. "Crowd-Assisted Polyp Annotation of Virtual Colonoscopy Videos"
    Ji Hwan Park, Saad Nadeem, Saeed Boorboor, Joseph Marino, Kevin Baker, Matthew Barish, Arie Kaufman
    SPIE Medical Imaging (oral presentation), 2018 PDF
  3. "Crowdsourcing Lung Nodules Detection and Annotation"
    Saeed Boorboor, Saad Nadeem, Ji Hwan Park, Kevin Baker, Arie Kaufman
    SPIE Medical Imaging , 2018 PDF
Video