Lab Note 11: And the crowd goes wild!!

 

Figure 1: Average OSATS sum scores by experts and the crowd. The graph depicts the average score assigned to each video (x axis) by the experts (blue) and by the lay raters (red). Spearman's rho for Crowd vs Expert scores = 0.879.

Well, the data are in... and the mean crowd scores (red) ARE correlated with the mean expert scores (blue)! It is pretty clear when you look at the graphic... the higher the expert score the higher the crowd score. The pattern is the same for both types of graders. 

We had pretest threshold of 0.7 for successful correlation between the experts and the crowd. We found that we met this threshold not only for the total OSATS score (Spearman's rho = 0.897) but also for all 5 component scores (see Figure 2).The correlation for each of the 5 domains was also pretty good, with respect for tissue performing the worst, but with a still very respectable 0.71.

 

Figure 2: Spearman's rho for expert vs crowd mean score for the OSATS sum score and for all 5 component areas examined.

We still only have average scores analyzed and will cull through more statistics and outcomes in the coming days to weeks.

But I'd say this counts as proof-of-concept and we have a jumping off point for more in-depth study of the developmental trajectory of surgical learning.

THANK YOU ALL for you support! We'll keep you posted on our further progress on this study as well as the outcomes from the ongoing longitudinal study that we are conducting in parallel.