Sustainable coordination in research labs via graph-enabled idea boards

Methods

Summary

Framed as a pilot study, we aim to collect different lines of evidence illustrating that our lab graph and issues board can be used for the regular exchange of ideas among different members of our research lab.  Most of these metrics are already extractable from the graph.  We aim to demonstrate that these metrics are useful and quantifiable, with the goal of spurring future in-depth studies that take advantage of these quantitative usage data to more rigorously test hypotheses about the impact of the tool on researchers.  

The types of evidence we intend to collect include the following:

Usage of issues: frequency with which researchers create issues

Proof of concept: ~100 Issues created in the lab have time stamps and user stamps, which can be plotted and correlated with metrics listed below. 

Claiming issues: frequency with which researchers claim issues (convert an issue into an experiment)

Proof of concept: Issues can be "claimed" with the click of a button, converting the Issue to an active experiment. Each has a time stamp and user information, allowing us to track their conversion from one type to the other. 

Contributing to experiment: length of time from claiming issue to registering first result or completion, as indicated by the creation of a conclusion

Proof of concept: The graph tracks the time the experiment was created as well as the time a result or conclusion was added:

Experience of users: qualitative interviews and ratings from questionnaire

Proof of concept: We have collected user feedback from rotation PhD students using the graph. This type of rating and rationale can be collated and reported over time.

Referencing of existing knowledge: number of references to existing discourse nodes (questions, hypotheses, results)

Proof of concept: the bidirectional nature of the graph shows all the links made between existing elements of the graph.

We expect that these metrics will help to illustrate the potential of this lab discourse graph software in distributing ideas, insights, and work between experienced and junior researchers.  We will consider this pilot study to be a success if these metrics provide evidence of regular exchange between experienced and junior researchers in the form of ideas, proposed experiments, and new research results.  We hope that these metrics will serve as a proof of concept that paves the way for a larger, more definitive study exploring the impact of this tool on collaboration and scientific progress. 

Challenges

Because we created a new mechanism for lab collaboration, in general we expect there to be no measurable baseline for how these aspects of research are usually done in our field in absence of these tools.  This makes it difficult to infer causality from these metrics.  So instead, we expect that these initial metrics may serve multiple purposes: 

• They may illustrate the benefits of the tool to other researchers in the field, who would compare these measurements to their anecdotal experience of the rate of collaboration within their own research groups; 
• Second, we'll have a proof of concept that these metrics can be collected, which will set the stage for future studies with more formal interventions (such as different sections of an undergraduate course, or longitudinal comparisons over time).