Measuring Efficiency of Use in a Web-based EMR Developed for Malawi: Lessons Learned From Performing a Preliminary CogTool Analysis
|
If you are the presenter of this abstract (or if you cite this abstract in a talk or on a poster), please show the QR code in your slide or poster (QR code contains this URL). |
Abstract
Background: Developing Web-based electronic medical record (EMR) systems is a fundamental step in enabling resource-constrained healthcare environments to benefit from Web 2.0 applications and services. Towards this goal, we have developed and implemented a Web-based EMR in Malawi, Africa. The EMR, built using Ruby on Rails and AJAX, features touchscreen workstations that collect and validate data from the clinician in real-time at the point of care. The system is deployed in high patient burden HIV/AIDS clinics that have frequent staff turnover. This context makes two aspects of system usability - learnability and efficiency of use[1] - important design considerations.
Objective: To determine how efficiently novice users complete tasks using the touchscreen interface of the EMR.
Methods: To predict how quickly a user could be expected to perform the EMR tasks, we used an established technique for predicting skilled performance time called the Keystroke-Level-Model (KLM)[2]. This technique models the low-level perceptual motor operators required to complete a specified task using a given interface. KLM calculations require the application of a significant amount of cognitive modeling information. However, a free software application called CogTool[3] has been developed to support cognitive modeling for KLM. In the first phase of our research, we selected thirty-one routinely performed EMR tasks and conducted a KLM analysis of the tasks using CogTool. The interfaces and user actions for each task were prepared within CogTool using a storyboard approach. CogTool analyzed the storyboards to predict performance time of a skilled user (i.e., one who performs without error or hestitation). In the second phase, we recruited four volunteers who had not previously used the EMR to complete the thirty-one tasks for three mock patient encounters (resulting in 372 task observations). We collected the task performance data by modifying the EMR to collect timestamp data for all interface events. In the third phase we compared the preliminary novice performance data with the CogTool predictions.
Results: Of the 372 EMR tasks, 77% (286) were performed without errors by the novices (errors defined as any deviation from the CogTool storyboard). Of these error-free tasks, the performance time for 68% (194) was faster than the CogTool prediction. We observed two unexpected user behaviors not captured by the initial CogTool models: (1) two-handed touchscreen interface use, and (2) prolonged dialogue during patient interaction (e.g. spelling verifications).
Conclusions: Although preliminary in nature, our initial data suggests that novices can sometimes achieve the efficiency level of a skilled user of the EMR. We found CogTool to be an effective software application that permitted us to rapidly develop predictions of skilled performance time. The ability of most novices to often outperform the KLM calculations was likely related to a documented weakness in CogTool regarding the insertion of "think steps" for all touch typing. In our case, for onscreen typing tasks, CogTool inserted "think steps" before each button press. However, upon review it was not possible to identify suggested inclusion criteria for all of these "think steps" [4]. We are currently revising the CogTool models before collecting additional empirical data.
References
1. Nielsen J. Usability Engineering. 1st edition. New York: Morgan Kaufman; 1993.
2. Card SK, Moran TP, Newell A. The Keystroke-Level Model for User Performance Time with Interactive Systems. Commun ACM. 1980 Jul;23(7):396-410.
3. John BE, et al. (2006). The CogTool Project. http://www.cs.cmu.edu/~bej/cogtool/index.html. Archived at: http://www.webcitation.org/5Xayftaav
4. Kieras D. (2001). Using the Keystroke-Level Model to Estimate Execution Times. ftp://www.eecs.umich.edu/people/kieras/GOMS/KLM.pdf
Objective: To determine how efficiently novice users complete tasks using the touchscreen interface of the EMR.
Methods: To predict how quickly a user could be expected to perform the EMR tasks, we used an established technique for predicting skilled performance time called the Keystroke-Level-Model (KLM)[2]. This technique models the low-level perceptual motor operators required to complete a specified task using a given interface. KLM calculations require the application of a significant amount of cognitive modeling information. However, a free software application called CogTool[3] has been developed to support cognitive modeling for KLM. In the first phase of our research, we selected thirty-one routinely performed EMR tasks and conducted a KLM analysis of the tasks using CogTool. The interfaces and user actions for each task were prepared within CogTool using a storyboard approach. CogTool analyzed the storyboards to predict performance time of a skilled user (i.e., one who performs without error or hestitation). In the second phase, we recruited four volunteers who had not previously used the EMR to complete the thirty-one tasks for three mock patient encounters (resulting in 372 task observations). We collected the task performance data by modifying the EMR to collect timestamp data for all interface events. In the third phase we compared the preliminary novice performance data with the CogTool predictions.
Results: Of the 372 EMR tasks, 77% (286) were performed without errors by the novices (errors defined as any deviation from the CogTool storyboard). Of these error-free tasks, the performance time for 68% (194) was faster than the CogTool prediction. We observed two unexpected user behaviors not captured by the initial CogTool models: (1) two-handed touchscreen interface use, and (2) prolonged dialogue during patient interaction (e.g. spelling verifications).
Conclusions: Although preliminary in nature, our initial data suggests that novices can sometimes achieve the efficiency level of a skilled user of the EMR. We found CogTool to be an effective software application that permitted us to rapidly develop predictions of skilled performance time. The ability of most novices to often outperform the KLM calculations was likely related to a documented weakness in CogTool regarding the insertion of "think steps" for all touch typing. In our case, for onscreen typing tasks, CogTool inserted "think steps" before each button press. However, upon review it was not possible to identify suggested inclusion criteria for all of these "think steps" [4]. We are currently revising the CogTool models before collecting additional empirical data.
References
1. Nielsen J. Usability Engineering. 1st edition. New York: Morgan Kaufman; 1993.
2. Card SK, Moran TP, Newell A. The Keystroke-Level Model for User Performance Time with Interactive Systems. Commun ACM. 1980 Jul;23(7):396-410.
3. John BE, et al. (2006). The CogTool Project. http://www.cs.cmu.edu/~bej/cogtool/index.html. Archived at: http://www.webcitation.org/5Xayftaav
4. Kieras D. (2001). Using the Keystroke-Level Model to Estimate Execution Times. ftp://www.eecs.umich.edu/people/kieras/GOMS/KLM.pdf
Medicine 2.0® is happy to support and promote other conferences and workshops in this area. Contact us to produce, disseminate and promote your conference or workshop under this label and in this event series. In addition, we are always looking for hosts of future World Congresses. Medicine 2.0® is a registered trademark of JMIR Publications Inc., the leading academic ehealth publisher.
This work is licensed under a Creative Commons Attribution 3.0 License.