Assessing Learner Growth on a Low-cost Laparoscopic Simulation Platform

Despite significant advancements in global health over the past few decades, a substantial gap persists in access to surgical care in low- and middle-income countries (LMICs). Approximately five billion people lack access to safe and affordable surgery and anesthesia, with over 90% of this population residing in Southern Asia and Africa[1,2]. Thus, an additional 143 million surgical procedures are needed annually to meet the unmet needs in LMICs[3]. This disparity is even more pronounced when considering advanced surgical techniques and innovative approaches. Laparoscopy, which is the standard approach for many common operations (e.g. cholecystectomy, appendectomy) in high-income countries (HIC), is rarely utilized in LMICs, due to a lack of equipment, infrastructure, and training opportunities with expert trainers[3,4]. A study conducted in the hospitals affiliated with the College of Surgeons of East, Central and Southern Africa showed that only 0.9% of the total cases were done laparoscopically[5]. Consequently, patients without access to laparoscopy face longer hospital stays, higher complication rates, and prolonged recovery [6]. This highlights the urgency for devising creative solutions to teach laparoscopy in LMICs.

In the past decade, there has been much motivation to increase laparoscopic training opportunities in LMICs. Early efforts focused on short-term support from HICs partnering with LMICs, through brief international trips or unilateral research initiatives. While such endeavors aim to advance medical knowledge and training globally, they raise ethical concerns about long-term sustainability and reinforce power dynamics between HICs and LMICs, providing limited agency for local providers to train independently[7-10]. Focusing on capacity building and local educational training through simulation-based self-learning, has demonstrated some success in LMICs[11].

One notable contribution in simulation-based surgical training in LMICs is the low-cost training platform developed by the ALL-SAFE team[12-15]. This open-source, web-based platform provides self-guided learning modules that support learning of the foundational concepts of laparoscopy through interactive case-based scenarios, followed by psychomotor training on a low-cost box trainer. Skill assessment is performed via video-based review, through self- and peer-assessments using a Verification of Proficiency (VOP) task list, a global rating, and a final competency rating. While prior studies have demonstrated the platform’s ability to differentiate participants’ experience levels within a single procedural module, further research is needed to assess its capacity to facilitate longitudinal skill development over time. This study evaluates how ALL-SAFE performance measures (1) predict participants’ baseline laparoscopic experience across multiple modules and (2) assess trends in skill progression over cumulative module engagement.

Platform Development

The ALL-SAFE training program is a free, online platform designed for global access to laparoscopic education and simulated hands-on practice[12, 14, 16-17]. With a low-cost cardboard box-trainer built from readily available materials costing a total <10 USD (Figure 1), each procedure can be tested with a distinct simulation model. Each module consists of a pre-test, case scenario, skills lessons (expert video filmed within the platform), and a simulation step that requires the participant to use their own video-capable cell phones to record and upload their performance using the box-trainer. Each uploaded video is assessed immediately by the participant and asynchronously by at least two other participants, of any training level, who are randomly selected by the platform to serve as peer assessors. The platform ends with a post-test, re-visiting the confidence and competence questions, and an exit survey.

The content for the clinical modules was developed by surgeons from partnering institutions in LMIC, through needs assessments and iterative expert review. Content was modeled after published materials from the Fundamentals of Laparoscopic Surgery (FLS), the American College of Surgeons (ACS), and the Association of Program Directors in Surgery (APDS). Each module was designed to teach foundational laparoscopic skills, such as instrument handling, tissue manipulation, knot tying, and dissection techniques[18]. At time of this study, the ALL-SAFE program included four modules: management of ectopic pregnancy (Module 1), appendicitis (Module 2), Meckel’s diverticulum (Module 3), and penetrating thoracoabdominal trauma (Module 4). As illustrated on the ALL-SAFE website (allsafe.education), the modules vary in difficulty, with the progression of difficulty categorized as follows: Module 3 (easy), Module 1 (moderate), Module 2 (moderate), and Module 4 (advanced).

Participants

This multisite study included participants from several countries, including LMICs and HICs. The African pilot sites included Mbingo Baptist Hospital in Cameroon, as well as Soddo Christian Hospital and Yekatit Hospital in Ethiopia. The US sites included Michigan Medicine, The MetroHealth System, and Case Western Reserve University. This study received Institutional Review Board exemption by the University of Michigan’s Institutional Review Board (HUM00199557), the site where the principal investigator is on faculty.

General surgery faculty members and resident physicians of varying training levels were recruited from study sites through surgeon champions (surgical trainees or attendings dedicated to leading the implementation of ALL-SAFE at each site), General Surgery Program Directors, Research Directors, and professional training and accrediting bodies, such as the West African College of Surgeons. Participants were categorized into one of three experience levels. Those with over 50 laparoscopic operations performed as the primary operating surgeon over their lifetime – including those performed as a trainee – were considered experts (E), while participants with between 10 and 50 lifetime laparoscopic operations were designated as intermediate (I), and <10 were considered novice (N). Participants independently completed the self-directed modules in their desired sequence and pace. Although the modules were released sequentially, starting from Module 1, participants were not required to start from Module 1.

Data collection

The retrospective study included performance data captured from August 2021 – December 2023 from surgical programs in Sub-Saharan Africa and in the US. Participants’ uploaded procedural videos were assessed using three performance measures: a procedure-specific Verification of Proficiency (VOP) task list, a global rating, and a final competency rating. The user data and performance metrics were retrieved from the ALL-SAFE online platform in April of 2024 from the module initiation and completion records.

Assessment Tools

Each module’s VOP task list was created based on expert consensus following iterative rounds of review, feedback, and revision, as described in prior studies[12, 14, 16, 19]. The task list was composed of step-wise procedural tasks adapted from the ACS and APDS online curriculum[20]. Task lists ranged from 10 items (e.g. Module 4) to 13 (e.g. Module 2). Each item was scored as 0 (not done) or 2 (done) with the exception of items critical to patient safety, marked by “avoids” language, were weighted and scored as 0 (not done) or 3 (done). To equate scores across different modules, task list scores were summed, and the total percent (%) completion was reported as the final score. For participants who completed more than one module, the total percent completion was averaged across all modules completed.

The 5-item GRS assessed the following domains: respect for tissue, economy of time and motion, instrument handling, flow of operation, and overall performance[21]. Each domain was scored on a 5-point scale using 1 (unacceptable), 2 (needs improvement), 3 (meets expectations), 4 (exceeds expectations), or 5 (outstanding).

The final competency rating evaluated the participant’s overall competency, as 1 (Not Competent), 2 (Borderline), or 3 (Competent). The mean rating from all three assessors (self-assessment and two peer assessments) was used as the primary metric in comparative analyses.

Data Analysis

Following confirmation of a non-parametric distribution, a Kruskal-Wallis test was used to evaluate the ability of the VOP performance measures to differentiate between the three experience levels. For participants who completed at least two of the four modules (n = 17), score progression was also examined using the Kruskal-Wallis test to evaluate if learned skills resulted in increased scores as participants completed subsequent modules. The number of total modules completed were analyzed, irrespective of the sequence of completion.

Finally, Spearman correlation (ρ) was used to estimate the correlations between the percent completion on the task list and the summed global rating scale and the task list and the final competency ratings. A ρ ≥ .50 was considered a strong correlation. A P ≤ 0.05 was considered statistically significant for all analyses. Statistical analyses were performed using IBM SPSS Statistics for Windows, version 28.0 (IBM Corp; Armonk, NY).

A total of 52 participants across four sites participated in at least one module, up to four modules (Table 1). Participants included medical students (n = 23), general surgery residents (n = 20), and faculty members (n = 9) who were categorized into novice, intermediate, and expert categories. Across all modules, the VOP task list score percents were lowest for novice participants and highest for expert participants, though the differences were not significant, P = 0.54 (Figure 2). Comparison of novice, intermediate and expert global ratings indicated novices had lowest scores and experts had highest scores, with statistically significant score differences across all experience levels, P < 0.001. Deeper pair-wise analyses confirmed statistically significant differences across novice and intermediate (P 0.05 (Figure 3). The mean final competency rating (M) effectively distinguished novices (M = 2.13) from intermediates (M = 2.59) and experts (M=2.76), P < 0.04 (Figure 4).

Correlation between Performance Measures
Correlations between the % VOP task list and the global ratings and between the % VOP task list and final competency ratings were moderate, ρ = .42, P < .001 for both pairings.

Trends Across Number of Modules
The VOP task list % completion increased from 87.30% to 90.97% from one module to four modules completed, P < 0.001 (Table 2). Similarly, all five global ratings and the final competency ratings increased with more modules completed, P < 0.001. Although deeper analyses indicated that many of the stepwise comparisons were not statistically significant, score differences between one versus four modules completed were statistically significant (Figure 5). All five domains of the global rating demonstrated an increase in scores with more module completion, P < 0.001. While certain domains demonstrated linear growth, other parameters did not. The final competency rating was unchanged for completing one to three modules but increased for four modules completed, P < .001.

This study evaluates several performance measures to assess learner progression and skill acquisition in a simulation-based laparoscopic training system. The summative results confirm individual findings previously published on the platform’s ability to assess psychomotor and cognitive skill growth within individual procedures, but with added findings that simulation-based practice of laparoscopy can help build an incremental foundational skill base for laparoscopy[12, 14, 16-17]. Global ratings effectively differentiated skill levels between novice and intermediate, and novice and expert participants, as did the final competency ratings. The VOP task list scores increased for novice, intermediate, and expert participants, respectively, though not statistically significant. For the subset of participants who completed more than one module (n = 17), performance measures positively correlated with the completion of more modules. These findings suggest that foundational psychomotor skills can increase with repeated exposure to different procedures within a simulation-based learning system. However, VOP task lists did not demonstrate similar trends in skill growth, indicating that procedure-specific knowledge may not be transferable across different modules, due to the unique anatomy and steps required for each procedure. These findings provide some evidence that supports the use of ALL-SAFE platform’s performance measures for LMICs.

Our findings highlight the ALL-SAFE platform’s ability to differentiate skill levels among participants and provide meaningful, constructive assessments, a critical feature for alternative training modalities in LMICs[22-25]. In settings where formal laparoscopic training is limited and accreditation through programs like FLS may be logistically challenges, the adoption of standardized scoring systems aligned with the content of FLS, ACS, and APDS, can play a pivotal role in establishing and reinforcing best practices in surgical education. Learners may also develop skills beyond the educational content directly provided on the platform through self-directed practice. For example, participants were allowed to freely repeat certain steps and assessments or even use the box-trainer and recording set-up for practicing other procedures and techniques. While asynchronous learning activities were not fully captured by the performance measures—since only the initial submissions were assessed—they likely contributed to further skill progression[26].

A major shortcoming of self-directed learning includes the difficulty establishing clear benchmarks for learner progression. Determining readiness from the transition from practice on simulation to operating on real patients remains a challenge within existing scoring paradigms, as evidence on this is limited. While simulation cannot replace in-person apprenticeship and feedback, observing trends of increasing scores across modules using standardized scoring systems provides valuable feedback to learners. In resource-limited settings, this may be the only form of quantitative and qualitative feedback available to independent learners, helping to identify educational gaps for further learning.

Another major challenge lies in sustaining motivation from learners, as witnessed by the low module completion rate, with only three participants (6%) completing all four modules. If motivation to practice independently diminishes after completing a single module, achieving the necessary repetitions to transition to real-life practice may become difficult. This trend parallels the patterns observed in performance measures over time, which exhibit non-linear growth—an initial spike, followed by regression, and subsequent improvement. Prior studies within other self-learning platforms have also showed similar findings[27-30], which can be attributed to the natural plateauing of skill acquisition in a limited learning environment[31]. While providing performance measures can play a role in incentivizing participation by creating aspirational targets and gamifying the learning process, this alone may be insufficient to sustain long-term engagement without additional strategies to foster motivation and progression, such as institutional backing and support from accrediting bodies. These partnerships can help integrate simulation-based training programs into surgical skills curricula, as exemplified by the recent endorsement from the West African College of Surgeons to incorporate ALL-SAFE into general surgery training programs across their networks. These collaborations help address barriers to self-learning and promote the gradual development of laparoscopic skills, ultimately contributing to a framework for lifelong surgical education.

Limitations

Several limitations must be acknowledged when putting our results into context. A small sample size across cohorts—only seven participants who completed three modules, and three who completed all four—limits our ability to draw conclusions on longitudinal trends. Therefore, statistically significant findings from these small subgroups should be interpreted cautiously, due to the elevated risk of Type I error. Additionally, our analysis did not control for participant-level factors—such as prior training, access to mentorship, and experience with live laparoscopic cases—which may have influenced the outcomes. Also, variation in module difficulty, noted in the introduction, was not adjusted for in the analysis. Thus, rising scores may be influenced by inherent module complexity depending on the sequence the modules were completed, rather than true skill acquisition. Although the assessment tools are grounded in established instruments such as FLS and previously validated ALL-SAFE assessments, we did not include data on interrater reliability or score reproducibility for peer assessments in this study. Our prior studies have suggested no rating differences or biases across expertise levels, continent, or site[12].

Future Directions

Future work should include larger studies with balanced module completion and longitudinal follow-up to assess sustained impact. Including control groups and adjusting for module difficulty and participant demographics will bolster validity evidence and better support any future inferences made. Further, estimating the assessment tools’ interrater reliability and alignment across peer-, self-, and faculty evaluations will improve scoring rigor. Incorporating qualitative data from interviews and surveys will also offer valuable insight into user experience. Lastly, supporting the transition from simulation to clinical practice through supervised opportunities and ongoing professional development will be essential for skill retention and application.

Our findings underscore the effectiveness of the ALL-SAFE assessment tools in discerning laparoscopic skill level among participants engaging in a low-cost simulation platform and measuring progression of scores with the completion of additional modules. While simulation cannot replace real-life experience and apprenticeship, our study suggests that the ALL-SAFE platform provides a valuable opportunity for participants to enhance their skills in a controlled, simulated environment, fostering progression that may serve as a foundation for future clinical practice.