Initial Qualitative Needs Assessment of Surgical Resident Case Logging in Low- and Middle-Income Countries Across Pan-African Training Programs

Formal case logging was introduced to surgical training in the United States throughout the 20th century. As surgical procedures and residency programs expanded, the demand for efficient and comprehensive case-logging systems drastically increased. Surgical case logging allows for training programs and trainees to track exposure and breakdown of types of surgical procedures and burdens of surgical disease throughout their residency training. The Accreditation Council for Graduate Medical Education (ACGME) developed a case logging system for general surgery residents in 1984 [1]. The initial paper-based logging system was digitized in 2000, since then, the software system has undergone significant automation, expansion, and development. The ACGME case logging system now incorporates cases of over 60 specialties and subspecialties and is utilized by a wide range of resident training programs to track experiences[2]. These developments have allowed the United States to consistently monitor the level of experience that residents are attaining and subsequently adjust resident requirements. While the accuracy and consistency of case logs vary between specialties, cohorts, and training programs, case logging software offers extraordinary value in the supervision of resident training[3].

The use of case logging systems for surgical education worldwide will likely help to ensure equitable training standards, however, implementation of formal systems faces several key obstacles. First, many low- and middle-income country (LMIC) training programs do not have access to electronic case logging systems. High surgical burden and limited technological resources can strain the ability of surgery training programs to incorporate and execute effective case logging. Even in high-income countries (HICs) where resources are available, the accuracy and effectiveness of case logs can be questioned due to delays in inputting data, the use of International Classification of Diseases (ICD) or Current Procedural Terminology (CPT) codes to categorize the diagnoses and procedures completed, and the increased burden it places on the clinical team [4]. Software accessibility, education systems, and language barriers further complicate the implementation of practical, understandable logging software in LMICs [5]. Emergency Medicine programs in rural America and India alike have acknowledged the great value of developing case logging systems for accurately capturing resident experiences and enabling curriculum development[6, 7]. These programs were successful in taking a customized approach to the rural-specific context of their training programs, which differed from the generalized ACGME case log previously offered to their trainees.

After the Association of Surgeons in East Africa (ASEA) was founded in 1950, residency training programs became established throughout Africa, and the need for accurate case-logging systems became evident [8]. As technology advanced in the late 20th century and the limitations of paper charting were exposed, organizations worked to establish accessible electronic logging systems in LMIC’s.  One of those organizations, the College of Surgeons of East, Central and Southern Africa (COSECSA), strived to this end by introducing an electronic logbook in 2015 [9].  This platform provided a framework for residents to track their surgical volume and align their experiences with COSECSA’s requirements for 500 major operative procedures (in addition to extensive research, board exam, and training-length requirements) [10].

Underneath the accreditation and oversight of COSECSA, another organization, the Pan-African Academy of Christian Surgeons (PAACS), aimed to implement a case logging system tailored to their own needs. PAACS is a surgical training organization with 24 training programs in 11 countries. Their residency programs include general surgery, orthopedic surgery, pediatric surgery, obstetrics and gynecology (OBGYN), neurosurgery, and anesthesia. PAACS also offers fellowship programs in cardiothoracic surgery, head and neck surgery, and surgical endoscopy [11, 12]. Given the breadth of practice throughout PAACS and the desire to utilize one case logging system, the current system serves all specialties and encompasses a wide range of cases. The system employed by PAACS boasts more volume and diversity than the ACGME; however, the obstacles that LMICs face in implementing software hold true for PAACS [13]. Attaining resource support, technological infrastructure, community interest, and sustainable workflows all emerge as obstacles to seamless software development. These obstacles must be addressed to standardize and enhance surgical education in LMICs.

In summary, case logs are an essential component of residents’ surgical education. However, there are key barriers to effective case log system implementation; this is especially true in LMIC contexts. There is a need for more research to better understand how current case log systems are perceived by residents and faculty and how they could be improved to better suit surgical educational needs in LMICs. A needs assessment was conducted to evaluate the current PAACS resident case log and to establish key parameters necessary for the development of a new resident case log system that could apply to wider LMIC contexts. Pioneering surgical training advancement, our needs assessment site visit in Kenya addresses the ongoing challenge of surgical resident case logging in LMICs by directly gaining the perspectives of trainees and faculty.

Multi-modal quality needs assessment

We conducted a multi-modal, qualitative needs assessment study with the combined use of focus group discussions, individual interviews, and a quantitative survey administered to participants online. This multi-modal approach is based on a grounded-theory approach and provides comprehensive and complementary qualitative data. We did this by asking open-ended questions, allowing the participants to bring about problems and issues with the current case logging system without being prompted by answer choices or direct questions. Interviews and discussions were held until there were no longer any novel concepts regarding the case-logs stemming from the discussions. In alignment with our qualitative approach, we deemed this to be sufficient data saturation. In reporting the qualitative findings, this study also follows the Standards for Reporting Qualitative Research (SRQR)[14].

Settings and participants

Focus groups and interviews were moderated by one author (MMM) and notes were kept by two authors (HER and BPC). The two note-keepers worked simultaneously to summarize opinions, key requests, and quotes mentioned during the focus groups and interviews. The focus groups and one-on-one interviews took place in person in Kenya at four locations: the PAACS Annual Basic Science Conference in Brackenhurst, AIC Kijabe Hospital, AIC Litein Hospital, and Tenwek Hospital. The researchers who conducted the needs assessment visits, interviews, and focus groups were not affiliated with PAACS. All residents and staff at each location were invited, but not required to participate given the nature of overburdened LMIC staff.

Focus group discussions and individual interviews

Through focus groups and interviews, stakeholders were provided open-ended opportunities to voice their concerns and suggestions regarding case logging within their respective contexts. Twenty-three faculty members, which included program directors, participated in a focus group at the PAACS Annual Basic Science Conference. The faculty focus group was semi-structured and lasted four hours spanning two days during the conference. Despite different structures, the interviews and focus group sessions ascertained common understanding of a case logging system and then deliberated on the important attributes of an ideal case log. The discussion included questions about key variables, compliance, reliability, ease of use, pitfalls, and improvements that can be made to the existing case logging system. At Tenwek Hospital, thirteen residents participated in a focus group conducted by MMM, HER, and BPC. At Litein and Kijabe Hospital, focus groups were not possible due to logistics and residents’ time constraints, so individual one-on-one interviews were completed with eight residents. Faculty at these hospitals had already been included in focus groups at the PAACS Annual Basic Science Conference.

Quantitative survey

Participants of the focus group or one-on-one interview were asked to complete a 26-item questionnaire. The survey was completed by faculty and residents at the PAACS Annual Basic Science Conference and field hospitals, respectively. A total of 39 physicians completed the survey, 21 residents and 18 faculty members. The survey was distributed before the initiation of focus groups and one-on-one interviews to encourage genuine responses and limit groupthink. Individuals were encouraged, but not required to participate in the survey due to conference time constraints, already overburdened responsibilities, and desire for well-developed answers. The survey included fields asking for the respondent’s position within PAACS, frequency of case log use, internet reliability, and Likert-scale questions regarding the case log’s impact on education, its need for modification/replacement, and the software’s reliability, performance, ease of use, look and feel, clarity, and security. The survey provided a standardized and collated view of the current problems of the case logging system as well as the best way forward in designing a new system.

Qualitative data analysis

After the completion of focus groups and interviews, data analysis was conducted on the responses in alignment with our grounded theory approach. To do so we transcribed and collated all the responses and analyzed them for recurring themes. The frequency of key themes was then quantified and the frequency for each specific variable requested was noted. All three on-site researchers (HER, BPC, and MMM) cross-checked the frequency of qualitative responses from the interviews and focus groups.

Data management

Questionnaire data were collected and managed using REDCap electronic data capture tools[15, 16]. REDCap is a secure, internet-based software designed for data capture for research projects. Participants accessed the questionnaire via a bar-code immediately prior to their respective focused group or interview session. All data collected were stored on secure servers maintained by Virginia Commonwealth University, with regular backups and restricted access to authorized personnel only.

Ethical considerations

The design and implementation of this study was informed by the Medical University of South Carolina (MUSC) Guidelines for Ethical Conduct of Research, the MUSC Global Surgery Program’s bi-directional research framework, and consultations with PAACS about research ethics and compliance. The study was exempted by the MUSC Institutional Review Board as a part of quality improvement.

Survey results

The survey was completed by 100% (21/21) of the residents and 78% (18/23) of the faculty who participated in focus groups or one-on-one interviews, with an overall response rate of 89%. Three survey responses, each from a resident, had partial completion: two failed to complete all Likert-scale questions regarding the software user interface and one did not answer the question regarding software security. Despite having incomplete data, their other recorded responses were still included in the analysis. A total of 12 different hospitals in Africa were represented in the survey (Table 1). Of the survey respondents, 38.5% (n=15) were from Tenwek Hospital (Kenya), 12.8% (n=5) were from Kijabe Hospital (Kenya), 10.3% (n=4) were from Litein Hospital (Kenya), and the remaining respondents were from Tanzania, Gabon, Madagascar, Togo, Burundi, Cameroon, Malawi, Niger, and Ethiopia. General Surgery represented the majority, 76.9% (n=30), of those surveyed with the remaining participants representing OBGYN (10.3%, n=4), orthopedic surgery (7.7%, n=3), anesthesia (2.6%, n=1), and pediatric surgery (2.6%, n=1).

The majority (76%) of residents did not agree that the current case log system improves resident education (Table 2). Regarding the case log’s need for modification, 38% of residents indicated a completely new software was needed, 33% chose a major restructuring of the current software, and only 10% were content with the current case logging system. Similarly, 72% of faculty did not agree that the current case log system improves resident education. Faculty were more ready to move to a new software with 56% expressing that a completely new software was needed, 17% selecting for a major restructuring of the current software, and only one individual (6%) desiring to stay with the current system.

Focus groups and interview results: Direct Quotes

When asked what the purpose of the case log is:

One resident identified the purpose of a case logging system as “track(ing) how much exposure we have.” (Q1)

Another resident commented, “the case logging system doesn’t show anything regarding the steps of the operation, what it shows is the demographics, disease, doctors, and complications. It doesn’t offer qualitative analysis but only shows the numbers.” (Q2)

When asked about areas of growth:

One resident lamented the ‘nightmare’ it was to enter disease codes reflecting the burden of an overabundance of data fields in their current case-logging software. (Q3)

Other residents added that it would “be nice to have a concise case-log” and that “we try to log every case immediately after, but it’s just not realistic.” (Q4)

When asked about complications:

Complications were deemed as unessential with the reasoning that, “hospitals can handle complications better through morbidity and mortality meetings and going through case-by-case basis.” (Q5)

Another resident noted that outcomes are “Not important for COSECSA to evaluate your operative skills.” (Q6)

“Leave it off, I only use the case logging system for COSECSA requirements. (Q7)

When asked about what features is needed in a future case-log:

One faculty member noted, “Residents should be able to log cases whenever and wherever.” (Q8)

Another voiced their desire for “a system that you can use immediately after the operation” such as a mobile phone application. (Q9)

Another faculty member said a future case logging system must be “something that doesn’t require local experts or technicians for maintenance.” (Q10)

Finally, participants stressed the importance of “…making sure that you at least accommodate COSECSA requirements;” “It still has to be in the context of meeting the COSECSA requirements.” (Q11)

When asked about variables to include in a future case-log:

Greater than 50% of faculty and residents agreed that the following trackable variables should be included in any new case-logging system that is designed for the LMIC context: procedure, post-operative diagnosis, a patient identifier (MRN, name), supervision level, the date of surgery, biological sex, age, hospital, and outcome (Table 3). Additionally, faculty members expressed that complications, surgical staff, and the staff surgeon were also important variables to include, however, residents disagreed.

Alternatively, residents found that classifying procedures as “major” or “minor” was important to include while faculty members did not find this distinction essential, noting that it is currently an unnecessary “hard stop” in the software causing delays or confusion. Upon discussion, the majority of residents concurred there was no standard definition for major vs minor classification[17]. Three additional variables that were deemed important by the residents were feedback from faculty, case classification as urgent/emergent/elective, and the name of the resident.

Our needs assessment site visit with PAACSis the first study in the literature that details the viewpoints of residents and faculty (Table 1) regarding surgical resident case logs in LMICs. We surveyed and interviewed residents and faculty from LMIC hospitals across the breadth of the PAACS network with a wide range of financial resources and experiences (Table 1). Both residents and faculty nearly unanimously concluded there was a need for either a completely new surgical resident case log or a completely reconstructed version to improve surgical resident education. Additionally, we found an essential need for offline accessibility with the ability to sync once online, a mobile version, and ensuring no undue burden to the already strained LMIC surgical residents. While both faculty and residents agreed on many of the technical and educational aspects of what an LMIC context-specific surgical case log should be, there were some differences in which variables the case log should capture. The consistencies and differences between residents and faculty members, highlighted in Table 2 and Table 3, started a dialogue that reinforced the core purpose of a surgical case log: enhancing resident education. This was made further apparent after sharing preliminary data with PAACS leadership. These conclusions about the unsustainable burden of the current detailed case logging system led the discussion to determine only the essential aspects required. After these discussions, the essential parameters for any new case-logging system became apparent: a mobile application, the ability to collect/input case data off-line, and adherence to the residency accreditation governing body (i.e., The College of Surgeons of East, Central and Southern Africa—COSECSA). These parameters, along with the top variables agreed upon, are summarized in Figure 1.

In addition to its role as a surgical case log, the current system utilized by PAACS has elements of an Electronic Health Record (EHR), CPT codes meant for billing , and remnants of research-specific fields. In a resource-constrained environment, the attempt to use a particular resource in as many ways as possible is innovative, admirable, and accepted. However, this study’s needs assessment shows that attempting to have a single system function as a case log, EHR, billing record, and research database ultimately can render the system overwhelming, and thus detrimental to its educational function. Referring to Q2 from our results, it can be seen that people may have different expectations about what a surgical case log should entail. Consequently, only 22.2% of faculty and 31.5% of residents endorsed satisfaction with the ease of use of the current system (Table 2). This alone can be enough to discourage consistent engagement with the surgical case log – especially with the immense clinical demands of surgical residency within the LMIC setting.

Additionally, the current case log was housed locally on a single server in each hospital, prohibiting residents from logging cases from personal computers in the hospital or from home. Given that some residents would have cases backlogged for months before inputting them into the case log, and that EHRs are scarce in some hospitals, the information that was ultimately input into the case log system was not always comprehensive nor completely accurate according to interviewees. This problem highlights the need for an efficient case logging system that can be accessed remotely. 

The biggest difference between resident and faculty responses in this study concerned which case-logging variables should be included in an ideal system. Both groups expressed interest in cutting down the existing variables. However, residents were more in favor of extensively reducing the number of variables. Faculty endorsed recording complications, surgical staff, staff surgeon, and patient names as essential aspects of the case-logging system while the residents did not (Table 3). Many of those who were against including a complication field did so because of the complexity and nuance of trying to collect accurate complication data linked to the appropriate index case. Capturing robust outcome data is difficult in the HIC context where additional resources optimize the ability to contact patients and electronically store patient information, only compounding the difficulty of obtaining this data in resource-constrained context. Though many residents felt it would be helpful to know, they did not believe it was possible nor reliable to collect complication data from a case log. When residents are required to separately track and retrospectively record patient complications it significantly increases the burden of a case logging system and likely influences both decreased compliance and accuracy of case log entries. Even if prompted at a specified time interval, the data may not be readily available or representative. This again highlights the potential consequences of using a single system for more than the designed goal of quantifying resident surgical experience.

One reason why more faculty tend to support keeping traditional variables may simply be to maintain the status quo. The interviews pushed the residents and faculty to think about every variable in the case log, as it pertains to a focus on surgical resident education. To maintain this focus, certain provoking questions surfaced: “Does it matter where or with whom a certain resident has logged an open appendectomy they were involved in?”; “Does the name of the patient matter, given that there is another patient identifier?”; “Or, does it only matter to capture the absolute necessary data to ensure that a surgical resident is doing enough cases, enough variety, with enough autonomy?” After discussing with faculty and residents, it was apparent that getting rid of extra, unnecessary variables will help encourage more consistent engagement with the case log, and thus longer-term educational value for its entries in resident training. The discussions with the residents and faculty provided insight into which variables should be included or excluded (Table 3). These decisions, of course, must also fall within the variables required by the surgical accrediting body (i.e., COSECSA in the case of the majority of PAACS programs).

An important concept that made this needs assessment trip successful is long-term, continuous collaboration. This study was predicated by a year-long dialogue and partnership-building process with PAACS to help the team better understand the LMIC contexts PAACS works in and the specific challenges and needs for their case logging systems. Additionally, close follow-up with the organization after the needs assessment trip has been ongoing and plans regarding the implementation of a new case-logging system are being developed. This long-term collaboration allows for everyone to understand the context in which they are working to a higher degree, builds trust amongst the faculty, residents, and needs assessment team, and overall will have a more impactful and effective intervention on the program.

Furthermore, much of the information gathered from this study can be related to the surgical case logs in HICs to increase consistency, efficiency, and satisfaction, and re-center the purpose of these case logs back to surgical resident education. An efficient and more accurate data-capturing system, like the one imagined in this study, could help identify reciprocal gaps between HICs and LMICs (for example, open vs minimally invasive approach exposure) that could support meaningful partnerships between respective countries. Additionally, an innovation in a new LMIC case logging system could inspire increased efficiency and compliance in the HIC setting. Conducting these discussions helped the authors of this study reconsider elements of the U.S. ACGME case logging system.

Looking forward to future software development, the three non-negotiable features determined by PAACS faculty and residents must be considered. Offline availability, mobile device accessibility, and inclusion of essential accreditation fields all frame the foundation for successful Pan-African case-logging systems. Even though computers and internet services can be scarce in remote settings, the smartphone penetration rate in Africa is over 100%[18]. Being able to log cases on a mobile device, whether or not the device is connected to the internet in real-time, will help alleviate a huge obstacle around technological resources. However, if cases are logged offline, there must be a feature to then synchronize with the main server or cloud once internet connectivity is reestablished. In the last few decades, significant efforts have been made to mitigate the discrepancies between HIC and LMIC data logging systems. Increased awareness, funding, commitment, and intentionality from national governments and international organizations have spurred the development of data-sharing technology in many LMICs. Throughout Thailand and Sub-Saharan Africa, researchers have worked to develop data-sharing programs in hopes of validating projects and quantifying success [19, 20]. Despite their differences in purpose, the foundational software referenced in these studies has significant implications for resident case logging and can be integrated similarly. Resident case logging could also benefit from automation and EHR integration, as residents using an automated system reported significantly more cases and higher satisfaction with the overall process [21]. The automation of case logging systems holds promise for the growth, development, and consistency of resident case logging [22].

Identifying problems is the first step towards sustainable change. This needs assessment study highlights what is important to include and possibly omit from a surgical case log in this LMIC setting, by working with physicians who practice in this specific setting. However, the lessons learned from this needs assessment project could also apply to a broad range of settings including the HIC context. Limiting case log data input to what is necessary, moving away from using CPT codes and other attributes geared towards billing, and improving mobile accessibility to case logging both online and offline will serve to improve resident education and encourage collaboration worldwide. As surgical case logs continue to evolve around the needs of LMICs, future studies should follow up and assess the impact and effectiveness of those changes.

Limitations:

One limitation of this work is the fact that this is a qualitative needs assessment, not a comprehensive systematic evaluation of the sites and case logging system. Another limitation is the fact that recent graduates were not included in the survey. This was not logistically possible as our participants only included people who were physically at the PAACS conference or at the hospital on the day of interviews. Another limitation is convenience sampling, given that everyone interviewed or who participated in focus groups were from the same training programs (PAACS). Additionally, the focus groups and interviews were semi-structured, not protocolized focus groups and interviews. All in all, study is an important first step to address the opportunity to improve surgical case logging in LMICs. Next steps include more systematic evaluations as well as working with PAACS and software developers to create a new, resident-focused surgical case logging system for LMIC contexts. Future studies will assess the implementation of the new system.

This study shared findings from a needs assessment conducted with twelve hospitals in ten different countries in coordination with PAACS. PAACS faculty and resident physician interviews, surveys, and discussions established that a new case logging system designed for the LMIC context is needed to encourage consistent surgical data entry. The new case log must accommodate LMIC contexts by being available on a mobile device, accessible offline, and fulfilling governing bodies’ requirements. This would not only benefit the surgical residents and programs in LMIC settings but strengthen understanding and comparability of the burden of surgical disease and procedures and training models worldwide.