Different methods for reporting major incidents exist, but a golden standard is still lacking. The study aimed to investigate the inter-rater reliability between EMS providers using a new major incident reporting template.

The exercise scenario was a bus collision in rural area. Four teams participated in the study. Cohen’s kappa analysis and thematic analysis of free-text fields were used to assess the inter-rater reliability between the reporting teams.

A total of 32 questions were selected with 16 binary and 16 open text answers. The general kappa value was rated at 0.30, indicating fair agreement. Qualitative data analysis showed similar themes emerging, but the findings varied by topic. The teams had different observational points and clinical tasks during the exercise.

There was only fair agreement between raters. Further studies using experienced observers without clinical tasks and an investigation of personal situational awareness are recommended.

Introduction

Major incidents, such as natural disasters, accidents, and terrorist attacks, affect millions of lives annually. In 2015, the international disaster database EM-DAT recorded 346 natural disasters that claimed almost 23 000 lives and affected over 98 million people (Guha-Spir 2016). By the end of the week 47, 2016, 319 disasters have been recorded (Centre for Research on the Epidemiology of Disasters 2016). Improved and systematic reporting of major incidents has become imperative (Bradt & Aitken 2010; Stratton 2012; Castrèn, Hubloue and Debacker 2012; Braun & Clarke 2006; Lockey 2012; Lennquist 2011; Fattah, Rehn, Lockey, Thompson, Lossius and Wisborg 2014). A systematic literature review identified ten different templates for reporting the pre-hospital medical response to major incidents (Chen, Hung and Sung 2015). These templates had been used on a limited scale, were not tested for feasibility, and lacked a specific focus on the pre-hospital phase of major incident management. Major incident experts applied a modified nominal group technique to design a new template that was easily accessible and focused on the pre-hospital phase. (Fattah, Rehn, Reierth and Wisborg 2013; Fattah, Rehn and Wisborg 2015). It has been feasibility tested and revised according to the findings (Fattah, Agledahl, Rehn and Wisborg 2016). The overall aim of the project is to create a global database where data can be aggregated to facilitate comparative analyses of several real life events and field exercises. The aim of the present study was to assess if the different teams report similarly using the consensus-based template during the Barents Rescue 2015 exercise.

Methods

Setting
This study was conducted during an international field disaster exercise held September 2015 in Kittilä, Northern Finland. The nearest central hospital is located in Rovaniemi, 145 km by road from the municipality center in Kittilä; while the university hospital in Oulu is situated 356 km by road from Kittilä. Distances and resources in the region are vast and scarce, so inter-regional cooperation is considered paramount. The Barents Rescue exercise is arranged by the Barents regional countries of Norway, Sweden, Finland, and Russia (Figure 1. Map of the Barents region). The exercise focuses on strengthening and maintaining preparedness for mutual assistance during major incidents. For this study, a scenario with a patient number estimated at 50 was selected (Incident scenario).

Incident scenario

Due to heavy rains in the area, the road network has suffered severe damages. A busy road has partly collapsed because of running water and a landslide has occurred in the same area. The poor condition of the roads has led to a traffic accident between a local bus full of tourists and a truck loaded with chemicals. Both vehicles are badly damaged and several woods nearby”. Total patient number is estimated at 50, with various injuries and chemical exposures.

Participants in the reporting teams

A total of four teams consisting of two emergency medical service (EMS) providers with a nursing degree in each team were recruited from the Laurea University of Applied Sciences and the Länsi-Pohja hospital district (Table 1).

Recruited teams received written and verbal information on the use of the major incident reporting template. Teams were instructed to use the majorincidentreporting.net webpage for submitting their reports. A total of 38 questions, regarding communication and triage, were selected from the template, and 22 binary (yes or no) statements were analyzed using Cohen’s kappa (Figure 2, Questions analyzed using Cohen’s kappa). The remaining 16 questions were reviewed and analyzed by thematic analysis of the content, searching for similar emerging themes. (Vaismoradi, Turunen and Bondas 2013; Braun & Clarke 2006.)

Statistical methods

Basic Cohen’s kappa (κ) measures the agreement between raters (Fattah et. al. 2017; Vaismoradi et. al. 2013). If the raters are in complete agreement, then κ = 1. If there is no agreement among the raters other than what would be expected by chance then κ ≤ 0. Cohen’s kappa values of 0–0.20 are considered as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect agreement. The statistical significance of κ can be tested. Values of κ with p<0.05 are considered statistically significant.

Qualitative method

The information derived from free-text questions was reviewed and analyzed using the principles of qualitative thematic analysis (Cohen 1960; Braun & Clarke 2006). Recurring themes and answers were searched for.

Results

Four teams submitted their reports to the majorincidentreporting.net database (Table 2, Questions in the major incident reporting template analyzed using Cohen’s kappa).

The agreement of raters was fair with  =0.30 (p<0.05). In free-text questions (Table 2, questions in the major incident reporting template subject to thematic analysis), the agreement and disagreement ratio varied; while, the answers were uniform for questions depicting chain of command and triage. Other topics, such as communication and public relations, showed moderately similar written answers; although, the same themes came up. Depending on the team, there was a lack of communication either with the rescue command, police, or international teams. The lack of similar communication devices such as Terrestrial Trunked Radio (TETRA) with common speech groups between the teams from different countries was also a concern.

Discussion

The present study found fair inter-rater reliability among teams reporting from a field disaster exercise using a major incident reporting template. Human factors influence inter-rater variability. (Norri-Sederholm, Paakkonen, Kurola and Saranto 2015; Endsley and Garland 2000; Endsley 1995.) and situational awareness is the key to achieving strong inter-rater agreement between observers. Situational awareness is affected by a person’s ability to perceive, comprehend, and structure information with past knowledge to project a possible future action (NHS England National Emergency Preparedness, Resilience and Response Unit 2015). The ability to observe and process large amounts of data is unreliable. Without proper training or experience, a person’s cognitive ability is insufficient to make reliable observations of the onset situation. A possible solution could be to recruit managers with a greater overview of the situation or use designated evaluators, as suggested in the National Health Services in the United Kingdom (Endsley 1995). Perhaps personnel with only clinical tasks should not report objective exercise data.

Field exercises are resource intensive events and it seems imperative that evaluation should be structured to ensure learning in the participating EMS services and the greater EMS community (Raatiniemi, Martikainen, Jama and Alahuhta 2016). Shortly after the current field exercise, the major incident reporting template was revised according to findings in a feasibility study (Fattah, Agledahl, Rehn and Wisborg 2016) and the majorincidentreporting.net was updated to also allow submitting field exercise reports (Raatiniemi et. al. 2016).  For comparative analysis on both field exercises and real life incidents, the non-profit major incident reporting project relies on the EMS community to contribute reports for further research. Given the number of lives lost and affected by major incidents every year, this contribution can be considered a necessary ethical obligation to both the patients and the EMS service providers.

Limitations

The total number of responding teams was low and the teams had different observational points and tasks during the exercise. However, this is the first pilot study aiming to investigate the inter-rater reliability of the template. The findings of the present study can be used to develop hypotheses for future studies.

Conclusions

In this study, the inter-rater variability for binary questions in a major incident reporting template was fair. Several factors such as situational awareness and language barriers may explain why the agreement between teams was not higher. Future studies are recommended to counteract limitations in situational awareness and we recommend using reporters with non-clinical roles and a broader perception of the incident.

Abbreviations

ALS, advanced life support; BLS, basic life support; EMS, emergency medical service; κ, basic Cohen’s kappa; RN, registered nurse.

Declarations

Ethics approval

No ethical approval was necessary for this study, but consent for participation and publication was acquired from the exercise administration (19.03.2015) (Finnish Advisory Board on Research Integrity 2016; Finnish Advisory Board on Research Integrity 2012).

Competing interests

JV, JJ, TW, and JE have no competing interest related to this study. SF, MR, and LR are voluntary members of the steering group of the Major Incident Reporting project and have solely an intellectual interest in disseminating knowledge of www.majorincidentreporting.net.

Funding

To execute this study, no funding was needed. The costs for travel and lodging during the Barents Rescue 2015 exercise was covered by the Laurea University of Applied Sciences. The Norwegian Air Ambulance Foundation covered the article processing charge.

Authors’ contributions

LR conceived the idea. JV, SF, JJ, TW, MR, and LR contributed to the planning of the study. JV, LR, and JJ recruited voluntary teams for reporting. Data were collected by the voluntary teams. Electronic collecting and extracting data from the major incident reporting database were facilitated by SF. JE and JV analyzed the data. All authors contributed to the writing of the manuscript and have approved the final version.

Acknowledgements

We acknowledge all the teams that participated in the study. Sincere thanks to Ellen Blix and Mari Salminen-Tuomaala for reviewing the manuscript. We sincerely thank Harri Antikainen, PhD, from the Geography Research Unit at the University of Oulu for preparing the map. We would also like to convey our gratitude to Petra Portaankorva, Chief EMS physician at Central hospital of Länsi-Pohja, for providing volunteers for this study.

Co-writers:

Jyri Viinamäki 1,2, Sabina Fattah 3,4, Torben Wisborg 4,5,6, Marius Rehn 3,7,8, Janne Engblom 9, Lasse Raatiniemi 10,11

1. Sector of Pre-hospital Emergency Care, Anesthesia, Intensive Care and Emergency Department, Emergency Department Acuta, Tampere University Hospital, Tampere, Finland
2. Master’s Degree Program in Crisis and Emergency Management, Laurea University of Applied Sciences, Vantaa, Finland
3. Department of Research. Norwegian Air Ambulance Foundation. Drøbak, Norway
4. Anestesia and Critical Care Research Group, Faculty of Health Sciences, University of Tromsø, Norway
5. Hammerfest Hospital, Department of Anesthesiology and Intensive Care, Finnmark Health Trust, Hammerfest, Norway
6. Norwegian National Advisory Unit on Trauma, Division of Emergencies and Critical Care, Oslo University Hospital, Oslo, Norway
7. Department of Health Studies, University of Stavanger, Stavanger, Norway
8. Division of Emergencies and Critical Care, Department of Anesthesia. Oslo University Hospital. Oslo, Norway
9. Department of Statistics, University of Turku, Finland
10. Centre for Pre-Hospital Emergency Care/FinnHEMS 50, Oulu University Hospital, Oulu, Finland
11. Anesthesia Research Group, MRC Oulu, University of Oulu, Oulu, Finland

References:

• Bradt DA, Aitken P (2010) Disaster medicine reporting: the need for new guidelines and the CONFIDE statement. Emerg Med Australas. 22 (6):483–487.
• Braun, V, Clarke, V (2006) Using thematic analysis in psychology. Qual Res Psychol 3 (2): 77-101.
• Castrèn M, Hubloue I, Debacker M. (2012) Improving the science and evidence for the medical management of disasters: Utstein style. Eur J Emerg Med. 19 (5):275–276.
• Centre for Research on the Epidemiology of Disasters (CRED). EM-DAT: International Disaster database. http://cred.be/. Accessed 27th of November 2016.
• Chen C-T, Hung C-T, Lin J-D, Sung P-H (2015) Application of a decision tree method with a spatiotemporal object database for pavement maintenance and management. Journal of Marine Science and Technology 23 (3):302-307.
• Cohen J (1960) A coefficient of agreement for nominal scales. Educational and psychological measurement 20 (1): 37-46.
• Endsley MR, Garland D (eds.) (2000) Situational awareness analysis and measurement. Mahwah (NJ), London: Lawrence Erlbaum Associates.
• Ensley, MR. (1995) Toward a Theory of Situation Awareness in Dynamic Systems. Human factors 37 (1995): 85-104.
• Fattah S, Agledahl KM, Rehn M, Wisborg T. (2016) Implementing a global open access template for reporting major incidents: qualitative feasibility study. Disaster Med. Public Health Prep. 11(4): 403-406.
• Finnish Advisory Board on Research Integrity. Responsible conduct of research and procedures for handling allegations of misconduct in Finland – RCR guidelines and ethical review in human sciences. http://www.tenk.fi/sites/tenk.fi/files/HTK_ohje_2012.pdf. Accessed 10 October 2016.
• Guha-Spir D 2016. Natural disasters in 2015. Disaster data: a balanced perspective. Cred Crunch 41. Brussels: Centre for Research on the Epidemiology of Disasters.
• Lennquist S. (2011) Introduction to the third “Focus-on” issue specially devoted to papers within the field of the ESTES section for Disaster and Military Surgery. Eur J Trauma Emerg Surg. 37 (1):1–2.
• Lockey DJ 2012. The shootings in Oslo and Utoya island July 22, 2011. Lessons for the International EMS community. Scand J Trauma Resusc Emerg Med. 20 (1):4.
• NHS England National Emergency Preparedness, Resilience and Response Unit. NHS England National Emergency Preparedness, Resilience and Response Framework. https://www.england.nhs.uk/wp-content/uploads/2015/11/eprr-framework.pdf. Accessed 18 November 2016.
• Norri-Sederholm T, Paakkonen H, Kurola J, Saranto K. 2015. Situational awareness and information flow in prehospital emergency medical care from the perspective of paramedic field supervisors: a scenario-based study. Scand J Trauma Resusc Emerg Med. 23 (1):4.
• Raatiniemi L, Martikainen M, Jama T, ja Alahuhta S. 2016. Miten voimme oppia suuronnettomuuksista ja harjoituksista? Lääketieteellinen Aikakauskirja Duodecim 132 (2):115-116.
• Smeeton NC. 1985. Early History of the Kappa Statistic. Biometrics. 41 (3): 795-795.
• Stratton SJ. 2012. The Utstein-style template for uniform data reporting of acute medical response in disasters. Prehosp Disaster Med. 27 (3): 219-219.