Original Article

 

Predictive validity testing of severely injured

limb scoring systems

 

Mohsen Karami,*^{, ‡} MD; Mir Mostafa Sadat,** MD; Hassan Tavakkoli,^† MD;

Morteza Taghavi,^† MD; and Mohammad Reza Golbakhsh,** MD

 

*Spine and Orthopedic Research Center (SSRC), **Department of Orthopedics, ^†Department of Surgery

Sina Hospital, Tehran University of Medical Sciences, Tehran, Iran

---

INTRODUCTION

 

Injuries to the musculoskeletal system often appear dramatic and occur in 85% of the patients, who sustain blunt trauma, but rarely cause a threat to life or limb. However, in some circumstances, the relevance of such injuries is of major importance. This study concentrates on certain limb injuries, which  present with crush  injury  and extensive soft tissue damage, concomitant vascular or nerve injury and major bony disruption.

During the past 2 decades, better understanding of the injury itself and technical advances in surgery (allowing revascularization of the extremity, stabilization of the complex fracture, and reconstruction of the soft tissues) medicine, and rehabilitation have led to an increased frequency of attempts at limb salvage. In some of these patients, however, limb salvage may have subsequent deleterious results, which is associated with a high morbidity and poor prognosis, and often requires late amputation (27 – 70%) despite initial success. In these cases, early or primary ablation might even be beneficial.

Attempts to qualify the severity of the trauma and to establish numerical guidelines to whether amputate or salvage the limb have been proposed by several authors. Published severity scoring systems of lower extremity injury include the Mangled Extremity Severity Scale (MESS) ¹; the Predictive Salvage Index (PSI) ²; the nerve injury, ischemia, soft tissue injury, skeletal injury, shock, age of patient (NISSSA)³ and Mangled Extremity Syndrome Index (MESI) score.⁴ The developers of these scoring systems attempted to validate them by demonstrating high rates of specificity and sensitivity in predicting limb salvage. However, independent testing of some of these scoring systems has not duplicated the success reported by the developers.^{5 – 7}

We shall attempt to establish guidelines for this group of patients in order to aid the surgeon in managing the options available. This study was approved by the ethics committee of Spine and Orthopedic Research Center of Tehran University of Medical Science.

 

Patients and METHODS

 

Subject

Over the past 3 years (1998 – 2001), every patient with open fracture that was referred to Sina Hospital and met the following criteria of the mangled extremity syndrome, was included in this study: (1) severe injury to three of four organ systems (integument, bone, nerve, and vessel) as described previously by Gregory et al;⁴ and (2) severe injury to two of the four organ systems with minor injury to two of four systems that require surgical intervention.⁴ Primary amputations were excluded from the analysis. A primary amputation was defined as a limb that underwent amputation as the primary treatment modality. Patients with traumatic amputation, isolated foot or hand crush injury, and all who died within 2 weeks after admission were also excluded from the study.

 

Study design

We modified NISSSA and PSI scores in order to use them in severely injured upper and lower extremity. To salvage each limb the results of MESS, MESI, modified NISSSA, and PSI scoring systems were calculated at the emergency room by one surgical resident.  The results of these scoring systems weren’t considered in either amputating or salvaging a limb (we amputated only the limbs that we couldn’t save, because of severe or life threatening infection and arterial graft failure which we couldn’t repair again). The four injury-severity scores of lower extremity vary in terms of the factors considered relevant to limb salvage and are summarized in Table 1. With the exception of the NISSSA  and  PSI,  the  component  levels  of   each

injury severity score were totaled according to the instructions specified by its developers. The NISSSA score was modified for use in other lower extremity trauma (in addition to tibial open fracture) and upper extremity trauma by considering: (1) sensory deficit at volar area of the hand, as plantar area of lower extremity; and (2) sensory deficit of dorsal area of the hand as dorsal area of lower extremity. This index was developed only for evaluating limb viability associated with open tibial fracture. The PSI score was also modified according to the level of vascular injury of upper extremity (brachial artery injury at elbow region was considered as popliteal artery injury at lower extremity).

General operative approach was divided into three phases: (1) reduction and fixation of fractures and dislocations (with priority as determined by vascular surgeon); (2) vascular (arterial and venous) injuries were defined and repaired and; (3) wound management was performed. Functional and residual deficiency score was obtained by patient interview and examination in a minimum six months after trauma (Table 2). Residual deficiency score (RSD) equal to or greater than 7, meant severe functional deficiency and useless limb.

 

Statistics

To examine the discriminated validity of the four injury-severity scores of lower extremity, sensitivity, specificity, and Yourdon’s J for predicting amputation were calculated. Yourdon’s J describes a statistics that combines sensitivity and specificity so that J = 1 indicates maximum sensitivity and specificity and J = 0 indicates no relation between the predicted outcomes of an index and the observed outcomes. The sensitivity is defined as the number of limbs amputated with scores, at or above the threshold, divided by the total number of limbs amputated in at least six- months of follow-up period. Specificity is defined as the number of salvaged limbs, with scores below the threshold, divided by  the total  number  of  salvaged limbs for at least six months after the injury. The recommended amputation-threshold scores published for the MESS, MESI, PSI, and NISSSA were used in calculating sensitivity, specificity and Yourdon’s J of all injured limbs and tibial fractures separately (the modified NISSSA and PSI had the same threshold score). The performance of these indices was also examined for limbs that were not functionally usable at the last follow-up (at least six months). Correlation between the results of these scoring systems and fate of the severely injured limb was calculated by Mann Whitney test.

 

Table 1. Variables of predictive indices.
Variables	MESI	PSI	MESS	NISSSA
Skin/muscle	√	√	√	√
Bone	√	√	√	√
Vessel	√	√	–	–
Nerve	√	–	–	√
Ischemia	√	√	√	√
Shock	√	–	√	√
Age	√	–	√	√
Mechanism of injury	√	–	√	√
Injury severity score	√	–	–	–
Comorbid disease	–	–	–	–
MESS = Mangled Extremity Severity Scale; PSI = Predictive Salvage Index; NISSSA = Nerve injury, ischemia, soft tissue injury, skeletal injury, shock, and age; MESI = Mangled Extremity Syndrome Index.

 

Table 2. Residual deficiency score (RDS).*

 

Loss of  sensation**	None = 0	Dorsal = 1	Planter(volar) = 2
Loss of  ROM*** (same or upper joint)	One third of  total ROM = 1	Two third of  total ROM = 2	Complete loss of ROM = 3
Loss of ROM (lower joint)	One third of total ROM = 1	Two third of  total ROM = 2	Complete loss of ROM = 3
Trophic ulceration	No = 0	Yes = 1
Use of crutches or orthosis	No = 0	Yes = 1

 

*Total scale ≥ 7 means functionless limb; **Protective sensation considered as –0.5 score on each session; ***ROM = range of motion.

 

 

RESULTS

 

Over the past 3 years (1998 – 2001), 324 trauma patients, with open fractures that were referred to Sina Hospital, participated in this project. Thirty-six of the patients met the criteria of the mangled extremity syndrome. Excluding primary amputations and all who died within 2 weeks after admission, there were 33 severely damaged limbs in these thirty-one patients (30% upper and 70% lower extremities). All of the patients had concomitant open fracture of tibia (Figure 1). The mean age of the patients was 28 years (range: 17 – 63); 28 of the 31 (90%) patients were male. The average injury severity score (ISS) was 9 points; only 2 patients (6%) had an ISS of more than 16 points.

The mean time of follow-up was 19 months (range: 6 – 40). The mean of residual deficiency score (RDS) at the final follow-up was 2.74 (range: 0 – 10, SD = 3.02).  Four of 25 remaining limbs had   

 

 

Figure 1. Diagram and distribution of patients.

 

RDS greater than or equal to 7, which meant severe functional deficiency and useless limb. However, all of these patients refused amputation at the last follow-up.

The number of amputations (all in the first month after trauma) and extremities that were salvaged but were not functional (according to the predicted outcome as determined with the threshold scores of the MESS, MESI, modified NISSSA, and PSI) are presented in Table 3. Eight (24.2%) of the limbs were amputated; seven (88%) amputations were lower extremities, and one (12%) was upper extremity. The cause of amputation of five limbs was severe infection and the others, unreconstructable graft failure.

The sensitivity, specificity, and Yourden’s J of the MESS, MESI, modified NISSSA, and PSI were determined for four groups: (1) all patients with upper or lower extremity trauma that had anatomic amputation; (2) all patients with upper or lower extremity trauma that had anatomic or functional amputation; (3) all patients with open tibial fracture that had anatomic amputation; and (4) all patients with open tibial fracture that had anatomic or functional amputation. The results are presented in Table 4.

The MESS and modified NISSSA demonstrated a high specificity (80% and 88%, respectively) and high sensitivity (87% each) for all of injured limb analysis. When only the open tibial fractures were considered, the performance of the MESS deteriorated to a specificity of 68%, but a sensitivity of 100%. The performance of the NISSSA in the evaluation of the injured lower extremity had similar sensitivity like MESS (100%), but better specificity     (81%).    The    result    of     predictive      amputation (anatomically or functionally) in each test was lower than anatomical amputations; the modified NISSSA score had greater value than the others (Table 4). Comparing the results of these scoring systems with fate of severely injured limbs (as amputation of all injured limbs and tibial open fractures, anatomically and functionally), revealed that there were significant correlations between the results of the modified NISSSA and fate of severely injured limbs (all p values = < 0.001) (Table 5).

 

Table 3. Number of amputated and salvaged limbs by threshold value of lower-extremity injury-severity scores.

 

	Number of  limbs
Scoring system	Amputated	Amputated + useless limb
	All              Tibial   limbs          fractures	All            Tibial     limbs        factures
MESS ≥ threshold < threshold PSI ≥ threshold < threshold NISSSA ≥ threshold < threshold MESI ≥ threshold < threshold	7                  7     1                  0       6                 6                               2                 1       7                 7     1                 0   6                       5 2                       2	8                       8      4                       2        9                       8      3                       2        8                       8      4                       2        9                      7      3                      3

 

 

 Table 4. Sensitivities and specificities of predictive indices.

 

	Number of limbs
Scoring system	Amputated	Amputated + useless limb
	All              Tibial   limbs         fractures	All            Tibial     limbs       fractures
MESS Sensitivity Specificity Yourden’s J Modified PSI Sensitivity Specificity Yourden’s J Modified NISSSA Sensitivity Specificity Yourden’s J MESI Sensitivity Specificity Yourden’s J	0.87             1    0.80             0.68    0.83             0.84        0.75             0.86             0.72             0.62    0.73             0.74        0.87             1    0.88             0.81    0.87             0.91      0.75             0.71    0.60             0.68    0.68          0.70	0.67             0.80     0.81             0.69     0.74             0.75          0.75             0.80     0.80             0.69     0.78             0.74         0. 67             0.80         0.90             0.85     0.79              0.82       0.75              0.70     0.67              0.77     0.71              0.74

 

MESS = Mangled Extremity Severity Scale; PSI = Predictive Salvage Index; NISSSA = Nerve injury, ischemia, soft tissue injury, skeletal injury, shock, and age; MESI = Mangled Extremity Syndrome Index.

 

 Table 5. Correlation between scoring systems and fate of severely injured limbs.

 

	Amputated	Amputation* + useless
Scoring system	All              Tibial limbs          fractures	All                Tibial limbs             fractures
PSI U rank Significance MESS U rank Significance NISSSA U rank Significance MESI U rank Significance	49               32.5 0.03          0.11   17               9.5 *L              0.0008   13.5            10  *L         0.0011   76               43 0.23        0.41	48                    22.5 0.003               0.0065   46                    22.5 0.002               0.0065   21                    11.5 *L                    0.0003   75                    37 0.06                 0.08

 

*less than 0.0001, MESS = Mangled Extremity Severity Scale; PSI = Predictive Salvage Index, NISSSA = Nerve injury, ischemia, soft tissue injury, skeletal injury, shock, and age; MESI = Mangled Extremity Syndrome Index.

DISCUSSION

 

The injury-severity scores of injured extremities were developed to assist the surgeon in making the initial decision on whether to salvage or amputate an

injured limb. Ideally, a trauma limb-salvage index would be 100% sensitive (all amputated limbs with trauma limb-salvage scores at or above the threshold), 100% specific (all salvaged limbs with scores below the threshold), and with Yourden’s J of 1 (perfect accuracy). Few clinical tests are performed ideally (Table 6). With the exception of the MESI and MESS, the developers of the NISSSA and PSI systems considered only lower-extremity injury evaluation. In the current study, we modified PSI and NISSSA so that these scores could be used for upper and lower extremity injury. Modified NISSSA score and then MESS had a high specificity and sensitivity in predicting fate of severely open tibial fractures and other lower and upper extremity injuries (Table 4).

Howe et al 2 reviewed 21 patients with pelvic or lower-extremity trauma with vascular injuries and proposed the predictive salvage index (PSI). Using the same limb cohort to develop and validate the PSI, Howe et al reported a sensitivity of 78% and a specificity of 100%. In the current study, we were not able to reproduce these findings (Table 4). There was not a significant correlation between this scoring and amputation of all the limbs and lower extremities (p value = 0.03 and 0.11), but in fate of the limb as anatomical and functional amputation, there was a significant correlation (p value = 0.003 in all limb and p value = 0.0065 in lower extremity damage) (Table 5).

The Mangled Extremity Severity Score (MESS) was proposed by Johansen et al¹ in 1990. The MESS was developed retrospectively in a study of twenty-five patients. The index was then validated in that same patient group and in a group of twenty-six additional limbs that were assessed prospectively. Johansen et al concluded that a MESS score of 7 or more was 100% predictive of amputation. The performance of the MESS in our prospective series did not duplicate these findings. If all of the limbs in the present study were considered, the sensitivity of the MESS was 87%. The sensitivity increased to 100% only if the severely open tibial fractures were considered (Table 4). The sensitivity decreased to 67% when the functional amputations were included in the analysis. The significance of correlation between this scoring system and fate of the limb was better than other scoring systems except for the modified NISSSA (p value = 0.002 and 0.0065) (Table 5).

 

Table 6. Sensitivities and specificities of predictive indices.