Philip R. Westbrook, MD, Michael J. Dickel, PhD, Dennis Nicholson, MD, Daniel J. Levendowski, MBA, Timothy Zavora, BS, Vladimir Simeunovic, BS and Roy Dalati, BS
Abstract
Objective: To compare two commercially available limited channel systems utilizing in-home self-applied recorders to each other and to standard in-laboratory attended nocturnal polysomnography (PSG) for the diagnosis of obstructive sleep apnea.
Study Design: A sequential trial with blinded comparison.
Methods:Each of 24 male patients suspected to have obstructive sleep apnea had three diagnostic studies performed. The participants first completed a self-administered NovaSom™ QSG™ (NovaSom), followed by a standard full-PSG, and then a self-administered Apnea Risk Evaluation System (ARES™) study. Both portable Level III systems were compared to PSG as the benchmark, using PSG AHI of 15 as the diagnostic cutoff.
Results: Twenty of 24 patients successfully completed the study. No failure of the in-home systems occurred. Both systems compared favorably in diagnostic accuracy to the PSG. The sensitivity, specificity, positive predictive value, and negative predictive value for the ARES were 100.0, 83.3, 93.3, 100.0; for the NovaSom–4% desaturation criteria were 78.6, 100.0, 100.0, and 66.7; and for the NovaSom–2% desaturation criteria were 85.7, 83.3, 92.3, and 71.4 respectively.
Conclusions:Both limited channel systems worked reliably when self-applied in the home and demonstrated acceptable accuracy when compared to polysomnography for diagnosing obstructive sleep apnea in this selected population.
Introduction
Obstructive sleep apnea (OSA) is a disorder characterized by frequent episodes of sleep induced obstructed breathing accompanied by hypoxemia and terminated with brief unremembered arousals. It has been documented as a prevalent chronic affliction of persons living in industrialized countries, causing disabling symptoms and potentially doubling healthcare utilization and cost1–6. The disorder has also been identified as a significant risk factor for accidents, cardiovascular disease and diabetes. In spite of the fact that effective treatment exists, in the United States it is estimated that only a small fraction of the 20 million7, 8 individuals estimated to suffer with OSA have been diagnosed. In part this may reflect ignorance about the disease on the part of health care providers. However, it has also been opined that the usual diagnostic evaluation is limited by cost and, in some cases, availability.9 The “gold standard” test, in-laboratory attended polysomnography (PSG), while expensive, is supported by current reimbursement policies.
In 2003, a joint task force of the American Thoracic Society the American Academy of Sleep Medicine, and the American Academy of Chest Physicians found insufficient medical evidence to support the efficacious use of portable monitoring.10–12 Medicare followed with a decision which disallowed reimbursement for continuous positive airway pressure (CPAP) if portable monitoring was used for the diagnosis of OSA. Following Medicare’s policy change, many managed care groups and insurers chose not to reimburse for an in-home unattended study. The only positive outcomes of the task force findings, with respect to portable monitoring, were recommended study design guidelines for assessing the efficacy of limited channel, unattended systems intended for the diagnosis of OSA in the home. These guidelines provide a blueprint for assessment of in-home OSA testing using an evidence-based approach.
During recruitment of patients for an ongoing study, an opportunity was provided to compare the diagnostic capabilities of two of the systems that meet the requirements for a Type 3 Device (Modified Portable Sleep Apnea Testing) when used unattended in-home. At that time, the VA Long Beach Healthcare System had just begun referring patients for NovaSom (NovaSom™ QSG™, Sleep Solutions, Pasadena, MD) in-home studies as a means to reduce its patient backlog. The VA Long Beach Healthcare System was a referral source of patients for the ARES™ (Apnea Risk Evaluation System, Advanced Brain Monitoring, Carlsbad, California) validation study. Subsequent to the acquisition of these data, both portable monitors were validated in the home13–15 in a combined total of 272 comparisons with PSG. This study provided an opportunity for an independent “real world” evaluation of how the NovaSom and the ARES compared to each other and how well they each compared to PSG.
Methods
Twenty-four consecutively recruited individuals scheduled for a NovaSom study by the Sleep Center at the Veteran’s Administration Long Beach Healthcare System participated in this study. All subjects completed the NovaSom in-home study first, followed by overnight PSG, and then the ARES Unicorder™ in-home study. Typically, a month lapsed before all three studies were completed. This study protocol was approved by the Institutional Review Boards of Advanced Brain Monitoring, with deferring approval by the Pomona Valley Hospital and the Veteran’s Administration Long Beach Healthcare System Institutional Review Boards. Informed consent was obtained from all subjects.
For the unattended NovaSom study, three nights of data were acquired in-home. The NovaSom acquires oxyhemoglobin saturation (SpO2) and pulse rate using a transmittance finger probe, snoring sounds and airflow derived from snoring sounds using a microphone, and respiratory effort by means of a chest piezo belt. The NovaSom includes audio alerts to notify the user when transducers have become detached. The signals are processed in real time to detect abnormal breathing and summary data is saved to a bedside recorder. The NovaSom system did not monitor head or body position. Patients were mailed the device and self-applied it using written and video instructions. A photo of the NovaSom System is presented in Figure 1.
The PSGs were performed at a community hospital-based American Academy of Sleep Medicine accredited Sleep Disorders Center using Sandman Elite V6.2 hardware and software (Nellcor Puritan Bennett, Kanata, ON). Signals included: electroencephalogram (EEG) (C4-A1, C3-A2, O2-A1, O1-A2), electrooculogram (EOG), submental and bilateral tibial electromyogram (EMG), electrocardiogram (ECG), airflow (nasal thermistor and nasal pressure (PTAF2, Pro-Tech Services, Woodinville, WA)), chest and abdominal motion (piezo bands), SpO2, and snoring intensity. An ARES Unicorder was worn concurrently during the PSG to obtain head position data.
For the unattended ARES study, two nights of data were acquired in-home with the ARES Unicorder. The Unicorder is a single, forehead site device that records the optical signals used to derive SpO2 and pulse rate, head movement and head position (accelerometers), and snoring sounds (microphone). (The model Unicorder used in this study did not include a measure of airflow). The Unicorder provides audio and visual alerts to notify the user when it is incorrectly affixed or when signal quality is poor. The full disclosure recording is downloaded and processed off-line to detect obstructive breathing during sleep. The Unicorder was mailed to the patient who self-applied it following written instructions. A photo of the Unicorder is presented in Figure 2.
Sleep Solutions and Advanced Brain Monitoring provided telephone technical support as necessary to patients using their respective devices. Of the twenty-four patients who were studied, the data from four subjects were unusable. Two cases were dropped because the patients were unable to sleep during their PSG (i.e., sleep time < 1.5 hours). One PSG record was unusable due to an incorrect setting of the averaging window of the pulse oximeter. The fourth patient was dropped from the study because of severe chronic obstructive pulmonary disease (COPD) which precluded him from being a candidate for unattended in-home OSA testing. No patients had to be dropped from the study because of failure of the in-home recording.
Apneas and hypopneas were visually scored in the PSG records by trained technologists and reviewed by a physician board-certified in sleep medicine. A desaturation threshold of 3% was required for all apnea and hypopneas used in the calculation of the apnea hypopnea index (PSG-AHI). The NovaSom data were analyzed using proprietary automated scoring algorithms. Because the NovaSom reports RDI based on a 2% and 4% desaturation level, both results were analyzed. The ARES-DI was computed using rules previously described13 relying primarily on the oximeter to determine abnormal breathing. A stepped approach that adapts based on the SpO2 level prior to the start of the desaturation was used to determine the necessary desaturation level (i.e., ranging from a desaturation plus resaturation of 2.2% and 2.2% when the start of the desaturation = 95% to 4.0% and 3.2% when the start of the desaturation was < 88%). Changes in heart rate and head movement indicative of arousals were used to confirm desaturations when the 2.2% threshold was applied. The definition for the PSG-AHI was the total number of abnormal breathing events divided by the hours of sleep, and the definition for the ARES-RDI and NovaSom-RDI was the total number of abnormal breathing events divided by the hours of recording time. The term RDI refers to the in-home studies and AHI refers to the PSG studies to distinguish between the use of total recording time and total sleep time. A clinical cut-off of 15 was used for the AHI/RDI. All parties involved in reviewing the data (PSG or in-home) were blinded to the results of the other systems. The average duration of the 20 completed PSG studies was 5.9 hours. The average valid recording time across both nights of the ARES studies was 11.4 hours. The average time-in-bed across the three nights of NovaSom studies was 18.1 hours.
The kappa coefficient was used to assess the beyond-chance agreement between the ARES-RDI, the NovaSom-RDI and PSG-AHI, without using the assumption that the PSG was the “gold-standard” comparison. Clinical cut offs of AHI = 10, 15, 20 and 30 were used to derive the Receiver Operator Characteristics of the two systems. Pearson correlations were used to compare the PSG-AHI vs. the ARES-RDI, NovaSom-2% RDI and Novasom-4% RDI.
Correlation analysis was also applied to the differences between the PSG-AHI and the ARES-RDI in all positions vs. only supine. It was also used to assess the relationship between the percentage of time supine during PSG compared to the percentage of time supine across both nights of ARES in-home recordings.
Results
Table 1 presents the cross tabulations of the ARES and NovaSom vs PSG comparisons used to compute the statistical results shown in Table 2. The kappa coefficients for the PSG vs. ARES (0.88), NovaSom-4% (0.69) and NovaSom-2% (0.66) indicate an excellent agreement beyond chance (Table 2). The ARES provided greater sensitivity than the NovaSom-4% or NovaSom-2% (100% vs. 78.6% and 85.7%, respectively). The NovaSom-4% provided better specificity than the ARES or NovaSom 2% (100% vs. 83.3% and 83.3% respectively). The NovaSom-4% provided a slightly better positive predictive value as compared to the ARES or NovaSom-2% (100% vs. 93.3 and 92.3, respectively). The ARES provided a substantial improvement in negative predictive value as compared to NovaSom-4% and NovaSom-2% thresholds (100% vs. 66.7% and 71.4%, respectively).
The Bland-Altman plots for the ARES-RDI, NovaSom-4% RDI and NovaSom-2% RDI vs. PSG-AHI are presented in Figures 3.a., 3.b., and 3.c., respectively. Novasom-4% exhibited the greatest bias (-5.6 events/hr) as compared to the ARES (-3.1 events/hr) and Novasom-2% (-2.8 events/hr). The ARES provided smaller standard deviations about the mean (6.3 events/hr) compared to NovaSom-4% (8.7 events/hr) and Novasom-2% (9.7 events/hr) (Figure 3). The correlation between the PSG-AHI and ARES-RDI was 0.93. The corresponding correlation for the NovaSom-4% RDI and NovaSom-2% RDI vs. PSG-AHI was 0.86 and 0.82, respectively.
A tabular presentation of the Receiver Operating Characteristics of the ARES and NovaSom vs. PSG is presented in Table 3 for clinical cut offs of AHI = 10, 15, 20, and 30. The NovaSom-2% provided perfect sensitivity and specificity with a clinical cut off of 10; the ARES provided a poorer specificity and the NovaSom-4% provided a poorer sensitivity. At all other clinical cut offs the ARES and NovaSom-2% and NovaSom-4% provided similar specificities while the ARES provided similar or better sensitivity.
To assess influence of position on the PSG-AHI and ARES-RDI, the differences between the ARES-RDI and PSG-AHI were computed across all positions and only in the supine position. The correlation between these two measures was 0.65. The correlation between the percentage of time supine during full-PSG (no split night studies) and across two nights of in-home recordings was 0.66.
Discussion
Recently a number of policy decisions have been published which, taken together, potentially impact the use of portable monitoring. First, the Institute of Medicine, in a published report, urged the evaluation and validation of existing diagnostic technologies to assist in the diagnosis of millions of undiagnosed individuals suffering from sleep disorders.16 Shortly thereafter, the American Academy of Sleep Medicine (AASM) issued practice parameters for Portable Monitoring in the Diagnosis of Obstructive Sleep Apnea.17 The AASM recommended that accredited sleep laboratories and board certified Sleep Specialists use portable monitoring when combined with a clinical assessment, and that the therapy decision be based on both the results of the study and knowledge of the individual patient’s symptoms. It appears that insurers are now following with modifications to their coverage position that includes the use of portable monitoring devices (level III) when PSG is not readily available, the patient is unable to be studied in the laboratory, or a home study is being used to evaluate the treatment rendered after the diagnosis of OSA has been established by a previous PSG or home sleep study.18
As recommended by the Institute of Medicine, both Type III devices used in this study were previously validated. The ARES was validated both concurrently with PSG 324 studies (261 patients and 63 presumably healthy controls) and in 227 unattended in-home study comparisons (172 patients and 55 controls).13,14 The NovaSom was previously validated concurrently with PSG in 101 patients and in 45 unattended in-home comparisons.19 Each system has been compared to PSG using procedures that minimized bias. Studies have been performed in an unattended setting and failure rates/lost data of the system were reported.
In this comparison study, both the ARES and NovaSom provided an effective alternative to laboratory-based PSG in a population of Veteran’s Administration patients. Differences in the sensitivity and specificity of the NovaSom and ARES (Table 2) as compared to previous reports 13, 14, 19 may be attributed to the small sample size in this study (n = 20). The ARES and NovaSom bias toward under-reported severity in patients with PSG-AHI levels above 60 is most likely attributed to additional recording time. The sensitivity of the NovaSom-4% was clearly influenced by the 4% desaturation requirement, as compared to the ARES stepped approach and the NovaSom-2%, as a result of the 3% desaturation requirement used to compute the PSG-AHI. In a previous report, the majority of the differences between the PSG-AHI and ARES-RDI that resulted in misclassifications were attributed to the influence of head position.13 Head position again appeared to explain some portion of the differences between the PSG-AHI and ARES-RDI. Analysis of the influence of position (head or body) on the NovaSom results was not possible because this measurement is not obtained
The benefit of using multiple testing nights as a means to improve the diagnostic accuracy of OSA is beginning to emerge. Recent studies suggest that at least 10% of sleep-disordered breathing cases would be misclassified based on only one night of data.20 Time-in-position was a primary source of misclassifications between in-home and PSG studies 13 and substantial night-to-night variability in head position during sleep was observed in that study. The use of automated scoring software reduces the labor required to hand-score multiple night of data, and eliminates inconsistencies introduced by human scoring. 21–23 The analytical techniques used for the ARES auto-scoring have been fully described;13 the NovaSom methods have not been described. An advantage of the ARES over the NovaSom is that full-disclosure recordings are acquired that can be visually inspected and hand-scored if necessary to verify the automated scoring algorithms. This capability is useful in the diagnosis of difficult cases or when less than optimal signal quality is acquired. Both the ARES and NovaSom include on-line monitoring and user warning alarms to assist the patient and ensure that high quality recordings are acquired, a feature unique to these two systems.
The patient with COPD provided a case study example of someone with a severe breathing impairment who is not a candidate for an for in-home OSA testing. During the PSG study the patient’s desaturation levels were below 50%, resulting in the attending technician calling emergency personnel. Patients hard of hearing (i.e., unable to respond to the alerts), or who suffer from severe arthritis, dementia, have cardiac arrhythmia, congestive heart failure or are on supplemental oxygen use at night are also not considered appropriate for unattended OSA diagnostic testing. Limited channel systems such as the ARES and NovaSom cannot be used to diagnose narcolepsy or sleep-related movement disorders.
This study confirms that both systems provide acceptable diagnostic accuracy in an unattended setting compared to laboratory PSG in a sample of VA patients suspected of having OSA.
Acknowledgements
The authors wish to thank Michele Burnett and George Wirtes from Pomona Valley Hospital Sleep Disorders Center, and Nancy Daily of the VA Long Beach Healthcare System for their assistance in managing the clinical studies and preparing the data.
References
1. Smith R, Ronald J, Delaive K, Walld R, Manfreda J, Kryger MH. What are obstructive sleep apnea patients being treated for prior to this diagnosis? Chest. 2002;121(1):164–172.
2. Kryger MH, Roos L, Delaive K, Walld R, Horrocks J. Utilization of health care services in patients with severe obstructive sleep apnea. Sleep. 1996;19(9 Suppl):S111–S116.
3. Kapur VK, Redline S, Nieto FJ, Young TB, Newman AB, Henderson JA. The relationship between chronically disrupted sleep and healthcare use. Sleep. 2002;25(3):289–296.
4. Cooney RN, Haluck RS, Ku J, et al. Analysis of cost outliers after gastric bypass surgery: what can we learn? Obes Surg. 2003;13(1):29-36.
5. Ronald J, Delaive K, Roos L, Manfreda JH, Kryger MH. Obstructive sleep apnea patients use more health care resources ten years prior to diagnosis. Sleep Res Online. 1998;1(1):71–74.
6. Bahammam A, Delaive K, Ronald J, Manfreda J, Roos L, Kryger MH. Health care utilization in males with obstructive sleep apnea syndrome two years after diagnosis and treatment. Sleep. 1999;22(6):740–747.
7. National Sleep Foundation. Sleep Facts and Stats. http://www.sleepfoundation.org/NSAW/pk_sleepfacts.cfm.
8. Young T, Palta M, Dempsey JA, Skatrud J, Weber SM, Badr MS. The occurrence of sleep-disordered breathing among middle-aged adults. N Engl J Med. 1993;328:1230–1235.
9. Pack AI. Sleep-disordered breathing: access is the issue. Am J Respir Crit Care Med. 2004;169(6):666–667.
10. Executive summary on the systematic review and practice parameters for portable monitoring in the investigation of suspected sleep apnea in adults. Am J Respir Crit Care Med. May 15 2004;169(10):1160–1163.
11. Chesson AL, Jr., Berry RB, Pack A. Practice parameters for the use of portable monitoring devices in the investigation of suspected obstructive sleep apnea in adults. Sleep. Nov 1 2003;26(7):907–913.
12. Flemons WW, Littner MR, Rowley JA, et al. Home diagnosis of sleep apnea: a systematic review of the literature. An evidence review cosponsored by the American Academy of Sleep Medicine, the American College of Chest Physicians, and the American Thoracic Society. Chest. Oct 2003;124(4):1543–1579.
13. Westbrook P, Levendowski D, Cvetinovic M, et al. Description and validation of the apnea risk evaluation system: a novel method to diagnose sleep apnea-hypopnea in the home. Chest. Oct 2005;128(4):2166–2175.
14. Reichert JA, Bloch BA, Cundiff E, Votteri BA. Comparison of the NovaSom QSG, a new sleep apnea home-diagnostic system, and polysomnography. Sleep Medicine. 2003;4:213–218.
15. Ayappa I, Rapoport D, Westbrook P, Levendowski D, Zavora T, Norman R. Validation of a Self-Applied Unattended Monitor for Sleep Disordered Breathing (SDB). Sleep. 2006;29(Abstract Supplement 29):A343.
16. Committee on Sleep Medicine and Research. Ensuring Adequate Diagnosis and Treatment: Access, Capacity, and Technology Development. In: Colten HR, Altevogt BM, eds. Sleep Disorders and Sleep Deprivation: An Unmet Public Health Problem. Washington, D.C.: The National Acadamies Press; 2006.
17. American Academy of Sleep Medicine. Portable Monitoring in the Diagnosis of Obstructive Sleep Apnea. Journal of Clinical Sleep Medicine. 2006;2(3):333.
18. CIGNA HealthCare Coverage Position: Home Sleep Studies for Obstructive Sleep Apnea: CIGNA Health Corporation; 2006.
19. Claman D, Murr A, Trotter K. Clinical validation of the Bedbugg in detection of obstructive sleep apnea. Otolaryngol Head Neck Surg. 2001;125(5):227–230.
20. Stepnowsky CJ, Orr WC, Davidson TM. Nightly variability of sleep-disordered breathing measured over 3 nights. Otolaryngology. 2004;131(6):834–843.
21. Norman RG, Pal I, Stewart C, Walsleben J, Rapoport DM. Interobserver agreement among sleep scorers from different centers in a large dataset. Sleep. 2000;23(7):901–907.
22. Whitney CW, Gottlieb DJ, Redline S, et al. Reliability of scoring respiratory disturbance indices and sleep staging. Sleep. 1998;21(7):749–757.
23. Collop NA. Scoring variability between polysomnography technologists in different sleep laboratories. Sleep Medicine. 2002;3
