Wireless, Web-Based Interactive Control of Optical Coherence Tomography with Mobile Devices

Abstract

Purpose: Optical coherence tomography (OCT) is widely used in ophthalmology clinics and has potential for more general medical settings and remote diagnostics. In anticipation of remote applications, we developed wireless interactive control of an OCT system using mobile devices.

Methods: A web-based user interface (WebUI) was developed to interact with a handheld OCT system. The WebUI consisted of key OCT displays and controls ported to a webpage using HTML and JavaScript. Client–server relationships were created between the WebUI and the OCT system computer. The WebUI was accessed on a cellular phone mounted to the handheld OCT probe to wirelessly control the OCT system. Twenty subjects were imaged using the WebUI to assess the system. System latency was measured using different connection types (wireless 802.11n only, wireless to remote virtual private network [VPN], and cellular).

Results: Using a cellular phone, the WebUI was successfully used to capture posterior eye OCT images in all subjects. Simultaneous interactivity by a remote user on a laptop was also demonstrated. On average, use of the WebUI added only 58, 95, and 170 ms to the system latency using wireless only, wireless to VPN, and cellular connections, respectively. Qualitatively, operator usage was not affected.

Conclusions: Using a WebUI, we demonstrated wireless and remote control of an OCT system with mobile devices.

Translational Relevance: The web and open source software tools used in this project make it possible for any mobile device to potentially control an OCT system through a WebUI. This platform can be a basis for remote, teleophthalmology applications using OCT.

Introduction

Acute care settings like the emergency department (ED) often serve as a care access point for patients seeking eye care.¹ Unfortunately, access to specialty ophthalmic services may be limited in these acute care settings, and inadequate ophthalmic assessment can put the patient at increased risk for delayed care and visual impairment.² Remote diagnostics such as teleophthalmology efforts (e.g., Parel JM, et al. IOVS 2012;53:ARVO E-Abstract 3633) could help increase access to specialty ophthalmic triage and care in these acute care settings.^3–5 

In addition to limited access to specialty ophthalmic care, another impediment to ocular examination in the acute care setting is that the major diagnostic device for the eye—the direct ophthalmoscope—is difficult to use for many general providers.⁶ As an alternative, studies have utilized fundus photography as a diagnostic device that is easier to use and has the additional potential for teleophthalmology support.^7–8 While promising, fundus photography is still a two-dimensional imaging modality. Important pathologies that may be seen in the acute setting, such as diabetic macular edema and papilledema, have a distinct elevation component that is more readily visualized using a three-dimensional modality such as optical coherence tomography (OCT). In the specialty ophthalmic setting, OCT is widely used, and patients with these pathologies would likely be imaged with OCT. In the acute care setting, though, availability of OCT is limited for various reasons. 

From a hardware standpoint, efforts have been made to make OCT more accessible for nonspecialty ophthalmic settings primarily by reducing the size of the OCT system. Portable OCT systems with handheld probes provide a format similar to handheld diagnostics already used in general and acute care settings (e.g., ophthalmoscope).^9–11 Our group also recently developed a handheld, high-speed swept-source OCT system probe capable of switchable anterior and posterior ocular imaging for use in acute care settings.¹² A key feature of these handheld OCT probes is on-probe display and control to further increase the usability of the device. Most of these handheld OCT probes have used customized LCD screens designed specifically for this purpose.^13,14 To provide a more general solution and to also offer the potential for telemedicine capabilities, we sought an on-probe display and control solution that could be used on any available portable, interactive screen. 

Our proposed solution was to create a novel web-based user interface (WebUI) for on-probe OCT display and control. The WebUI can be accessed from any available mobile device such as a cellular phone, tablet, or laptop. This mobile device (cellular phone in our case) can be mounted onto the handheld OCT probe to display and control the system. Communication between the WebUI and the OCT system takes place across the internet using standard communications protocols. Because of this architecture, additional mobile devices can simultaneously access the OCT session, making this a promising option for telemedicine purposes. In this work, we demonstrate the successful use of this platform-neutral WebUI to enable multisite collaborative viewing and control of handheld OCT imaging sessions in an acute care setting. 

Methods

A WebUI was developed to provide wireless control of an OCT system from any mobile device and to display live OCT images on that or any other mobile device. The specific OCT system used in this work was a previously developed swept-source OCT system with a handheld, switchable anterior/posterior imaging probe and integrated iris aiming camera.¹² This previously developed system was an investigational device used under a research protocol. 

Three major design specifications guided the WebUI development as follows. 

1. Enable Multisite Collaborative Viewing and Control of the OCT Imaging Session

Multisite collaborative viewing allows other (remotely located) individuals to observe an OCT imaging session, provide assistance with controls if needed, and review the acquired image data with the device operator. To accomplish this, we chose to develop an interactive webpage that would serve as the display and user interface for the system and then use existing networking infrastructure (internet) for wireless, remote interactions with the OCT system computer. Because the WebUI is a webpage, it can be accessed on any device capable of rendering a webpage. This setup is analogous to other interactive client–server relationships on the internet such as accessing e-mail from any device over the internet. In our case, any mobile device can use the WebUI client and wirelessly interact with the OCT system computer server regardless of physical location. 

2. Develop the WebUI Using Open Source Software Tools and Standard Communication Protocols

Use of open source software tools and standard communication protocols make the WebUI a more replicable general solution accessible to others. The WebUI interface itself was created using HTML, JavaScript, and cascading style sheets (CSS), all common tools used to create any interactive page on the web today. Communication from the WebUI to the OCT system computer was accomplished through the WebSocket protocol, which is recognized in all modern web browsers. Communication over the internet used standard internet protocols and transmission control protocol/internet protocol, and both the WebUI viewing device and OCT system computer were connected by their wireless IEEE 802.11n radios to the Duke University Medical Center local area network. 

3. Minimize Any Latency from the WebUI to the OCT System

Because the WebUI is not hardwired to the OCT system computer and is physically separate from the OCT system computer, there is potential for system delay between the WebUI and OCT system that could interfere with live viewing and control. For instance, there is a live iris aiming camera that shows the device operator where the OCT probe is currently aiming. If the time to acquire this image on the OCT system and send it to the WebUI is prolonged, then the device operator would be viewing a delayed aiming image that could adversely affect image acquisition. Similarly, if the device operator pushed the button to acquire the OCT image on the WebUI, but the time to communicate this to the OCT system computer was prolonged, then this would also adversely affect image acquisition. To reduce system latency, we used image and event compression techniques. Image compression was used to efficiently transfer probe data by conversion of the live frame buffer on the OCT system computer to the JPEG image format. The live frame buffer received the incoming stream of processed intensity data from the iris camera and OCT computer. This intensity data was then compressed using JPEG to reduce the size of the camera images sent to the WebUI. Event compression was used to act only on the last action request generated by the user rather than acting continuously on every generated request. As an example, if the user wanted to access a specific OCT B-scan image within a volume on the WebUI, the user might move the slider quickly from frame 1 and end on frame 50. Instead of sending every image from 1 to 50 from the OCT system computer to the WebUI, the system intelligently sends only the frame on which the slider ended because this final position is the desired frame. Network speed and the time to regenerate an image are slower than the user's ability to manipulate the graphical user interface. Hence, the user may build up a queue of submitted input events from the WebUI to the OCT system computer. By discarding event requests that have already been superseded, we could generate images and responses to the user that are more responsive to user intent and decrease latency. 

WebUI Implementation on the Handheld OCT Probe System and Subject Imaging

As described earlier, WebUI implementation was done with a previously developed handheld OCT system.¹² The desktop OCT control and processing software was developed using C++ with graphics processing unit (GPU) acceleration capable of live rendering of OCT B-scans and volumes.¹⁵ The OCT computer was equipped with a 3.20-GHz central processing unit (Intel i7; Intel Corp., Hillsboro, OR), 64 GB RAM, and a GPU (nVidia Titan X; NVIDIA, Santa Clara, CA), and the OCT engine had an A line rate of 100 kHz. 

To this base OCT system, we added the WebSocket code to the OCT control and processing software that allowed communication to and from our WebUI. Key OCT displays and controls from the desktop computer user interface were ported to the WebUI webpage. The WebUI HTML, JavaScript, and CSS files were then hosted on a Duke University server. A basic consumer grade cellular smartphone (Microsoft Lumia 640; Microsoft, Redmond, WA) was attached to the handheld probe using adhesive mounts, and the WebUI was accessed from this cellular phone to aim and control the OCT system for patient imaging. To demonstrate simultaneous interactivity, a laptop at a separate physical location logged into the imaging session via the WebUI, and the laptop was used to monitor and trigger acquisitions occurring at the handheld OCT probe and its attached cellular phone. 

Under a protocol that adhered to the tenets of the Declaration of Helsinki and that was approved by the Duke University Medical Center Institutional Review Board and the U.S. Army Medical Research and Materiel Command Office of Research Protections, we imaged 20 normal subjects with the described system. Eighteen subjects were first imaged under controlled conditions in a laboratory setting at Duke University Biomedical Engineering, and then two subjects were imaged in the ED at the Duke University Medical Center to assess performance in our target setting. 

Characterizing Latency from the WebUI

To characterize latency, the OCT system described above was used to image a flashlight turning on and off. A 240-Hz camera (Apple iPhone 6 Plus capturing video at 240 frames per second; Apple, Inc., Cupertino, CA) was used to monitor the flashlight, desktop monitor, and cellular phone monitor. Video frame analysis was used to determine the time between the flashlight turning on (or off) and when that event appeared on the iris camera window on the desktop computer monitor (OCT system computer latency) and when the same event finally appeared on the iris camera window of the cellular phone screen (added WebUI latency). Ten trials were performed, and descriptive statistics were used to report the OCT system computer latency, the added WebUI latency, and the combined latency of the entire system. 

This latency experiment was conducted under three different methods of connecting the smartphone to the internet: (1) a local Duke University 802.11n wireless radio connection only; (2) a cellular network only (Verizon with five bars on signal strength indicator); and (3) a local 802.11n wireless radio connection but with an additional virtual private network (VPN) connection to the University of Michigan. This last connection (wireless with VPN through a remote site) was performed to test the latency associated with use over a geographic distance of over 500 miles. 

Results

Demonstration of the Use of Our WebUI Technology to Enable Simultaneous, Multilocation Viewing and Control of OCT Imaging Session

The video Supplementary Materials Video 1 demonstrates the WebUI being used to image one subject in the ED setting, and Figure 1 presents select screenshots from the video. Briefly, the cellular phone accessing the WebUI is mounted to the OCT handheld probe, and the device operator is actively using the cellular phone to wirelessly aim the probe on the subject's eye and acquire the image (Fig. 1A). Another video camera shows a second user at a separate location accessing the same OCT imaging session via a wireless laptop (top left inset of screenshots). This second user is then shown simultaneously viewing the same information as the device operator during acquisition and reviewing the acquired scan (Fig. 1B). Finally, the remote user triggers a switch on the probe from posterior to anterior segment mode and initiates an acquisition (Fig. 1C). 

Figure 1

 

Screenshots from video demonstrating WebUI control of OCT system and a simultaneous remote interaction. (A) Overview showing the OCT device operator imaging a subject's eye using a black OCT handheld probe with attached cellular phone running the WebUI. The bottom left inset is a clearer view of the cellular phone screen taken from a second video camera located over the device operator's left shoulder. Note that the WebUI on the cellular phone mirrors the OCT system's computer monitor (to the left of the operator's face in this screenshot). The top left inset (in gray scale) is a third video camera in the room, which will travel to the remote user over the course of this video. (B) By this point in the video, the device operator has triggered a retinal scan from the WebUI on the cellular phone. Simultaneously, the remote user using a laptop (top left inset) is viewing the same session and can review the scanned image on his laptop through the WebUI. The remote user can also control the OCT probe through the WebUI, such as switching anterior/posterior modes and triggering scans. (C) At this point in the video, the remote user has used the WebUI to switch the OCT probe to anterior segment mode and trigger an anterior segment acquisition that the on-site device operator can observe.

Figure 2 shows screenshots comparing the WebUI to the desktop OCT system computer control software. As seen in Figure 2, the WebUI is a selective port of the OCT system computer user interface. Key displays ported to the WebUI are the live iris camera view (lateral aiming; not shown in this screenshot but present in Fig. 1); the live B-scan view (axial aiming); and the updating summed voxel projection (OCT field of view). Key controls ported to the WebUI are a button to toggle between anterior and posterior modes, a large multistate button to trigger and stop the acquisition, and a slider to change the displayed B-scan within the OCT volume. Other useful controls such as image contrast and brightness and login fields and buttons are placed farther down the page. The live WebUI displays and controls can be seen used locally on a cellular phone at the probe by the device operator (Fig. 1) and on a laptop computer by a remote user (Fig. 1B,C). 

Figure 2

 

Comparison between displays and controls on the full OCT system desktop user interface (top) and the more compact WebUI as accessed on a cellular phone (bottom). Selected key displays such as B-scans and summed voxel projections are mirrored in both interfaces used (blue boxes). Important controls for acquisition, such as toggling between retina and anterior segment mode, starting acquisition, and sliding to choose B-scans, were ported from the larger, full set of controls on the main OCT system to the WebUI (orange boxes). Additional controls from the main OCT system, which are less frequently used, are located farther down the WebUI page (not shown here). The left side of the WebUI would show a live iris camera view during imaging; it is not seen here because these screenshots are of a WebUI session reviewing previously acquired data.

Measured System Latency

Supplementary Materials Video 2shows one cycle (flashlight on/off) of our latency experiment using only a local 802.11n wireless connection. Measured OCT system computer latency was 148 ± 20 ms (mean ± 1 SD), and added WebUI latency was 58 ± 69 ms. Combined, the overall end-to-end latency (camera to OCT system computer to computer monitor to smartphone) was 206 ± 64 ms (Fig. 3). 

Figure 3

 

System latency measurements using different connection types from video frame analysis of the OCT system and WebUI responding to the flashlight impulse. Each bar represents the mean of 10 measurements (Error lines: ±1 SD). Using a wireless-only connection, the mean baseline latency of the OCT system alone was 148 ± 20 ms, the WebUI added 58 ± 69 ms, and the combined end-to-end latency was 206 ± 64 ms. Using wireless and VPN to a remotely located server, the mean latency for the OCT system along was 165 ± 14 ms, the WebUI added 95 ± 34 ms, and the combined end-to-end latency was 260 ± 34 ms. Using a cellular connection, the mean latency for the OCT system alone was 145 ± 15 ms, the WebUI added 170 ± 117 ms, and the combined end-to-end latency was 314 ± 114 ms. Qualitatively, the device operator did not perceive any delay while aiming and acquiring images using the WebUI with wireless only.

Using the cellular network, the measured OCT system computer latency was 145 ± 15 ms, and added WebUI latency was 170 ± 117 ms. Combined, the overall end-to-end latency was 314 ± 114 ms. 

Using a local 802.11n wireless connection and then further connecting to a VPN server approximately 500 miles away from our physical location, the measured OCT system computer latency was 165 ± 14 ms, and added WebUI latency was 95 ± 34 ms. Combined, the overall end-to-end latency was 260 ± 34 ms. 

Representative Images Captured Using the WebUI

WebUI was used to capture posterior eye OCT images in 20 subjects. Figure 4 shows representative optic nerve head OCT images acquired under WebUI control. The OCT probe uses a compact, nonlinear microelectromechanical systems (MEMS) scanning mirror, which introduces distortions in the live image. However, these distortions can be removed in postprocessing¹² and the B-scan images further averaged to produce the images shown in Figure 4. 

Figure 4

 

Representative images of the optic nerve of two separate subjects captured using the handheld OCT probe with WebUI on a cellular phone in the ED showing a distinct elevation difference between the normal optic nerve (left column) and the pseudopapilledema (right column). The summed voxel projection represents the OCT analogue of the typical two-dimensional view afforded by direct ophthalmoscopy of fundus photography. The B-scan and volume renders show depth information that allows ready identification of a normal optic nerve compared to one with elevation. These images have been postprocessed to remove scan distortions and to average the B-scans to improve signal-to-noise ratio.

Discussion

This article describes the successful implementation of a novel WebUI for OCT, enabling multiple users to visualize and control the same OCT imaging session with readily available mobile devices such as cell phones, tablets, and laptops. As seen in Supplementary Video 1, the remote user was able to review the data acquired by the on-site device operator, monitor the session live, and remotely trigger a change in imaging mode. Modern smartphones are also able to transmit both data and voice. This feature would further enhance communication between the remote and local users, though we did not take advantage of the voice capabilities in our demonstration. 

The platform-neutral and flexible nature of the WebUI was made possible by purposely using established and available software languages and communications protocols such as HTML, JavaScript, and WebSocket. As long as the WebSocket code can be added into the OCT system's control software, the WebUI could be used to interact with any OCT system over the internet from any device capable of accessing a webpage. For example, though a noncommercial research OCT system was used in the current work, there are handheld OCT probes that are commercially available. The proprietary nature of commercial OCT control software does present a limitation that would need vendor collaboration. Once the WebSocket code is added, though, the WebUI could similarly be used with those systems on any handheld screen of the user's choice. The WebUI can also be readily customized for specific end users (e.g., more detailed interface for experienced users or a simplified interface for less experienced users). The only requisite skill needed to customize the WebUI is the ability to edit a webpage. 

Given the accessibility that the WebUI provides, a natural concern would be the security of the system. This work was primarily focused on the development necessary to implement the WebUI and its interactions with the OCT system. We did ensure that all our devices were connected through the institutional network, which was protected by an institutionally maintained firewall and required credentials to access the network. Additionally, the WebUI required manual input of the OCT system computer's intranet IP address to complete the connection. While the cellular phone was fully capable of connecting via cellular network to the OCT computer, we wanted to keep our connections behind the institutional firewall for this current work. Overall, this provided an initial degree of security, but larger scale, real-world implementation would require a dedicated security plan to safeguard data transfers and comply with pertinent regulatory policies. 

Limiting our wireless communication to the institutional network could also have affected our latency because the Duke University network uses IEEE 802.11n. Networks utilizing faster wireless protocols could potentially have improved our latency from the addition of the WebUI. Even as implemented on our network, the WebUI added a mean latency of only 58 ms. This meant that the overall end-to-end system latency was 206 ms (58 ms + mean inherent system latency of 148 ms), which is above the generally preferred value of 100 ms for human–computer interactions.¹⁶ However, this latency ultimately did not affect actual operator usage as operator perception is largely dependent on variability in latency. Other wireless conditions—for example, implementations on cellular networks or less robustly performing networks—also affect the remote performance of the WebUI. Using the system remotely over a geographic distance (experimentally tested by using VPN to a physically remote server) increased the WebUI latency time from 58 to 95 ms, and using a cellular network also increased the WebUI latency (from 58 to 170 ms). Shorter distances and additional refinement of the software and network conditions could also potentially improve latency. 

Overall, by enabling wireless viewing and control of an OCT imaging session by multiple users, the WebUI provides a promising platform to develop remote OCT applications. This is particularly important for eye care delivery in acute or general care settings, which may have limited access to specialty ophthalmic care. The WebUI has the potential to enable primary, nonophthalmologist providers to receive real-time feedback and input from remotely located specialists simultaneously during OCT imaging sessions (or asynchronously for stored OCT data) and ultimately improve eye care for those patients with ocular pathology who present to nonspecialty acute care settings. Though there have been prior works in store-and-forward teleophthalmology using OCT,^17–19 the presented work is, to the best of our knowledge, the first demonstration of live, remote interactivity and control of ocular imaging with an OCT system. 

Acknowledgments

The authors thank Ana Garcia-Turner (Duke Ophthalmology) for her assistance in coordinating subject imaging; Francesco LaRocca (Duke Biomedical Engineering) for his assistance in removing MEMS-related distortions from the raw acquisitions; Christian Viehland (Duke Biomedical Engineering) for the ray-casted 3D volume renders; and Ryan McNabb and Pam Bhullar (Duke Ophthalmology) for their assistance with videography. 

Supported by the U.S. Army Medical Research and Material Command (Ft. Detrick, MD, USA). 

Disclosure: R. Mehta, None; D. Nankivil, Johnson & Johnson Vision Care Inc. (E); D.J. Zielinski, None; G. Waterman, None; B. Keller, None; A.T. Limkakeng Jr, None; R. Kopper, None; J.A. Izatt, None; A.N. Kuo, None  

Supplementary Material Video 1 (01:39, 18.4 Mb). Screenshot of a video demonstration of the WebUI in use with a handheld OCT probe as well as simultaneous interaction by a remote user through the WebUI. Video 1 is available in the Supplementary Materials. 

Display Formula

Supplementary Material Video 2 (00:53, 6.23 Mb). Screenshot of a video of system latency measurements in response to a flashlight impulse (video only, no sound). Video 2 is available in the Supplementary Materials. 

Display Formula