Uptime . . . All the Time

University of Virginia Creates an Enterprise PACS that Just Won’t Quit

By Sean J. Moynihan, Director of Information Systems for University of Virginia Health Science System Department of Radiology, and
Matthew J. Bassignani, M.D, Medical Director for Radiology Information Systems, University of Virginia Health Sciences Center

Any image, anywhere, any time. Imagine a PACS that never crashes.

Simply stated but difficult to achieve, 100 percent uptime is an ambitious goal for a digital image system—especially one that manages over 300,000 cases and 10 terabytes (TB) of data each year. However, this was a key objective behind the development of University of Virginia Health System’s (UVaHS) enterprise-wide PACS, which brought together the expertise of a broad range of Health System information technology experts, department administrators, physicians and marketing-leading vendors. The system has been operating successfully since 2001 without any major failure. If you’re considering implementing a PACS that works as long and hard as your radiology department, then you’ll want to learn about UVaHS’s advanced disaster recovery (DR) plan and experience.

Recognized as one of the nation’s top 100 hospitals by Solucient for the past four years, UVaHS maintains an unwavering commitment to clinical and academic excellence and state-of-the-art patient care. Given this, we wanted an enterprise-wide PACS that would support our cutting-edge clinical expertise with equally cutting- edge digital image management technology.

A tertiary referral center that draws patients from throughout the region and a level one trauma center, UVaHS provides vital medical care, including organ transplants and open heart surgery, in an energetic academic environment for a significant number of critically ill patients. Our clinicians felt that even an hour or two of system downtime delaying care would have a significant negative impact. An extremely fault-tolerant PACS, we believed, would support the UVaHS mission and personal conviction of our dedicated physicians to provide the highest quality of care 24/7.

Additionally, a PACS engineered for uptime makes good business sense. A system failure would leave our staff of 60 highly skilled radiologists pacing the hallways waiting and would mean additional economic loss due to longer patient hospital stays, increased malpractice risks, scheduling changes and cross-departmental inefficiencies.

By contrast, a reliable system would pay dividends in goodwill from surgeons, oncologists and other referring physicians as well as enhance the international reputation of our medical center.

System selection and specification was a group effort. Contributing significantly to its success were UVaHS experts Barbara Baldwin, Chief Information Officer, Cindy Perry, Associate Chief Information Officer, and Mark Monroe, Networking Manager. When it comes to PACS, radiology should not go it alone.

Our goals were demanding, from superb image quality and robust user-friendly features to ultra-fast communication of vast amounts of data throughout the enterprise. But system reliability—now more specifically defined as the user’s ability to continue working without interruption under the most adverse circumstances—was certainly one of our major challenges.

UVaHS adopted a multi-pronged approach to meet this goal. Our DR plan had to cover all types of system failures—network, software and hardware as well as any possible resulting data corruption. Also important was protecting UVaHS from the effects of environmental disasters, such as fire and flood. Naturally, shaping the entire project was our conviction that DR should extend well beyond mere data recovery to ensure complete business continuity—keeping the PACS and our doctors working 24/7, no matter what.

Clearly, this was a tall order. It called for duplication not only of images but also of applications, databases, user preferences and more—and it meant doing so in multiple locations. It also demanded planning for a wide range of disaster scenarios with a multitude of safety nets and fail-over options covering a broad range of situations. And it called for automatic activation of recovery plans as potential problems developed.

Our goals also created demanding vendor requirements. We needed a PACS supplier offering advanced technology, scaleability and open architecture that at the same time provided the flexibility and commitment to leverage its technology to address our unique and uncompromising demands. Additionally, the PACS had to function flawlessly throughout the enterprise over the hospital network, while accommodating the large volume of data generated in radiology. Since this was an ambitious and potentially costly project, we also needed a vendor that could deliver all this within our budget. To that end, we wanted to combine sophisticated software with affordable standard hardware and network equipment.

UVaHS considered RFP responses from all major vendors but found none could reference an installation already in place meeting our stringent requirements. Algotec came the closest, and our experience with the Algotec ICU/ER PACS we had implemented several years previously, convinced us that its underlying architecture would support our significant demands. What’s more, we knew they offered the flexibility to tailor a system to meet our needs.

For data storage and archiving, Algotec offered a two-tiered approach with all data stored online on magnetic disks and backed up to more affordable tape. Each tier had built-in redundancy. Together with the PACS vendor, we selected EMC for online storage because its Symmetrix networked storage systems offered benefits including high availability and flexibility. Market leader StorageTek was selected for tape. Close cooperation among all vendors was vital to the project’s success.

Early on, we decided to implement complete system redundancy through a “hot” backup site, with full mirroring of all data on an ongoing basis to ensure system continuity even with complete loss of our main data center due to an environmental disaster.

The system specifications were comprehensive, calling for two online storage systems, four servers controlled by automated clustering software and tape backup with an integrated cloning mechanism. This hardware was divided between two locations about 3/4 of a mile a part to create two interchangeable data centers.

To speed data access during possible failure, we implemented an innovative configuration of two fully redundant EMC RAID storage systems both configured to function as active devices, instead of as more typical primary and secondary archives. In fact, the PACS can run seamlessly accessing either storage device. The RAID storage system not currently in use runs in a frozen state, ready to spring into action immediately if necessary. Fail-over is so fast it is barely perceptible.

Complete, timely synchronization of the two Symmetrix storage systems, with a total capacity of 15 TB of online storage, is enabled by EMC’s innovative SRDF remote replication software through a fiber channel with bi-directional communication, allowing data to flow immediately from the active archive to the second storage device and vice versa.

As part of the DR plan, Algotec’s PACS application also resides on the RAID storage systems, taking advantage of its built-in redundancy and not on the server as is usually the case. Duplication of PACS on both storage systems provides an additional level of backup to ensure the application is always available.

When operating, the Algotec PACS is accessed through the RAID storage system by two of the four SunFire 4800 servers from Sun Microsystem servers. Working behind the scenes, Veritas’ clustering software tracks server activities. In the case of failure, it initiates the PACS applications on the two alternate devices transparently to both the imaging modalities and PACS users.

Backed up with the same level of redundancy as the image files, the PACS database provides access to appropriate images, DICOM headers and such essential information as patient demographics and exam dates. Immediate access to this information is essential to keep information flowing. Therefore, Algotec also designed the system so that the database is stored on the RAID storage system as part of the PACS application with all the same backup levels.

Likewise, all our PACS customized settings, such as user profiles and protocols, also reside in the database, stored in multiple locations. Immediate access to these means that users continue to work in the same familiar environment, whatever copy of the application is running.

Deep archiving and an extra level of system security is provided by the tape backup at the main location. It duplicates the entire content of the RAID storage systems, including applications and system files. A cloned tape is stored at a secure off-site location.

A word to the wise—to ensure success whatever your backup plan, test it, and test it again. Despite elaborate and costly planning, this is too often forgotten. UVaHS executes its recovery scenarios on a monthly basis to ensure flawless fail-over. As part of testing, we continue to run the system off the alternate RAID system for an extended period of time.

Also remember, if your PACS is running on a hospital-wide network, your DR is only as good as its network host. Hospital networks need DR too--so plan accordingly.

What does the future hold for UVaHS? As our current online storage system reaches capacity, we may introduce an intermediate level of online storage on new lower priced spinning media, such as EMC’s Centera content addressed storage system, rather than archiving to tape only. We are keeping an eye on our options. Whatever course we pursue, UVaHS is confident our Algotec PACS and its technology infrastructure will deliver the flexibility to handle it.