The need for clustered storage in large-scale video surveillance deployments
By Lee Caswell As organizations’ video surveillance requirements become more complex, storage requirements increase. The digital video surveillance market has some unique requirements, and a new form of clustered storage offers benefits that are especially helpful when upgrading from small-scale recording environments to large-scale surveillance systems that support thousands of cameras and petabytes of storage.
Data Set Size: how much raw data needs to be stored? A large database today is roughly 100 gigabytes and only very large companies create one terabyte (1,000 gigabytes) of general business data. By contrast, a video surveillance system with just a few hundred cameras can easily generate more than one terabyte of data in a single day. Read vs. write activity: Normal business data, whether it is payroll database entries, legal file updates, or email messages, generally follows the 80/20 rule, where 80 percent of the time data is being read and only 20 percent of the time is dedicated to write activity. Video data has nearly the exact opposite characteristics, with data being written 100 percent of the time and read activity happening only rarely when incidents occur and the relevant footage needs to be retrieved and reviewed. Random vs. sequential: A storage administrator would look at how data is sent to the storage system and how this handles that traffic. In the normal business case, data is written randomly in short bursts in what is termed ‘small block transfers’. Video data, once again, is radically different as it arrives in a sequential and constant stream of dense information. As camera resolutions increase, this density grows geometrically as more pixels are used to convey each image. Planned downtime: business data generally follows a cycle with a natural rhythm of downtime when maintenance, upgrades, and changes can be made to the system without affecting business activity. Video data has no such downtime cycle since the incoming stream of video is unrelenting and any changes or maintenance must be made dynamically without affecting the incoming stream. A traditional large-scale storage system is designed to accommodate business data. The architecture of these systems is well-established and is centred around the concept of a master RAID controller with a fixed amount of network bandwidth that controls a fixed maximum number of drives. Traditionally, this master RAID controller has acted as a funnel for incoming video data and limited the maximum capacity of the system. More advanced technology has introduced a secondary controller for redundancy, but the maximum bandwidth and capacity remain fixed. Video data presents quite a challenge to this traditional storage set up. The funnel design, which is rarely overwhelmed for business data, is poorly suited for the 100 percent write and sequential nature of video data and introduces data loss risks. Traditional designs tend to need either next-generation networks such as 10 Gigabit Ethernet, or proprietary, expensive storage protocols such as Fibre Channel, in order to meet the bandwidth requirements of large-scale video installations. Similarly, the master controller architecture is problematic for field maintenance since capacity, bandwidth, and performance changes are difficult, if not impossible, to dynamically upgrade in the field. Newer clustered storage architectures aggregate a set of distributed RAID controllers to provide large-capacity storage. While providing the same data protection as traditional designs, the clustered systems introduce a number of innovations that are particularly useful for video. For example, data is spread automatically across the cluster, performance is allocated dynamically without user intervention, and the cluster manages failure conditions for drives, network connections, and complete controllers. In storage terms, these features are termed dynamic provisioning, automatic load balancing, and multi-pathing support. In the best cluster implementations, the bandwidth of all the clustered RAID controllers is aggregated together and available to all the incoming video data, thereby removing the traditional funnel limitation using the cost-effective Gigabit Ethernet technology available today. For video, this is the optimal architecture for providing the high-bandwidth that is essential for the write-mostly, sequential nature of this type of data traffic. Clustering provides high bandwidth by aggregating several network links, and the architecture can deploy off-the-shelf, Gigabit Ethernet switches and cables for low installation and support costs. Dynamic field upgrades are easily maintained by adding further clustered components. A side benefit of the architecture is that automated system rebuilds as in the case of a hard disk replacement, are accelerated over traditional designs because all system components contribute to a massively parallel recovery. Total Cost of Ownership considerations for large-scale video deployments For someone who is not a storage expert, there are management and support differences between traditional and clustered storage systems that should also be thought through before selecting a video storage system. First and foremost, traditional large-scale storage systems were designed for specially-trained, dedicated storage administrators in Fortune 100 companies. Configuring these systems requires extensive upfront planning and training, often at the user’s expense. Because of the fixed nature of the underlying RAID controller design, users generally purchase the maximum configuration at the outset since early configuration decisions cannot be easily changed in the field. Like the most sophisticated scientific calculators, these traditional systems have a lot of specialised commands and features to learn. In most cases, field support is best left to the manufacturer and annual maintenance fees can easily be astronomical. In contrast, a storage cluster manages sophisticated tasks without user intervention and requires only a high-level understanding of capacity requirements. The expectation is that existing staff with basic PC server skills will be able to manage the system on a daily basis alongside any required field upgrades or replacements. The same customer or integrator that handles gigabytes of data today can effectively manage hundreds of terabytes with these new systems. What to look for when evaluating clustered storage for video surveillance Once you have made the decision to look at clustered storage systems, it is useful to understand differences in the available choices. Here is a quick guide on key features to look for when evaluating clustered storage systems for large-scale video surveillance installations: * Complete node protection for high availability – A clustered storage system should eliminate all single points of failure including the failure of an entire distributed controller node; Summary Not all large-scale storage solutions are necessarily well-suited to store and retrieve video data. Newer clustered storage systems provide enhancements over traditional storage designs that are especially helpful given the characteristics of video data itself, and also meet the total cost of ownership requirements in the exploding video market. Overlooking these considerations can result in a serious mismatch of storage performance, cost, and long-term manageability. Author: Lee Caswell, Co-Founder and Chief Marketing Officer, Pivot3 www.pivot3.com
•Date:24th June 2009• Region:World •Type: Article •Topic: IT continuity |
