Coping with the Video Data Bulge - Tape Technology to the Rescue

An explosion in the amount of data being collected for critical applications like video surveillance is driving organizations to consider new cost-effective and reliable ways to store and efficiently access vital information. Increasingly, organizations are considering updated tape technology to provide the right storage solution.

With the increasing use of video surveillance cameras, Information Technology departments are being overwhelmed with recorded video data, which needs to be stored securely and managed for longer periods of time than ever before. It’s the lethal three-way combo of more cameras, higher resolution images that demand more computer storage space, and the need to keep data almost forever.

Is your IT department prepared to meet this oncoming data bulge? There are a number of options available, from storing the data on a hard disk to archiving files through the cloud or a tape backup system. One solution that is getting more attention lately is a familiar one: magnetic tape data storage in the form of Linear Tape Open (LTO) and Linear Tape File System (LTFS) which offer an option that is a fraction of the cost of disk storage, easily accessible and has the ability to retain images for a long time. Here is a look at the video surveillance storage challenges, potential solutions, and how LTO Technology together with LTFS provides fast access to high volume of archived data while making discovery faster, easier, less time consuming and less costly.

The Video Data Bulge

The amount of all data needed to be captured and stored is growing exponentially. There are currently 4.4 zettabytes of data being created in the world; this is estimated to grow 10 times by 2020, according to the latest IDC’s Digital Universe report.  However the anticipated growth in data is not matched by storage capacity. In 2013, available storage capacity could hold just 33% of the digital universe.

The data explosion for video applications like surveillance is even more acute. Originally created in the 1940s for security in the banking industry, video surveillance has evolved quickly to become the most accepted form of security. Such cameras are ubiquitous.

This ability to install inexpensive camera systems and do higher-quality video surveillance on every city street corner, or in every high-value location across an enterprise, has created a “data bulge”: mountains of video data that needs to be cost-effectively stored and efficiently accessed. The average amount of digital data generated per day by a single surveillance camera is 40 gigabytes at basic compression rates. A site with 100 cameras, like a small jail or casino, would be creating four terabytes of data per day. When you multiply this by the average number of days data can be required to be saved, 30 to 90 days, companies are looking at a total storage capacity to be installed and managed of 120 to 360 terabytes, just from a single camera system.

Faced with this deluge of data, storage management departments are often in permanent reactive and firefighting mode due to storage issues. Change comes slowly because risk- averse storage administrators and conservative storage working practices are often a barrier to the adoption of the latest technologies. That’s understandable because data cannot be replaced once it is lost or destroyed. Although programs can be rerun and data can be re-entered, lost surveillance video data cannot be reincarnated. Plus the new video formats, like HD and 4K, require significantly more space that the old standard definition video.

Video surveillance Storage Solutions

There is not a single solution to store data throughout all the different stages of its lifecycle, and while keeping all data online on high-speed disks might be a good solution for access purposes, in practice it can become very expensive and difficult to manage. Video Surveillance Systems have traditionally been solely based on spinning disks, high-speed disk has been used for data ingest and the long term retention being written to slow spinning disk or to optical.

Now that modern Digital Video Surveillance systems are producing high-resolution video that generates terabytes of data per week, the need for high performance, cost-effective and reliable long-term storage is more critical than ever.

It is important to be able to identify data that is unlikely to be used again in the near future but that needs to be kept for compliance or for future repurposes. This data can then be moved to a more cost-effective storage medium, such as tape, and be kept off-line, freeing up expensive disk storage space and reducing overall storage costs.

Video solutions such as Milestone Systems and Genetec can interface with the tape as a file system. Data that is placed into the archive category in the applications can now see the LTO tape as a standard directory structure. LTO tape technology with LTFS means that the need for tape specific software and user interfaces are no longer needed, data on tape can be kept near-line or even, in some cases, online as LTFS allows applications to access data on tape as they would if the data resided on disk. As archiving medium LTO technology has long been the choice, now with LTFS it presents all the advantages of tape while also minimizing dependencies, maximizing recoverability and making the use of LTO tape for long-term data retention easier than ever before.

The ability to access data directly from any tape in the directory structure means that forensic investigations do not have to impact the Digital Surveillance ingest system. The tapes can not only be viewed off-line, but also copied and sent for further investigation.

As storage systems grow, LTFS solutions are easily upgraded; data does not have to be migrated from tape to tape as regularly as it is in disk systems. LTFS tapes can also be integrated into higher-level systems that expand the performance and usability of the LTO technology with LTFS tapes. The advanced LTFS solutions allow the Digital Video Surveillance applications to expand to support enterprise level storage with unlimited retention at less than two cents per gigabyte acquisition cost and fractions of a penny per gigabyte per month for the unlimited life data.

The Cloud Alternative for Video Surveillance Storage

One trendy option to store data is cloud storage. While cloud storage sounds like something from the night’s television weather forecast, it actually refers to saving data to an off-site storage system maintained by a third party. Cloud storage is a subcategory of cloud computing. Cloud computing offer users access to not only storage, but also processing power and computer applications. Instead of storing information within an on-site computer’s hard drive or other local storage device, cloud storage users save it to a remote database.

At first blush, cloud storage appears appealing. Cloud data is available from any location that has Internet access, there is no need for multiple physical storage devices, and users access and save information to the cloud from the same workplace computer.

However, upon further review, cloud storage raises a number of prickly issues. One has to do with the concept of data ownership and who really controls the data stored in a cloud system. Experts disagree over whether it actually belongs to the client who originally saved the data to the hardware or the company that owns the physical equipment storing the data.

Public Safety Video Storage Challenges

Whether it’s tracking down criminals in an urban center or watching over an officer as he makes a traffic stop, video cameras are an integral part of today’s law enforcement and security efforts. Typically, departments upload all the video feeds, from stationary pole cameras, body cams and dashboard video recorders, to redundant, encrypted, on-site and off- site data centers via a high-speed Internet connection through the cruiser’s Mobile Data Terminal. There, the data is indexed and stored. Some larger departments, like the Los Angeles Police Department, connect all those feeds into a central command center, to process the data in real-time. The LAPD is at the forefront of a new anticrime technique called video harvesting, which allows control centers to grab video from remote sites over the Internet. This allows the police department to triage incoming calls. It’s also a jump toward predictive policing, which uses video images and data to calculate where crimes will occur. The downside of all that captured video data is that it must be stored somewhere.

John Powell, a retired Los Angeles County Sheriff’s Department deputy, said the department was overwhelmed by the video data bulge when it installed surveillance cameras in its jails and detention areas several years ago.

The sheriff’s department patrols the unincorporated parts of Los Angeles County and also runs the urban center’s huge jails. Cameras were installed after complaints surfaced about abuse in the jail system. The camera plan was not well-thought-out, falling short on key “back end” components such as storing, filing and retrieving video evidence. Soon the LASD was overwhelmed with all the jail video recordings.

“We installed hundreds of cameras and really the amount of storage we had wasn’t sufficient,” Powell said. “It became a huge problem because they didn’t think it all the way through.” Instead of buying more storage space, the sheriff ordered the camera frame rate to be reduced to a point where some of the recordings became unusable.

The Ultrium LTO Solution

The sheriff’s department may have been better served had it explored the LTO option. Magnetic tape data storage can be used as near-line storage as well as off-line storage and it’s been around since the advent of computers. Linear Tape-Open (LTO) marked a turning point for tape storage. It was originally developed in the late 1990s as an open standards alternative to the proprietary magnetic tape formats. HP, IBM and Quantum, through “The LTO Program,” spurred development in the magnetic tape upgrade.

LTO generations 5 and 6 work with the Linear Tape File System, which allows the storage to be indexed. LTFS, released in 2010, divides the tapes into two segments called partitions. One partition holds directory structures and pointers that let the tape drive quickly seek specific data and the data itself is stored in the other partition. By applying a file system to a tape, users can organize and search its contents as they would on a hard disk. Users can also copy the media by using a simple copy command without having to wait for the entire tape cartridge to be scanned. Video files like those taken from inside the county jails are good candidates for this solution because they don’t require immediate access and can be kept for longer periods of time at a much lower cost.

One of the problems the sheriff’s department faced was the massive cost of storing the influx of data on county hard drives. The lower expense of LTO certainly makes it attractive compared to other media. The Clipper Group Calculator report “Revisiting the Search for Long-Term Storage – A TCO Analysis of Tape and Disk” (May 13, 2013) examined total costs of ownership for archiving data over nine years in a SATA disk system compared to an LTO tape library system. The study included all hardware acquisition costs, maintenance costs, floor space costs and energy costs. They found that the disk system was about 26 times more costly than the LTO tape system and the energy alone to power and cool the disk system was more than the total cost of the tape system. 

The Los Angeles Sheriff’s Department is not alone. Every law enforcement agency is struggling to keep up with the technology explosion.

“The scope and nature of law enforcement is changing,” said Mike Fergus, program manager, Technology Center, International Association of Chiefs of Police. “It’s becoming more of an information business. For years, the chiefs have been talking about ways to handle the IT needs. It’s a tough sell to add five IT guys, if that’s what they really want, as opposed to putting more cops on the streets. It’s a very tough sell to the city council.”

Law enforcement and private security are constantly searching for ways to keep video tapes even longer, especially if the video is evidence of a crime. Deletion of data is generally not a widely accepted solution. And any new storage solution must be easy to use.

“Anything that can simplify the process and make it more user-friendly to the law enforcement community would be very welcome,” Fergus said. But there are other factors. The magic of LTO Technology combined with LTFS is that it helps data access; the LTO file system structure allows applications to directly write and read to tape without knowledge that the storage device is tape.

Summary

Tape will not replace disk for instant access to data. But the LTO solution is more cost-effective for long-term storage and more reliable when the data “must” be preserved beyond the near horizon. It is a good fit for file system access of data that is infrequently retrieved or modified. Because storage is a horizontal technology, businesses should be able to gain benefits across the board by better storage management. It means businesses can manage data smarter while also meeting their budget demands.

"When buying a system, agencies or departments must look at the total cost of ownership,” said a forensic video analyst who uses LTO technology to archive his lab’s work. He teaches at the FBI National academy in Quantico. “Our policy is to hold data for 10 years,” he said. “In reality we keep it forever. The cloud is not designed for archiving. When you are talking about archiving, LTO should always be part of the discussion.”