Recertified LTO Ultrium Media Quality - Things you should be aware of!

Introduction

Even before the pandemic, IT and Procurement organizations were being challenged to squeeze every last drop of value from their operational budgets. Under these pressures, it makes sense to look at a routine task, like replacing old data cartridges, to see if there are opportunities for cost savings. After all, with growth rates in excess of 40-50% per year, the amount of tape media required for backup and archiving processes will also increase rapidly. Therefore, the savings on a large quantity of data cartridges could be substantial for savvy buyers who shop around.

One of the advantages of the LTO open standard is that it supports a healthy and competitive marketplace, with numerous vendors and suppliers offering LTO products and solutions. But even though LTO technology offers outstanding value, it is also a rigorously controlled, standards-based format, with minimum requirements for performance and interchangability that all licensees must meet in order to sell their solutions.

Outside of this authorized market, however, the same standards and requirements do not apply and this is particularly true of the recertified, second-hand, tape media market.

The study reveals that customers who purchase (or trade in) recertified media could be vulnerable to unauthorized data sharing, encounter quality and reliability issues, put their brand reputation or their customers at risk and ultimately, may not actually save any money compared to buying original, authorized media in the first place!

The Report

The LTO Consortium (HPE, IBM and Quantum) asked OvationData to follow-up on previous studies and source quantities of LTO data cartridges in the open market to specifically validate the claims of the “recertified” market.

OvationData provide a range of IT services and solutions, from scalable data management products to data storage, networking, security services and cloud-based solutions.

Within the research, three key areas of focus were identified to scrutinize the claims being made by recertified media suppliers.

• Security - would primarily affect the original owner of the media who sold the tapes back into the recertified marketplace. Does data remain on the tape when it is sold on the secondary market?
• Environment - would primarily affect the secondary purchaser of these recertified cartridges. How reliably will the media perform given that it has been used before?
• Economics - would affect the entire supply chain for recertified media. Economic considerations – e.g. the time taken to perform an activity - might entice the recertified media vendor to cut corners in the erasure process, which in turn affects the security of both the original owner and the secondary purchaser.

Methodology

The original objective was to source approximately 100 LTO data cartridges from multiple sources but – perhaps owing to the constraints of the pandemic – it was only possible to obtain 86 cartridges during the timeframe for the study in mid 2020.

As far as possible, vendors were not made aware that they were supplying a research company instead of a conventional end user customer, although this could have been established from the ‘Ship To’ and other contact information.

A limited number of cartridges were obtained randomly from the open market (specifically Ebay), but the majority were acquired by searching the web and purchasing from vendors who specifically advertised their products as being “recertified”. A total of 15 purchases were attempted but in the end, OvationData only received product from 11 suppliers. Of these 11 organizations, 5 suppliers were general eBay resellers who made no claims of selling recertified tape.

Note: When ordering from suppliers of recertified media, it was implicit that the researchers were specifically asking for recertified tape media. Also, where possible, random media manufacturers were chosen within a specific order and sometimes within the same generation.

Delivering Against Expectations

One interesting aspect of the purchasing process was the number of resellers who were selling “recertified tape” that had never actually been used. Of the samples purchased by OvationData, 30% of the tapes received had never been mounted in a tape drive and, therefore, these tapes were not included in any of the metrics that follow.

Example:

One recertified media reseller asked at the time of purchase, if it was “Okay to only do a ‘Quick Erase’ on the tapes they would be sending. It is assumed that this would have saved the reseller considerable time, expense and wear-and-tear on their equipment. Instead of having to spend a minimum of twenty six hours for a Full Erase procedure, a Quick Erase on the ten tapes ordered could be completed in just a couple of minutes. The problem with this method, however, is that it does not completely erase the data and still leaves open the possibility that information could be recovered by the new owner or other third parties.

Within the recertified reseller community, there does not seem to be any consistent process for recertifying tape media for resale. OvationData were unable to find any publication (e.g. on a website) describing the process used to prepare the tapes for resale.

Some of the paperwork supplied with the order (for example, goods invoice) did state “Certified Error Free”. But a fundmental question for buyers of recertified media should be to ask: what does ‘recertfied’ or ‘certified error free’ actually mean; to whose specification has this been carried out; and is the process consistent when measured against new media, media from other used tape suppliers or even previous orders from the same, recertified vendor?

A common feature, although not entirely consistent across all media samples, was the stamping of a white “L” on the bottom of the cartridge case. OvationData speculate that most of the “recertified” media is actually being done by only a small number of companies (possible two or three) who then they resell to other suppliers who trade in recertified media. This “L” marking was only observed on media that appeared have undergone a Full Erase process.

Focus Area Findings

Security:

The reality of buying data cartridges from a recertified media supplier is that you could be buying anything. Unfortunately, this is the inevitable conclusion of this investigation.

For example, the website of one of the vendors claimed that the media they were supplying was “Complete Chain-of- Custody, 100% Data Erasure. 100% Secure, 100% Compliant”. But one of the tapes received from this seller still contained readable data – e.g. no special tools were required to access the information!

This exposes an obvious data security vulnerability. By examining the ‘mount’ data in the LTO Cartridge Memory of each tape, it was obvious that the original owner of the media did not properly erase their information before releasing the tapes to the recertified reseller.

On the evidence of these findings, if you intend to resell old or unwanted tapes into the secondary market place, you cannot assume that the recertified reseller will properly erase the information for you before selling the goods on (can we do a quick erase etc). Depending on the nature of the contents, there is a huge risk of exposing confidential information relating to your business, intellectual property or the personal details of your customers, which may in turn be within the scope of privacy legislation like GDPR, CCPA or LGPD.

Accidental disclosure of sensitive information could carry significant financial penalties, to say nothing of the damage to your competitive position and brand reputation.

The presence of data on the reused tape can be discerned quite easily from the LTO Cartridge Memory. Although this is a relatively advanced technique, it is perfectly feasible for anyone with the right equipment and motivation to scan through reused media in the hope of finding information that could be put to unauthorized use. When one considers $136,000, speculatively purchasing a few recycled tapes in the hope of finding some confidential data, is arguably a relatively low risk and easier method of extortion! The lower wrap pass value of 1366 compared to the previous wrap pass values of between 2139 and 2144, strongly suggests an incomplete erasure process – e.g. that the last write process covered a smaller area of the tape, thereby leaving data in the sections between 1366 and 2144 that could potentially be read and recovered.

It was observed within the LTO Cartridge Memory of all the recertified LTO-6 and LTO-7 tapes sampled that they did not appear to have been over-written to the very end, usually short of the last 1-3 wrap passes of every tape. It is hard to be certain as to why this is. It could be a calculation error regarding how much data the device performing the over-writing was set to. But the clear implication of this phenomenon is that data could still exist on 100% of LTO-6 and LTO-7 tape samples that were sold as “fully erased”.

Equally concerning, OvationData also found that many recertified media resellers do not erase any previous written user information that can be found in the Cartridge Memory, thus leaving potential critical information exposed. This user information includes VolSer and Media Asset Management (MAM) information.

The MAM page information is a freeform area where software applications can write various information, such as the name of the application that wrote the tape, software versions, date recorded, owner’s name, system name, directory paths and backup set name. These details could be valuable for cyber criminals as they would offer valuable insight into the organization’s network environment, as well as point to the likely return on investment for spending more time trying to recover data from an insecure tape – e.g. if the previous owner was a sophisticated business or Enterprise more likely to have valuable data assets.

Environmental

While it is possible that recertified media will work adequately, it’s more likely that users of these tapes will experience higher error rates, lower reliability, and in the worst case scenario, complete product failure. These assumptions appear to be borne out by the OvationData investigation.

Renovation or Ruin?

Ordinarily, there is no way to tell the difference in error rate between new and reused data cartridges because such distinctions can only be observed when the drive is mounted and the media performance is compared. But even to the naked eye, there were some obvious signs of reuse and renovation in the cartridges within the sample that could cause a number of concerns for the reliability and safety of backup and archive data.

For example, many of the recertified tapes had visual signs of the outside of the cartridges being cleaned. This begs the question: what kind of chemical solvents were used? Could they have leaked into the tape area and possibly degraded the tape media itself? Worse, might these foreign agents have created debris or left residue which might cross-contaminate the read/write head of the tape drive, or even more drastically, the entire library if other tapes came into contact with these cleaning agents via media rotation inside the library drives? The cost of this kind of damage could be very significant.

It was also noted that several cartridges had the manufacturer’s imprinted batch code (a form of serialisation found on the bottom of each cartridge) completely expunged (e.g. no longer visible) which would make it harder to identify when and where the tape was produced.

What Came Before?

The absence of standards means that unfortunately, purchasers of recertified media can have no guarantee of quality and reliability when they procure secondhand tape. This is because they cannot be certain of the conditions under which the tapes were previously used and/or re-engineered to be sold again as recertified media. Users are literally flying blind.

Each tape has a specified life that relates to in-drive performance and is often measured by the number of head passes, load/unload cycles, and the number of backups it has experienced. All of these everyday activities will, to some degree, shorten the life of the product. No amount of cosmetic surgery will ever make it as good as new.

And OvationData found that the devices (tape drives) that are commonly being used to recertify media appear to be located in very different environments to a controlled and highly regulated data center.

This means that buyers of recertified media should give serious consideration as to what kind of environmental conditions the recertified media is being processed. Also, how much expertize do the operators really have to carry out this activity? For example, in some cases, the environmental values returned by examination of the LTO Cartridge Memory data would suggest the activity occurred in non-airconditioned site like an open warehouse.

More than 40% of the recertified tapes were exposed to a temperature of over 90° F and 18% were exposed to a temperature of over 100° F – the maximum value was 117°!

Temperature, humidity, and air quality can all affect media quality. Some chemical processes can cause pigment binders to degrade and reduce the durability of tape. In addition, this degradation can increase the amount of debris piling up to clog the read/write head later on. This can lead to data loss with further use but only after the cartridge has been reconditioned, recertified and sold ‘as good as new’.

Other physical factors of concern that any purchaser or pre-owned tape media should be aware of relate to the number of physical tape mounts. OvationData found that 30% of the sample media had more than 50 mounts; several had over 100 mounts and the maximum was 947 mounts. Disturbingly, 4% of the sample displayed a history of severe problems like Fatal Writes or Servo Errors.

These findings are relevant because they implicate aspects of media performance. Magnetic media recording is a mechanical process that involves friction between the tape head and the surface of the media when the tape is mounted and in use. The tape itself passes through reels and rollers within the tape path as it is pulled through the drive, and over time its physical characteristics will be affected. This is perfectly normal. The problem with a recertified tape, however, is that it’s difficult for buyers to know if previous use has exceeded the recommended lifecycle for that particular cartridge.

For example, if the tape edge or servoband is damaged, it cannot be repaired through re-conditioning. A worn tape edge will contribute to error rates and degraded performance long before the actual media surface itself wears out.

And what about transportation and handling? New, original data cartridges are cushioned by several protective layers to prevent damage from the shock, vibration and environmental extremes that the package may encounter during shipping. Product packaging is routinely tested by manufacturers to ensure factory-validated media performance is not upset by the shipping process. Recertified media vendors may – or may not – do any of these things.

In the case of the sample scrutinised by OvationData, other unknown factors, come into play, such as the age of the media. Of the media sampled, the oldest was almost 12 ½ years old. The average age of the LTO-4 and LTO-5 media were almost identical (8.5~8.6 y.o.). But the youngest LTO-5 media was older than the youngest LTO- 4 by over a year. Ordinarily, for tapes bought from an authorized source, this would not necessarily be a concern because LTO media is warranted for 30 years. The supply chain for recertified media is far more opaque which makes it almost impossible to know how older media has been handled and stored during its working life, and whether it may have passed through several hands before reaching its current owner.

All of which must raise powerful questions about recertified media integrity when it comes to the vital purpose of backing up or archiving valuable business data.

Economic

It was noted that almost 40% of the recertified tapes had been processed using a “Quick Erase” method, which takes less than twenty seconds to complete because it only overwrites a small amount of index information at the beginning of the tape. This means the bulk of data on the media is potentially retrievable for an unauthorized person with sufficient motivation and resources to extract it.

The safest but most time-consuming process is a full erase or overwrite: but this could take anywhere from two to six hours per tape. Clearly, the full erase is more time and resource intensive, which is why so many recertified suppliers prefer to cut corners by using the quick erase method. But this clearly exposes customers to all kinds of legal and security vulnerabilities. It is also worth saying that this kind of short cut is not something that is being highlighted when the customer purchases the media – “Chain-of-Custody, 100% Data Erasure” etc.

The other strange aspect of the OvationData study is that although many customers seem drawn to recertified media for economic reasons, in a number of instances, the actual purchase price of the secondhand cartridges was more expensive than brand new media purchased from a verifiable source. Even where recertified media was cheaper, the saving (in comparison to the risk of loss of data, the risk of security or privacy breaches, and the potential damage to brand reputation) seemed hardly sufficent to justify some of these risks.

In 50% of the recertified media purchased by OvationData, there was no more than a 23% saving over the cost of like brand-new media. And as mentioned, 16% of the recertified media samples were actually more expensive than brand new media (genuine, unused media costs were based on OvationData’s cost to acquire these products).

Conclusion

In the current economic climate, exacerbated by the Covid-19 pandemic, everyone understands the desire to reduce costs and find greater efficiencies in IT spending. But on the evidence of this study, purchasing recertified media is very unlikely to help businesses achieve their procurement goals.

Instead, selling used and unwanted media risks putting your business critical information into the hands of unauthorized third parties. And if you purchase these cartridges, there seems to be a very high chance that you are buying product that has surpassed its recommended working life; been processed in excessively harsh conditions outside of recommended environmental limits and which may expose your business to punitive costs and sanctions that far outweigh any possible media savings.

Worst of all, you may do so unwittingly having been reassured by marketing claims that cannot be substaniated and, in the worst case, bear no resemblance to what has actually happened to the cartridges that you plan to install in your data center.

In summary, selling recertified media is not an illegal trade but customers who purchase this product should be very careful indeed that they understand what they are letting themselves in for. The phrase caveat emptor or buyer beware has rarely seemed so appropriate.

 

Link: Original White paper PDF Click here