Big Data Storage
Thursday, October 27 2005 @ 01:54 PM PDT
Back when I began, most available micro-computers had only floppy disks for storage. Hard drives were available but they were thousands of dollars for a few megabytes (e.g. $5,000 for 8 Megs for a Tandy Model II hard disk drive in 1980.) But the programs of the day stored mostly compressed numeric data that took up only 2-4 bytes for numbers of any reasonable size, and minimal text data for comments and such. The programs worked hard at getting the maximum out of the little storage available.
Today, we deal with images, faxes, music and video on our systems. The amount of storage we use is incredible compared to that of 25 years ago - yet the cost is so minimal that many people think little of adding another 100+ Gigabytes of storage or creating a CD or DVD of their myriads of information.
This article will cover some of the options available for adding such vast amounts of storage to a home or business computer system. Other articles will go into more depth on some of the options presented here, especially for storing and cataloguing images.
Lots of people today are accumulating pictures, video and music on their systems and are running out of space. They copy some to their CD or DVD burner to free up hard disk space but then they lose the immediacy of having all of it online. They also run the risk of losing their data, even though it may be on a CD/DVD, since these media may not last as long as you might think, and in any case are subject to "bit rot" which is an interesting problem more to do with the fast pace of technology than with actual decomposition of the medium.
Bit Rot
-
This describes the loss of data due to any one of a number of phenomena but is typified by the inability of today's generation of computer systems to read the product of yesterday's and the extension of this to anything digital, including recordings of audio and video.
- incompatible media types - try finding something that will read a DEC-Tape today (or a 3200BPI 1/2" tape recorded in EBCDC or a BETA video tape)
- incompatible data formats
- impermanence of "permanent" media (if it isn't carved into rock or bronze, it won't last centuries) There are other reasons too, but these are the main ones.
There are all sorts of reasons why today's systems might not be able to read yesterday's data:
The problem today is that just adding more or bigger hard disks to your computer is not really the best idea. While they are really quite reliable in general, hard drives have been known to fail and when they do they take your valuable data with them. If you don't have the data backed up onto some other device or medium, you lose much or all of it forever (and that medium still needs to be readable in the future or you again have bit-rot)
There are a number of things that can be done, but my favourite today is to add more disk storage to a system using RAID (Redundant Array of Inexpensive Disks) technology.
The two main ways of adding storage to your computer system:
For another thing, I firmly believe in the saying "If it ain't broke, don't fix it," as there is always the possibility that something will go wrong with the addition and you'll lose data and time.
An added computer should use one of the RAID storage types that guard against data loss rather than just making bigger amounts of space available. The two most typical are:
There are other RAID levels which guard against multi-drive failures, but for most people these two, and in particular the RAID-5 array is more than adequate if a spare drive is included.
Build vs. Buy
Build
-
Today, 10+ years after the advent of the Internet and 20+ years after the large-scale availability of local area networking technologies for the microcomputer the fact is that if you are viewing this you likely have the beginnings of a LAN at least, so adding more storage in a separate computer is really quite straight forward. Additionally, there is a good chance you have an older computer around that is either not being used much or is in fact just sitting in a corner. If you don't, you can probably purchase one for a few dollars at a yard sale or a shop that deals in used computers. These older computers typically had more room in their cases and in fact are easier to work with than many of the newer ones - and for many people their older, slower CPUs are more than adequate to act as the core to a storage system.
Systems that have CPUs such as the Intel PIII and speeds between 400MHz and 1GHz are fine for this job. Such systems usually are the newer ATX style boards and have RAM from 64Megs to 512Megs (the more the better). Many even have an Ethernet port already on them or included as an add-in card. Most cases also have places where extra fans can be added to aid in cooling additional drives, and have power supplies that are more than adequate to drive today's smaller form-factor and less power hungry hard disks.
With 3-6 hard drives, 2-3 drive interface cards, a couple of fans and some free software these old computers can be turned into a storage platform that will add hundreds of Gigabytes of storage for home or office in a manner that can contain your precious images and other data for many, many years in an ongoing, expandable, extendable way.
I've had just such a system sitting downstairs in my home for a number of years now. It has over 600 Gigabytes of RAID-5 storage with a spare drive "just in case" and has images and data from the time of my first microcomputer over 20 years ago.
-
Of course if you don't want to put something together yourself, you can now purchase pre-created RAID-arrays that can be simply plugged into the network, turned on and used. These "data appliances" come in flavours from home-network to high-availablity business systems with storage from a few hundred Gigs to thousands of Gigs (Terabytes) and even thousands of Terabytes (Petabytes) Search your favourite search engine for the words "Network Attached Storage" or "NAS Storage"
The next article will deal with how to organize all this space, not a trivial task - but one that there are more and more solutions to.
Tag: richard pitt business digital storage media disks drives dollars programs cd dvd backup computer raid
What's Related