Why an all-flash storage system isn’t what your business needs
I’ve been involved in IT for a lot longer than I like to admit - over 30 years! In that time, I’ve seen storage media move from cassette tapes (familiar to anyone who owned a ZX Spectrum or Commodore 64), floppy disks, laserdiscs and WORM storage to high-performance devices such as SCSI, IDE, SATA and SAS drives which get bigger and faster with each generation.
The biggest change over the past five years has been the move away from traditional spinning disks to solid-state drives (SSD) and flash drives. This is digital storage with no moving parts, essentially, it’s high-speed memory.
The first ‘supercomputer’ with a hard disk drive (HDD) was launched by IBM in 1956. The 305 RAMAC came with a drive, that alone weighed over a ton and stored only 5MB of data.
SSD and flash technology has been around longer than most people think, originally developed in the 1950s by IBM in the form of card capacity read-only store and magnetic core memory. Yet it wasn’t until the late 1970s when supercomputers such as the Cray were developed that the technology was refined into being highly reliable and practical.
1 The central mechanism of the IBM RAMAC’s storage unit, the world’s first hard disk. Photo by vnunet.com, CC BY-SA 2.5, https://commons.wikimedia.org/w/index.php?curid=4085202
However, it was eye-wateringly expensive with StorageTek’s SSD effectively costing $8,800 per MB upon its release in 1978. That works out to be around $9 million for 1GB!
Isn’t SSD the same thing as flash storage?
Many people assume flash and SSD are synonymous names for a certain type of storage technology. It’s an understandable expectation, and in a few ways, it is true.
Of course, when it comes to IT terms, there is some nit-picking to be done. I’ll explain the differences briefly.
SSD technology was originally based on volatile RAM (i.e. the same memory as in your PC or laptop) and needed constant power in order to store data. Losing power on this meant you lost your data, so it was a bit of a challenge for long term storage.
Flash, on the other hand, was designed to permanently store data even when no power was present. It used much faster memory than SSDs but had a downside of only allowing the memory to be written to a limited number of times before that part of the memory became unusable. This is still the case today, but it’s less of an issue as the technology improves and capacities increase.
Today, flash and SSDs are essentially the same things, but the drives used in enterprise-class storage systems are of much higher quality and performance compared to an SSD made to sit inside a laptop. Naturally, they cost a lot more too.
Price is also a factor when comparing SSD/flash storage to hard drives. The technology is still significantly more expensive than using traditional spinning disks. When IBM originally introduced enterprise-class flash drives on their IBM Power systems around seven years ago, a 300GB drive had a list price of around £13,000!
Thankfully, prices have reduced, and capacities and performance have increased exponentially, proving Moore’s Law isn’t quite dead yet.
What are NVMe drives?
The latest flash storage technology is NVMe (which stands for Non-Volatile Memory Express). This was developed back in 2011 by a consortium of Intel, Samsung, SanDisk, Dell, and Seagate. It’s a storage protocol created to accelerate the transfer of data between the host system and SSD/flash drives.
NVMe provides the benefit of significantly reduced latency, higher input/output operations per second (IOPS), and potentially lower power consumption. It’s designed to replace SAS/SATA interfaces.
The main attraction of NVMe drives is that they are insanely fast. To give you an idea, a typical hard disk transfers data at around 200Mbps, a SATA SSD at 550Mbps, and an NVMe SSD transfers data at over 3Gbps, so over six times faster than typical desktop/laptop SSDs.
In another echo of Moore’s Law, costs for NVMe have significantly reduced, capacities increased, and performance is constantly improving. We’ve personally seen many customers revitalise the performance of ageing IBM Power Systems by simply replacing the old storage with flash drives.
IBM NVMe FlashCore Modules (FCM)
IBM FlashCore Modules (FCMs) are NVMe flash drives that are unique to IBM storage offerings and are a game-changer when it comes to extreme performance, very high-density storage solutions.
All storage vendors have solutions that support real-time storage compression and encryption but depending on the solution, switching this on can have a detrimental effect on the performance of the storage as the compression/encryption is typically performed by the storage controller itself.
When IBM released the initial generation of Storwize V7000 SAN controllers, they had a ‘first to market’ claim on real-time compression with zero performance impact, but there were some caveats around this. For example, some other controller features could not be used at the same time.
This is typically the case with all storage vendors where the controller is doing the heavy lifting, so the more powerful the SAN controller is (processor, memory, cache, etc) the less performance impact there is when compression is enabled.
Why IBM FlashCore Modules are hugely different to standard NVMe drives
Each FlashCore drive incorporates its own hardware compression and encryption, so if you have a SAN with 300 FlashCore drives then you have 300 individual compression/encryption engines taking the load off the SAN controller.
FlashCore Modules are available as 4.8TB, 9.6TB, and 19.2TB devices in 2.5” form factor, allowing you to cram a jaw-dropping amount of storage into a very small space.
As an example, 24 x 19.2TB FCMs will give a raw capacity of 460.8TB in just 2U of rack-space, giving you around 383TB of usable capacity with DRAID 6. However, using the in-built hardware compression and a guaranteed minimum compression ratio of 2:1, that increases to almost 800TB usable effective capacity in 2U. Turn on de-duplication, and 1PB of usable storage in 2U is easily achievable!
Additionally, FlashCore Modules are extremely fast. For the techies among you, a single 2U storage solution using FCMs will achieve 2.5 million IOPs, scaling to 10 million IOPs in 8U guise, with latencies as low as 100us. That’s fast!
All-flash storage and the 80/20 rule
Storage vendors such as Dell, HP, Pure, Nimble, NetApp and IBM are currently pushing all-flash storage solutions that are designed to only work with SSD/flash devices and won’t support traditional spinning disks. Many see this as the inevitable march of new technology and while all-flash storage is the future, is it the right thing to do today when the costs are still relatively high?
The 80/20 rule, also known as the Pareto principle, states that 80% of results in a system come from 20% of the causes. As with many things in life, the 80/20 rule also applies to storage and research shows that only around 20% of what you’re storing is used on a day to day basis and the remaining 80% tends to be static data that is rarely if ever, accessed.
If you invest in an all-flash storage system, you will potentially be storing static data on very expensive high-performance flash drives. Some will argue that the solution to this is to archive data to a legacy, cloud or object storage solution, but are you certain about the part of your data that will remain static? Manually archiving this adds complexity and administration overhead when your time is probably better spent focusing on ways to make your business more productive.
Spend 20% and save 80%!
As highlighted, flash storage is still significantly more expensive per byte than traditional spinning drives, so it doesn’t really make financial sense to use it for storing a mass of static data.
A more effective solution would be to deploy a hybrid flash storage solution such as IBM’s Storwize V5000 or the V7000 SANs with NVMe, allowing you to mix both flash and spinning disks with data automatically moved the most appropriate storage. The static data go to low-cost spinning drives and heavily accessed data for business-critical applications goes to your high-performance flash storage.
This approach will provide your business with the same real-world performance as an all-flash system but will save you around 80% of the costs of adding future storage, while still allowing you to move to an all-flash solution as the costs come down.
If you’re still unsure on about the best way to refresh or expand your existing storage, then come and talk to us as we’d be delighted to help.