RESEARCH PAPER: Datacenter Memory and Storage: Past, Present, Future

The server industry is in the middle of a once-every-20-years shift. What has happened in the past is a good indicator of what is about to happen. Given radical past transitions, it is very likely that many of today’s key technologies probably will fade out of the market by the year 2020.

This paper focuses on server primary system memory and secondary storage technologies spanning the past, present, and future. We begin with an overview of historical server architectures going all the way back to the 1960s. Then we outline the current state of server hardware, software, and how they are used today. We conclude with a description of potentially important technologies upcoming in the next decade.

Summary
History of Server Architecture
Hardware Technologies
Integration
Memory vs. Storage
Registers
Primary Memory
Secondary Storage
Tertiary Storage
Software Technologies
Physical Memory Model
Virtual Memory Model
Non-Uniform Memory Access (NUMA) Memory Model
RDMA (Remote Direct Memory Access) Memory Model
Virtual Machines (VM) Model
Usage Models
Current State of Server Architecture
Hardware Technologies
Application Specific Acceleration and Shared Primary System Memory
Memory and Storage Access Over Networks
Non-Volatile Chip Technologies Move onto Primary Memory Bus
Software Technologies
Solid State Drives (SSD) Will Replace Hard Disk Drives (HDD)
Impact of NVDIMM on the Operating System
Impact of NVDIMM on the BIOS
Usage Models
Future Datacenter Architecture Directions
Hardware Technologies
Integration Continues: Packaging Multiple Chips
Rack Scale Architecture
Non-Volatile Memories
Software Technologies
Usage Models
Conclusion
Glossary
Figure 1: Major Parts of a Server
Figure 2: Late 1980s Separate Chips
Figure 3: 1990s x86 Integration
Figure 4: Current Server SoC and Package Integration
Figure 5: Possible Future of Server SoC Integration
Figure 6: Mainframe Data Pyramid, Circa 1965
Figure 7: Updated Data Pyramid
Figure 8: DIMM (Dual In-Line Memory Module)
Figure 9: A Server Motherboard with DIMMs and Other Components
Figure 10: A Storage Server Motherboard with Many DAS Devices
Figure 11: Physical Memory Model
Figure 12: Virtual Memory Model
Figure 13: NUMA Memory Model
Figure 14: RDMA Memory Model
Figure 15: A Physical Server, Not Virtualized
Figure 16: A Virtualized Server
Figure 17: A. PCI-Express Processor Card, B. PCI-Express Storage Card, & C. PCI-Express Network Card
Figure 18: Server Processor Shipments Over Time
Figure 19: Micron Automata Processor Attaches to DIMM Connector
Figure 20: A. Common "Flash Drive", B. Flash Drive with Exposed Flash Storage Chip, & C. Flash Storage Chip
Figure 21: Battery Backed Up DRAM General Architecture
Figure 22: NVDIMM-N General Architecture
Figure 23: NVDIMM-F General Architecture
Figure 24: NVDIMM-unnamed General Architecture
Figure 25: Overview of BIOS Boot Procedure
Figure 26: Competing Memory Technologies: Relative Performance
Figure 27: Competing Storage Technologies: Relative Performance
Figure 28: Potential Future Data Pyramid

You can download the paper here.

Companies Cited

Amazon
AMD
Dell
Diablo Technologies
Facebook
Google
HP
IBM
Intel
Lenovo
Micron
Microsoft
SanDisk
Storage Networking Industry Association (SNIA)
VMware

Patrick Moorhead

+ posts

Patrick founded the firm based on his real-world world technology experiences with the understanding of what he wasn’t getting from analysts and consultants. Ten years later, Patrick is ranked #1 among technology industry analysts in terms of “power” (ARInsights) in “press citations” (Apollo Research). Moorhead is a contributor at Forbes and frequently appears on CNBC. He is a broad-based analyst covering a wide variety of topics including the cloud, enterprise SaaS, collaboration, client computing, and semiconductors. He has 30 years of experience including 15 years of executive experience at high tech companies (NCR, AT&T, Compaq, now HP, and AMD) leading strategy, product management, product marketing, and corporate marketing, including three industry board appointments.