Computer I/O and Storage Systems

Computer I/O and Storage Systems are fundamental components of any computing architecture, dictating how a computer interacts with the outside world and manages its data. Understanding these systems is crucial for comprehending overall system performance, reliability, and design. This article will delve into the various aspects of I/O and storage, from performance measurement to advanced storage solutions like RAID, and common pitfalls.

What are Computer I/O and Storage Systems?

I/O (Input/Output) devices facilitate communication between a computer and the user or other machines, while storage systems provide nonvolatile data persistence. I/O devices are characterized by their behavior (input, output, storage), partner (human or machine), and data rate (bytes/sec, transfers/sec). Both I/O and storage systems rely on bus connections for data transfer.

Key Characteristics of I/O Systems

Dependability is paramount, especially for storage. Performance is measured by:

• Latency (Response Time): The time it takes for an operation to complete.
• Throughput (Bandwidth): The rate at which data can be processed or transferred.

Desktops and embedded systems often prioritize response time and device diversity, while servers emphasize throughput and expandability.

Understanding System Dependability

Dependability focuses on maintaining service accomplishment without interruption. Key metrics include:

• Fault: A component failure that may or may not lead to system failure.
• Service Accomplishment: Service delivered as specified.
• Service Interruption: Deviation from specified service, leading to a failure.

Measures of Dependability:

• Reliability: Measured by Mean Time To Failure (MTTF).
• Service Interruption: Measured by Mean Time To Repair (MTTR).
• Mean Time Between Failures (MTBF): Calculated as MTTF + MTTR.
• Availability: The fraction of time a system is operational, calculated as MTTF / (MTTF + MTTR).

Availability can be improved by increasing MTTF (through fault avoidance, tolerance, or forecasting) and reducing MTTR (through better diagnosis and repair tools).

Deep Dive into Storage Devices

Storage is where your data lives, persistently. We'll explore two primary types: disk and flash.

Disk Storage: The Rotating Magnetic Medium

Disk storage refers to nonvolatile, rotating magnetic storage devices, commonly known as Hard Disk Drives (HDDs). Each disk sector records a Sector ID, Data (e.g., 512 or 4096 bytes), Error Correcting Code (ECC), and synchronization fields/gaps.

Accessing a Sector Involves:

• Queuing delay: If other accesses are pending.
• Seek time: Moving the read/write heads to the correct track.
• Rotational latency: Waiting for the desired sector to rotate under the head.
• Data transfer time: Reading or writing the data.
• Controller overhead: Time spent by the disk controller.

Example Disk Access: For a 512B sector, 15,000rpm, 4ms average seek, 100MB/s transfer, and 0.2ms controller overhead on an idle disk, the average read time is approximately 6.2ms (4ms seek + 2ms rotational latency + 0.005ms transfer + 0.2ms controller delay).

Disk Performance Issues: Manufacturers often quote average seek time based on all possible seeks, but actual average seek times are usually smaller due to locality and OS scheduling. Modern disk controllers use logical block addresses and include caches to prefetch sectors, reducing seek and rotational delays.

Flash Storage: Nonvolatile Semiconductor Power

Flash storage is a nonvolatile semiconductor storage technology that is 100x – 1000x faster than traditional disks. It offers advantages in smaller size, lower power consumption, and increased robustness, though at a higher cost per gigabyte compared to HDDs.

Types of Flash:

• NOR Flash: Bit cells similar to a NOR gate, offering random read/write access. Commonly used for instruction memory in embedded systems.
• NAND Flash: Bit cells similar to a NAND gate, denser (more bits per area), but accesses data block-at-a-time. Cheaper per GB, found in USB keys and media storage.

Flash memory cells wear out after thousands of accesses. To mitigate this, wear leveling techniques remap data to less-used blocks, extending the life of the drive.

Interconnecting Components: The Role of Buses

To connect the CPU, memory, and I/O controllers, systems use buses, which are shared communication channels. Historically, these were parallel sets of wires, but increasingly, high-speed serial connections with switches are used, similar to networks.

Types of Buses

• Processor-Memory Buses: Short, high-speed, and designed to match the memory organization.
• I/O Buses: Longer, allowing multiple connections, specified by standards for interoperability. They connect to the processor-memory bus through a bridge.

Bus Signals and Synchronization

Buses use various signals for operation:

• Data Lines: Carry address and data, often multiplexed.
• Control Lines: Indicate data type and synchronize transactions.

Synchronization Methods:

• Synchronous: Uses a bus clock for timing.
• Asynchronous: Uses request/acknowledge control lines for handshaking.

I/O Bus Examples

Modern systems utilize a variety of I/O buses, each with specific applications and characteristics:

• Firewire (IEEE 1394): External, 63 devices per channel, 50-100MB/s, hot-pluggable.
• USB 2.0: External, 127 devices per channel, up to 60MB/s, hot-pluggable.
• PCI Express (PCI-E): Internal, 250MB/s/lane (up to 32x lanes), hot-pluggable depending on implementation.
• Serial ATA (SATA): Internal, 1 device per channel, 300MB/s, some hot-pluggable.
• Serial Attached SCSI (SAS): External, 4 devices per channel, 300MB/s, hot-pluggable.

Managing I/O Operations

The operating system (OS) plays a critical role in mediating I/O, ensuring multiple programs can share resources and protecting system integrity. I/O operations often trigger asynchronous interrupts, similar to exceptions, but not tied to instruction execution.

I/O Controller Hardware and Commands

I/O devices are managed by I/O controller hardware, which transfers data and synchronizes operations with software. Controllers use specific registers:

• Command Registers: Cause the device to perform an action.
• Status Registers: Indicate the device's current state and report errors.
• Data Registers: Used to transfer data to or from the device.

I/O Register Mapping

There are two main ways to access I/O registers:

• Memory-Mapped I/O: Registers are addressed within the same space as memory. The OS uses address translation to make them kernel-only accessible.
• I/O Instructions: Separate instructions specifically designed for I/O register access, executable only in kernel mode (e.g., on x86 architectures).

Polling vs. Interrupts

Polling involves the CPU periodically checking an I/O status register. If the device is ready or an error occurs, the CPU takes action. This is common in small, low-performance embedded systems due to predictable timing and low hardware cost, but wastes CPU time in other systems.

Interrupts occur when a device is ready or an error happens, causing the controller to signal the CPU. The CPU then invokes an interrupt handler. Priority interrupts allow more urgent devices to interrupt lower-priority handlers.

I/O Data Transfer: CPU vs. DMA

In polling and interrupt-driven I/O, the CPU often transfers data between memory and I/O data registers, which can be time-consuming for high-speed devices. Direct Memory Access (DMA) offers a solution:

• The OS provides the starting memory address.
• The I/O controller autonomously transfers data to/from memory.
• The controller interrupts the CPU only upon completion or error.

DMA/Cache Interaction: DMA operations can cause cache coherence issues if DMA writes to cached memory (stale cached copy) or if a write-back cache has dirty blocks that DMA reads (reads stale data). Solutions include flushing blocks from the cache or using non-cacheable memory locations for I/O.

DMA/VM Interaction: The OS uses virtual addresses for memory. If DMA uses physical addresses, transfers may need to be broken into page-sized chunks, chained, or require contiguous physical page allocation. Controllers that handle virtual addresses would need translation capabilities.

Measuring I/O Performance

I/O performance is influenced by hardware (CPU, memory, controllers, buses), software (OS, DBMS, application), and workload (request rates and patterns). I/O system design often involves a trade-off between response time and throughput, with throughput measurements frequently constrained by response time limits.

I/O Performance Measures

Performance benchmarks are crucial for evaluating I/O systems:

• Transaction Processing Benchmarks: Focus on small data accesses to a Database Management System (DBMS). Key metrics include I/O rate, throughput (subject to response time limits), and handling of ACID properties (Atomicity, Consistency, Isolation, Durability). The Transaction Processing Council (TPC) provides benchmarks like TPC-APP, TPC-C, TPC-E, and TPC-H.
• File System & Web Benchmarks:
• SPEC System File System (SFS): A synthetic workload for NFS servers, measuring throughput (operations/sec) and response time (average ms/operation).
• SPEC Web Server Benchmark: Measures simultaneous user sessions, subject to required throughput per session, with workloads like Banking, Ecommerce, and Support.

I/O vs. CPU Performance: Amdahl's Law in Action

Amdahl's Law reminds us not to neglect I/O performance, especially as compute performance increases with parallelism. For example, if a benchmark takes 90s CPU time and 10s I/O time, doubling CPUs every two years will progressively increase the percentage of time spent on I/O, making I/O the dominant factor over time.

Enhancing I/O with RAID

RAID (Redundant Array of Inexpensive/Independent Disks) uses multiple smaller disks instead of one large disk. This approach improves performance through parallelism and provides fault-tolerant storage via extra disks for redundant data, especially with hot-swappable drives.

RAID Levels Explained

• RAID 0 (Striping): Data is striped across multiple disks without redundancy, purely for performance improvement. No fault tolerance.
• RAID 1 (Mirroring): N + N disks; data is replicated (mirrored) onto a separate set of disks. On disk failure, data is read from the mirror.
• RAID 2 (Error Correcting Code - ECC): N + E disks (e.g., 10+4) where data is split at a bit level, and ECC bits are generated. Too complex for practical use.
• RAID 3 (Bit-Interleaved Parity): N + 1 disks; data is striped across N disks at a byte level, with one dedicated disk storing parity. Not widely used.
• RAID 4 (Block-Interleaved Parity): N + 1 disks; data is striped across N disks at a block level, with one dedicated disk storing parity for a group of blocks. Suffers from parity disk being a bottleneck for writes. Not widely used.
• RAID 5 (Distributed Parity): N + 1 disks; similar to RAID 4 but parity blocks are distributed across all disks, avoiding the parity disk bottleneck. Widely used.
• RAID 6 (P + Q Redundancy): N + 2 disks; similar to RAID 5 but with two independent parity schemes, offering greater fault tolerance against multiple disk failures. Often implemented as nested RAID levels (e.g., RAID 1+0, RAID 0+1).

RAID significantly improves both performance and availability. High availability often requires hot swapping capabilities and assumes independent disk failures.

I/O System Design and Examples

Designing an I/O system involves satisfying latency requirements for time-critical operations and maximizing throughput by identifying and addressing the