Storage performance compared

All these tests were performed on an HP DL360G5 with local storage. The windows VMs tested were 32 bit Windows Server 2003 Standard with all updates installed. The goal was to get an overview of the storage virtualization overhead between different products and also to get a picture of the effect on having an array accelerator and also the difference between raid 1 and raid 1+0.  This test also shows the difference between an accelerated disk controller (cache+battery) and a plain one.

VMs were installed from scratch and vendor native drivers (VMware Tools, VS Tools, Virtual Machine Additions) was installed before running iometer.


HP tools and drivers were also installed on windows hosts (non HP native cciss disk drivers were used by the Debian install and Virtual Iron).


These results was first posted to the  Open inofficial storage performance thread @ VMTN

The results below revealed that there's quite a big impact on what product that is chosen and how it configured.

SERVER TYPE: Physical Windows 2003R2sp2 (not a virtual machine)
CPU TYPE / NUMBER: 8 cpu cores, 2 sockets
HOST TYPE: HP DL360G5, 4GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs block size (4096)

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

3.18

18530

579

RealLife-60%Rand-65%Read

78.6

739

5.7

Max Throughput-50%Read

3.74

15579

486

Random-8k-70%Read.

72.7

787

6.1

SERVER TYPE: Virtual Windows 2003R2sp2 on VMware Server 1.0.4 on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

0.5

10900

340

RealLife-60%Rand-65%Read

156

368

2.8

Max Throughput-50%Read

1.22

7472

233

Random-8k-70%Read.

88.1

630

4.9

EXCEPTIONS: CPU Util. 99% 17% 98% 22%

SERVER TYPE: Virtual Windows 2003R2sp2 on VMware Server on Debian Linux 4.0 2.6.18 x64
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default jfs (4096)

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

0.5

8550

267

RealLife-60%Rand-65%Read

79

747

5.8

Max Throughput-50%Read

0.63

3804

237

Random-8k-70%Read.

97

609

4.7

EXCEPTIONS: CPU Util. 100% 17% 98% 16%

SERVER TYPE: Virtual Windows 2003R2sp2 on VMware Player 2.0.1 on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

0.5

9920

310

RealLife-60%Rand-65%Read

139

411

3.2

Max Throughput-50%Read

3.1

2656

83

Random-8k-70%Read.

93.3

632

4.9

EXCEPTIONS: CPU Util. 99% 17.5% 98% 23%

SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual Iron 3.7
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

16.2

3732

116

RealLife-60%Rand-65%Read

169

353

2.75

Max Throughput-50%Read

15.2

3940

123

Random-8k-70%Read.

177

337

2.6

EXCEPTIONS: CPU Util. 39% 17% xx% 17%

SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual Server 2005r2sp1 (1.1.603.0 EE R2 SP1) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

15.5

3860

120

RealLife-60%Rand-65%Read

159

374

2.9

Max Throughput-50%Read

17.3

3444

107

Random-8k-70%Read.

198

300

2.3

EXCEPTIONS: CPU Util. 58% 17% 57% 16%

SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual Server 2005r2sp1 (1.1.603.0 EE R2 SP1) (VT enabled) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

15.9

3773

117

RealLife-60%Rand-65%Read

159

375

2.9

Max Throughput-50%Read

17.5

3420

106

Random-8k-70%Read.

199

299

2.3

EXCEPTIONS: CPU Util. 58% 17% 55% 16%

SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual PC 2007 (6.0.156.0) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

16.7

3571

111

RealLife-60%Rand-65%Read

161

371

2.9

Max Throughput-50%Read

18.6

3219

100

Random-8k-70%Read.

200.5

298

2.3

EXCEPTIONS: CPU Util. 53% 16% 54% 15%

SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual PC 2007 (6.0.156.0) (VT enabled) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

15.2.

3948

123

RealLife-60%Rand-65%Read

148.

403

3.2

Max Throughput-50%Read

16.8.

3561

111

Random-8k-70%Read.

184.

324

2.5

EXCEPTIONS: CPU Util. 56% 16% 51% 15%

SERVER TYPE: Virtual Windows 2003R2sp2 on XenServer release 4.0.1-4249p (xenenterprise)
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

5

10445

326

RealLife-60%Rand-65%Read

44

810

6.3

Max Throughput-50%Read

6.46

8896

278

Random-8k-70%Read.

55.9

811

6.3

EXCEPTIONS: CPU Util. 92% 52% 83% 37%

SERVER TYPE: Virtual Debian Linux  4.0, kernel 2.6.18.xs4.0.1.900.5799 on XenServer release 4.0.1-4249p (xenenterprise)
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

0.36

2773

86.6

RealLife-60%Rand-65%Read

3.04

328

2.6

Max Throughput-50%Read

1.38

724

22.6

Random-8k-70%Read.

3.3

302

2.36

EXCEPTIONS: CPU Util. 0% 0% 0% 0%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service disabled
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

5.3

9711

303

RealLife-60%Rand-65%Read

43

786

6.1

Max Throughput-50%Read

6.4

8796

274

Random-8k-70%Read.

55

778

6

EXCEPTIONS: CPU Util. 73% 55% 56% 41%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

5.4

9887

308

RealLife-60%Rand-65%Read

42

777

6.0

Max Throughput-50%Read

6

8987

280

Random-8k-70%Read.

53

539

6

EXCEPTIONS: CPU Util. 67% 56% 67% 43%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled, arrayaccelerator=disable
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB arrayaccelerator=disable / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

24

2384

74.5

RealLife-60%Rand-65%Read

96

607

4.7

Max Throughput-50%Read

76

758

23.7

Random-8k-70%Read.

87

671

5.2

EXCEPTIONS: CPU Util. 27% 17% 20% 17%

SERVER TYPE: Physical Windows 2003R2sp2 (not a virtual machine), raid 1+0
CPU TYPE / NUMBER: 2x quad core
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

2.95

19932

622

RealLife-60%Rand-65%Read

46

1209

9.4

Max Throughput-50%Read

5

11272

352

Random-8k-70%Read.

39

1391

10.8

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

4.3

9976

311

RealLife-60%Rand-65%Read

30

1439

11

Max Throughput-50%Read

5.5

8779

274

Random-8k-70%Read.

30

1431

11

EXCEPTIONS: CPU Util. 92% 46% 89% 45%

SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled. Cache on individual disks enabled.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB

TEST NAME

Av. Resp. Time ms

Av. IOs/sec

Av. MB/sec

Max Throughput-100%Read.

5.4

9681

302

RealLife-60%Rand-65%Read

34

1353

10.5

Max Throughput-50%Read

6.1

8763

273

Random-8k-70%Read.

35

1412

11

EXCEPTIONS: CPU Util. 71% 40% 72% 33%

Breaking down the numbers
The most important values from these tests are the mixed workload tests (RealLife & Random). These are the workloads that multiple VMs on the same storage will generate. Also, only during large file operations (such as backup) we will see workloads that will be similar to the max  throughput tests.

Compared to a physical install, the best virtual max throughput tests were ~50% of the physical one, while the mixed workload tests were fairly equal in both these environments.

From the above tests we have seen that the virtualization product choise will indeed affect the storage performance. The best performer here was Citrix'es XenServer, but VMware's ESX Server was also a close follower (XenServer: 10445 810 8896 811 vs ESX:9887 777 8987 539)

For hosted VMware products on Windows, the RealLife performance is about half the performance of VMware Server running on Debian Linux.  Compared to ESX, VMware Server on Linux were generally a bit slower, but it's also not that bad (ESX:9887 777 8987 539 vs Server:8550 747 3804 609)

The big disappointments were however Virtual Iron and the Microsoft products. They showed fairly lousy performance and overall performance of less than half the performance of the above mentioned products in all the tests. Enabling VT didn't have much notable effect in the MS products either.


The different disk setup tests were all done on ESX server and showed the importance of using a good disk controller in an optimal RAID setup.


Disabling the array accelerator had the biggest impact on the max throughput tests (9887vs2384 and 8987vs758) while the mixed workload tests, differed only marginally. This will probably affect backup time and other large file operations quite extensively.

Using RAID 1+0 across 4 disks instead of RAID 1 across 2 disks also showed that more disks is having a very positive impact in such an environment. The biggest difference here was in the mixed workload tests where we got RealLife 1439iops (vs 777) and Random 1431iops (vs 539). The max throughput tests were not largely affected.

Disk cache on individual disks is disabled by default (atleast in HP servers), and if you enable it you get a warning that you shouldn't do this unless your server is powered by an UPS. Enabling this cache didn't give us any better (or worse) performance, but instead lowered the cpu usage consumed by this virtual machine. This might have an effect on the consolidation ratio since many VMs are sharing the same cpu resources.