All these tests were performed on an HP DL360G5 with local storage. The windows VMs tested were 32 bit Windows Server 2003 Standard with all updates installed. The goal was to get an overview of the storage virtualization overhead between different products and also to get a picture of the effect on having an array accelerator and also the difference between raid 1 and raid 1+0. This test also shows the difference between an accelerated disk controller (cache+battery) and a plain one.
VMs were installed from scratch and vendor native drivers (VMware Tools, VS Tools, Virtual Machine Additions) was installed before running
iometer.
HP tools and drivers were also installed on windows hosts (non HP native cciss disk drivers were used by the Debian install and Virtual Iron).
These results was first posted to the
Open inofficial storage performance thread @ VMTN
The results below revealed that there's quite a big impact on what product that is chosen and how it configured.
SERVER TYPE: Physical Windows 2003R2sp2 (not a virtual machine)
CPU TYPE / NUMBER: 8 cpu cores, 2 sockets
HOST TYPE: HP DL360G5, 4GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs block size (4096)
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
3.18
|
18530
|
579
|
|
RealLife-60%Rand-65%Read
|
78.6
|
739
|
5.7
|
|
Max Throughput-50%Read
|
3.74
|
15579
|
486
|
|
Random-8k-70%Read.
|
72.7
|
787
|
6.1
|
SERVER TYPE: Virtual Windows 2003R2sp2 on VMware Server 1.0.4 on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
0.5
|
10900
|
340
|
|
RealLife-60%Rand-65%Read
|
156
|
368
|
2.8
|
|
Max Throughput-50%Read
|
1.22
|
7472
|
233
|
|
Random-8k-70%Read.
|
88.1
|
630
|
4.9
|
EXCEPTIONS: CPU Util. 99% 17% 98% 22%
SERVER TYPE: Virtual Windows 2003R2sp2 on VMware Server on Debian Linux 4.0 2.6.18 x64
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default jfs (4096)
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
0.5
|
8550
|
267
|
|
RealLife-60%Rand-65%Read
|
79
|
747
|
5.8
|
|
Max Throughput-50%Read
|
0.63
|
3804
|
237
|
|
Random-8k-70%Read.
|
97
|
609
|
4.7
|
EXCEPTIONS: CPU Util. 100% 17% 98% 16%
SERVER TYPE: Virtual Windows 2003R2sp2 on VMware Player 2.0.1 on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
0.5
|
9920
|
310
|
|
RealLife-60%Rand-65%Read
|
139
|
411
|
3.2
|
|
Max Throughput-50%Read
|
3.1
|
2656
|
83
|
|
Random-8k-70%Read.
|
93.3
|
632
|
4.9
|
EXCEPTIONS: CPU Util. 99% 17.5% 98% 23%
SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual Iron 3.7
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
16.2
|
3732
|
116
|
|
RealLife-60%Rand-65%Read
|
169
|
353
|
2.75
|
|
Max Throughput-50%Read
|
15.2
|
3940
|
123
|
|
Random-8k-70%Read.
|
177
|
337
|
2.6
|
EXCEPTIONS: CPU Util. 39% 17% xx% 17%
SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual Server 2005r2sp1 (1.1.603.0 EE R2 SP1) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
15.5
|
3860
|
120
|
|
RealLife-60%Rand-65%Read
|
159
|
374
|
2.9
|
|
Max Throughput-50%Read
|
17.3
|
3444
|
107
|
|
Random-8k-70%Read.
|
198
|
300
|
2.3
|
EXCEPTIONS: CPU Util. 58% 17% 57% 16%
SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual Server 2005r2sp1 (1.1.603.0 EE R2 SP1) (VT enabled) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
15.9
|
3773
|
117
|
|
RealLife-60%Rand-65%Read
|
159
|
375
|
2.9
|
|
Max Throughput-50%Read
|
17.5
|
3420
|
106
|
|
Random-8k-70%Read.
|
199
|
299
|
2.3
|
EXCEPTIONS: CPU Util. 58% 17% 55% 16%
SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual PC 2007 (6.0.156.0) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
16.7
|
3571
|
111
|
|
RealLife-60%Rand-65%Read
|
161
|
371
|
2.9
|
|
Max Throughput-50%Read
|
18.6
|
3219
|
100
|
|
Random-8k-70%Read.
|
200.5
|
298
|
2.3
|
EXCEPTIONS: CPU Util. 53% 16% 54% 15%
SERVER TYPE: Virtual Windows 2003R2sp2 on Virtual PC 2007 (6.0.156.0) (VT enabled) on Windows Server 2003R2sp2
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default ntfs 4096
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
15.2.
|
3948
|
123
|
|
RealLife-60%Rand-65%Read
|
148.
|
403
|
3.2
|
|
Max Throughput-50%Read
|
16.8.
|
3561
|
111
|
|
Random-8k-70%Read.
|
184.
|
324
|
2.5
|
EXCEPTIONS: CPU Util. 56% 16% 51% 15%
SERVER TYPE: Virtual Windows 2003R2sp2 on XenServer release 4.0.1-4249p (xenenterprise)
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
5
|
10445
|
326
|
|
RealLife-60%Rand-65%Read
|
44
|
810
|
6.3
|
|
Max Throughput-50%Read
|
6.46
|
8896
|
278
|
|
Random-8k-70%Read.
|
55.9
|
811
|
6.3
|
EXCEPTIONS: CPU Util. 92% 52% 83% 37%
SERVER TYPE: Virtual Debian Linux 4.0, kernel 2.6.18.xs4.0.1.900.5799 on XenServer release 4.0.1-4249p (xenenterprise)
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
0.36
|
2773
|
86.6
|
|
RealLife-60%Rand-65%Read
|
3.04
|
328
|
2.6
|
|
Max Throughput-50%Read
|
1.38
|
724
|
22.6
|
|
Random-8k-70%Read.
|
3.3
|
302
|
2.36
|
EXCEPTIONS: CPU Util. 0% 0% 0% 0%
SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service disabled
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
5.3
|
9711
|
303
|
|
RealLife-60%Rand-65%Read
|
43
|
786
|
6.1
|
|
Max Throughput-50%Read
|
6.4
|
8796
|
274
|
|
Random-8k-70%Read.
|
55
|
778
|
6
|
EXCEPTIONS: CPU Util. 73% 55% 56% 41%
SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
5.4
|
9887
|
308
|
|
RealLife-60%Rand-65%Read
|
42
|
777
|
6.0
|
|
Max Throughput-50%Read
|
6
|
8987
|
280
|
|
Random-8k-70%Read.
|
53
|
539
|
6
|
EXCEPTIONS: CPU Util. 67% 56% 67% 43%
SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled, arrayaccelerator=disable
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB arrayaccelerator=disable / 2xSAS 15k rpm / raid 1 / 128KB stripe size / default vmfs 1MB
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
24
|
2384
|
74.5
|
|
RealLife-60%Rand-65%Read
|
96
|
607
|
4.7
|
|
Max Throughput-50%Read
|
76
|
758
|
23.7
|
|
Random-8k-70%Read.
|
87
|
671
|
5.2
|
EXCEPTIONS: CPU Util. 27% 17% 20% 17%
SERVER TYPE: Physical Windows 2003R2sp2 (not a virtual machine), raid 1+0
CPU TYPE / NUMBER: 2x quad core
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
2.95
|
19932
|
622
|
|
RealLife-60%Rand-65%Read
|
46
|
1209
|
9.4
|
|
Max Throughput-50%Read
|
5
|
11272
|
352
|
|
Random-8k-70%Read.
|
39
|
1391
|
10.8
|
SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
4.3
|
9976
|
311
|
|
RealLife-60%Rand-65%Read
|
30
|
1439
|
11
|
|
Max Throughput-50%Read
|
5.5
|
8779
|
274
|
|
Random-8k-70%Read.
|
30
|
1431
|
11
|
EXCEPTIONS: CPU Util. 92% 46% 89% 45%
SERVER TYPE: Virtual Windows 2003R2sp2 on ESX 3.0.2. Descheduled time service enabled. Cache on individual disks enabled.
CPU TYPE / NUMBER: VCPU / 1
HOST TYPE: HP DL360G5, 4 GB RAM; 2x XEON E5345, 2,33 GHz, QC
STORAGE TYPE / DISK NUMBER / RAID LEVEL: P400i 256MB 50% read cache / 4xSAS 15k rpm / raid 1+0 / 128KB stripe size / default vmfs 1MB
|
TEST NAME
|
Av. Resp. Time ms
|
Av. IOs/sec
|
Av. MB/sec
|
|
Max Throughput-100%Read.
|
5.4
|
9681
|
302
|
|
RealLife-60%Rand-65%Read
|
34
|
1353
|
10.5
|
|
Max Throughput-50%Read
|
6.1
|
8763
|
273
|
|
Random-8k-70%Read.
|
35
|
1412
|
11
|
EXCEPTIONS: CPU Util. 71% 40% 72% 33%
Breaking down the numbers
The most important values from these tests are the mixed workload tests (RealLife & Random). These are the workloads that multiple VMs on the same storage will generate. Also, only during large file operations (such as backup) we will see workloads that will be similar to the max throughput tests.
Compared to a physical install, the best virtual max throughput tests were ~50% of the physical one, while the mixed workload tests were fairly equal in both these environments.
From the above tests we have seen that the virtualization product choise will indeed affect the storage performance. The best performer here was Citrix'es XenServer, but VMware's ESX Server was also a close follower (XenServer: 10445 810 8896 811 vs ESX:9887 777 8987 539)
For hosted VMware products on Windows, the RealLife performance is about half the performance of VMware Server running on Debian Linux. Compared to ESX, VMware Server on Linux were generally a bit slower, but it's also not that bad (ESX:9887 777 8987 539 vs Server:8550 747 3804 609)
The big disappointments were however Virtual Iron and the Microsoft products. They showed fairly lousy performance and overall performance of less than half the performance of the above mentioned products in all the tests. Enabling VT didn't have much notable effect in the MS products either.
The different disk setup tests were all done on ESX server and showed the importance of using a good disk controller in an optimal RAID setup.
Disabling the array accelerator had the biggest impact on the max throughput tests (9887vs2384 and 8987vs758) while the mixed workload tests, differed only marginally. This will probably affect backup time and other large file operations quite extensively.
Using RAID 1+0 across 4 disks instead of RAID 1 across 2 disks also showed that more disks is having a very positive impact in such an environment. The biggest difference here was in the mixed workload tests where we got RealLife 1439iops (vs 777) and Random 1431iops (vs 539). The max throughput tests were not largely affected.
Disk cache on individual disks is disabled by default (atleast in HP servers), and if you enable it you get a warning that you shouldn't do this unless your server is powered by an UPS. Enabling this cache didn't give us any better (or worse) performance, but instead lowered the cpu usage consumed by this virtual machine. This might have an effect on the consolidation ratio since many VMs are sharing the same cpu resources.
