I'm just getting started with SmartOS so don't expect any deep tuning of the file system. This is just to show a few different "out-of-the-box" disk throughput tests to compare KVM guests to native SmartOS guests. The host has six 2.5" consumer grade disks, 5 in raid and 1 SSD for read-cache.
NAME STATE READ WRITE CKSUM zones ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c0t0d0 ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 c0t3d0 ONLINE 0 0 0 c0t4d0 ONLINE 0 0 0 cache c0t5d0 ONLINE 0 0 0
These are Advanced Format disks so they are natively 4K but SmartOS picked them up as 512K, so some write penalty is expected.
[root@las-node4 ~]# zdb | egrep 'ashift| name' name: 'zones' ashift: 9
Windows 7 Baremetal (single SATA disk):
Just to set a credible baseline I benchmarked a single drive in a clean Win7 install in the same host hardware. There is only one disk and no SSD cache so the expectation is the SmartOS ZFS raid volume will be a number of times faster on read and write.
write = 101.6 MB/s read = 108.8 MB/s
Nothing too earth shattering going on here. The drive was picked up as 512K by Windows 7 so there is likely some performance penalty there also.
[root@las-node4 /opt]# dd if=/dev/zero of=testfile bs=4096k count=10000 41943040000 bytes transferred in 103.821261 secs (403992783 bytes/sec)
[root@las-node4 /opt]# dd of=/dev/zero if=testfile bs=4096k count=10000 41943040000 bytes transferred in 13.238225 secs (3168328000 bytes/sec)
write = 385 MB/s read = 3.0 GB/s
[root@base-64-test1 ~]# dd if=/dev/zero of=testfile bs=4096k count=10000 41943040000 bytes (42 GB) copied, 113.349 s, 370 MB/s
[root@base-64-test1 ~]# dd of=/dev/zero if=testfile bs=4096k count=10000 41943040000 bytes (42 GB) copied, 18.5145 s, 2.3 GB/s
write = 370 MB/s read = 2.3 GB/s
CentOS 6 KVM Guest (virtio):
[root@centos-6-test1 data]# dd if=/dev/zero of=testfile bs=4096k count=10000 41943040000 bytes (42 GB) copied, 1158.3 s, 36.2 MB/s
[root@centos-6-test1 data]# dd of=/dev/zero if=testfile bs=4096k count=10000 41943040000 bytes (42 GB) copied, 51.2937 s, 818 MB/s
write = 36.2 MB/s read = 818 MB/s
Windows 2008 Guest (virtio):
write = 3.1 MB/s read = 290.8 MB/s
Read speed is much slower than SmartOS or KVM CentOS, but the real issue here is the anemic write speed. Lots of people seem to be having issues with the Joyent signed virtio drivers when running on guests not provisioned by Joyent SDC.
SmartOS native guests are remarkably fast, with close to baremetal performance. Each guest was only provisioned with 2 cores and default I/O priority, so that probably explains why the SmartOS guest is a little slower than the SmartOS global zone (baremetal).
Overall KVM on SmartOS seems to have some performance issue with write speed for both Linux and Windows guests. I assume this can be tuned out of the system with optimal driver selection and vdisk provisioning. I will update this port as I learn more...