Address

Los Angeles, California

Phones

1-844-CLOUDLA 1-844-256-8352

Hours

Mon-Fri 8:00 am to 4:00 pm

SmartOS Disk Speed – Native vs. KVM

I'm just getting started with SmartOS so don't expect any deep tuning of the file system. This is just to show a few different "out-of-the-box" disk throughput tests to compare KVM guests to native SmartOS guests. The host has six 2.5" consumer grade disks, 5 in raid and 1 SSD for read-cache.

 

 

NAME        STATE     READ WRITE CKSUM
          zones       ONLINE       0     0     0
            raidz1-0  ONLINE       0     0     0
              c0t0d0  ONLINE       0     0     0
              c0t1d0  ONLINE       0     0     0
              c0t2d0  ONLINE       0     0     0
              c0t3d0  ONLINE       0     0     0
              c0t4d0  ONLINE       0     0     0
          cache
            c0t5d0    ONLINE       0     0     0

These are Advanced Format disks so they are natively 4K but SmartOS picked them up as 512K, so some write penalty is expected.

  [root@las-node4 ~]# zdb | egrep 'ashift| name'
      name: 'zones'
              ashift: 9

Windows 7 Baremetal (single SATA disk):

Just to set a credible baseline I benchmarked a single drive in a clean Win7 install in the same host hardware. There is only one disk and no SSD cache so the expectation is the SmartOS ZFS raid volume will be a number of times faster on read and write.

Write Test

win-disk-write-test

Read Test

win-disk-read-test

write = 101.6 MB/s  read = 108.8 MB/s

Nothing too earth shattering going on here. The drive was picked up as 512K by Windows 7 so there is likely some performance penalty there also.


SmartOS Host:

[root@las-node4 /opt]# dd if=/dev/zero of=testfile bs=4096k count=10000
  41943040000 bytes transferred in 103.821261 secs (403992783 bytes/sec)
[root@las-node4 /opt]# dd of=/dev/zero if=testfile bs=4096k count=10000
  41943040000 bytes transferred in 13.238225 secs (3168328000 bytes/sec)

write = 385 MB/s  read = 3.0 GB/s


SmartOS Guest:

[root@base-64-test1 ~]# dd if=/dev/zero of=testfile bs=4096k count=10000
  41943040000 bytes (42 GB) copied, 113.349 s, 370 MB/s
[root@base-64-test1 ~]# dd of=/dev/zero if=testfile bs=4096k count=10000
  41943040000 bytes (42 GB) copied, 18.5145 s, 2.3 GB/s

write = 370 MB/s  read = 2.3 GB/s


 CentOS 6 KVM Guest (virtio):

[root@centos-6-test1 data]# dd if=/dev/zero of=testfile bs=4096k count=10000
  41943040000 bytes (42 GB) copied, 1158.3 s, 36.2 MB/s
[root@centos-6-test1 data]# dd of=/dev/zero if=testfile bs=4096k count=10000
  41943040000 bytes (42 GB) copied, 51.2937 s, 818 MB/s 

write = 36.2 MB/s  read = 818 MB/s


Windows 2008 Guest (virtio):

Write Test

win-vm-write-test

Read Test

win-vm-read-test

write = 3.1 MB/s  read = 290.8 MB/s

Read speed is much slower than SmartOS or KVM CentOS, but the real issue here is the anemic write speed. Lots of people seem to be having issues with the Joyent signed virtio drivers when running on guests not provisioned by Joyent SDC.


SmartOS native guests are remarkably fast, with close to baremetal performance. Each guest was only provisioned with 2 cores and default I/O priority, so that probably explains why the SmartOS guest is a little slower than the SmartOS global zone (baremetal).

Overall KVM on SmartOS seems to have some performance issue with write speed for both Linux and Windows guests. I assume this can be tuned out of the system with optimal driver selection and vdisk provisioning. I will update this port as I learn more...

3 Responses

  1. I'll soon be adding a write cache disk and repeating these tests to see if that's the cause of the performance issues with KVM writes.
  2. By default all writes to KVM branded zones are synchronous writes, a design choice from Joyent. You can disable synchronous writes on the zvol, but there are safety issues. With a ZIL on SSD you should be fine. (see wesolows comment https://github.com/joyent/smartos-live/issues/320)
    • Hi Luc, Thanks for the reply. I realized there would be some write penalty for not having a ZIL, but I wasn't expecting 2MB/s :)

Leave a comment