ASP Developer Network - Brought to you by Steve Radich and BitShop, Inc. - http://www.bitshop.com

ASPDeveloper

ASPDeveloper.Net / VirtualServerFAQ.COM

Make sure you go to our home page for:

* Virtual Server 2005 FAQ
* ASP.Net 2.0 Information
* ASP.Net 2.0 Streaming Tutorials
* ASP.Net 2.0 HTML based Tutorials

Made possible by Business Internet Technology Shop (BitShop)

Virtual Server 2005

Recent Virtual Server 2005 posts

* Automatic start of W2003 guest does not work....
* Host networking issues on Windows Vista Business
* Tips on Virtual Server
* Network Adapter disconnected
* Scripts in host don"t run
* authentication and API
* Upgradeing Integration Services on Hyper-V in "08
* Virtual Server Admin - Access was denied
* Hosting O.S.
* How to install MS virtual server 2005
* Which user account to run virtual machine?
* Virtual server Admin page issue
* Virtual DC (Windows Server 2003)
* Transfer File from host to virtual machines
* Problems installing vs 2005r2 enterprise on windows xp/sp2 in NT domain
* Windows 2008 core
* Licensing information for hosted VMs...
* How can you tell if you"re running Virtual Server 2005 R2 SP1?
* Convert XEN guest to VHD?
* VMRC Client List Size

Add Solution

Abysmal Network Performance

Source: microsoft.public.virtualserver
Sent: 02/17/2008
From: At the end of my rope
Message:

Performance of virtualized file server is horrible. I've tried everything I
can think of. Anybody out there have any ideas?

Here's my configuration.

Server cluster, three identical HP Proliant DL380 G4 servers:
2 3.6GHZ 1M processors
12GB RAM
2 72GB SCSI drives for host OS
Dual integrated gigabit NICs
Intel Dual-Port PRO/1000MT Server Adapter
HP NC1020 Gigabit Server Adapter
Alacritech SE2002XT Dual-Port iSCSI Accelerator
Windows Server 2003 Enterprise x64 Edition SP2
Virtual Server 2005 R2 Enterprise SP1

iSCSI SAN
2 Netgear GS516 gigabit switches
Promise m300i RAID Storage Array
2 WD 7200 500GB SATA
10 WD 10K 150GB SATA
Promise m310i RAID Storage Array
2 WD 10K 150GB SATA

I've already tried turning off tcp offloading. I isolated the file server
by moving all of the other virtual machines to other cluster nodes, but that
didn't have any noticable impact on performance. Disk read performance in
the VM is not great, 33MB/sec. Read performance over the network from a
high-end workstation (with a gigabit NIC) with nobody else on the network
drops to 11MB/sec. Performance between virtual machines on the same host is
roughly the same--even using the MS loopback adapter. Disk read performance
on the host is great--even from the iSCSI SAN: I transferred 3GB off a SAN
disk onto the host disk in about 5 seconds.

I have hyperthreading turned on. Does anybody out there think that might be
a problem?

Post Reply | This is Spam! | Mark as Spam

Related Messages

RE: Abysmal Network Performance

Sent: 02/17/2008
From: Jesper Arnecke
Message:

Hiya,

First of all regarding hyperthreading, read this:
http://support.microsoft.com/default.aspx/kb/867586?lc=2057

Secondly read this
http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?&lang=en&cr=US&guid=&sloc=en-us&dg=microsoft.public.virtualserver&p=1&tid=87d9f0f9-bcfe-4acf-8fb4-0d0406fd60ac&mid=87d9f0f9-bcfe-4acf-8fb4-0d0406fd60ac

And third note that the guest systems can only emulate a 10/100 network
card. (which is usually good enough, as the host will run 1000, and that way
one guest system wont eat up the entire network. - it's a balance of good and
bad.
I cant see from your setup where you have your virtual files located either?

I hope it helps you getting a conclusion on your problem, else ask again :)

Re: Abysmal Network Performance

Sent: 02/18/2008
From: Steve Jain <(email address - cut out)>
Message:

On Sun, 17 Feb 2008 23:16:00 -0800, Jesper Arnecke
<(email address - cut out)> wrote:

Which isn't a limit in itself. The only limit is the actual host's
connection, and the CPU overhead. A VM can get up to about 500Mb
throughput if the host is robust enough.

--
Cheers,
Steve Jain, Virtual Machine MVP
http://vpc.essjae.com/
I do not work for Microsoft.

Show quoted text

>And third note that the guest systems can only emulate a 10/100 network
>card.

Re: Abysmal Network Performance

Sent: 02/18/2008
From: Jesper Arnecke
Message:

"Steve Jain" wrote:

Hey Steve,

I am sure when you say it, it holds truth. But to my head it doesnt really
makes sense that a specific 10/100 NIC can deliver more, if located in a vm.
As far as I understand, what happends is that it will emulate a specific
piece of hardware, leaving the boundries of a vm within that piece of
hardware...
I would very much like to read more on this overclocking of a NIC, as my
knowledge ends about the 100 :)

Show quoted text

> Which isn't a limit in itself. The only limit is the actual host's
> connection, and the CPU overhead. A VM can get up to about 500Mb
> throughput if the host is robust enough.

RE: Abysmal Network Performance

Sent: 02/19/2008
From: At the end of my rope
Message:

"Jesper Arnecke" wrote:

The virtual hard disks are fixed size SCSI, located on the iSCSI SAN, and
have to be in order for the VMs to be clustered. But the system isn't disk
bound, it's network bound. When I run a process directly on the VM, I get
about 33MB per second throughput. When I run the same process across the
network I get about 11MB per second throughput--despite there being gigabit
NICs and no other network traffic except between the high-end workstation and
the VM. That just doesn't make any sense. I've tried dedicating a 1GB
NIC--and it doesn't matter whether I use one of the embedded HP NICS or one
of the Intel NICS, or even the Microsoft Loopback adapter. The performance
is still roughly 1/3 of what in my opinion it should be. Something is screwy.

I tried disabling hyperthreading, but it didn't have any effect. The system
isn't by any means CPU bound.

Show quoted text

> Hiya,
>
> First of all regarding hyperthreading, read this:
> http://support.microsoft.com/default.aspx/kb/867586?lc=2057
>
> Secondly read this:
> http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?&lang=en&cr=US&guid=&sloc=en-us&dg=microsoft.public.virtualserver&p=1&tid=87d9f0f9-bcfe-4acf-8fb4-0d0406fd60ac&mid=87d9f0f9-bcfe-4acf-8fb4-0d0406fd60ac
>
> And third note that the guest systems can only emulate a 10/100 network
> card. (which is usually good enough, as the host will run 1000, and that way
> one guest system wont eat up the entire network. - it's a balance of good and
> bad.
> I cant see from your setup where you have your virtual files located either?
>
> I hope it helps you getting a conclusion on your problem, else ask again :)

Re: Abysmal Network Performance

Sent: 02/19/2008
From: "Bill Grant" <not.available@online>
Message:

The number that appears in the vm is just that- a number. Under Hyper-V
it is 10Gb (probably to avoid discussions like this) !

The rate at which the data is actually transferred depends on the NIC in
the host (and its driver) and the network to which it is attached, not to
anything in the vm.

"Jesper Arnecke" <(email address - cut out)> wrote in message
news:(email address - cut out)...

Show quoted text

> "Steve Jain" wrote:
>> Which isn't a limit in itself. The only limit is the actual host's
>> connection, and the CPU overhead. A VM can get up to about 500Mb
>> throughput if the host is robust enough.
>
> Hey Steve,
>
> I am sure when you say it, it holds truth. But to my head it doesnt really
> makes sense that a specific 10/100 NIC can deliver more, if located in a
> vm.
> As far as I understand, what happends is that it will emulate a specific
> piece of hardware, leaving the boundries of a vm within that piece of
> hardware...
> I would very much like to read more on this overclocking of a NIC, as my
> knowledge ends about the 100 :)

Re: Abysmal Network Performance

Sent: 02/20/2008
From: Jesper Arnecke
Message:

"Bill Grant" wrote:

Bill,
What your saying is that the virtual guest will run directly on the hardware
provided by the host. As this is almost the case with 2008 Hyper-V, it is not
the case for VS 2005.
Please, can you provide any MS documentation, white papers, test scenariors
etc. that can confirm what your saying?
We are facing similar issues the ones described earlier in this thread, so
anything to narrow it down would be high appreciated..

Show quoted text

> The number that appears in the vm is just that- a number. Under Hyper-V
> it is 10Gb (probably to avoid discussions like this) !
>
> The rate at which the data is actually transferred depends on the NIC in
> the host (and its driver) and the network to which it is attached, not to
> anything in the vm.

RE: Abysmal Network Performance

Sent: 02/20/2008
From: Jesper Arnecke
Message:

> The virtual hard disks are fixed size SCSI, located on the iSCSI SAN, and

If my assumption is correct, and the virtual guest only emulates a 10/100
NIC, then your max throughput will be around 12MB/sec.
So having a throughput of 11MB/sec is expected. there is a difference
between 100Mbit and 100Mbyte.
100Mbit per sec / 8 = speed in Mbyte per sec.- Approximately.

If possible, try and test your process from the host machine, to see if it
will perform accordingly. 1000Mbit per second.

Being about 89% on this part, I would still recommend that you check up on
it :)

Show quoted text

> have to be in order for the VMs to be clustered. But the system isn't disk
> bound, it's network bound. When I run a process directly on the VM, I get
> about 33MB per second throughput. When I run the same process across the
> network I get about 11MB per second throughput--despite there being gigabit
> NICs and no other network traffic except between the high-end workstation and
> the VM. That just doesn't make any sense. I've tried dedicating a 1GB
> NIC--and it doesn't matter whether I use one of the embedded HP NICS or one
> of the Intel NICS, or even the Microsoft Loopback adapter. The performance
> is still roughly 1/3 of what in my opinion it should be. Something is screwy.
>
> I tried disabling hyperthreading, but it didn't have any effect. The system
> isn't by any means CPU bound.

Re: Abysmal Network Performance

Sent: 02/20/2008
From: Jesper Arnecke
Message:

"Bill Grant" wrote:

Hey Bill,

First of all, thank you for your patience and assistance in inlighting me :)
Secondly, what he describes is exactly what I am saying, at least in my head.
Server A -> Physical NIC -> 'Virtual Machine Network Services' -> Emulated
NIC -> Server B
Considering the above, they emulated NIC is running 10/100, as per 2005
design, so no matter how fast your Physical NIC, you will not get speeds of
more than 100Mbit.
Unless it's because this emulation is not "true", and thus allowing the VM
NS to actually be the one controlling the speed(which is dependant on the
physical NIC), however that sounds a bit far fetched.

If I am still off in the understanding, I'll quit it and buy you a beer if I
ever bump into you ; )

Show quoted text

> What do you believe happens with VS2005? The data coming over a network
> can only be handled by a physical device connected to the network. That is
> your NIC. Virtual Machine Network Services works with the NIC driver to
> split off traffic for the virtual machine (and delivers packets to the NIC
> driver to send in the other direction).
> All of this is controlled by the physical NIC, not by anything inside the
> vm.
> See this article in Ben's blog.
> http://blogs.msdn.com/virtual_pc_guy/archive/2005/04/01/404816.aspx

Re: Abysmal Network Performance

Sent: 02/21/2008
From: "Chuck Cook" <(email address - cut out)>
Message:

> The virtual hard disks are fixed size SCSI, located on the iSCSI SAN, and

You say you're not CPU-bound, but have you measured processor activity
during heavy I/O? If you're using the Microsoft iSCSI initiator, you might
be surprised at how much of a hit you take. This white paper
[http://www.sanstor.info/5iSCSI%20software%20initiators%20vs.pdf] implies
that a software initiatior "may consume up to 500MHz of CPU for 1 Gigabit
Ethernet line rate" - which is a heavy hit on a virtual machine that can
only utilize host one processor.

Almost all of my production servers are VS2003R2 guests, using the MS iSCSI
initiator to connect to EMC arrays, and I've had less-than-stellar
throughput for machines with heavy I/O.

Plus the emulated Intel/DEC NIC in the VM can't do TCP offloading or take
advantage of jumbo frames. I really wish Microsoft had offered an emulated
Gigabit NIC in R2 as an optional addition so we could have used jumbos -
even though it might not have been "faster" in terms of wire speed (and by
the way, I'm very skeptical of the claim that the emulated NIC can run 'four
to six times faster' than 100Mbps in real-world scenarios. I'd buy it when a
guest is talking to another guest on the same host, but I've never seen that
kind of performance when the guest is accessing a physical LAN.)

I will be very happy when I can move my production systems to Veridian.

Chuck

Show quoted text

> have to be in order for the VMs to be clustered. But the system isn't
> disk
> bound, it's network bound. When I run a process directly on the VM, I get
> about 33MB per second throughput. When I run the same process across the
> network I get about 11MB per second throughput--despite there being
> gigabit
> NICs and no other network traffic except between the high-end workstation
> and
> the VM. That just doesn't make any sense. I've tried dedicating a 1GB
> NIC--and it doesn't matter whether I use one of the embedded HP NICS or
> one
> of the Intel NICS, or even the Microsoft Loopback adapter. The
> performance
> is still roughly 1/3 of what in my opinion it should be. Something is
> screwy.
>
> I tried disabling hyperthreading, but it didn't have any effect. The
> system
> isn't by any means CPU bound.

Re: Abysmal Network Performance

Sent: 02/21/2008
From: "Bill Grant" <not.available@online>
Message:

"Jesper Arnecke" <(email address - cut out)> wrote in message
news:(email address - cut out)...

What do you believe happens with VS2005? The data coming over a network
can only be handled by a physical device connected to the network. That is
your NIC. Virtual Machine Network Services works with the NIC driver to
split off traffic for the virtual machine (and delivers packets to the NIC
driver to send in the other direction).
All of this is controlled by the physical NIC, not by anything inside the
vm.

See this article in Ben's blog.

http://blogs.msdn.com/virtual_pc_guy/archive/2005/04/01/404816.aspx

Show quoted text

> "Bill Grant" wrote:
>> The number that appears in the vm is just that- a number. Under
>> Hyper-V
>> it is 10Gb (probably to avoid discussions like this) !
>>
>> The rate at which the data is actually transferred depends on the NIC
>> in
>> the host (and its driver) and the network to which it is attached, not to
>> anything in the vm.
>
> Bill,
> What your saying is that the virtual guest will run directly on the
> hardware
> provided by the host. As this is almost the case with 2008 Hyper-V, it is
> not
> the case for VS 2005.
> Please, can you provide any MS documentation, white papers, test
> scenariors
> etc. that can confirm what your saying?
> We are facing similar issues the ones described earlier in this thread, so
> anything to narrow it down would be high appreciated..

Re: Abysmal Network Performance

Sent: 02/21/2008
From: "Bill Grant" <not.available@online>
Message:

"Jesper Arnecke" <(email address - cut out)> wrote in message
news:(email address - cut out)...

No, there is no speed limit set inside the vm. If the physical NIC is fast
enough it can run at four to six times faster than a 10/100 device, as I
think Steve Jain said much earlier in this thread.

Show quoted text

>
> "Bill Grant" wrote:
>> What do you believe happens with VS2005? The data coming over a network
>> can only be handled by a physical device connected to the network. That
>> is
>> your NIC. Virtual Machine Network Services works with the NIC driver to
>> split off traffic for the virtual machine (and delivers packets to the
>> NIC
>> driver to send in the other direction).
>> All of this is controlled by the physical NIC, not by anything inside the
>> vm.
>> See this article in Ben's blog.
>> http://blogs.msdn.com/virtual_pc_guy/archive/2005/04/01/404816.aspx
>
> Hey Bill,
>
> First of all, thank you for your patience and assistance in inlighting me
> :)
> Secondly, what he describes is exactly what I am saying, at least in my
> head.
> Server A -> Physical NIC -> 'Virtual Machine Network Services' -> Emulated
> NIC -> Server B
> Considering the above, they emulated NIC is running 10/100, as per 2005
> design, so no matter how fast your Physical NIC, you will not get speeds
> of
> more than 100Mbit.
> Unless it's because this emulation is not "true", and thus allowing the VM
> NS to actually be the one controlling the speed(which is dependant on the
> physical NIC), however that sounds a bit far fetched.
>
> If I am still off in the understanding, I'll quit it and buy you a beer if
> I
> ever bump into you ; )

Post Reply | This is Spam! | Mark as Spam

Other groups

Virtual PC (Mac)(3592)	Virtual PC(12844)	Virtual PC (French)(496)
Virtual Server 2005(6245)

Our Sponsors

Virtual Server 2005 Info Here!
http://www.aspdeveloper.net

Wiki

Blogs

Articles

Image galleries

File galleries

Forums

Maps

Directories

ASP Developer Network (since 1996) - Brought to you by Steve Radich and BitShop, Inc. - http://www.bitshop.com
Copyright © 1997-2005 by BitShop, Inc. All Rights Reserved.
No content may be reproduced in any form without written permission from BitShop, Inc.
We will be honored to assist with reproduction rights on our material if you contact us.
We also can provide print articles for your magazine, private training seminars at your site, and conference presentations.
Thank you for visiting. Please bookmark our site if you found it useful!
If you wish to make a donation it will help further the development of these free resources: