Pages Menu
TwitterRssFacebook
Categories Menu

Posted by on Nov 3, 2014 in Infrastructure, Storage, Uncategorized | 0 comments

VSAN and Jumbo Frames

yuno

Error Message:

Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 5419e4e5-daed-b7e8-3a07-00266cf65634. An unknown error has occurred. Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 53229f42-40ff-aa65-8444-00266cf89748

 

Do you know the feeling of when you experience a real awesome technology and you become so excited you just can’t hide it? Yeah..so for me that is VMware’s VSAN.  I was an early adopter of VSAN and grabbed onto those beta bits as soon as it hit the wire.  During the beta it was advised to use Jumbo Frames (aka increasing the MTU size past 1500) for best performance.  I took the bait and why not? It was an easy enough to setup.

This past weekend, I migrated my VSAN cluster over to a Dell PowerConnect 6248 Layer 3 switch.  There are two places to configure Jumbo Frames on the Dell Powerconnect: The VLAN and the physical interface.

To recap, I have set:

  1. 9000 MTU on the VSAN VMkernel on the vSphere host.
  2. 9000 MTU on the VSAN VLAN on the Dell PowerConnect
  3. 9000 MTU on the VSAN Physical Interface the Dell PowerConnect

Lastly, I tested VSAN out by writing a folder to the VSAN datastore but get returned a nasty error which read

Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 5419e4e5-daed-b7e8-3a07-00266cf65634. An unknown error has occurred. Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 53229f42-40ff-aa65-8444-00266cf89748

 

Great, now I can’t even write a folder to the VSAN datastore so I figuratively yell into the sky, Come on VSAN Gods! Why!??

Once I got over the self pity stage, I noticed Dell uses a different type of MTU logic on the physical switch interface.  While in the rest of the world, when you configure a MTU of 9000, you mean it.  However, with Dell switches a MTU 9000 is actually (9 * 1024) = 9216.  Ugh, so now I have set

  1. 9000 MTU on the VSAN VMkernel on the vSphere host.
  2. 9000 MTU on the VSAN VLAN on the Dell PowerConnect
  3. 9216 MTU on the VSAN Physical Interface the Dell PowerConnect

BAM! The VSAN datastore is rocking and I can now write to it.  However, this scenario posed some reflection on how MTU actually impacts VSAN.  I did some further testing and came up with the follow conclusions:

  1. If any host in a VSAN cluster has a mismatched MTU size, NOTHING can write to the VSAN datastore. Even if one host with the wrong MTU is set then it will prevent VSAN from working.
  2.  Even with mismatched MTU’s when one verifies the Network Status (vCenter > Virtual SAN > General) it will show Normal. However, this doesn’t verify MTU, just IP connectivity.  To test if the MTU is correct then use the MTU of the VSAN VMkernel’s MTU size and issue a vmkping -s <VSANvmkernel_mtu_setting> <Other-VSANvmkernel-interfaces-in-cluster>
  3. VSAN performances about the same with or without Jumbo Frames configured.

In conclusion, would I advise configuring Jumbo Frames with VSAN?  No.  Unless you’re the type who prefers all risk and no reward…

 

 

Post a Reply

Your email address will not be published. Required fields are marked *