Pages Menu
TwitterRssFacebook
Categories Menu

Posted by on Nov 3, 2014 in Infrastructure, Storage, Uncategorized | 0 comments

VSAN and Jumbo Frames

yuno

Error Message:

Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 5419e4e5-daed-b7e8-3a07-00266cf65634. An unknown error has occurred. Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 53229f42-40ff-aa65-8444-00266cf89748

 

Do you know the feeling of when you experience a real awesome technology and you become so excited you just can’t hide it? Yeah..so for me that is VMware’s VSAN.  I was an early adopter of VSAN and grabbed onto those beta bits as soon as it hit the wire.  During the beta it was advised to use Jumbo Frames (aka increasing the MTU size past 1500) for best performance.  I took the bait and why not? It was an easy enough to setup.

This past weekend, I migrated my VSAN cluster over to a Dell PowerConnect 6248 Layer 3 switch.  There are two places to configure Jumbo Frames on the Dell Powerconnect: The VLAN and the physical interface.

To recap, I have set:

  1. 9000 MTU on the VSAN VMkernel on the vSphere host.
  2. 9000 MTU on the VSAN VLAN on the Dell PowerConnect
  3. 9000 MTU on the VSAN Physical Interface the Dell PowerConnect

Lastly, I tested VSAN out by writing a folder to the VSAN datastore but get returned a nasty error which read

Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 5419e4e5-daed-b7e8-3a07-00266cf65634. An unknown error has occurred. Call “FileManager.MakeDirectory” for object “FileManager” on vCenter Server “vCenter” failed. Failed to connect to component host 53229f42-40ff-aa65-8444-00266cf89748

 

Great, now I can’t even write a folder to the VSAN datastore so I figuratively yell into the sky, Come on VSAN Gods! Why!??

Once I got over the self pity stage, I noticed Dell uses a different type of MTU logic on the physical switch interface.  While in the rest of the world, when you configure a MTU of 9000, you mean it.  However, with Dell switches a MTU 9000 is actually (9 * 1024) = 9216.  Ugh, so now I have set

  1. 9000 MTU on the VSAN VMkernel on the vSphere host.
  2. 9000 MTU on the VSAN VLAN on the Dell PowerConnect
  3. 9216 MTU on the VSAN Physical Interface the Dell PowerConnect

BAM! The VSAN datastore is rocking and I can now write to it.  However, this scenario posed some reflection on how MTU actually impacts VSAN.  I did some further testing and came up with the follow conclusions:

  1. If any host in a VSAN cluster has a mismatched MTU size, NOTHING can write to the VSAN datastore. Even if one host with the wrong MTU is set then it will prevent VSAN from working.
  2.  Even with mismatched MTU’s when one verifies the Network Status (vCenter > Virtual SAN > General) it will show Normal. However, this doesn’t verify MTU, just IP connectivity.  To test if the MTU is correct then use the MTU of the VSAN VMkernel’s MTU size and issue a vmkping -s <VSANvmkernel_mtu_setting> <Other-VSANvmkernel-interfaces-in-cluster>
  3. VSAN performances about the same with or without Jumbo Frames configured.

In conclusion, would I advise configuring Jumbo Frames with VSAN?  No.  Unless you’re the type who prefers all risk and no reward…

 

 

Read More

Posted by on Nov 2, 2014 in Infrastructure, Networking | 0 comments

Increasing the number vNetwork Distributed Switch (vDS) Uplink Ports

“Call “HostNetworkSystem.UpdateNetworkConfig” for object “networkSystem-27” on vCenter Server “vCenter” failed.  Network configuration change disconnected the host ‘x.x.x.x’ from vCenter server and has been rolled back.  A change in the network configuration disconnected the host ‘x.x.x.x’ from vCenter Server. The change has been rolled back.”

Let me guess, were you adding a host to a vDS and all of sudden you received this cryptic message?  If so, you may have ran out of vDS uplink ports. You may be reflecting on why you were so ungenerous to my vDS uplink port allocation in the first place?  Unfortunately I can’t answer that but I can tell you how to fix it!

Running out of vDS uplink ports will put the breaks on adding anymore hosts to your vDS.  However,  it is down right embarrassing when you find out that you can’t increase the number of ports through vCenter.  This bad boy parameter is all grayed.  WHAT!

vds1

 

When this occurred to me, I made a mad rush to Google and realized I couldn’t locate any steps to increase the number of uplink ports.  I was disappointed when I came to realization that I was the only stingy vDS uplink port allocator on the entire World Wide Web.  Or maybe I am just inefficient with my Google search engine expressions.

Regardless, I captured below on how you can get out of this sticky situation:

  1. In a web browser, go to http://vCenterServer-ip-address/mob/
  2. When prompted, enter your vCenter Server username and password.
  3. Click the content link.
  4. In the left pane, search for the row with the word rootFolder.
  5. Open the link in the right pane of the row. The link is similar to group-d1 (Datacenters).
  6. In the left pane, search for the row with the word childEntity. In the right pane, you see a list of datacenter links.  Click the datacenter link where the vDS is defined.
  7. In the left pane, search for the row with the word networkFolder, and open the link in the right pane. The link is similar to group-n (network).
  8. In the left pane, search for the row with the word childEntity. You see a list of vDS and distributed port group links in the right pane.  Select the dvportgroup that is for the Uplink.  
  9. In the left pane, search for the row with the word config, and click the link in the right pane.
  10. In the left pane, search for the row with the word configVersion. It is normally the first row.  WRITE THIS NUMBER DOWN!
  11. Go back one page and select ReconfigureDVPortgroup_Task
  12. Delete the contents that you see under Value and replace it with the syntax below.  Fill in configVersion with what was captured in Step 10.  The field for numPorts is what you want to increase the number of uplink ports to.

    <spec><configVersion>0</configVersion><numPorts>20</numPorts></spec>

  13. Select Invoke Method
  14. Optionally, if you don’t trust what you just did, scroll up and select the config link and see if your numPorts field increased
  15. You will notice that if you go back into vCenter, the number of Uplink ports will be reflected.  However, don’t get too excited yet.  You’re not out of the woods until you have restarted the vCenter Server.

vds2

Read More

Posted by on Oct 4, 2014 in Automation, Compute, Infrastructure | 0 comments

DPM – A vSphere Feature That Can Save Your Home Lab From Thermonuclear Meltdown

Screen shot 2011-08-03 at 1.09.18 PM

 

Florida summers are hot.  Especially when you work out of a small home office and have a ton of equipment running non-stop.  This year I finally had enough and was motivated to do something about it. My home lab consists of two Dell PowerEdge C6100 rack mount servers running two nodes each and a Synology DS1813+ NAS as well as all of your typical networking gear all tucked away in a tiny little closet.

I started playing around with a lesser used feature in vSphere called ‘Dynamic Power Management’ or DPM.  DPM works with DRS to monitor the workloads across hosts, consolidate them down to run on as few hosts as possible and then put the vacant hosts in standby mode thereby saving power.  When the workloads spike, DPM reverses the process, turns on hosts and DRS redistributes the loads amongst the hosts.  It truly is as simple as that.

DPM is a simple cluster setting.  Open up your vCenter web or C+ client, right click the cluster and select ‘edit settings’.  Highlight vSphere DRS and choose ‘Automatic’ from the Power Management drop down.

DPM

 

After turning this on and waiting around a few minutes…nothing happened.  So I set the DPM Threshold to the most aggressive setting in the C+ client to try to get it to react.

DPM Threshold

After waiting a few minutes with no vMotion activity or hosts beginning to go into standby mode, I started to research custom settings.  I found the following that I applied under the ‘Advanced Options’ button at the bottom of the DRS configuration screen.

Advanced Options DRS

And voila!  A few seconds after I hit okay, vMotion started consolidating VMs and eventually put three of four hosts in standby (I don’t run HA in my lab as none of my environment is mission-critical).  Now, depending on the workloads, I can usually get a 10:1 consolidation ratio which is about what I need to be running in my day-to-day activities to remain productive.  As soon as I launch more VMs, within a few seconds I see another host come out of standby mode and DRS load balances my environment as expected.

My office is A LOT cooler now and my power bill saw a noticeable drop thanks to DPM.

Meltdown averted!!!

images

 

 

Disclaimer

Read More

Posted by on Oct 1, 2014 in Infrastructure, Storage, Uncategorized | 4 comments

Solved: The Case of the Missing Snapshot and the Failed VDP Backup

You’ve been working in the lab creating a plethora of virtual goodness and figure you’ll do a backup so generations will be able to enjoy the fruits of your labor.  You download the vSphere Data Protection appliance and deploy it, schedule a backup job and get mixed success.  Looking through your log of failed jobs you see “VDP: Operation failed due to existing snapshot”.

VDP error message

You check the snapshot manager, but find no snapshots.  You even click ‘Consolidate’ – which completes successfully and relaunch the backup job, but it fails again.

“NOOOOO!!!!!” You cry out in unimaginable frustration.

Take a deep breath.  This is a quick and easy fix and you’ll soon be the virtualization hero again.  There are just some extra files in your VM folder that VDP doesn’t know what to do with.

  • Open your web or C+ vCenter client and find the datastore your failing VM is attached to.
  • Open the datastore browser for that datastore and find your failing VM folder.
  • Leave the datastore browser window open and go back to the web or C+ client and perform a storage vMotion on the failing VM to another datastore.
  • Upon completion, refresh your datastore browser.  You should see at least one file left over.
  • If the folder fails to delete, shutdown the VM and try again.
  • Once the original folder has been successfully deleted, kick off the backup job again.

VDP error

 

Congratulations!  You are a true virtualization ninja.

CASE CLOSED.

 

 

Disclaimer

Read More