Solving WinRM Host connection errors in SCVMM - System Center Virtual Machine Manager 2012 R2

I had a few Hyper-V 2012 R2 hosts that didn't want to talk to my VMM 2012 R2 server, so I started down the long road of tracking down the problems.  Sorry if some of this seems disjointed or out of order.  I'm trying to document this all after a week of troubleshooting errors.

Here is my advice for tracking down the problems. Most are related to WinRM.

Step 1: Check the Health status of the Host
To check the Health status of a host, perform the following steps:

1. Open the VMM Console.

2. Select the Fabric view, right-click the host that’s experiencing issues and chose properties.

3. Within the host properties, select Status.

4. Select the category that has the Red exclamation to view the error details.

In my case, the third category down was WinRM that was giving me an error.

Lets start with the easy stuff.
Make sure WMI is working...
net start winmgmt
If you think the WMI repository is corrupted, you could try this:
winmgmt /salvagerepository
There is more detail on fixing WMI here:
http://social.technet.microsoft.com/Forums/windows/en-US/8ed26d46-9994-4052-a307-5b071805aea8/wmi-corrupt-how-to-reinstallrepair

After checking WMI, I moved on to finding solutions for WinRM problems.

When troubleshooting WinRM the process is like running a simple ping test.
Type the command below at an elevated command prompt:
winrm id

This should produce output similar to below:
IdentifyResponse
    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
    ProductVendor = Microsoft Corporation
    ProductVersion = OS: 6.1.7201 SP: 0.0 Stack: 2.0

If you get an error instead of the output above, perform a Quick Configuration of WinRM:
winrm qc

You should see something like this below. Answer ‘Yes’ to any questions.
WinRM already is set up to receive requests on this machine.
WinRM is not set up to allow remote access to this machine for management.
The following changes must be made:
Enable the WinRM firewall exception.
Make these changes [y/n]?    
WinRM has been updated for remote management.
WinRM firewall exception enabled.

WinRM can now be tested again
winrm id

If "winrm qc" fails with this error:
WSManFault Message = The system cannot find the file specified.
Error number: -2147024894 0x80070002

Then try this:

sc config WinRM start= delayed-auto 
net start WinRM
winrm create winrm/config/listener
Address=*+Transport=HTTP
netsh firewall add portopening TCP 80 "Windows Remote Management" 

Sources:
Articles for configuring the WINRM service manually:
http://adminontherun.blogspot.com/2011/07/winrm-qc-fails-with.html
Installation and Configuration for Windows Remote Management
http://msdn.microsoft.com/en-us/library/aa384372(VS.85).aspx

I checked my hosts and WinRM seemed to be running fine, but this article went into more detail:
How to troubleshoot the “Needs Attention” and “Not Responding” host status in System Center 2012 Virtual Machine Manager http://support.microsoft.com/kb/2742246

Highlights from that Knowledge Base Article are as follows:

• Use the Virtual Machine Manager Configuration Analyzer (VMMCA)
• Verify the VMM service account is a member of the local administrators group on the host
• Check for corrupted performance counters
• Check the Svchost.exe process of the Windows Remote Management service

This step is related to the error 0x8033803b found in the VMM Console and more detail can be found here:

• Increase the default values for WinRM (Shown in this code sample)

winrm set winrm/config @{MaxTimeoutms = "1800000"}
winrm set winrm/config/Service @{MaxConcurrentOperationsPerUser="400"}
net stop winrm 
Y
net start winrm 
net start scvmmagent

I also found some reference to SPN issues.

Issue: The Service Principal Names (SPNs) that VMM requires were not correctly registered when the VMM management server was set up on the specified server. This is a known issue in System Center 2012 Virtual Machine Manager.

Impact: The VMM console may not be able to connect to the VMM management server and/or deploying a Hyper-V host to a bare-metal computer may not work.

Resolution: Manually add the SPNs that are required by VMM by running the following commands as an administrator at an elevated command prompt:

setspn -S SCVMM/yourmachine domain\account
setspn -S SCVMM/yourmachine.fqdn.com domain\account 

Still having SPN Problems?

Have you done the items below? These need to be done as a domain admin.

The Service Principal Name (SPN) could not be registered in Active Directory Domain Services (AD DS) for the VMM management server.

• Use setspn.exe to create SPN for vmmserver using following command "C:\Windows\system32\setspn.exe  -S SCVMM/hostname.domain accoutname"
• Add SPN values to following registry key "Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\Setup\VmmServicePrincipalNames"
• Run "C:\Program Files\Microsoft System Center 2012\Virtual Machine Manager\setup\ConfigureSCPTool.exe -install" to configure SCP.

Next on the Agenda, see if Group Policy can help with fixing Powershell Remoting.
Enable Powershell Remoting via Group Policy
http://www.briantist.com/how-to/powershell-remoting-group-policy/

After all that it turns out that our problem was due to a certificate error with PowerShell on the host servers.  We fixed it by removing the certificate from PowerShell on those servers.

If your getting this error, "The WinRM client received an HTTP server error status (500)", then these two articles should help.

Article talking about WINRM IIS Extensions

http://social.technet.microsoft.com/Forums/exchange/en-US/5b82f131-b469-4661-9d6c-1c1c7939b73a/the-winrm-client-received-an-http-server-error-status-500?forum=exchange2010

Article about the certificate issue with PowerShell

http://measureofchaos.wordpress.com/2011/11/02/powershell-remoting-and-certificates/