EXCHANGE 2010 PERFORMANCE SLOW WITH CPU UTILIZATION 100% [SOLVED]

I was facing this issue for the last two days ago, one of my customer complained one of their exchange mailbox server performance slow and the CPU utilization is 100%. Our customer is using VMware and all the VMs server running on top of the VMware.

When checking on the server that has performance issue, I found out all the services running as usual, the fail-over cluster up and running, but the connectivity to this server seems very slow.  When open the EMC we got this error below;

Processing data for a remote command failed with the following error message: The WinRM client cannot complete the operation within the time specified. Check if the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled. For more information, see the about_Remote_Troubleshooting Help topic.

After deeply checking with the VMware team, we found out that both exchange mailbox servers require to perform Disk Consolidation. It is because their backup for VMware has a problem and too many "VM snapshot" created for these two servers and its impact to server performance. After the VMware team finished performing the disk consolidation, the issue has resolved and the mailbox server back to normal.

INSTALL ACTIVE DIRECTORY MANAGEMENT TOOLS ON CLIENT COMPUTER

First we need to download the Remote Server Administration Tools (RSAT) for from Microsoft's web site:
http://www.microsoft.com/en-us/download/details.aspx?id=7887

After downloading and installing the tool on your computer, use the Turn Windows features on or off function to enable AD management tools.

1. Go to the Control Panel, click on Programs & Features.
2. On Programs and Features, select Turn Windows features on or off.
3. Under Remote Server Administration Tools > Role Administration Tools, select AD DS and AD LDS Tools. Click Ok. Done.

GENERATE EXCHANGE ENVIRONMENT REPORTS

Great Power Shell script to generate exchange environment reports.  Refer to this URL

http://www.stevieg.org/2011/06/exchange-environment-report/

GET SERIAL NUMBER COMPUTER USING COMMAND PROMPT

This is how to get the serial number computer using command prompt.  The command as below;

wmic bios get serialnumber

HOW TO REMOVE SAVED LOGS IN THE EVENT VIEWER

1. Open command prompt “CMD” with “Run as administrator”.

2. Run the following command as below and press enter to proceed.

del /s /q %programdata%\microsoft\eventv~1\extern~1

3. You will see the saved logs deleted.

Deleted file - C:\ProgramData\microsoft\eventv~1\extern~1\Log_0.xml

Deleted file - C:\ProgramData\microsoft\eventv~1\extern~1\Log_1.xml

Deleted file - C:\ProgramData\microsoft\eventv~1\extern~1\Log_10.xml

…….

4. Done.

More details may refer to http://support.microsoft.com/kb/2489761

TOOLS FOR THE NETWORK TROUBLESHOOTING

There is a some useful tools for the network troubleshooting purpose. 

1.       Network Monitor 3.4.

http://www.microsoft.com/en-us/download/details.aspx?id=4865 

2.       Wireshark

https://www.wireshark.org/download.html 

3.       PortQuery/PortqueryUI

http://www.microsoft.com/en-us/download/details.aspx?id=24009 

4.       RpcDump : this tool is contained in Windows resource pack and it is useful to dump all the RPC interface information on a machine.

http://www.microsoft.com/en-us/download/details.aspx?id=17657 

5.       Fiddler : this tool is very useful for HTTP related connection issue)

http://www.telerik.com/download/fiddler 

6.       Microsoft Remote Connectivity Analyzer

https://www.testexchangeconnectivity.com  

7.       Ping/Telnet/tracert/Netstat/nslookup: All these are Windows OS built-in tools. 

8.       TCPView : TCPView is a Windows program that will show you detailed listings of all TCP and UDP endpoints on your system

https://technet.microsoft.com/en-us/sysinternals/bb897437 

9.       Whois : Whois performs query against the registration record for the domain name or IP address that you specify

https://technet.microsoft.com/en-us/sysinternals/bb897435

Thanks to Premier Support for this info.

FIND YOUR CA (CERTIFICATE AUTHORITY) SERVER IN YOUR ACTIVE DIRECTORY ENVIRONMENT

This utility tool able to identify CA server in AD environment.

certutil –config - -ping

Thanks to David Rowe for this article.

http://blogs.catapultsystems.com/drowe/archive/2012/11/15/how-to-find-a-certificate-authority-in-your-active-directory-environment.aspx

THE WINRM CLIENT RECEIVED AN HTTP STATUS CODE OF 502 FROM THE REMOTE WS-MANAGEMENT SERVICE.

Found error when open the Exchange Management Shell on exchange 2013.

Connecting to remote server failed with the following error message : The WinRM client received an HTTP status code of 502 from the remote WS-Management service.

Solution: 
Open command prompt and run the command below;

netsh winhttp reset proxy

https://technet.microsoft.com/en-us/library/bb430772(v=exchg.141).aspx

THE REMOTE SESSION WAS DISCONNECTED BECAUSE THERE ARE NO TERMINAL SERVER LICENSE SERVERS AVAILABLE

I found this issue when try to remote desktop (RDP) to the production server, Windows Server 2012.   My workaround is using mstsc /admin can RDP without any issue.

Another method, you may need to touch the Registry on the server. Refer to this article below:

http://www.stopdoingitwrong.co.uk/server-2012-rds-there-are-no-remote-desktop-license-servers/

PERFORMANCE MONITOR UNABLE TO ADD THESE COUNTERS

When open the Performance Monitor (PerfMon) in Windows, you may found this alert "Unable to add these counters" as below;

 In my situation, I just perform few steps and it's solved my problem.

Steps:
1. Open the command prompt (CMD) with run as Administrator.
2. Go to the path C:\Windows\System32\ 
3. Type this command lodctr /R
4. Done.

Ref. URL: http://support.microsoft.com/kb/300956