Microsoft recently released an update which causes DFSRS.exe CPU usage to hit 100%.
https://support.microsoft.com/en-us/kb/3156418
I wrote about this problem on the following blog post:
http://clintboessen.blogspot.com.au/2016/07/dfsr-service-100-cpu-usage.html
Since installing this update and removing this update, one of our spoke servers failed to replicate back to the primary hub server. All data from the spoke server replicated correctly to the hub, however data from the hub was not propagating down to the spoke.
The backlog count continued to increase.
DFSR Hotfix https://support.microsoft.com/en-us/kb/2996883 was not installed on the affected hub server.
We knew there was no issue with our hub server as this was replicating successfully to 16 other DFSR spoke servers.
On the affected spoke server, I worked with the Directory Services team from Microsoft who made the following changes to the Network Interface.
C:\Windows\system32>netsh int tcp set global chimney=disabled
Ok.
C:\Windows\system32>netsh int tcp set global rss=disabled
Ok.
C:\Windows\system32>netsh int ip set global taskoffload=disabled
Ok.
C:\Windows\system32>netsh int tcp set global autotuninglevel=disabled
Ok.
C:\Windows\system32>netsh int tcp set global ecncapability=disabled
Ok.
C:\Windows\system32>netsh int tcp set global timestamps=disabled
Ok.
C:\Windows\system32>netsh advf set allp state off
Ok.
Microsoft then disabled the following Offload features on the HP Network Interface card on the spoke server:
https://support.microsoft.com/en-us/kb/3156418
I wrote about this problem on the following blog post:
http://clintboessen.blogspot.com.au/2016/07/dfsr-service-100-cpu-usage.html
Since installing this update and removing this update, one of our spoke servers failed to replicate back to the primary hub server. All data from the spoke server replicated correctly to the hub, however data from the hub was not propagating down to the spoke.
The backlog count continued to increase.
DFSR Hotfix https://support.microsoft.com/en-us/kb/2996883 was not installed on the affected hub server.
We knew there was no issue with our hub server as this was replicating successfully to 16 other DFSR spoke servers.
On the affected spoke server, I worked with the Directory Services team from Microsoft who made the following changes to the Network Interface.
C:\Windows\system32>netsh int tcp set global chimney=disabled
Ok.
C:\Windows\system32>netsh int tcp set global rss=disabled
Ok.
C:\Windows\system32>netsh int ip set global taskoffload=disabled
Ok.
C:\Windows\system32>netsh int tcp set global autotuninglevel=disabled
Ok.
C:\Windows\system32>netsh int tcp set global ecncapability=disabled
Ok.
C:\Windows\system32>netsh int tcp set global timestamps=disabled
Ok.
C:\Windows\system32>netsh advf set allp state off
Ok.
Microsoft then disabled the following Offload features on the HP Network Interface card on the spoke server:
- IPv4 Large Send Offload
- Large Send Offload V2 (IPv4)
- Large Send Offload V2 (IPv6)
- TCP Connection Offload (IPv4)
- TCP Connection Offload (IPv6)
- TCP/UDP Checksum Offload (IPv4)
- TCP/UDP Checksum Offload (IPv6)
After disabling these components of the network interface card and restarting the DFS Replication service, the backlog count from the hub server to the spoke server slowly started decreasing.
TCP offload engine is a function used in network interface cards (NIC) to offload processing of the entire TCP/IP stack to the network controller. By moving some or all of the processing to dedicated hardware, a TCP offload engine frees the system's main CPU for other tasks
Excellent article. Did anyone from MS explain why we still need to carry out the same TCP offload engine mods (basically turn it all off) from the era of the Server 2003 scalable networking fiasco over 10 years later? I personally have never had to change a Server 2012 R2 network stack in this manner, but have seen this problem in 2003 and 2008.
ReplyDeleteI'm curious what program you're using to show you your backlog. I've only done so through PowerShell.
ReplyDeleteWOW!! This is the most wonderful thing i have ever experienced. I visited a forum here on the internet on the 17 APRIL 2016, and i saw a marvelous testimony of Tracie Aldana from United States on the forum about the good works DR OSEMU. I never believed it, because have never heard anything about such miracle before. No body would have been able to convince me about it not until DR OSEMU did a marvelous work for me that restored my marriage of 4 years by getting back my divorced wife just as i read on the internet. I was truly shocked when my wife knelt down pleading for forgiveness to accept her back. I am really short of words to use to show my appreciation to DR OSEMU. For his a God sent to me and my entire family for divine restoration of marriage. Here is his email if you need any kind of help. ( Doctorokpamenspelltemple@hotmail.com ), Website: doctorokpamenpowerfulspelltemple.webs.com, call and whats App him on +2348135254384.
ReplyDelete