Data Deduplication Enhancement in Windows Server 2012 R2

Windows Server 2012 R2 has a new feature which I can see very handy in the real world especially with VDI environments which have lots of Virtual Hard Disk files (VHD's) of similar nature.

In Server 2012 R2, Data Deduplication is now supported on VHD data stores.  This was not supported with the initial release of Server 2012.

Data Deduplication is also supported on Cluster Shared Volumes (CSVs) with file servers configured in scaled-out for high availability.

For companies that run a Microsoft-based VDI pool with multiple hosts, Data Deduplication can reduce the storage requirements of the VDI environment up to 90%.

The Dirty Little Secret about P2V Migration with System Center Virtual Machine Manager

Physical to Virtual Migration has been around for a long time ever since companies started making the transition to Virtualisation as a standard back in 2008 with the release of VMware ESX 3.x quickly followed by 4.x and vSphere.

There a many tools on the market for Physical to Virtual migration of machines with the most common being "VMware vCenter Converter: P2V Virtual Machine Converter", "Microsoft Virtual Machine Converter 3.0" and the handy little tool from sysinternals "Disk2vhd".

In the brand new shiny System Center Virtual Machine Manager (VMM) 2012 R2, this tool also supports Physical to Virtual migration of workstations as an easy transition to a virtual platform for physical servers.

However if you look at the fine print in the "prerequisites" you will see:

"Cannot have any volumes larger than 2040 GB"

https://technet.microsoft.com/en-au/library/hh427293.aspx

What the @$%@!!!

Very disappointing seeming this is the latest release of VMM and this limitation is still around... this would trip up many companies who are still looking to virtualise that legacy file server or mail server sitting around on their network!

Event Viewer Tasks

I just want to touch against a feature in Windows Server 2008 R2 - 2012 which I believe is very cool.  Windows Event Viewer has the ability to launch tasks automatically when a particular error occurs.  This is great for companies that do not have System Center (or similar) tools in the environment to perform remediation tasks when problems occur on server infrastructure.

The button in Event Viewer is called "Attach Task to This Event"

Clicking it we can see that it actually relies on the Task Scheduler service to monitor the event logs.

Select Start a Program then associate it with cscript.exe or powershell.exe to launch a script that performs remediation tasks whenever the event error reoccurs.  You can instruct your script to also notify administrators via email which is very easy with PowerShell using the Send-MailMessage cmdlet.

Common DCDIAG Error with NCSecDesc

When running a DCDiag on 2008 or 2008 R2 domain controllers, it is very common to see the following error when running a dcdiag.exe.

Starting test: NCSecDesc
   Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have Replicating Directory Changes In Filtered Set access rights for the naming context:
   DC=DomainDnsZones,DC=domain,DC=local
   Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have Replicating Directory Changes In Filtered Set access rights for the naming context:
   DC=ForestDnsZones,DC=domain,DC=local
   ......................... DC1 failed test NCSecDesc

This is caused on Active Directory domains which have not prepared Active Directory for read only domain controllers with "adprep /rodcprep".

Server 2012 / 2012 R2 domain controllers do not receive this error for NCSecDesc.

Also it is recommended you do not prepare you domain for RODC unless you intend to deploy Read Only Domain Controllers provided you have the requirement for specific branch locations from a physical security perspective.

MSExchange ADAccess EventID 4027

A customer of mine contacted me today regarding an EventID 4027 from MSExchange ADAccess they wanted resolved.  This error was being generated on all Exchange 2013 servers in their cluster.

After looking into the issue, I found that legacy Cross-Forest configuration remained in configuration partition from a previous Cross-Forest Exchange Migration.  This is located under the Configuration --> Services --> Microsoft Exchange Autodiscover.

Simply remove the additional references to the legacy forest (the ones highlighted in yellow above).

Do not remove "Microsoft Exchange Online".  This is a default entry and is used when you create a Hybrid deployment with Office 365.

Outloook 2010 Starting in Safe Mode?

Microsoft recently released a bad Windows Update (KB3114409) which caused Outlook 2010 to start loading in safe mode for multiple clients of mine.  This update has recently been recalled by Microsoft due to the number of issues it caused.

In the event your company installed it across multiple workstations, you can quietly uninstall it across all computers.

One of the easiest ways to do this is creating a Startup Script with Group Policy and create a batch script with the following command:


C:\Windows\System32\wusa.exe /uninstall /kb:3114409 /quiet /norestart


Make sure the batch file is launched via a Startup Script and not a logon script.  Logon Scripts in Group Policy require users to have local administration rights to make system wide changes (something which is not best practice).  Startup Scripts will run under the SYSTEM account with administrative rights.

Active Directory Topology Diagrammer Error

I'm running Windows 10 with Visio 2016 (both x64 builds).  I needed to draw out the topology from a customers Active Directory domain however when attempting to draw the topology using the free Microsoft tool I received the following error message:

Could not open the Visio Stencil !!!
ADTD can not continue with the drawing.
The current drawing will be canceled !

 

To resolve this issue, in Visio click File --> Options.

 

Open the "Trust Center" then click "Trust Center Settings".

Untick all Open and Save boxes as shown below:

After making this change the topology should generate without issues:

 

Active Directory Issues - Network Drives Not Mapping

A customer of mine raised an issue in regards network drives not being mapped for users.  This includes drives mapped via Group Policy and Home Drives mapped via the NT4 Home Drive option of the Active Directory user account.

When users attempt to navigate to the UNC paths manually or map a drive manually it works as expected, however mapping network drives automatically upon logon simply did not work.

Also users were unable to navigate to the domain name "\\domain.local".  However users could navigate to "\\domain.local\netlogon" and "\\domain.local\sysvol".  Navigating to the domain root resulted in this error being generated:

\\domain.local is not accessible.  You might not have permissions to use this network resource.  Contact the administrator of this server to find out if you have access permissions.

Logon Failure: The target account name is incorrect.

This issue with not being able to navigate to the domain root UNC share occurred on all member workstations and servers throughout the organisation.  Domain Controllers were not effected by the issue.

The following three event logs were also found throughout the SYSTEM event log on all client workstations throughout the companies domain.

Log Name:      System

Source:        NETLOGON

Date:          26/10/2015 7:56:15 AM

Event ID:      5719

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      COMPUTER.domain.local

Description:

This computer was not able to set up a secure session with a domain controller in domain DOMAIN due to the following: 

There are currently no logon servers available to service the logon request. 

This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.  

Log Name:      System

Source:        Microsoft-Windows-Security-Kerberos

Date:          23/10/2015 4:02:46 PM

Event ID:      4

Task Category: None

Level:         Error

Keywords:      Classic

User:          N/A

Computer:      COMPUTER.domain.local

Description:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server candc1$. The target name used was cifs/domain.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (DOMAIN.LOCAL) is different from the client domain (DOMAIN.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Log Name:      System

Source:        Microsoft-Windows-GroupPolicy

Date:          28/10/2015 6:18:58 PM

Event ID:      1006

Task Category: None

Level:         Error

Keywords:      

User:          DOMAIN\UserAccount

Computer:      COMPUTER.domain.local

Description:

The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description.

Some computers were also receiving:

The processing of Group Policy failed. Windows attempted to read the file \\domain.local\sysvol\domain.local\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini from a domain controller and was not successful. Group Policy settings may not be applied until this event is resolved. This issue may be transient and could be caused by one or more of the following: 

a) Name Resolution/Network Connectivity to the current domain controller. 

b) File Replication Service Latency (a file created on another domain controller has not replicated to the current domain controller). 

c) The Distributed File System (DFS) client has been disabled.

This GUID referenced the Default Domain Policy (the first policy in the domain).  There was nothing wrong with the Default Domain Policy on the customers network, policy was simply not applying to domain members due an issue in Active Directory.

One issue which I focused on was the Kerberos error "KRB_AP_ERR_MODIFIED".  I have seen this issue before when the same SPN was registered on at least two accounts. For example, a SPN was registered on two accounts: A and B. What happens is that KDC will generate a service ticket that may be encrypted with password of account A. Then, when the client sends that ticket to the service during authentication, the service may try to decrypt this using account B.

I searched the customers domain for duplicated SPN's using "setspn.exe -X" and found some however they were not related to the error received.  Also I have never seen the "KRB_AP_ERR_MODIFIED" error generated on EVERY domain member workstation/server in an Active Directory environment.

Looking further into the SPN's we decided to dump the entire domain with every object/attribute to a text file using:

ldifde -f out.txt -d dc=domain,dc=local

Generally SPN against a user account reference a member server on the network running a particular service account such as SQL  However as this issue was affecting the entire domain and KRB_AP_ERR_MODIFIED refers to duplicated SPN records, we looked to see if there were any SPNs set at domain level on an account by searching our output from ldifde for "host/domain.local".

We found an SPN set to the root of the domain from the search results.  Important content from output below is blurred to protect the privacy of the customer.

We went and removed the incorrectly setup SPN record from the problematic service account svc_adfs using Active Directory Users and Computers with Advanced Features turned on then forced replication with "repadmin /syncall /APeD".

After removing the incorrectly configured SPN, we purged the kerberos tickets off a workstation then attempted to start explorer at the root of the domain "\\domain.local".  We were able to navigate to this share successfully.

The issues with network drives not mapping on logon were also resolved.

The SVC_ADFS account was created as part of an AD FS deployment for federation with applications and Microsoft Cloud Services.  AD FS backend roll was installed on two corporate domain controllers and two proxy servers were deployed in a DMZ setup to process the authentication requests from external services.  This is the Microsoft Best Practice for corporate organisations under 1000 seats as it reduces the amount of servers required and provides high redundancy by leveraging NLB on both the backend and frontend AD FS servers as per:

https://msdn.microsoft.com/en-us/library/azure/dn151324.aspx

I went over the engineers build documentation who was in charge of implementing AD FS and could not see how the SPN was set, he did not manually set it.

Hope this post helps someone who experiences this same issue.

Enable Firewall Logging on Windows

Are your packets being dropped by Windows Firewall?  Want an insight into what is going on?  Simply open local group policy on a workstation / server (gpedit.msc) or configure a GPO in Group Policy Management Console (GPMC).  Under Windows Firewall with Advanced Security, go to the general properties.  Select the profile --> Logging and enable Logging on the set profile.  The log file by default goes to:

%windir%\system32\logfiles\firewall\pfirewall.log

Very handy for troubleshooting.

Exchange 2013 POP3 Proxy Inactive

A customer complained that POP3 was no longer working.  After looking into this, it turned out that PopProxy was Inactive on the Exchange 2013 server.  As to why it was inactive is unknown.

To start the PopProxy was challanging, generally you change ServerComponentState using the maintenance requester for most components.  However running the following command did nothing:

Set-ServerComponentState -State Active -Requester Maintenance -Component PopProxy -Identity AB-EXCH-01

To start the PopProxy component, I needed to use the Exchange 2013 Health API as a requester.

Set-ServerComponentState -State Active -Requester HealthAPI -Component PopProxy -Identity AB-EXCH-01

As shown below:

Exchange 2013 421 4.3.2 Service not active

A customer of mine upgraded an Exchange 2013 cluster node from Exchange 2013 CU7 to Exchange 2010 CU10.  After the upgrade, emails failed to come in on the cluster node with the following SMTP error being generated "421 4.3.2 Service not active".

This can be reproduced by simply telneting the faulty Exchange 2013 server.  After entering MAIL FROM: into the SMTP syntax, the error occurs and is shown numerous times throughout the receive connector protocol logs on the frontend transport stack.

After further investigation I found out that majority of the Server Components for the faulty Exchange Server were in an inactive state.

To bring the server back to an active state, ServerWideOffline was set to Active which resumes all services using a requester of Maintenance.  This was done with the following command:

Set-ServerComponentState -State Active -Requester Maintenance -Identity AB-EXCH-02 -Component ServerWideOffline

Note: ForwardSyncDaemon and ProvisioningRps is Inactive by default.

After running this command all Exchange 2013 components were back to an active state apart from components disabled by default.

Exchange 2013 Cluster Issues 0x80071736

A customer of mine went upgraded one node of a two node DAG from Exchange 2013 CU7 to Exchange 2013 CU10.  After installing the update on the first cluster node, they contacted us complaining complaining that the DAG was no longer available online in Failover Cluster Manager.

The following error was presented when attempting to bring the cluster online:

Failed to bring the resource 'Cluster Name' online.
Error code 0x80071736
The resource failed to come online due to the failure of one or more provider resources.

The resource it was complaining about was the Cluster IP address was unavailable (10.10.0.245).

In addition to this error, the following errors were logged in event viewer on a regular basis.

Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          26/10/2015 8:45:01 AM
Event ID:      10028
Task Category: None
Level:         Error
Keywords:      Classic
User:          DOMAIN\administrator
Computer:      Exchange1.domain.local

Description:
DCOM was unable to communicate with the computer DAG1.domain.local using any of the configured protocols; requested by PID 6bc4 (C:\Windows\system32\ServerManager.exe).

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          26/10/2015 9:58:05 AM
Event ID:      1223
Task Category: IP Address Resource
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      AB-EXCH-01.domain.local
Description:
Cluster IP address resource 'Cluster IP Address' cannot be brought online because the cluster network 'Cluster Network 1' is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.

These cluster nodes both had two interfaces:

• Mapi Interface
• Replication Interface

After patching one of the Exchange 2013 servers the MapiDagNetwork got disabled on the with IgnoreNetwork set to true on the DagGroupNetwork.  As this DAGNetwork contained the DAG cluster IP address 10.10.0.245 (within the 10.10.0.0/20 subnet), it forced the cluster offline.  To re-enable the cluster we simply needed to set IgnoreNetwork to false.

After setting IgnoreNetwork to false, we were able to manually start the DAG in FailoverManager by right clicking and clicking Bring Online.

Removing Large Amounts of Spam from Exchange Server Queue Database

A customer running Exchange 2010 experienced a large number of spam emails in the submission queue (over 80,000).

All the Spam Emails started with "ID#ALERT#", as a result we ran the following command to clean up any emails with "ID#ALERT#" in the subject.

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"} | Remove-Message

Due to the significant amount of spam, the command failed to remove the messages and just hung for hours.  However we could remove messages individually.

We could however list the emails with the following command:

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"}

This gave us an idea to write a foreach command to remove each email individually from a CSV file.  First we needed a detailed list with the identity of each spam email so we ran the following command:

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"} | select Identity > C:\output.txt

We then formatted the CSV file to ensure we specified a name for the Identity column.  As you see we added "Identity" to the top of the file.

We then ran the Remove-Message command multiple times for each Identity in the CSV file using the following command:

Import-Csv "C:\output.txt" | ForEach-Object {Remove-Message -Identity $_.Identity -confirm:$false}

This ran the Remove-Message command over 80,000 times for each message in the queue and was able to clean it up successfully.

Websense Appliance Services Not Starting after Hard Shutdown

When a Websense v5000 or v10000 appliance is forcefully shutdown due to power loss or hard system failure, upon restart the Filtering service, Policy service, User service and Usage monitor can fail to start in the Websense Appliance Manager portal.  This is an issue I have seen more then once and as a result decided to do a write-up.

This issue occurs due to a number of temporary files which are not cleaned up (a process that occurs during a graceful shutdown).  To remove these files manually, we must connect to the appliance using an SSH shell session.

To connect to a shell session you need to login to Websense Appliance Manager first then under Administration --> Toolbox, click Technical Support Tools and find the passcode under "Websense Remote Access".  This is the password used for SSH which is randomly generated by the appliance.

 

Next login to the IP address of the Websense Appliance (the same IP you used for the Appliance Manager web interface).

 

The username is "websense-ts" and the password is the one obtained above.

 

 

Navigate to /opt/Websense/bin and remove all temporary p12 files which were not deleted due to an incorrect shutdown.

 

rm -f *.p12

 

 

 Also remove the journal.dat file under the /opt/Websense/bin using the following command:

 

rm -f 'journal.dat

After the temporary files have been removed, restart all websense services.  This can be done from the Websense Appliance Manager website or from the shell by running the following command:

./WebsenseAdmin restart

Now all the Websense services on the appliance have returned to a running healthy state.

 

VawTrak Trojan

Today I was diagnosing why a clients Internet was running so slow.  After tracing the traffic I found it was one Windows 7 PC which was infected with a virus.  The following processes were running on the machine all communicating with various Internet IP addresses.

• conhost.exe
• cmd.exe
• ctfmon.exe
• dllhost.exe
• msiexec.exe
• notepad.exe
• presentationhost.exe

Note: Use Windows Resource Monitor and navigate to the Network tab to find out which processes are communicating with Internet resources.

When killing one of these processes, they would simply respawn.  The computer was also running very slow and sluggish with web browsers and windows explorer constantly hanging and freezing.

 

These symptoms are related to Trojan.VawTrak which the computer was infected with.  Trojan.VawTrak copies it self into C:\ProgramData and spawns these processes with its malicious code.

 

Trojan.VawTrak can be cleaned up with Malware Bytes or manually.

 

Trojan.VawTrak is a virus you definitely want to get rid of as it is designed to steal online banking information.  Some of the common tasks it performs are:

• Disables antivirus protection.
• Communicates with remote C&C servers – executes commands from a remote server, sends stolen information, downloads new versions of itself and web-injection frameworks.
• Hooks standard API functions, injects itself into new processes.
• Steals passwords, digital certificates, browser history, and cookies.
• Logs keystrokes.
• Takes screenshots of desktop or particular windows with highlighted mouse clicks.
• Captures user actions on desktop in an AVI video.
• Opens a VNC11 (Virtual Network Computing) channel for a remote control of the infected machine.
• Creates a SOCKS12 proxy server for communication through the victim's computer.
• Changes or deletes browser settings (e.g. disable Firefox SPDY13) and history. Vawtrak supports three major browsers to operate in – Internet Explorer, Firefox, and Chrome. It also supports password stealing from the other browsers.
• Modifies browser communication with a web server.
• Stores internal settings into encrypted registry keys.

Due to the severity of this Trojan and the rate it is spreading, AVG has done a detailed writeup which is available here:

http://now.avg.com/wp-content/uploads/2015/03/avg_technologies_vawtrak_banking_trojan_report.pdf

Quick access in Windows 10 with Direct Access

Quick access is a feature in Windows 10 which lets you quickly view recently opened documents and folders.  This is handy for users to gain access to files they access on a regular basis.

 

On the Avantgarde Technologies network all our employees use Direct Access to provide seamless connectivity back to resources in the office.  We found on links with poor bandwidth and high latency, Quick access causes performance issues and causes Windows Explorer to hang up to 10 seconds every time a user tries to save a file or open a new Explorer window.

 

To ensure employees which are outside the office are not affected with performance issues, we disabled this technology on all our Windows 10 workstations.  The two registry keys you want to modify are under this location:

 

HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer

 

ShowFrequent

ShowRecent

 

Both are REG_DWORD values.

 

Simply modify both these registry keys to 0.

 

 

 Deploy these registry keys to all your users using Group Policy Preferences.

After making this change, Windows Explorer will be a lot more snappy for remote users connecting via Direct Access or another VPN technology.

Remove all Printers Deployed from a specific Print Server with Powershell

I had a customer which had deployed printers from a legacy print server utilising scripts.  They have recently built a new 2012 R2 print server where they deployed the printers utilising Print Management Console and Group Policy.

All printers were redeployed from the new print server.

The customer however had a number of printers setup on workstations still pointing to the legacy print server.  As such they wanted to remove all printers deployed from the hostname of the legacy print server.

The following PowerShell script achieves this and can be easily deployed with Group Policy.  Simply replace "PRINTSERVER" with the name of your print server and then deploy the PowerShell script.

$PrintServer = "\\PRINTSERVER"
$Printers = Get-WmiObject -Class Win32_Printer
ForEach ($Printer in $Printers) {
If ($Printer.SystemName -like "$PrintServer") {
(New-Object -ComObject WScript.Network).RemovePrinterConnection($($Printer.Name))
}
}

 

Regaining Access to an SQL Instance

After a previous employee left an organisation, no one had access to an SQL Instance and the SA Password was unknown.  In this article I will show you how to regain access to an SQL Instance.

This process was performed in SQL Server 2012 Enterprise Edition by booting the SQL server into Single User Mode.

First stop the SQL service for which we need to recover the password.

Then start the service by entering a Start parameter of "-m" in the services window in Control Panel.

Next connect to the instance with SQLCMD.exe -S "SERVERNAME\Instance".

To grant sysadmin to a user or entire group such as Domain Admins, run the following command:

EXEC sp_addsrvrolemember 'DOMAIN\Domain Admins', 'sysadmin'; 

GO

Next run "quit" to close SQLCMD.

After this, remove the -m from the SQL Instance and start the instance normally.  Now anyone in the Domain Admins group will have sysadmin rights to the instance.  Login with a Domain Admin account and reset the SA password (provided your Instance is setup for Mixed authentication).

Enter the new password and click OK.

Howto Decrypting an Active Directory Password

In this post I will show a tool which makes decrypting Active Directory passwords easy.  It is important to note decrypting highly secure passwords takes a long time and is not always achievable within a reasonable period of time based on the complexity of the password however I have had success recently using this product.

Find a Windows Computer where the user has logged into recently and has their password cached.  Next obtain the Network Password Recovery Wizard (NPRW) tool from:

http://www.passcape.com/network_password_recovery

After the tool simply use the GUI for performing the password encryption.  I found this video very helpful.

http://www.passcape.com/download/swf/nprw-domain-cached-passwords.swf

The SMTP availability of the Receive connector Default was low

In Exchange 2013 CU9 you may start seeing the following error message more often.

Log Name:      Application
Source:        MSExchangeTransport
Date:          10/09/2015 12:15:21 PM
Event ID:      1040
Task Category: SmtpReceive
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CCEX2.toph.local
Description:
The SMTP availability of the Receive connector Default was low (98 percent) in the last 15 minutes.

This is due to a new Transport Agent introduced by CU9 called the System Probe Drop Smtp Agent - undocumented, but apparently responsible for dropping probe emails - which does not "drop the email" in a traditional sense, but removes all email recipients instead. Alas, this method of stopping an email from getting delivered has an adverse effect, namely the email still gets passed down the pipeline, but now with the recipient information destroyed.

This results in the email delivery failing only for the System Mailbox "probe drop" emails as shown in the following screenshot.

As a result, Event ID 1040 for the MSExchangeTransport service is logged saying that a small percentage of emails are failing.  These are generally the probe messages from Exchange 2013 CU9.

I do recommend you run the above Get-MessageTrackingLog command with an EventId of FAIL to determine that there is no other emails which have failed. 

Enable Split Tunneling on Windows 10 VPN Connections

In previous versions of Windows Server, Split Tunneling was enabled by removing the default gateway from the IPv4 settings under the properties of a Windows PPTP, L2TP or SSTP VPN connection.  This was done on the Networking tab and selecting Properties on the Internet Protocol 4 (TCP/IPv4) settings.

 

In Windows 10 if you click properties on the Internet Protocol Version 4 (TCP/IPv4) settings, nothing happens the button has no code behind it.

 

On Windows 10, to enable Split Tunneling this must now be done with PowerShell. 

 

Set-VpnConnection "VPN Connection Name" -SplitTunneling $true

 

 

 After enabling the VPNConnection for Split Tunneling this achieves the same affect as the "Use as Default Gateway" button from the IPv4 properties dialog box.

MAPI NSPI Issues with Exchange 2013

A customer running two Exchange 2013 CU9 multi-role servers in a cluster with an F5 load balancer contacted me this morning reporting issues where they could not create new Outlook Profiles for users.  Users with existing Outlook profiles were not affected by the issue.

Users primarily connect with MAPI over HTTP which is enabled on the Exchange Organization configuration.

When a user attempted to create a new profile, they would receive the following error:

The connection to Microsoft Exchange is unavailable.  Outlook must be online or connected to complete this action.

When running the Microsoft Exchange Remote Connectivity Analyzer against the Exchange organisation, the following errors were captured:

Testing MAPI over HTTP connectivity to server webmail.company.com
MAPI over HTTP connectivity failed.
  Additional Details
  HTTP Response Headers:
request-id: 1f79419f-0c42-4a38-bf07-99b1c6882928
Set-Cookie: ClientId=MUZOXYKGXN9GLA; expires=Sat, 13-Aug-2016 02:50:02 GMT; path=/; HttpOnly
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
X-FEServer: SERVER2
Date: Fri, 14 Aug 2015 02:50:02 GMT
Content-Length: 0
Elapsed Time: 7764 ms. 

Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
  Additional Details
  Elapsed Time: 5784 ms. 

  Test Steps
  Testing the address book "Check Name" operation for user TestUser@company.com against server webmail.company.com.
  An error occurred while attempting to resolve the name.
  Additional Details
  A protocol layer error occured. HttpStatusCode: 500
FailureLID: 47372
FailureInfo: 

###### REQUEST [2015-08-14T02:50:03.1064218Z] ######

POST /mapi/nspi/?mailboxId=e110343c-c351-4dac-b066-3b552417a51b@company.com HTTP/1.1
Content-Type: application/octet-stream 
User-Agent: MapiHttpClient 
X-RequestId: 57959cb1-fd5d-4ae1-ac55-4494f0a61748:1 
X-ClientInfo: 6016fb10-ca94-4a67-847c-549dd99bacb8:1 
X-ClientApplication: MapiHttpClient/15.1.243.0 
X-RequestType: Bind 
Authorization: Negotiate [truncated] 
Host: webmail.company.com 
Cookie: ClientId=SGMTABEYKDADLXOTCG 
Content-Length: 45 

--- REQUEST BODY [+0:05.159] ---
..[BODY SIZE: 45]

--- REQUEST SENT [+0:05.159] ---

###### RESPONSE [+0:05.781] ######

HTTP/1.1 500 Internal Server Error
request-id: 9653be79-d895-4cf3-8169-98ceeb218197 
X-CalculatedBETarget: Server1.company.local 
X-DiagInfo: Server1 
X-BEServer: Server1 
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError; 
Persistent-Auth: true 
X-FEServer: Server1 
Transfer-Encoding: chunked 
Cache-Control: private 
Content-Type: text/html; charset=utf-8 
Date: Fri, 14 Aug 2015 02:50:08 GMT 
Set-Cookie: X-BackEndCookie=e110343c-c351-4dac-b066-3b552417a51b=u56Lnp2ejJqBz5nPyM2cxpnSncebmdLLyZue0sfJnJ7SnM2Znc7Iys7LnJnMgYHNz87K0s/G0s7Mq8/NxcrPxc/H; expires=Sun, 13-Sep-2015 02:50:08 GMT; path=/mapi; secure; HttpOnly 
Server: Microsoft-IIS/8.5 
X-AspNet-Version: 4.0.30319 
X-Powered-By: ASP.NET 

Also on only on Server1, the following error was experienced in the event log.

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          14/08/2015 5:54:03 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Server1
Description:
Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 14/08/2015 5:54:03 PM 
Event time (UTC): 14/08/2015 9:54:03 AM 
Event ID: 6e8a15487387414b9279078f9f6aba51 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 0 

Application information: 
    Application domain: /LM/W3SVC/2/ROOT/mapi/nspi-3615-130840196323691514 
    Trust level: Full 
    Application Virtual Path: /mapi/nspi 
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\nspi\ 
    Machine name: Server1

Process information: 
    Process ID: 13528 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\SYSTEM 

Exception information: 
    Exception type: HttpException 
    Exception message: RpcObjectSetType
   at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

RpcObjectSetType
   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.ProcessAccess.ProcessAccessRpcServer.RegisterInterface(Void* ifSpec, ValueType mgrTypeGuid, _GUID* pMgrTypeUuid, Void* pMgrEpv, UInt32 flags, UInt32 maxCalls)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)

Request information: 
    Request URL: https://localhost:444/mapi/nspi 
    Request path: /mapi/nspi 
    User host address: ::1 
    User:  
    Is authenticated: False 
    Authentication Type:  
    Thread account name: NT AUTHORITY\SYSTEM 

Thread information: 
    Thread ID: 81 
    Thread account name: NT AUTHORITY\SYSTEM 
    Is impersonating: False 
    Stack trace:    at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

After further investigation, we discovered that only mailboxes which resided on an Active database on Server1 experienced the issue creating Outlook profiles, the same server which experienced EventID 1309.  After failing mailbox databases over to Server2, users were able to login to Outlook without issues.

After this we identified the issue was only present with Server1.

On Server1 we tested the NSPI website by navigating to the Exchange backend in IIS Manager and selecting Browse:444 (https).

The browse failed on Server1 with an "unhandled exception error".  This was aligned with the HTTP 500 exception shown in the Event Logs on Server1 and the Exchange Remote Connectivity Analyzer.

RpcObjectSetType 

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Exchange.Rpc.RpcException: RpcObjectSetType

Source Error: 

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 

On Server2 when repeat the same process.

On Server2 it prompted as authentication as normal.

This shows the MAPI/NSPI results as expected.

Resolution:

We rebooted Exchange Server 1 and after a reboot, it started responding to MAPI/NSPI requests as expected.  We believe it was a side affect after the CU9 upgrade, however will monitor the situation and update this blog thread in the event the issue reoccurs.