Troubleshooting Account Lockouts in Active Directory

In this post we will look into troubleshooting Account Lockouts in Active Directory. From my experience identifying the source of an Account Lockout can often be easy, or extremely difficult.

When an authentication attempt hits a domain controller that is incorrect, a second authentication attempt will always hit the Primary Domain Controller (PDC). This is due to Active Directory replication intervals, if you reset a users password, it may take a few hours for the password change to propagate across the network based on how you have configured your Inter-Site Transport links in AD Sites and Services. The Primary Domain Controller (PDC) always has the latest list of passwords (one of the many things the PDC emulator role performs).

The easiest way to troubleshoot Account Lockouts is simply login to the Primary Domain Controller and review the security log, as this will have a list of all account lockouts that have occured across all domain controllers. The EventID that represents an account lockout is EventID 4740.

The screenshot below shows a typical Account Lockout event on the PDC.  It will display the account name that was locked out, and the computer in which the account was locked out on.

However this is where it can get more complicated.  If the lockout came from a system not in Active Directory, the "Caller Computer Name" value will always be blank.  This can include numerous scenarios such as:

• Mobile Devices (Android / IOS) that are authenticating against an Exchange Server through Active Sync.
• Proxy Servers
• Java Applications
• UNIX/Linux Systems
• Other non-domain joined computer objects.

For these events we cannot simply only look at the PDC Emulator role, we need to look at where the "Bad Pwd Count" is incrementing the the "Last Bad Pwd" timestamp across all domain controllers in the Active Directory Domain.  Microsoft has a tool called lockoutstatus.exe that allows us to easily the domain controller which the lockout occurred so we can review the security logs on the domain controller in question.

Download this tool from here:

https://www.microsoft.com/en-au/download/details.aspx?id=15201

If you want to see it utilised, refer to this post:

https://clintboessen.blogspot.com/2018/08/troubleshooting-account-lockouts.html

The EventID Security logs you want to filter are as follows:

• Event ID 4625 - An account failed to log on
• Event ID 4776 - The computer attempted to validate the credentials for an account.
• Event ID 4771 - Kerberos pre-authentication failed

Note: The Event ID's for Windows Server 2000/2003 are different but I assume your not running this anymore!

These Events ID's will reveal the source IP address the authentication attempt failed from.

For many companies still running on-premises Exchange, Mobile Devices on Active Sync are a common cause for account lockouts.  If you have confirmed using the EventID's above that the account lockout is coming from your Exchange Server, you can utilise the IIS Logs to identify the device and external IP address that caused the lockout.  You need to check the IIS Logs on the Exchange Server for a HTTP 401 "Unauthorized" error for the user in question.  IIS Logs can easily be imported into Excel for easy formatting/review.  Also check out the following blog post that has a Logparser.exe query that lets you quickly search IIS Logs for the account lockout.  

http://messagingadmins.blogspot.com/2014/08/troubleshoot-exchange-cas-server-is-lockout-source.html

In addition to the HTTP 401 "Unauthorized" error, the Exchange IIS Logs also can provide additional information in the Sc-Win32-status code:

• 1326 - The user name or password is incorrect.
• 1330 - The password for this account has expired.
• 1331 - This user can't sign in because this account is currently disabled.
• 1909 - The referenced account is currently locked out and may not be logged on to.

Note: I have seen an account lockout event at a customer site where 4771 Event ID's were logged pointing at the Exchange Server, however I could not see the user account in the IIS Logs with the 401. 

More Reading / References:

https://www.business.com/articles/powershell-active-directory-lockouts/

https://social.technet.microsoft.com/Forums/en-US/735602f0-3ddc-4bb4-b6ba-dffcb7605ca1/account-lockout-on-windows-2008-r2-and-windows-7?forum=winserverDS

https://www.windowstricks.in/2016/06/account-lockout-caller-computer-name-blank-cisco-workstation-domain-controller.html

https://kb.netwrix.com/2744

I also recommend reading the following article "The Low Down on Password Policies".

https://clintboessen.blogspot.com/2009/12/low-down-on-password-policies.html

Hopefully this post has been informative for you in troubleshooting Account Lockouts.

Exchange Not Functional after installing Patches

We had an Exchange 2016 server running Exchange 2016 Cumulative Update 15 where as part of Windows Updates it rendered the server not functional.  The update that we installed was the Security Update for Exchange 2016 - KB4536987.

After the update, the following symptoms were experienced.

All Outlook clients received the following message when attempting to open Outlook.

Cannot start Microsoft Outlook. Cannot open the Outlook window. The set of folders cannot be opened. The attempt to log on to Microsoft Exchange has failed.

Users could log onto Outlook Web App (OWA) however could not send email.  All messages sent would go straight to Draft folder with the message:

Something went wrong and we haven't been able to send your message yet.

The Exchange Remote Connectivity Analyzer tool passed all tests under "Outlook Connectivity".

Attempting to connect to the SMTP service on localhost results in the SMTP Error Code "421 4.3.2 Service not active".

This lead me to a previous blog post:

http://clintboessen.blogspot.com/2015/10/exchange-2013-421-432-service-not-active.html

The Server Component States were inactive on all Exchange Services after the patch.

In my previous blog post, I used the following command to activate the component states.

Set-ServerComponentState -State Active -Requester Maintenance -Identity SERVER -Component ServerWideOffline

This however did not work... All components remained in the state Inactive.

To change the component states to Active, I needed to change the requester to Functional.

Boom we were back online!

Note: If Requester Maintenance doesn't work, try Functional istead.

Set-ServerComponentState -Identity Bentley-MAIL -Component ServerWideOffline -State Active -Requester Functional

SMTP Namespace Sharing - How it differs in the Cloud

When you enter Hybrid with Microsoft Office 365, you establish a concept called SMTP Namespace sharing to allow mailboxes in the cloud and on-premises to share the same address space such as @contoso.com.

Microsoft Exchange handles SMTP Namespace Sharing with Office 365 in a different manner to implementing SMTP Namespace Sharing with another Exchange Server or third party mail system.

In this blog post we are going to run through how to configure SMTP Namespace Sharing between an Exchange Server and another mail system, and then look at how SMTP Namespace Sharing works with Office 365.

Configuring SMTP Namespace Sharing

Before we talk about SMTP Namespace Sharing, we must first understand the concept of an Exchange Accepted Domain.  There are three types of Accepted Domains:

• Authoritative Domains - This tells an Exchange Server that the "Exchange Organisation" has a recipient (in particular a mailbox) for that inbound email such as recipient@contoso.com and it needs to deliver it locally.  All email address namespaces that your Exchange Servers receive email for need to be setup as Authoritative Accepted Domains.
• Internal Relay Domains - Internal Relay Domains are used when you share a domain between the Exchange organization and a third-party messaging system.  When an Exchange Server receives an email for an email address such as user@contoso.com, it will look to see if a mailbox exists, else send it out a Send Connector.
• External Relay Domains - No mailboxes exist on the Exchange Environment, send it out a send connector matching the address space.  You would use these if your Exchange Server is setup to perform mail cleansing at a transport layer most likely with a third party Transport Agent, then send the email on its way.

For more information please refer to https://docs.microsoft.com/en-us/exchange/mail-flow/accepted-domains/accepted-domains

To configure SMTP Namespace Sharing, you would change the Accepted Domain to an "Internal Relay Domain" and then create a send connector.  An example config would look like this:

Set-AcceptedDomain contoso.com -DomainType InternalRelay

New-SendConnector -Name "SMTP Namespace Sharing for Consoto.com" -Custom -AddressSpaces contoso.com -SmartHosts 10.1.1.54 -SourceTransportServers ExchangeServerFQDN

This will tell all Transport Servers in the Exchange Organisation to perform the following:

1. Look in Active Directory to see if a mailbox exists, if it does route the email to the Mailbox Server and deliver it.
2. If a mailbox doesn't exist, route it out a Send Connector to a remote mail environment for delivery instead of sending the typical 550 5.1.10 RESOLVER.ADR.RecipientNotFound error.

Configuring SMTP Namespace Sharing with Office 365

When you enable Hybrid Exchange using the Hybrid Configuration Wizard (HCW), SMTP Namespace sharing is setup differently.  After you move a mailbox to Exchange Online, it replaces the mailbox on-premises with a "Remote Mailbox" object.  These mailboxes can be viewed in the Exchange Admin Center as "Office 365" Mailboxes or with the following on-premises Exchange cmdlets:

• Get-RemoteMailbox
• Set-RemoteMailbox

Each of these Remote Mailboxes have a "Remote Routing Address" which looks like alias@tenant.mail.onmicrosoft.com.  If we look at one of the Remote Mailboxes in Exchange Admin Centre on-premises, we can see there is a new box to specify the Remote Routing Address.

Note: This can also be set using Set-RemoteMailbox "mailbox name" -RemoteRoutingAddress "avantgarde@tenant.mail.onmicrosoft.com"

In addition, SMTP Namespace Sharing does not require the Accepted Domain be changed to an "Internal Relay Domain".  It should stay "Authoritative" due to these Remote Mailbox objects.

How the Remote Routing Address is Set

By default, when you run Hybrid Configuration Wizard and move a mailbox to Office 365, the Remote Routing Address is set automatically and you never need to touch/look at this value.

We had an incident however for a customer where the Remote Routing Address was getting automatically set to the users primary domain name, not the Office 365 tenant.mail.onmicrosoft.com SMTP namespace.

How Exchange Server sets the Remote Routing Address automatically is unclear.  I can confirm it does not get set by an Email Address Policy.

We have reached out to Microsoft to find an answer, as it is important this is set to the correct value automatically!

Configuring Centralized Transport with Office 365 Exchange Online

I was called in to assist a customer make changes to their Office 365 mail routing.  The customer had a requirement to implement a Centralized Transport Model to ensure all email routes through on-premises so that a custom Transport Agent could run across all email and stamp the companies email signature.  Before we get into the changes needed to setup Centralized Transport, lets run through what the customer currently had in place.

• All mail from the Internet was routed to a cloud emailing service known as Forcepoint.
• Force point delivers all inbound email to the on-premises server.
• The on-premises server has a send connector to Forcepoint for all outbound Internet email "*".
• The on-premises has a second send connector to Office 365 for cloud based mailboxes
• There is a route from Office 365 cloud to Forcepoint for all internal mail recipient domains (no idea why this was done, Office 365 EOP should route directly to on-premises Exchange for any internal emails. Luckily Forcepoint was able to route email back to on-premises.).
• When Office 365 needed to email any Internet Recipients, it routed directly to the Internet bypassing the Forcepoint cloud.

I drew up in Visio a quick overview of what the mail routing looked like.

Now the customer needed to enable a "Centralized Transport Model" to ensure all email from Office 365 was routed through the on-premises server whether it is destined for an external recipient or not.  This is because the on-premises server has a custom Exchange Transport Agent which was responsible for stamping signatures on all outbound email.

Note: A Centralized Transport Model is a very bad configuration and should be avoided at all costs as it means the cloud cannot send/receive email in the event the on-premises Exchange environment is down.  It should only be used when there is a technical requirement such as this one.

I drew up what the mail routing will look like in a Centralized Transport Model:

It is important to note, you can configure a Centralized Transport Model using the Hybrid Configuration Wizard.  This was not an option for this customer as they were on Exchange 2013 CU13 and wanted to address patching at a later date despite being vulnerable to zero day cve-2020-0688 but that's another story. Hybrid Configuration Wizard is constantly updated by Microsoft and is only supported on the two latest CU updates.  Make sure your Exchange environment is running one of the latest CU updates as Microsoft does not test HCW on old releases of Exchange.

"Hybrid deployments require the latest Cumulative Update (CU) or Update Rollup (RU) that's available for your version of Exchange. If you can't install the latest update, the immediately previous release is also supported."

Source: https://docs.microsoft.com/en-us/exchange/hybrid-deployment-prerequisites

Manually Enabling Centralized Transport

To get this customer into a Centralized Transport Model we must do two things:

1. Create a new "Inbound from Office 365" Receive Connector and i'll explain why this is required in a minute.
2. Modify the Outbound connector in Office 365 to route all email "*" to the on-premises Server.

Create a Inbound from Office 365 Receive Connector

By default, all email from Office 365 enters through the Default Frontend Receive connector.  The only change made by the Hybrid Configuration Wizard needed to receive email from Office 365 is by modifying the "TlsCertificateName" attribute on the Default Frontend Receive connector so that SMTP TLS can be established between all emails from Office 365 to the on-premises environment.

In a Centralized Transport Model we can no longer use the Default Frontend Receive Connector.  The Default Frontend Receive Connector can receive email for all "Accepted Domains" domains which have mailboxes on the on-premises server.  The Default Frontend Receive connector by default cannot accept email for external Internet Domain Names then route the email to a remote server on the Internet via a Smart Host or MX records for a good reason - this would make the mail server an open relay!

As a result, if you have a requirement to configure a Centralized Transport Model, you will need to create a new receive connector with a name such as "Inbound from Office 365".

As a side note, if you don't do this any emails trying to route out to the Internet through your on-premises server from Office 365 will bounce with the following error:

“550 5.7.1 Unable to relay”

To create the new "Inbound from Office 365" connector required for a Centralized Transport model we need to do the following things:

1. Create the Frontend Receive Connector and call it "Inbound from Office 365" on TCP25
2. Configure the Authentication and Permission Groups
3. Lock the Receive Connector down to the Office 365 IP ranges.
4. Configure SMTP TLS on the Receive Connector required for Office 365
5. Configure the ExtendedRight MS-Exch-SMTP-Accept-Any-Recipient so that the Receive Connector can route email out a Send Connector for internet bound emails it receives from Office 365.

When you create the new Frontend Receive Connector enable TLS required for Office 365 and Anonymous Users as we are accepting emails destined for external recipients on the Internet.

Lock it down to the Exchange Online Protection IP Ranges documented here:

https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges

40.92.0.0/15, 40.107.0.0/16, 52.100.0.0/14, 52.238.78.88/32, 104.47.0.0/17

To configure the SMTP TLS on the receive connector refer to the following article by Paul Cunningham.  It should look like my screenshot below.

https://practical365.com/exchange-server/configuring-the-tls-certificate-name-for-exchange-server-receive-connectors/

Lastly you need to allow the receive connector to accept and relay email for non authoritative domain names (domains that are not an Accepted Domain in your Exchange environment) by adding the ExtendedRight MS-Exch-SMTP-Accept-Any-Recipient.  A command similar to the one below will achieve this.

Get-ReceiveConnector "Inbound from Office 365" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient

Cool now we have a receive connector on-premises locked down to the Office 365 ranges that will be able to relay email for internet-bound receipts through the Send Connector marked with "*" for all internet recipients.

Modify the Outbound Connector in Office 365

Next we need to modify the Outbound Connector in Office 365 to route all email through the on-premises server in a "Centralized Transport Model" configuration.  I blocked out information below for privacy reasons and using the Microsoft test domain contoso.com instead of my customers.

Get-OutboundConnector "Outbound to GUID" | Set-OutboundConnector -RecipientDomains "*" -RouteAllMessagesViaOnPremises:$true -SmartHosts mail.contoso.com

This will ensure every email leaving Office 365 will go to the on-premises server "mail.contoso.com.

Mail Loop Issue

Despite the configuration for Centralized Transport being correct, we experienced a Mail Loop when routing emails from the on-premises environment to Office 365.  This is what we experienced.

The bounce back we received from the mail loop was as follows, the NDR shows the email bouncing back and forth between Office 365 and Exchange 2013 until Loop Detection kicks in and blocks the email generating an NDR.

Delivery has failed to these recipients or groups:
Test User
A problem occurred during the delivery of this message. Please try to resend the message later. If the problem continues, contact your email admin.
The following organization rejected your message: Exchange2013.domain.local.


Diagnostic information for administrators:
Generating server: SYBPR01MB4362.ausprd01.prod.outlook.com
Test.User@contoso.com
Exchange2013.domain.local
Remote Server returned '554 5.4.6 '
Original message headers:
Received: from ME2PR01CA0046.ausprd01.prod.outlook.com (2603:10c6:201:14::34)
 by SYBPR01MB4362.ausprd01.prod.outlook.com (2603:10c6:10:56::21) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.27; Sun, 24 May
 2020 10:03:55 +0000
Received: from ME1AUS01FT014.eop-AUS01.prod.protection.outlook.com
 (2603:10c6:201:14:cafe::d5) by ME2PR01CA0046.outlook.office365.com
 (2603:10c6:201:14::34) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend
 Transport; Sun, 24 May 2020 10:03:54 +0000
Authentication-Results: spf=softfail (sender IP is 203.54.134.98)
 smtp.mailfrom=avantgardetechnologies.com.au; contoso.mail.onmicrosoft.com;
 dkim=none (message not signed) header.d=none;contoso.mail.onmicrosoft.com;
 dmarc=none action=none
 header.from=avantgardetechnologies.com.au;compauth=none reason=405
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 avantgardetechnologies.com.au discourages use of 203.54.134.98 as permitted
 sender)
Received: from Exchange2013.domain.local (203.54.134.98) by
 ME1AUS01FT014.mail.protection.outlook.com (10.152.232.114) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.3021.23 via Frontend Transport; Sun, 24 May 2020 10:03:53 +0000
Received: from Exchange2013.domain.local (192.168.0.13) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6; Sun, 24 May 2020 18:00:02 +0800
Received: from AUS01-SY3-obe.outbound.protection.outlook.com (104.47.117.51)
 by Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6 via Frontend Transport; Sun, 24 May 2020 18:00:02 +0800
Received: from SY4P282CA0010.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:a0::20) by
 SYCPR01MB5248.ausprd01.prod.outlook.com (2603:10c6:10:84::23) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3021.27; Sun, 24 May 2020 09:59:56 +0000
Received: from SY3AUS01FT014.eop-AUS01.prod.protection.outlook.com
 (2603:10c6:10:a0:cafe::23) by SY4P282CA0010.outlook.office365.com
 (2603:10c6:10:a0::20) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend
 Transport; Sun, 24 May 2020 09:59:56 +0000
Authentication-Results-Original: spf=softfail (sender IP is 203.54.134.98)
 smtp.mailfrom=avantgardetechnologies.com.au; contoso.mail.onmicrosoft.com;
 dkim=none (message not signed) header.d=none;contoso.mail.onmicrosoft.com;
 dmarc=none action=none
 header.from=avantgardetechnologies.com.au;compauth=none reason=405
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 avantgardetechnologies.com.au discourages use of 203.54.134.98 as permitted
 sender)
Received: from Exchange2013.domain.local (203.54.134.98) by
 SY3AUS01FT014.mail.protection.outlook.com (10.152.234.114) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.3021.23 via Frontend Transport; Sun, 24 May 2020 09:59:55 +0000
Received: from Exchange2013.domain.local (192.168.0.13) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6; Sun, 24 May 2020 17:59:22 +0800
Received: from AUS01-SY3-obe.outbound.protection.outlook.com (104.47.117.55)
 by Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6 via Frontend Transport; Sun, 24 May 2020 17:59:22 +0800
Received: from SYBPR01CA0077.ausprd01.prod.outlook.com (2603:10c6:10:3::17) by
 SY3PR01MB1738.ausprd01.prod.outlook.com (2603:10c6:0:1e::9) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3021.27; Sun, 24 May 2020 09:59:16 +0000
Received: from SY3AUS01FT016.eop-AUS01.prod.protection.outlook.com
 (2603:10c6:10:3:cafe::2f) by SYBPR01CA0077.outlook.office365.com
 (2603:10c6:10:3::17) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend
 Transport; Sun, 24 May 2020 09:59:15 +0000
Authentication-Results-Original: spf=softfail (sender IP is 203.54.134.98)
 smtp.mailfrom=avantgardetechnologies.com.au; contoso.mail.onmicrosoft.com;
 dkim=none (message not signed) header.d=none;contoso.mail.onmicrosoft.com;
 dmarc=none action=none
 header.from=avantgardetechnologies.com.au;compauth=none reason=405
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 avantgardetechnologies.com.au discourages use of 203.54.134.98 as permitted
 sender)
Received: from Exchange2013.domain.local (203.54.134.98) by
 SY3AUS01FT016.mail.protection.outlook.com (10.152.234.71) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.3021.23 via Frontend Transport; Sun, 24 May 2020 09:59:14 +0000
Received: from Exchange2013.domain.local (192.168.0.13) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6; Sun, 24 May 2020 17:59:06 +0800
Received: from cluster-m.mailcontrol.com (116.50.58.190) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server id
 15.0.1156.6 via Frontend Transport; Sun, 24 May 2020 17:59:06 +0800
Received: from mail.avantgardetechnologies.com.au (mail.avantgardetechnologies.com.au [59.167.109.99])
by rly15m.srv.mailcontrol.com (MailControl) with ESMTPS id 04O9x0Xr067289
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK)
for ; Sun, 24 May 2020 10:59:01 +0100
Received: from Bentley-MAIL.at.local (10.1.30.18) by Bentley-MAIL.at.local
 (10.1.30.18) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.529.5; Sun, 24 May 2020
 17:58:53 +0800
Received: from Bentley-MAIL.at.local ([fe80::fc2d:dec2:9b22:4f2a]) by
 Bentley-MAIL.at.local ([fe80::fc2d:dec2:9b22:4f2a%3]) with mapi id
 15.02.0529.008; Sun, 24 May 2020 17:58:47 +0800
From: Clint Boessen
To: Test User
Subject: RE: Test External Email through new Receive Connector
Thread-Topic: Test External Email through new Receive Connector
Thread-Index: AQHWMbHMsLmpvI1RokaklktqWBJNSKi3AGdQ
Date: Sun, 24 May 2020 09:58:46 +0000
Message-ID:
References:
In-Reply-To:
Accept-Language: en-AU, en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
x-originating-ip: [10.2.10.104]
Content-Type: multipart/related;
boundary="_006_db8ba13106a042f59f1b4cc7154685ffavantgardetechnologiesc_";
type="multipart/alternative"
MIME-Version: 1.0
X-Modified-HTML: 6
X-Mailcontrol-Inbound: VasuXYiFy3qCtpA1zsc0iLOFFGXQvwm!nVYR6ThwFMIidec+qhp6ZSi!Mnl8Fsmw0sAYU+ZMD9zpwrW47BjDsw==
X-Spam-Score: -2.9
X-MailControl-ReportSpam: https://www.mailcontrol.com/sr/4vHmKlvfRcfGX2PQPOmvUtMaodt6qto8zwizKSQwqRUN24RXJczj00urrTqFWStbC6mdJoQP7nEhR_AbTPK7tQ==
X-Scanned-By: MailControl 44278.2096 (www.mailcontrol.com) on 10.77.0.125
Return-Path: clint.boessen@avantgardetechnologies.com.au
X-EXCLAIMER-MD-CONFIG: 659f567a-0ab7-4104-a8ab-4b8b5d34a680
X-OrganizationHeadersPreserved: Exchange2013.domain.local
X-EOPAttributedMessage: 2
X-EOPTenantAttributedMessage: 7e0e266c-0475-408b-9eb7-cb0cf8b31e59:2
X-CrossPremisesHeadersFiltered: SY3AUS01FT016.eop-AUS01.prod.protection.outlook.com
X-Forefront-Antispam-Report-Untrusted: CIP:203.54.134.98;CTRY:AU;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:Exchange2013.domain.local;PTR:abi3057722.lnk.telstra.net;CAT:NONE;SFTY:;SFS:(53546011)(2616005)(86362001)(33964004)(19627405001)(26005)(15974865002)(36756003)(356005)(83080400001)(66574014)(166002)(5660300002)(81166007)(24736004)(6916009)(1096003)(336012)(44832011)(108616005)(8676002)(36906005)(19607625011);DIR:INB;SFP:;
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 8134a206-5e1e-4c8c-50b8-08d7ffc9c425
X-MS-TrafficTypeDiagnostic: SY3PR01MB1738:|SYCPR01MB5248:|SYBPR01MB4362:
X-MS-Oob-TLC-OOBClassifiers: OLM:6790;OLM:6790;OLM:6790;
X-Microsoft-Antispam-Untrusted: BCL:0;
X-Microsoft-Antispam-Message-Info-Original: =?us-ascii?Q?iA2wyOTRoRwZLE3dlGXOPjuc/V5nlVGk0qx3c3cVWVq6DeIGm1BllHTCVSiP?=
 =?us-ascii?Q?WdgykixTEOiu5qMNLlDbyvS2B3E3qFunMZOmsip1wW+2+EYpZWRl3h3p1qMe?=
 =?us-ascii?Q?gyHb+gYsWcLLLO/d2oyvKNCusbLDzM+doRdC675aXRpQLJVN0nkIvoeTlN8q?=
 =?us-ascii?Q?uAxB3OZWclI1iz0LWyGk7q7bEL0cf1B7+L5gEvxv+6ccBVjGxDCUARbDvLnH?=
 =?us-ascii?Q?l0az2fYOYGteZOTgZkL/QA4P+UdTnL/ZJFD524SawSpoIipAp4/jd7q4HCdF?=
 =?us-ascii?Q?ii+IYIBvHSb+Ib07PQyBHWQZWcetnQvin6wKe4zFmVxal3O331OUhADCrNX2?=
 =?us-ascii?Q?dABvHfIW5pde9NNG+gtPKa4839nANzsibjIrC4S5/myohU3D0tfQNc21sV9c?=
 =?us-ascii?Q?X8ajwUqsg8oiwXaJEXeZqyS9+em0DhmjpML3/H8aLUa9krvS1hjk/T/95gaU?=
 =?us-ascii?Q?9JlTkvW7otLXXP3scvj9VPvP45JDQ0T4/UV+tS8NciviPlgMPcPHSck6tAE1?=
 =?us-ascii?Q?Bbvs5/mIP8ExRjpHLOrAP1bnFdotp7LAeP9zDvnQ4noFncY7VDCGA7aRAYvp?=
 =?us-ascii?Q?gzi5CYKQc6W7E7vMJaQ4poq567KhRJOgBiLy7/P6LwTaGAQDkCJtydjtCdMX?=
 =?us-ascii?Q?D3gaOUKUGptP07VQXA5kSOqYbXIIzkLuMAzUMpdJ9sUMCuYqZntL7oEJs5Vz?=
 =?us-ascii?Q?7ya/OmOTaSGyWqXURw6jyt7s3sPxXRwZYrszcZx/fTi7CpgMzhgNsVvlyplQ?=
 =?us-ascii?Q?DxgEO7zetqixpz+MfGCpO4oqxlnEaBAcLf7ml8A/UsavMy3MCnghUtFKfZzy?=
 =?us-ascii?Q?C2fXr0H5Q7dm0GS2UA4UwIwGx27nU2ah2542blE+41kzfT2yj0Pya36mePgi?=
 =?us-ascii?Q?Eie4ZoX7lxJ5Ob1l5b6g4Fn6cd2iOVwZw8ldz7TVZEus/r9HbKD96H7E+0gO?=
 =?us-ascii?Q?Jzr37suMHpaKJdTwrQUvXPff7rt3NhglfVoFFc0YaEiJqUKK8YvoyDDqriBg?=
 =?us-ascii?Q?7mSl3nudaNDXcsrZwMH39Ee8glstaFgu3Iht1G5bXjIjzAnu4d8DAONd/Vuw?=
 =?us-ascii?Q?lmIf3jkbeczjusoRukL7qLwFHx766qhLVbNT5ublsgPn8Ed2qUeJcPawb8gf?=
 =?us-ascii?Q?SYCLxnFICDvNWpMogjNuqJr1Vl6dJWtQRruNF0kUNm4rlJnuVXtQ0+B4eeyL?=
 =?us-ascii?Q?Hi2Xi4zvRHnA4EezF7lkDe4XE4/XN74zr1l2KXbTC2qFyAX1UUuKWST9INZU?=
 =?us-ascii?Q?bzlyIW+f4GylBSwuFknr?=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SY3PR01MB1738
X-OrganizationHeadersPreserved: SY3PR01MB1738.ausprd01.prod.outlook.com
X-CrossPremisesHeadersFiltered: Exchange2013.domain.local
X-OrganizationHeadersPreserved: Exchange2013.domain.local
X-CrossPremisesHeadersFiltered: SY3AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-MS-Exchange-Transport-CrossTenantHeadersStripped: SY3AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-Forefront-Antispam-Report-Untrusted: CIP:203.54.134.98;CTRY:AU;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:Exchange2013.domain.local;PTR:abi3057722.lnk.telstra.net;CAT:NONE;SFTY:;SFS:(36756003)(336012)(2616005)(44832011)(6916009)(15974865002)(8676002)(1096003)(36906005)(2160300002)(108616005)(86362001)(5660300002)(53546011)(166002)(26005)(33964004)(24736004)(81166007)(356005)(83080400001)(19627405001)(66574014)(19607625011);DIR:INB;SFP:;
X-MS-Office365-Filtering-Correlation-Id-Prvs: 88288457-f197-4473-a57d-08d7ffc91d97
X-Microsoft-Antispam-Untrusted: BCL:0;
X-Microsoft-Antispam-Message-Info-Original: =?us-ascii?Q?HDO3W+nr056uwZuOZ7TYTG2KW9HPxRSExlC/7yQL77Lh8xd5toPU9ZMXlt+/?=
 =?us-ascii?Q?q8S3K0REYizRMHkthA3wNJb6MU4vgHvsPbRJNHjBeJw08xJepEx7YRJklYdx?=
 =?us-ascii?Q?nSegk5JXfkz3FnYHpVhO1t7qoKwwcMFHifB69QFoA0V6ggrIjDJEwNq27sTb?=
 =?us-ascii?Q?u2vrAss33OY+1dUUp1GiiBZzT4DpL16QotqSIEfiRFWWXS/KqbwJEJ0+wp7l?=
 =?us-ascii?Q?ujFm3W1ndL5XWmRlztxS3+/E4mHNVqAj0hgoUu97adO/E0+oqbYn61mqNOH0?=
 =?us-ascii?Q?FkDij6ZoUw3XFYNu0hTpsUoGBEWCu9aX1/cz2TVmAGN5DBXfYI9+ZgLeXOF2?=
 =?us-ascii?Q?7/5m+E9c6zQLpPRWNj+0q821bxNzE4yhUB+C+K5PNqCpg3YAd2kws7e2+HjT?=
 =?us-ascii?Q?ybEvvGfrX/JVlhTA+5yK0YQwNm0caLgx/BTKx0dEFUoHGaDqg0LRQlnEXr9y?=
 =?us-ascii?Q?zPftt3dapzVbKQdVp15Ae1ErOleQLDRZEpQ/m50pQji37iHH/jpf84LthOQB?=
 =?us-ascii?Q?NX/oUDEZsyj83RowWP6sSAZ7Z9NFSoFKkubvDqMeXflQbW3vz+oL/cJfuSoZ?=
 =?us-ascii?Q?eC12mwEvYibjhWqUsYB5NnSQ1DmvW/WGZNioxbinDOlxrclSm/g7RwKdkkl1?=
 =?us-ascii?Q?NADfKmk3dN5roljcmd/MrBszuC4nigH7USGCCVMDTPSmoP9I+TAPhhKaRP26?=
 =?us-ascii?Q?LNQh5Fe50kxOxwdUGSDoQTr56sUkaGEIsoiP/ue4PMRRXZMjPGTbCI6eCRde?=
 =?us-ascii?Q?hdfTd9fMIeSBjKhNXDtc5P63e9fhp8SXVeT5T4MycPOs+VKEgL9K8cassjbr?=
 =?us-ascii?Q?KmbSdhp00ftDoi9Q8of6jV/Ve6CSYgFIqi9MYd5EUwGOYTK6hcAiU4B1oe5G?=
 =?us-ascii?Q?sVMksEY0SMKTcHn3gnQVuWRXKpjDMU5nYVh2SklhVa1sgo2SR+9i2dDw8Wsd?=
 =?us-ascii?Q?TVt3bMWhm33WMVi79TZla9z3DOqzS9HRZ8lpJKzHS/iO3KMDPq1GwryMRu63?=
 =?us-ascii?Q?PVjn+KYYvEs+l1c5/Qbgb64uwjoL+VZ76x4E4YvR6WaSScFLtuqs6pwAzoYB?=
 =?us-ascii?Q?hFRxmXcdjdnPGh3klPWyqBARJOlKMTR3tzcA7g6MFm7wp4lmKh7PwwPLJIkA?=
 =?us-ascii?Q?QVIN591rFcQ8RQIj47kf5BaJRj3GoosyZsnilb8L2wIwRiQjSsmUmk+cqQ3G?=
 =?us-ascii?Q?M3UtTLIA53MQk6UL1Z1Sd9iXZDdZwY8p9r7vi5lPWmtjQ9Aeo/AomnwY71N3?=
 =?us-ascii?Q?eZ6E2tP8thFY//uSKB7l?=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SYCPR01MB5248
X-OrganizationHeadersPreserved: SYCPR01MB5248.ausprd01.prod.outlook.com
X-CrossPremisesHeadersFiltered: Exchange2013.domain.local
X-OrganizationHeadersPreserved: Exchange2013.domain.local
X-CrossPremisesHeadersFiltered: ME1AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-MS-Exchange-Transport-CrossTenantHeadersStripped: ME1AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-Forefront-Antispam-Report: CIP:203.54.134.98;CTRY:AU;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:Exchange2013.domain.local;PTR:abi3057722.lnk.telstra.net;CAT:NONE;SFTY:;SFS:(24736004)(36756003)(36906005)(19627405001)(26005)(53546011)(5660300002)(33964004)(108616005)(44832011)(6916009)(15974865002)(2616005)(86362001)(1096003)(166002)(336012)(356005)(66574014)(83080400001)(8676002)(81166007)(19607625011);DIR:INB;SFP:;
X-MS-Office365-Filtering-Correlation-Id-Prvs: 2bb48326-def0-492c-8752-08d7ffc935e8
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vyA9jQ5YP4tbiJQ1N2ff25VgnM+SJiFhE3Yln6tQVaileyKRkO+BuzPHYj6Q?=
 =?us-ascii?Q?R+XgxcBUeNcGuLQOtJZJD6XEXu0UbScHjmvFzBtKM6xT7AC9PD1aTcQ364rJ?=
 =?us-ascii?Q?c8SxdVfBKMuOWjAKK/SHGyuZdAvObLEbgloQS/lG3UEJkCUTEHm8n2WqXggb?=
 =?us-ascii?Q?IZiBNwqd4iWWStR+UyIM/h3t/TIToNx5FHbPD4fPAvZG/L+lCasiGPkaw7sf?=
 =?us-ascii?Q?ZQJfKrllu9oRIjrxw4NYTtDfQdr2OJ+IA606UhprZ7tmrlBIoid+NsFDXB62?=
 =?us-ascii?Q?G/9VH6IvGRNxA445F6764dIyXifRsuaGYs/T11WSIzAJ/5ArCemQuAf+1Ek6?=
 =?us-ascii?Q?KyhsNPSmh9+rGGmNuvV5RXh2E4aqDhZWS6ZGK1aMSTXO6KHeQd+FrEZHM3nf?=
 =?us-ascii?Q?Xl5OdF6OPKdLlMv1nEa1UVpMi52rkMvxXp7DRmWxspsi9JkIhlU/OBdMjOmC?=
 =?us-ascii?Q?Us+aJLNXT9O/JIc/7mPwNE3e8+Hj9zIHxNIcrYelEoopTGXSmcdTP0eOslts?=
 =?us-ascii?Q?Ap1JQ0sEeODnnNs4Z0r2NjzrHOOKRY8YjLhXz4vE+iWzzMhLUgW+17Lc9cHG?=
 =?us-ascii?Q?/z9uQm800gOM7k6By+jhkxzM8Ho49Uin65W3YMI31VqWQzTgg6kBxoYgNqls?=
 =?us-ascii?Q?VylLdyHSa8rjX3n31TNqzFxN7DHZRLL+0Ar/53pw9yj6YETYgoBZX8b4zQCD?=
 =?us-ascii?Q?KZl6GY+7WSjn1MlecYlPAlCCY3rSkwPWGaP5nb1OMPryBPa/hCna5UQJpQK0?=
 =?us-ascii?Q?NKCSLmg374I4/+jrylQ/fR1upcDBJwDfX0c28v3kzCxv1wlGwWK4LSn5SWrM?=
 =?us-ascii?Q?GZOW6Cc3K+/G/prlpPrCPsRHdNNGVaGWHqoaaFrLgnFeJQiKDu+afKXAaSem?=
 =?us-ascii?Q?yqCF551HYU+ItucZDwzF2L2yAu0hX3PZohye+tT4ErFrGAtYrZdAWHr23wL4?=
 =?us-ascii?Q?+LH54epp83Xrgl0aZm4N1XomI0G94LY+vBQiNK3W075GoK2bhNvYYmVGR03q?=
 =?us-ascii?Q?w0uD8TzzRXd7MSOC6nwwKCJDiovNYq3ms4AsY4B/t4UQyf/mhTr9h24yuBCa?=
 =?us-ascii?Q?1GUkuH/ta2m/4NfpUefV0dlBJtCS1QWEZsCClWHo0UPUb+PwtPNO7atyR4vj?=
 =?us-ascii?Q?hmIkERJe2CYWSVa3B8mKM+Md70ZSPNIdSCHM+FXhM28DAntZmBt3Ox8zVl5Y?=
 =?us-ascii?Q?2yDsCXoYhGM9UByN5JXDkMe3WwH14Mm++895YPqAMT9u5Zd8OeWAw7WztPC3?=
 =?us-ascii?Q?RDfJkidLJlLE5XVEABX7?=
X-OriginatorOrg: contoso.onmicrosoft.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2020 10:03:53.9414
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 8134a206-5e1e-4c8c-50b8-08d7ffc9c425
X-MS-Exchange-CrossTenant-Id: 7e0e266c-0475-408b-9eb7-cb0cf8b31e59
X-MS-Exchange-CrossTenant-FromEntityHeader: Internet
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SYBPR01MB4362
X-OrganizationHeadersPreserved: SYBPR01MB4362.ausprd01.prod.outlook.com
X-CrossPremisesHeadersFilteredByDsnGenerator: SYBPR01MB4362.ausprd01.prod.outlook.com

Reverting the change we needed to troubleshoot.  With Centralized Transport disabled on the Outbound connector we saw that any emails passing from on-premises to Office 365 had in the header:

X-MS-Exchange-Organization-AuthAs: Anonymous

Office 365 should be receiving any emails from the on-premises Exchange Server as "Internal" if they are in the same tenant.  This means the emails were not hitting an Inbound Connector in Office 365 and were coming into Office 365 as anonymous.  This can also cause the SMTP error "451 4.7.500 Server busy" for very large tenants as Microsoft throttles emails from anonymous sources to limit spam in Office 365 - for more information see .

https://www.alitajran.com/error-code-451-4-7-500-server-busy-email-delivery-issues/

After researching the issue and a call with Microsoft, we saw that there was a problem with the Certificate configuration on the Inbound connector.  The common name on the certificate - lets say "mail.contoso.com" was correct on both ends however the Organisation Name on the certificate was different which was enough to cause inbound SMTP email from on-premises to be not identified by the Inbound Connector and get flagged as anonymous.

To fix this issue, we needed to modify the Inbound Connector from having multiple attributes of the certificate as shown below.

To only having the "Common Name" of the on-premises certificate which for argument sake lets call it "mail.contoso.com".

After fixing the certificate details on the Inbound Connector, emails from on-premises to the cloud were identified as Internal.  This can be verified by looking in the message header in one of the emails.

Note: You need to wait an hour for any changes to Exchange Online to propagate to Exchange Online Protection unless you speak to Microsoft Support, they have a script that forces it on the backend!

830fecf9-8ece-4d66-2536-08d81e5f643a
X-EOPAttributedMessage: 0
X-MS-Exchange-Organization-MessageDirectionality: Originating
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: Exchange2013.domain.local
X-MS-Exchange-Organization-SCL: -1
X-CrossPremisesHeadersPromoted:
SY3AUS01FT006.eop-AUS01.prod.protection.outlook.com
X-CrossPremisesHeadersFiltered:
SY3AUS01FT006.eop-AUS01.prod.protection.outlook.com

After this, we re-instated the Centralized Mail Configuration and it worked perfectly.

Direct Access not working with Windows Firewall Enabled on Client

We had an interesting issue with Microsoft Direct Access on Windows 10 latest build 1909.

• When Windows Firewall is disabled, Direct Access works on the client.
• Enabling Windows Firewall on the client breaks Direct Access resulting in no connection.

Symptoms of Issue

Here is the symptoms we were receiving:

Get-DAConnectionStatus returned "CouldNotContactDirectAccessServer"

netsh interface httpstunnel show interface shows the Interface is Active

IPHTTPS interface active
Last Error Code 0x0

The issue appears to be with Name Resolution Policy Table (NRPT).  When Windows Firewall was enabled on the client, you were able to ping IPv6 addresses on the Direct Access server through the tunnel but all name resolution failed.

For example, you can ping the tunnel endpoints discoverable using the Get-DAClientExperianceConfiguration cmdlet on the client.  You can also ping IPv6 addresses of hostnames previously resolved when the Windows Firewall was in a disabled state.  For example, whilst Windows Firewall was in a disabled state, ping domain.local (your AD forest name), copy the IPv6 address, re-enable Windows Firewall and you will notice you can resolve it by the IPv6 address, not by hostname using NRPT.

Resolution

The issue was caused by Windows Firewall set to a disabled state on the server.

What we found was when the Windows Firewall is disabled on the server, Windows Firewall must be disabled on the client.

If DA Server Firewall is disabled, but DA client Firewall is enabled - Direct Access breaks.
If DA Server Firewall is enabled, but DA client Firewall is disabled - Direct Access breaks

You must have Windows Firewall disabled on both the DA Server and DA Client, or enabled on both the DA Server and DA Client.

We also identified another issue on the DA server with regards to Windows Firewall being disabled.  In our environment, disabling Windows Firewall on the DA Server breaks Direct Access reporting in the "Remote Access Management" console.

As a result, it is strongly recommended to keep Windows Firewall enabled on both the DA Server and DA clients.