Thursday, July 2, 2020

Configuring Centralized Transport with Office 365 Exchange Online

I was called in to assist a customer make changes to their Office 365 mail routing.  The customer had a requirement to implement a Centralized Transport Model to ensure all email routes through on-premises so that a custom Transport Agent could run across all email and stamp the companies email signature.  Before we get into the changes needed to setup Centralized Transport, lets run through what the customer currently had in place.
  • All mail from the Internet was routed to a cloud emailing service known as Forcepoint.
  • Force point delivers all inbound email to the on-premises server.
  • The on-premises server has a send connector to Forcepoint for all outbound Internet email "*".
  • The on-premises has a second send connector to Office 365 for cloud based mailboxes
  • There is a route from Office 365 cloud to Forcepoint for all internal mail recipient domains (no idea why this was done, Office 365 EOP should route directly to on-premises Exchange for any internal emails. Luckily Forcepoint was able to route email back to on-premises.).
  • When Office 365 needed to email any Internet Recipients, it routed directly to the Internet bypassing the Forcepoint cloud.
I drew up in Visio a quick overview of what the mail routing looked like.


Now the customer needed to enable a "Centralized Transport Model" to ensure all email from Office 365 was routed through the on-premises server whether it is destined for an external recipient or not.  This is because the on-premises server has a custom Exchange Transport Agent which was responsible for stamping signatures on all outbound email.

Note: A Centralized Transport Model is a very bad configuration and should be avoided at all costs as it means the cloud cannot send/receive email in the event the on-premises Exchange environment is down.  It should only be used when there is a technical requirement such as this one.

I drew up what the mail routing will look like in a Centralized Transport Model:



It is important to note, you can configure a Centralized Transport Model using the Hybrid Configuration Wizard.  This was not an option for this customer as they were on Exchange 2013 CU13 and wanted to address patching at a later date despite being vulnerable to zero day cve-2020-0688 but that's another story. Hybrid Configuration Wizard is constantly updated by Microsoft and is only supported on the two latest CU updates.  Make sure your Exchange environment is running one of the latest CU updates as Microsoft does not test HCW on old releases of Exchange.

"Hybrid deployments require the latest Cumulative Update (CU) or Update Rollup (RU) that's available for your version of Exchange. If you can't install the latest update, the immediately previous release is also supported."

Source: https://docs.microsoft.com/en-us/exchange/hybrid-deployment-prerequisites

Manually Enabling Centralized Transport

To get this customer into a Centralized Transport Model we must do two things:
  1. Create a new "Inbound from Office 365" Receive Connector and i'll explain why this is required in a minute.
  2. Modify the Outbound connector in Office 365 to route all email "*" to the on-premises Server.

Create a Inbound from Office 365 Receive Connector

By default, all email from Office 365 enters through the Default Frontend Receive connector.  The only change made by the Hybrid Configuration Wizard needed to receive email from Office 365 is by modifying the "TlsCertificateName" attribute on the Default Frontend Receive connector so that SMTP TLS can be established between all emails from Office 365 to the on-premises environment.

In a Centralized Transport Model we can no longer use the Default Frontend Receive Connector.  The Default Frontend Receive Connector can receive email for all "Accepted Domains" domains which have mailboxes on the on-premises server.  The Default Frontend Receive connector by default cannot accept email for external Internet Domain Names then route the email to a remote server on the Internet via a Smart Host or MX records for a good reason - this would make the mail server an open relay!

As a result, if you have a requirement to configure a Centralized Transport Model, you will need to create a new receive connector with a name such as "Inbound from Office 365".

As a side note, if you don't do this any emails trying to route out to the Internet through your on-premises server from Office 365 will bounce with the following error:

“550 5.7.1 Unable to relay”

To create the new "Inbound from Office 365" connector required for a Centralized Transport model we need to do the following things:
  1. Create the Frontend Receive Connector and call it "Inbound from Office 365" on TCP25
  2. Configure the Authentication and Permission Groups
  3. Lock the Receive Connector down to the Office 365 IP ranges.
  4. Configure SMTP TLS on the Receive Connector required for Office 365
  5. Configure the ExtendedRight MS-Exch-SMTP-Accept-Any-Recipient so that the Receive Connector can route email out a Send Connector for internet bound emails it receives from Office 365.
When you create the new Frontend Receive Connector enable TLS required for Office 365 and Anonymous Users as we are accepting emails destined for external recipients on the Internet.


Lock it down to the Exchange Online Protection IP Ranges documented here:

https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges

40.92.0.0/15, 40.107.0.0/16, 52.100.0.0/14, 52.238.78.88/32, 104.47.0.0/17


To configure the SMTP TLS on the receive connector refer to the following article by Paul Cunningham.  It should look like my screenshot below.



Lastly you need to allow the receive connector to accept and relay email for non authoritative domain names (domains that are not an Accepted Domain in your Exchange environment) by adding the ExtendedRight MS-Exch-SMTP-Accept-Any-Recipient.  A command similar to the one below will achieve this.

Get-ReceiveConnector "Inbound from Office 365" | Add-ADPermission -User 'NT AUTHORITY\Anonymous Logon' -ExtendedRights MS-Exch-SMTP-Accept-Any-Recipient


Cool now we have a receive connector on-premises locked down to the Office 365 ranges that will be able to relay email for internet-bound receipts through the Send Connector marked with "*" for all internet recipients.

Modify the Outbound Connector in Office 365

Next we need to modify the Outbound Connector in Office 365 to route all email through the on-premises server in a "Centralized Transport Model" configuration.  I blocked out information below for privacy reasons and using the Microsoft test domain contoso.com instead of my customers.

Get-OutboundConnector "Outbound to GUID" | Set-OutboundConnector -RecipientDomains "*" -RouteAllMessagesViaOnPremises:$true -SmartHosts mail.contoso.com


This will ensure every email leaving Office 365 will go to the on-premises server "mail.contoso.com.

Mail Loop Issue

Despite the configuration for Centralized Transport being correct, we experienced a Mail Loop when routing emails from the on-premises environment to Office 365.  This is what we experienced.


The bounce back we received from the mail loop was as follows, the NDR shows the email bouncing back and forth between Office 365 and Exchange 2013 until Loop Detection kicks in and blocks the email generating an NDR.

Delivery has failed to these recipients or groups:
Test User
A problem occurred during the delivery of this message. Please try to resend the message later. If the problem continues, contact your email admin.
The following organization rejected your message: Exchange2013.domain.local.


Diagnostic information for administrators:
Generating server: SYBPR01MB4362.ausprd01.prod.outlook.com
Test.User@contoso.com
Exchange2013.domain.local
Remote Server returned '554 5.4.6 '
Original message headers:
Received: from ME2PR01CA0046.ausprd01.prod.outlook.com (2603:10c6:201:14::34)
 by SYBPR01MB4362.ausprd01.prod.outlook.com (2603:10c6:10:56::21) with
 Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.27; Sun, 24 May
 2020 10:03:55 +0000
Received: from ME1AUS01FT014.eop-AUS01.prod.protection.outlook.com
 (2603:10c6:201:14:cafe::d5) by ME2PR01CA0046.outlook.office365.com
 (2603:10c6:201:14::34) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend
 Transport; Sun, 24 May 2020 10:03:54 +0000
Authentication-Results: spf=softfail (sender IP is 203.54.134.98)
 smtp.mailfrom=avantgardetechnologies.com.au; contoso.mail.onmicrosoft.com;
 dkim=none (message not signed) header.d=none;contoso.mail.onmicrosoft.com;
 dmarc=none action=none
 header.from=avantgardetechnologies.com.au;compauth=none reason=405
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 avantgardetechnologies.com.au discourages use of 203.54.134.98 as permitted
 sender)
Received: from Exchange2013.domain.local (203.54.134.98) by
 ME1AUS01FT014.mail.protection.outlook.com (10.152.232.114) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.3021.23 via Frontend Transport; Sun, 24 May 2020 10:03:53 +0000
Received: from Exchange2013.domain.local (192.168.0.13) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6; Sun, 24 May 2020 18:00:02 +0800
Received: from AUS01-SY3-obe.outbound.protection.outlook.com (104.47.117.51)
 by Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6 via Frontend Transport; Sun, 24 May 2020 18:00:02 +0800
Received: from SY4P282CA0010.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:a0::20) by
 SYCPR01MB5248.ausprd01.prod.outlook.com (2603:10c6:10:84::23) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3021.27; Sun, 24 May 2020 09:59:56 +0000
Received: from SY3AUS01FT014.eop-AUS01.prod.protection.outlook.com
 (2603:10c6:10:a0:cafe::23) by SY4P282CA0010.outlook.office365.com
 (2603:10c6:10:a0::20) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend
 Transport; Sun, 24 May 2020 09:59:56 +0000
Authentication-Results-Original: spf=softfail (sender IP is 203.54.134.98)
 smtp.mailfrom=avantgardetechnologies.com.au; contoso.mail.onmicrosoft.com;
 dkim=none (message not signed) header.d=none;contoso.mail.onmicrosoft.com;
 dmarc=none action=none
 header.from=avantgardetechnologies.com.au;compauth=none reason=405
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 avantgardetechnologies.com.au discourages use of 203.54.134.98 as permitted
 sender)
Received: from Exchange2013.domain.local (203.54.134.98) by
 SY3AUS01FT014.mail.protection.outlook.com (10.152.234.114) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.3021.23 via Frontend Transport; Sun, 24 May 2020 09:59:55 +0000
Received: from Exchange2013.domain.local (192.168.0.13) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6; Sun, 24 May 2020 17:59:22 +0800
Received: from AUS01-SY3-obe.outbound.protection.outlook.com (104.47.117.55)
 by Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6 via Frontend Transport; Sun, 24 May 2020 17:59:22 +0800
Received: from SYBPR01CA0077.ausprd01.prod.outlook.com (2603:10c6:10:3::17) by
 SY3PR01MB1738.ausprd01.prod.outlook.com (2603:10c6:0:1e::9) with Microsoft
 SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id
 15.20.3021.27; Sun, 24 May 2020 09:59:16 +0000
Received: from SY3AUS01FT016.eop-AUS01.prod.protection.outlook.com
 (2603:10c6:10:3:cafe::2f) by SYBPR01CA0077.outlook.office365.com
 (2603:10c6:10:3::17) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3021.23 via Frontend
 Transport; Sun, 24 May 2020 09:59:15 +0000
Authentication-Results-Original: spf=softfail (sender IP is 203.54.134.98)
 smtp.mailfrom=avantgardetechnologies.com.au; contoso.mail.onmicrosoft.com;
 dkim=none (message not signed) header.d=none;contoso.mail.onmicrosoft.com;
 dmarc=none action=none
 header.from=avantgardetechnologies.com.au;compauth=none reason=405
Received-SPF: SoftFail (protection.outlook.com: domain of transitioning
 avantgardetechnologies.com.au discourages use of 203.54.134.98 as permitted
 sender)
Received: from Exchange2013.domain.local (203.54.134.98) by
 SY3AUS01FT016.mail.protection.outlook.com (10.152.234.71) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id
 15.20.3021.23 via Frontend Transport; Sun, 24 May 2020 09:59:14 +0000
Received: from Exchange2013.domain.local (192.168.0.13) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server (TLS) id
 15.0.1156.6; Sun, 24 May 2020 17:59:06 +0800
Received: from cluster-m.mailcontrol.com (116.50.58.190) by
 Exchange2013.domain.local (192.168.0.13) with Microsoft SMTP Server id
 15.0.1156.6 via Frontend Transport; Sun, 24 May 2020 17:59:06 +0800
Received: from mail.avantgardetechnologies.com.au (mail.avantgardetechnologies.com.au [59.167.109.99])
by rly15m.srv.mailcontrol.com (MailControl) with ESMTPS id 04O9x0Xr067289
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK)
for ; Sun, 24 May 2020 10:59:01 +0100
Received: from Bentley-MAIL.at.local (10.1.30.18) by Bentley-MAIL.at.local
 (10.1.30.18) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.529.5; Sun, 24 May 2020
 17:58:53 +0800
Received: from Bentley-MAIL.at.local ([fe80::fc2d:dec2:9b22:4f2a]) by
 Bentley-MAIL.at.local ([fe80::fc2d:dec2:9b22:4f2a%3]) with mapi id
 15.02.0529.008; Sun, 24 May 2020 17:58:47 +0800
From: Clint Boessen
To: Test User
Subject: RE: Test External Email through new Receive Connector
Thread-Topic: Test External Email through new Receive Connector
Thread-Index: AQHWMbHMsLmpvI1RokaklktqWBJNSKi3AGdQ
Date: Sun, 24 May 2020 09:58:46 +0000
Message-ID:
References:
In-Reply-To:
Accept-Language: en-AU, en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
x-originating-ip: [10.2.10.104]
Content-Type: multipart/related;
boundary="_006_db8ba13106a042f59f1b4cc7154685ffavantgardetechnologiesc_";
type="multipart/alternative"
MIME-Version: 1.0
X-Modified-HTML: 6
X-Mailcontrol-Inbound: VasuXYiFy3qCtpA1zsc0iLOFFGXQvwm!nVYR6ThwFMIidec+qhp6ZSi!Mnl8Fsmw0sAYU+ZMD9zpwrW47BjDsw==
X-Spam-Score: -2.9
X-MailControl-ReportSpam: https://www.mailcontrol.com/sr/4vHmKlvfRcfGX2PQPOmvUtMaodt6qto8zwizKSQwqRUN24RXJczj00urrTqFWStbC6mdJoQP7nEhR_AbTPK7tQ==
X-Scanned-By: MailControl 44278.2096 (www.mailcontrol.com) on 10.77.0.125
Return-Path: clint.boessen@avantgardetechnologies.com.au
X-EXCLAIMER-MD-CONFIG: 659f567a-0ab7-4104-a8ab-4b8b5d34a680
X-OrganizationHeadersPreserved: Exchange2013.domain.local
X-EOPAttributedMessage: 2
X-EOPTenantAttributedMessage: 7e0e266c-0475-408b-9eb7-cb0cf8b31e59:2
X-CrossPremisesHeadersFiltered: SY3AUS01FT016.eop-AUS01.prod.protection.outlook.com
X-Forefront-Antispam-Report-Untrusted: CIP:203.54.134.98;CTRY:AU;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:Exchange2013.domain.local;PTR:abi3057722.lnk.telstra.net;CAT:NONE;SFTY:;SFS:(53546011)(2616005)(86362001)(33964004)(19627405001)(26005)(15974865002)(36756003)(356005)(83080400001)(66574014)(166002)(5660300002)(81166007)(24736004)(6916009)(1096003)(336012)(44832011)(108616005)(8676002)(36906005)(19607625011);DIR:INB;SFP:;
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 8134a206-5e1e-4c8c-50b8-08d7ffc9c425
X-MS-TrafficTypeDiagnostic: SY3PR01MB1738:|SYCPR01MB5248:|SYBPR01MB4362:
X-MS-Oob-TLC-OOBClassifiers: OLM:6790;OLM:6790;OLM:6790;
X-Microsoft-Antispam-Untrusted: BCL:0;
X-Microsoft-Antispam-Message-Info-Original: =?us-ascii?Q?iA2wyOTRoRwZLE3dlGXOPjuc/V5nlVGk0qx3c3cVWVq6DeIGm1BllHTCVSiP?=
 =?us-ascii?Q?WdgykixTEOiu5qMNLlDbyvS2B3E3qFunMZOmsip1wW+2+EYpZWRl3h3p1qMe?=
 =?us-ascii?Q?gyHb+gYsWcLLLO/d2oyvKNCusbLDzM+doRdC675aXRpQLJVN0nkIvoeTlN8q?=
 =?us-ascii?Q?uAxB3OZWclI1iz0LWyGk7q7bEL0cf1B7+L5gEvxv+6ccBVjGxDCUARbDvLnH?=
 =?us-ascii?Q?l0az2fYOYGteZOTgZkL/QA4P+UdTnL/ZJFD524SawSpoIipAp4/jd7q4HCdF?=
 =?us-ascii?Q?ii+IYIBvHSb+Ib07PQyBHWQZWcetnQvin6wKe4zFmVxal3O331OUhADCrNX2?=
 =?us-ascii?Q?dABvHfIW5pde9NNG+gtPKa4839nANzsibjIrC4S5/myohU3D0tfQNc21sV9c?=
 =?us-ascii?Q?X8ajwUqsg8oiwXaJEXeZqyS9+em0DhmjpML3/H8aLUa9krvS1hjk/T/95gaU?=
 =?us-ascii?Q?9JlTkvW7otLXXP3scvj9VPvP45JDQ0T4/UV+tS8NciviPlgMPcPHSck6tAE1?=
 =?us-ascii?Q?Bbvs5/mIP8ExRjpHLOrAP1bnFdotp7LAeP9zDvnQ4noFncY7VDCGA7aRAYvp?=
 =?us-ascii?Q?gzi5CYKQc6W7E7vMJaQ4poq567KhRJOgBiLy7/P6LwTaGAQDkCJtydjtCdMX?=
 =?us-ascii?Q?D3gaOUKUGptP07VQXA5kSOqYbXIIzkLuMAzUMpdJ9sUMCuYqZntL7oEJs5Vz?=
 =?us-ascii?Q?7ya/OmOTaSGyWqXURw6jyt7s3sPxXRwZYrszcZx/fTi7CpgMzhgNsVvlyplQ?=
 =?us-ascii?Q?DxgEO7zetqixpz+MfGCpO4oqxlnEaBAcLf7ml8A/UsavMy3MCnghUtFKfZzy?=
 =?us-ascii?Q?C2fXr0H5Q7dm0GS2UA4UwIwGx27nU2ah2542blE+41kzfT2yj0Pya36mePgi?=
 =?us-ascii?Q?Eie4ZoX7lxJ5Ob1l5b6g4Fn6cd2iOVwZw8ldz7TVZEus/r9HbKD96H7E+0gO?=
 =?us-ascii?Q?Jzr37suMHpaKJdTwrQUvXPff7rt3NhglfVoFFc0YaEiJqUKK8YvoyDDqriBg?=
 =?us-ascii?Q?7mSl3nudaNDXcsrZwMH39Ee8glstaFgu3Iht1G5bXjIjzAnu4d8DAONd/Vuw?=
 =?us-ascii?Q?lmIf3jkbeczjusoRukL7qLwFHx766qhLVbNT5ublsgPn8Ed2qUeJcPawb8gf?=
 =?us-ascii?Q?SYCLxnFICDvNWpMogjNuqJr1Vl6dJWtQRruNF0kUNm4rlJnuVXtQ0+B4eeyL?=
 =?us-ascii?Q?Hi2Xi4zvRHnA4EezF7lkDe4XE4/XN74zr1l2KXbTC2qFyAX1UUuKWST9INZU?=
 =?us-ascii?Q?bzlyIW+f4GylBSwuFknr?=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SY3PR01MB1738
X-OrganizationHeadersPreserved: SY3PR01MB1738.ausprd01.prod.outlook.com
X-CrossPremisesHeadersFiltered: Exchange2013.domain.local
X-OrganizationHeadersPreserved: Exchange2013.domain.local
X-CrossPremisesHeadersFiltered: SY3AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-MS-Exchange-Transport-CrossTenantHeadersStripped: SY3AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-Forefront-Antispam-Report-Untrusted: CIP:203.54.134.98;CTRY:AU;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:Exchange2013.domain.local;PTR:abi3057722.lnk.telstra.net;CAT:NONE;SFTY:;SFS:(36756003)(336012)(2616005)(44832011)(6916009)(15974865002)(8676002)(1096003)(36906005)(2160300002)(108616005)(86362001)(5660300002)(53546011)(166002)(26005)(33964004)(24736004)(81166007)(356005)(83080400001)(19627405001)(66574014)(19607625011);DIR:INB;SFP:;
X-MS-Office365-Filtering-Correlation-Id-Prvs: 88288457-f197-4473-a57d-08d7ffc91d97
X-Microsoft-Antispam-Untrusted: BCL:0;
X-Microsoft-Antispam-Message-Info-Original: =?us-ascii?Q?HDO3W+nr056uwZuOZ7TYTG2KW9HPxRSExlC/7yQL77Lh8xd5toPU9ZMXlt+/?=
 =?us-ascii?Q?q8S3K0REYizRMHkthA3wNJb6MU4vgHvsPbRJNHjBeJw08xJepEx7YRJklYdx?=
 =?us-ascii?Q?nSegk5JXfkz3FnYHpVhO1t7qoKwwcMFHifB69QFoA0V6ggrIjDJEwNq27sTb?=
 =?us-ascii?Q?u2vrAss33OY+1dUUp1GiiBZzT4DpL16QotqSIEfiRFWWXS/KqbwJEJ0+wp7l?=
 =?us-ascii?Q?ujFm3W1ndL5XWmRlztxS3+/E4mHNVqAj0hgoUu97adO/E0+oqbYn61mqNOH0?=
 =?us-ascii?Q?FkDij6ZoUw3XFYNu0hTpsUoGBEWCu9aX1/cz2TVmAGN5DBXfYI9+ZgLeXOF2?=
 =?us-ascii?Q?7/5m+E9c6zQLpPRWNj+0q821bxNzE4yhUB+C+K5PNqCpg3YAd2kws7e2+HjT?=
 =?us-ascii?Q?ybEvvGfrX/JVlhTA+5yK0YQwNm0caLgx/BTKx0dEFUoHGaDqg0LRQlnEXr9y?=
 =?us-ascii?Q?zPftt3dapzVbKQdVp15Ae1ErOleQLDRZEpQ/m50pQji37iHH/jpf84LthOQB?=
 =?us-ascii?Q?NX/oUDEZsyj83RowWP6sSAZ7Z9NFSoFKkubvDqMeXflQbW3vz+oL/cJfuSoZ?=
 =?us-ascii?Q?eC12mwEvYibjhWqUsYB5NnSQ1DmvW/WGZNioxbinDOlxrclSm/g7RwKdkkl1?=
 =?us-ascii?Q?NADfKmk3dN5roljcmd/MrBszuC4nigH7USGCCVMDTPSmoP9I+TAPhhKaRP26?=
 =?us-ascii?Q?LNQh5Fe50kxOxwdUGSDoQTr56sUkaGEIsoiP/ue4PMRRXZMjPGTbCI6eCRde?=
 =?us-ascii?Q?hdfTd9fMIeSBjKhNXDtc5P63e9fhp8SXVeT5T4MycPOs+VKEgL9K8cassjbr?=
 =?us-ascii?Q?KmbSdhp00ftDoi9Q8of6jV/Ve6CSYgFIqi9MYd5EUwGOYTK6hcAiU4B1oe5G?=
 =?us-ascii?Q?sVMksEY0SMKTcHn3gnQVuWRXKpjDMU5nYVh2SklhVa1sgo2SR+9i2dDw8Wsd?=
 =?us-ascii?Q?TVt3bMWhm33WMVi79TZla9z3DOqzS9HRZ8lpJKzHS/iO3KMDPq1GwryMRu63?=
 =?us-ascii?Q?PVjn+KYYvEs+l1c5/Qbgb64uwjoL+VZ76x4E4YvR6WaSScFLtuqs6pwAzoYB?=
 =?us-ascii?Q?hFRxmXcdjdnPGh3klPWyqBARJOlKMTR3tzcA7g6MFm7wp4lmKh7PwwPLJIkA?=
 =?us-ascii?Q?QVIN591rFcQ8RQIj47kf5BaJRj3GoosyZsnilb8L2wIwRiQjSsmUmk+cqQ3G?=
 =?us-ascii?Q?M3UtTLIA53MQk6UL1Z1Sd9iXZDdZwY8p9r7vi5lPWmtjQ9Aeo/AomnwY71N3?=
 =?us-ascii?Q?eZ6E2tP8thFY//uSKB7l?=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SYCPR01MB5248
X-OrganizationHeadersPreserved: SYCPR01MB5248.ausprd01.prod.outlook.com
X-CrossPremisesHeadersFiltered: Exchange2013.domain.local
X-OrganizationHeadersPreserved: Exchange2013.domain.local
X-CrossPremisesHeadersFiltered: ME1AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-MS-Exchange-Transport-CrossTenantHeadersStripped: ME1AUS01FT014.eop-AUS01.prod.protection.outlook.com
X-Forefront-Antispam-Report: CIP:203.54.134.98;CTRY:AU;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:Exchange2013.domain.local;PTR:abi3057722.lnk.telstra.net;CAT:NONE;SFTY:;SFS:(24736004)(36756003)(36906005)(19627405001)(26005)(53546011)(5660300002)(33964004)(108616005)(44832011)(6916009)(15974865002)(2616005)(86362001)(1096003)(166002)(336012)(356005)(66574014)(83080400001)(8676002)(81166007)(19607625011);DIR:INB;SFP:;
X-MS-Office365-Filtering-Correlation-Id-Prvs: 2bb48326-def0-492c-8752-08d7ffc935e8
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?vyA9jQ5YP4tbiJQ1N2ff25VgnM+SJiFhE3Yln6tQVaileyKRkO+BuzPHYj6Q?=
 =?us-ascii?Q?R+XgxcBUeNcGuLQOtJZJD6XEXu0UbScHjmvFzBtKM6xT7AC9PD1aTcQ364rJ?=
 =?us-ascii?Q?c8SxdVfBKMuOWjAKK/SHGyuZdAvObLEbgloQS/lG3UEJkCUTEHm8n2WqXggb?=
 =?us-ascii?Q?IZiBNwqd4iWWStR+UyIM/h3t/TIToNx5FHbPD4fPAvZG/L+lCasiGPkaw7sf?=
 =?us-ascii?Q?ZQJfKrllu9oRIjrxw4NYTtDfQdr2OJ+IA606UhprZ7tmrlBIoid+NsFDXB62?=
 =?us-ascii?Q?G/9VH6IvGRNxA445F6764dIyXifRsuaGYs/T11WSIzAJ/5ArCemQuAf+1Ek6?=
 =?us-ascii?Q?KyhsNPSmh9+rGGmNuvV5RXh2E4aqDhZWS6ZGK1aMSTXO6KHeQd+FrEZHM3nf?=
 =?us-ascii?Q?Xl5OdF6OPKdLlMv1nEa1UVpMi52rkMvxXp7DRmWxspsi9JkIhlU/OBdMjOmC?=
 =?us-ascii?Q?Us+aJLNXT9O/JIc/7mPwNE3e8+Hj9zIHxNIcrYelEoopTGXSmcdTP0eOslts?=
 =?us-ascii?Q?Ap1JQ0sEeODnnNs4Z0r2NjzrHOOKRY8YjLhXz4vE+iWzzMhLUgW+17Lc9cHG?=
 =?us-ascii?Q?/z9uQm800gOM7k6By+jhkxzM8Ho49Uin65W3YMI31VqWQzTgg6kBxoYgNqls?=
 =?us-ascii?Q?VylLdyHSa8rjX3n31TNqzFxN7DHZRLL+0Ar/53pw9yj6YETYgoBZX8b4zQCD?=
 =?us-ascii?Q?KZl6GY+7WSjn1MlecYlPAlCCY3rSkwPWGaP5nb1OMPryBPa/hCna5UQJpQK0?=
 =?us-ascii?Q?NKCSLmg374I4/+jrylQ/fR1upcDBJwDfX0c28v3kzCxv1wlGwWK4LSn5SWrM?=
 =?us-ascii?Q?GZOW6Cc3K+/G/prlpPrCPsRHdNNGVaGWHqoaaFrLgnFeJQiKDu+afKXAaSem?=
 =?us-ascii?Q?yqCF551HYU+ItucZDwzF2L2yAu0hX3PZohye+tT4ErFrGAtYrZdAWHr23wL4?=
 =?us-ascii?Q?+LH54epp83Xrgl0aZm4N1XomI0G94LY+vBQiNK3W075GoK2bhNvYYmVGR03q?=
 =?us-ascii?Q?w0uD8TzzRXd7MSOC6nwwKCJDiovNYq3ms4AsY4B/t4UQyf/mhTr9h24yuBCa?=
 =?us-ascii?Q?1GUkuH/ta2m/4NfpUefV0dlBJtCS1QWEZsCClWHo0UPUb+PwtPNO7atyR4vj?=
 =?us-ascii?Q?hmIkERJe2CYWSVa3B8mKM+Md70ZSPNIdSCHM+FXhM28DAntZmBt3Ox8zVl5Y?=
 =?us-ascii?Q?2yDsCXoYhGM9UByN5JXDkMe3WwH14Mm++895YPqAMT9u5Zd8OeWAw7WztPC3?=
 =?us-ascii?Q?RDfJkidLJlLE5XVEABX7?=
X-OriginatorOrg: contoso.onmicrosoft.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2020 10:03:53.9414
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 8134a206-5e1e-4c8c-50b8-08d7ffc9c425
X-MS-Exchange-CrossTenant-Id: 7e0e266c-0475-408b-9eb7-cb0cf8b31e59
X-MS-Exchange-CrossTenant-FromEntityHeader: Internet
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SYBPR01MB4362
X-OrganizationHeadersPreserved: SYBPR01MB4362.ausprd01.prod.outlook.com
X-CrossPremisesHeadersFilteredByDsnGenerator: SYBPR01MB4362.ausprd01.prod.outlook.com

Reverting the change we needed to troubleshoot.  With Centralized Transport disabled on the Outbound connector we saw that any emails passing from on-premises to Office 365 had in the header:

X-MS-Exchange-Organization-AuthAs: Anonymous

Office 365 should be receiving any emails from the on-premises Exchange Server as "Internal" if they are in the same tenant.  This means the emails were not hitting an Inbound Connector in Office 365 and were coming into Office 365 as anonymous.  This can also cause the SMTP error "451 4.7.500 Server busy" for very large tenants as Microsoft throttles emails from anonymous sources to limit spam in Office 365 - for more information see .

After researching the issue and a call with Microsoft, we saw that there was a problem with the Certificate configuration on the Inbound connector.  The common name on the certificate - lets say "mail.contoso.com" was correct on both ends however the Organisation Name on the certificate was different which was enough to cause inbound SMTP email from on-premises to be not identified by the Inbound Connector and get flagged as anonymous.


To fix this issue, we needed to modify the Inbound Connector from having multiple attributes of the certificate as shown below.


To only having the "Common Name" of the on-premises certificate which for argument sake lets call it "mail.contoso.com".


After fixing the certificate details on the Inbound Connector, emails from on-premises to the cloud were identified as Internal.  This can be verified by looking in the message header in one of the emails.

Note: You need to wait an hour for any changes to Exchange Online to propagate to Exchange Online Protection unless you speak to Microsoft Support, they have a script that forces it on the backend!

830fecf9-8ece-4d66-2536-08d81e5f643a
X-EOPAttributedMessage: 0
X-MS-Exchange-Organization-MessageDirectionality: Originating
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 04
X-MS-Exchange-Organization-AuthSource: Exchange2013.domain.local
X-MS-Exchange-Organization-SCL: -1
X-CrossPremisesHeadersPromoted:
SY3AUS01FT006.eop-AUS01.prod.protection.outlook.com
X-CrossPremisesHeadersFiltered:
SY3AUS01FT006.eop-AUS01.prod.protection.outlook.com

After this, we re-instated the Centralized Mail Configuration and it worked perfectly.

Tuesday, June 30, 2020

Hybrid Configuration Wizard unable to Validate Hybrid Agent for Exchange Use

When running a "Modern Exchange" Hybrid Deployment with deploying Hybrid Agents for a customer, we ran into an issue where the Hybrid Configuration Wizard (HCW) failed on "Validate Hybrid Agent for Exchange usage".

Symptoms of Issue


In the Hybrid Configuration Wizard log we had the following error:

FINISH Time=320.5s Results=Failed The connection to the server 'CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error: (503) Server Unavailable.., The HTTP service located at https://CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the sevice is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later., The remote server returned an error: (503) Server Unavailable.
2020.06.11 07:18:48.539 *ERROR* 10349 [Client=UX, Page=HybridConnectorInstall, Thread=8] 

The connection to the server 'CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error: (503) Server Unavailable.., The HTTP service located at https://CUSTOMER-APPID-GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later., The remote server returned an error: (503) Server Unavailable.
2020.06.11 07:18:48.550         10390 [Client=UX, Page=HybridConnectorInstall, Thread=8] Test duration 00:05:20.5283461
2020.06.11 07:19:37.075         10062 [Client=UX, Page=HybridConnectorInstall, Thread=1] [Close/Cancel] Pushed: True
2020.06.11 07:19:37.092         10266 [Client=UX, Page=HybridConnectorInstall, Thread=1] FINISH Time=369.1s

When HCW installs the Hybrid Agent on a server, it also looks at the URL set on the EWS virtual directory and sets this in the Azure AD Application Proxy cloud.

The value set on the EWS virtual directory internally will look something like this:

"https://FQDN/EWS/Exchange.asmx"

HCW will strip off "EWS/Exchange.asmx" and set the value in the cloud as:

"https://FQDN/" for example... https://mail.contoso.com/

This is what Hybrid Agent utilises by default for all communication with on-premises Exchange.

When the Hybrid Agent service starts for the first time, it downloads its configuration from Azure AD Application Proxy cloud through a  JSON Graph call.  This is also shown in the Hybrid Configuration Wizard log and resulted in "Application_NotFound".

2020.06.16 05:03:31.533         10332 [Client=UX, fn=SendAsync, Thread=23] START GET https://graph.microsoft.com/edu/d2a627a8-8704-4164-88d6-1355bb35ee0e/applications/9062c321-7c9b-415e-ab46-a19e92ebdf68/onPremisesPublishing
2020.06.16 05:03:32.604         10333 [Client=UX, fn=SendAsync, Thread=23]
                                      FINISH Time=1070.9ms Results=NotFound {
                                        "error": {
                                          "code": "Application_NotFound",
                                          "message": "Application '9062c321-7c9b-415e-ab46-a19e92ebdf68' not found or OnPremisesPublishing is not enabled for your tenant.",
                                          "innerError": {
                                            "date": "2020-06-16T05:03:32",
                                            "request-id": "a6b5704d-09ef-4def-a473-c1f82ae835b7"
                                          }
                                        }
                                      }

Utilizing the HybridManagement.psm1 PowerShell Module we can also manually check this with the Get-HybridApplication cmdlet.


Note: The HybridManagement module is located under ‘C:\Program Files\Microsoft Hybrid Service\HybridManagement.psm1’ and can be loaded with...

Import-Module Azure
Import-Module ‘C:\Program Files\Microsoft Hybrid Service\HybridManagement.psm1’

We were able to validate requests from "Exchange Online" to "Azure AD Application Proxy" to the "Hybrid Agent" were working correctly.  This is validated using the PerfMon counters.  This is done by performing the following.

1. Open up "perfmon" on the Hybrid Agent Server
2. Add the "Microsoft AAD App Proxy Connector" #requests counter.


3. Connect to Exchange Online via PowerShell
4. Run the Test-MigrationServerAvailability cmdlet to force Exchange Online to validate the migration endpoint (same test the Hybrid Configuration Wizard is attempting to perform).  We can see it fail.


5. The test hits from the Exchange Online were connecting to the Hybrid Agent and we were able to validate this by watching the #requests increment each time the Test-MigrationServerAvailability cmdlet attempted the connection and failed.  The Test-MigrationServerAvailability cmdlet attempted the test 6 times before failing resulting in the #requests incrementing from 128 to 134.



What this validates is the requests from Azure AD Application Proxy are hitting the on-premises Hybrid Agent but failing to get to the MRSProxy endpoint on Exchange.


On the Hybrid Agent server itself we were able to access the Exchange Cluster nodes through the load balancer via https://LoadBalancedFQDN/owa and also access the MRSProxy endpoint in Internet explorer with https://LoadBalancedFQDN/EWS/mrsproxy.svc (it throws a bad request but as expected as Internet Explorer will not know how to interact with the MRSProxy web application).  This validated network communication on TCP443 and TCP80 was working correctly.  We also validated that TCP5985 and TCP5986 were open as per Microsoft documentaiton:

https://docs.microsoft.com/en-us/exchange/hybrid-deployment/hybrid-agent

Issue Cause

After working with the product team we were able to identify the issue.  The customer had a free open-source load balancing solution called HAProxy in front of an Exchange 2016 cluster with multiple nodes.  All requests from Hybrid Agent were hitting the HAProxy as this is is what that the DNS record was pointing to set on the EWS virtual directories which Hybrid Agent discovered (as documented above).



As part of troubleshooting, the product team showed me how to manually overwrite which internal endpoint the Hybrid Agent connects to using the HybridManagement.psm1 PowerShell Module.

To see what the InternalUrl the Hybrid Agent is connecting to, use the Get-HybirdApplication cmdlet.  You will need to put the Application ID "CUSTOMER-APPID-GUID" into a variable along with some Global Administrator credentials in a variable.  This is the thing in front of ".resource.mailboxmigration.his.msappproxy.net".
  • The externalURL points at the URL that Exchange Online hits via Azure AD Application Proxy.
  • The internalURL points to the internal URL of the on-premises Exchange servers.  This was pointing at the HA Proxy load balance.


We updated the HybridApplication to bypass the HA Proxy Load Balancer and hit one of the back-end servers directly.  This was done with the Update-HybridApplication cmdlet.


After manually updating Hybrid Agent to communicate directly with one of the backend servers instead of HA Proxy, we used the Test-MigrationServerAvailability cmdlet from Exchange Online.  

Test-MigrationServerAvailability cmdlet was successful!


Very interesting point however which I do not understand yet... When we we manually hit https://LoadBalancerFQDN/EWS/mrsproxy.svc using Internet Explorer from the Hybrid Agent server, we can see the hit in the IIS Logs... however when Hybrid Agent attempted to reach the same URL from the same server, the request never reached the IIS Logs.

What we have done is isolate the issue to the free open-source load balancing solution HA Proxy...

What is interesting though is all other Exchange Load Balancing traffic is transversing the HA Proxy without issues including:

  • MAPI over HTTPS
  • RPC over HTTPS
  • OWA
  • ActiveSync
  • OAB Distribution
  • Other EWS calls such as Free/Busy and the Availability Service
It is very interesting that the MRSProxy call from Hybrid Agent never reaches the IIS Logs on the backend server when going through HAProxy!

Thursday, May 21, 2020

Unable to Install Azure Powershell Module

I had an issue where i was unable to install the Azure PowerShell Module on Windows Server 2016.  When attempting to install the module the following error was experianced.

PackageManagement\Install-PackageProfider : No match was found for the specified search criteria for the provider 'NuGet'. The package provider requires 'PackageManagement' and 'Provider' tags. Please check if the specified package has the tags.


To fix this issue i needed to enable Strong Crypto on the .NET framework to ensure TLS 1.3 was supported.  This was done with the following commands.


Set-ItemProperty -Path 'HKLM:\SOFTWARE\Wow6432Node\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type Dword
Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\.NetFramework\v4.0.30319' -Name 'SchUseStrongCrypto' -Value '1' -Type Dword

Before:


After:



After making this change, we were successfully able to install the Azure PowerShell module.


Tuesday, May 5, 2020

Direct Access not working with Windows Firewall Enabled on Client

We had an interesting issue with Microsoft Direct Access on Windows 10 latest build 1909.

  • When Windows Firewall is disabled, Direct Access works on the client.
  • Enabling Windows Firewall on the client breaks Direct Access resulting in no connection.
Symptoms of Issue

Here is the symptoms we were receiving:

Get-DAConnectionStatus returned "CouldNotContactDirectAccessServer"



netsh interface httpstunnel show interface shows the Interface is Active

IPHTTPS interface active
Last Error Code 0x0



The issue appears to be with Name Resolution Policy Table (NRPT).  When Windows Firewall was enabled on the client, you were able to ping IPv6 addresses on the Direct Access server through the tunnel but all name resolution failed.

For example, you can ping the tunnel endpoints discoverable using the Get-DAClientExperianceConfiguration cmdlet on the client.  You can also ping IPv6 addresses of hostnames previously resolved when the Windows Firewall was in a disabled state.  For example, whilst Windows Firewall was in a disabled state, ping domain.local (your AD forest name), copy the IPv6 address, re-enable Windows Firewall and you will notice you can resolve it by the IPv6 address, not by hostname using NRPT.

Resolution

The issue was caused by Windows Firewall set to a disabled state on the server.

What we found was when the Windows Firewall is disabled on the server, Windows Firewall must be disabled on the client.

If DA Server Firewall is disabled, but DA client Firewall is enabled - Direct Access breaks.
If DA Server Firewall is enabled, but DA client Firewall is disabled - Direct Access breaks

You must have Windows Firewall disabled on both the DA Server and DA Client, or enabled on both the DA Server and DA Client.

We also identified another issue on the DA server with regards to Windows Firewall being disabled.  In our environment, disabling Windows Firewall on the DA Server breaks Direct Access reporting in the "Remote Access Management" console.

As a result, it is strongly recommended to keep Windows Firewall enabled on both the DA Server and DA clients.

Friday, November 1, 2019

Patching old Exchange 2013 or Exchange 2016 servers

When patching Exchange 2013 or Exchange 2016 from an old Cumulative Update version, it is common to need to hop through multiple versions of Exchange Server.

For example, if your on Exchange CU11... to remain in a supported state you will need to patch as follows:
  • Upgrade Exchange 2013 from CU11 to CU15.
  • Upgrade the .NET Framework to 4.6.2
  • Upgrade from Exchange 2013 CU15 à CU20
  • Upgrade .NET Framework to 4.7.1
  • Upgrade from Exchange 2013 CU20 à CU22
  • Upgrade .NET Framework to 4.7.2
  • Upgrade from Exchange 2013 CU22 à CU23
See the support matrix here:

https://docs.microsoft.com/en-us/exchange/plan-and-deploy/supportability-matrix?view=exchserver-2019



The problem that most people on old patch levels find, is Microsoft has removed many of the older CU updates which makes moving from an old patch level a difficult task.

Luckily we have a legend that has posted all the Exchange updates here:


Very handy link to keep in your favorites.

Wednesday, September 25, 2019

Basic Authentication being Disabled in Exchange Online

On the 13th of October 2020, Microsoft announced they are turning of Basic Authentication across all protocols in Exchange Online apart from SMTP.  Basic Authentication will be turned of on all web services on 13th of October 2020 including POP and IMAP.  This was published here:

https://techcommunity.microsoft.com/t5/Exchange-Team-Blog/Improving-Security-Together/ba-p/805892

Microsoft are pushing people to use Modern Authentication (OAUTH2) which provides numerous advantages over basic authentication.  Basic Authentication is secure provided it is encrypted with TLS and has been used since Exchange 5.5 and is still heavily used now even in Exchange Sever 2019, however there are more secure ways which provide support for additional security such as Multi Factor Authentication (MFA).

I have issues with this announcement - many customers that have enterprise applications which connect to Exchange via basic authentication using POP or IMAP4 over TLS - and this is the only connectivity option these applications support.

Microsoft say in the announcement that they know this will cause potential disruption but they want to force companies to adopt the new authentication technology.  I have many customers with help desk ticketing systems, ERP solutions, document management, life-cycle management system etc that only support basic authentication.  Not to mention, as of 26/09/2019 Microsoft still doesn't even support Modern Authentication on POP or IMAP (commonly used for applications to connect) and say in their article "we are planning on adding OAuth Support to both POP and IMAP in next few months".  Great - gives application vendors lots of time to prepare!

The benefit of Exchange Server on-premises is you can control your own destiny and if you have applications you have invested 10 million+ into developing or rolling out, you wont expect that your cloud vendor will suddenly flick a switch and cause your application to stop functioning correctly.

If your using Basic Authentication in O365 - and I know many of you reading this article would be in some extent (most likely mobile phones) - make sure you address this, install the Outlook for Mobile application, upgrade your enterprise applications to ensure your ready for this significant change.

Monday, August 26, 2019

Preparing Exchange Topology - PrepareAD, PrepareSchema, PrepareDomain etc

I'm about to sit the Exchange Upgrade Exam (MS-202: Microsoft 365 Messaging Administrator Certification Transition) on Friday and currently doing a last minute brush up.  Having done all the certifications in Exchange Server since MCSE: Messaging 2003 all the way through to the latest Exchange 2016 exam (70-345) I know the types of Questions Microsoft has asked in the past.

One area tested that I recall from all previous exams was the process for preparing schema and domain...

Whilst the latest Exchange 2019 Exams are definitely going to be heavily focused on Exchange Online in Office 365 given this is Microsoft's primary drive, they also test on all the on-premises content.  I wouldn't be surprised if topology preparation is tested once again... even though in my professional opinion it is not an item of significant importance given that over 99% of businesses simply rely on the Cumulative Update wizard to automatically extend the schema and prepare the domains.

In Exchange 2003 we had the setup.exe /forestprep and setup.exe /domainprep.

From Exchange 2007 all the way to Exchange 2019 we have a number of commands now including:
  • Setup /PrepareAD
  • Setup /PrepareSchema
  • Setup /PrepareDomain
  • Setup PrepareAllDomains
I remember there use to be a fantastic article on TechNet which gave a breakdown of exactly what each of these commands did - however after spending a good 5 minutes on Google i came up short trying to find the article and not sure if it is still published.

I could find no "clear" breakdown of each of these commands and the descriptions given on the installer help is useless as shown below:


I did come across the book however Exchange Server 2010 Administration: Real World Skills for MCITP Certification and Beyond (Exams 70-662 and 70-663) published by Joel Stidley and Erik Gustafson that touched on these commands in more detail.

Given the lack of content covering these commands, I decided to do a quick blog post.

Setup /PrepareSchema

This command does one thing, prepares the schema (additional class objects and attributes required for Exchange Server).  It must be run in the same Active Directory site as the Schema master.

To run this command you must be a Schema Admin.

Setup /PrepareDomain

This command must be run in each domain within an Active Directory forest.  This command simply creates special domain accounts and security groups in each domain for hosting Exchange Servers.  Thing of it as creating some additional "Active Directory" objects, no schema extensions within the "Domain Partition" only.

To run this command you must be an Enterprise Admin.

Setup /PrepareAD

The PrepareAD command performs three things:
  • Prepares the Schema if not done already (same as PrepareSchema)
  • Prepares the Domain (for the forest root domain only in a multi-domain forest)
  • Creates the Global Exchange Objects in the Configuration Partition.
It is important to note, PrepareAD runs the PrepareSchema command for the forest and PrepareDomain for the forest root domain only (if not done already).

For a single forest, single domain environment - PrepareAD is the only command you need to run.

If you have multiple child domains or new tree domains in the same Forest, after you run /PrepareAD in the forest root domain, you will need to /PrepareDomain for each of the additional domains within the forest.

To run this command you must be an Enterprise Admin and a Schema Admin.

Setup /PrepareAllDomain

If you have multiple domains in an Active Directory forest and you wish to run /PrepareDomain across all domains at the same time, this is what the /PrepareAllDomains command is for.

Hopefully this post has been useful.

Exchange RBAC Example - Provide User access to manage Contacts

In June 2010, I wrote an article explaining how Exchange Role Based Access Control (RBAC) works - a new feature released with Exchange 2010.  RBAC is still heavily utilised today with Exchange 2019 and Office 365 following these principals.

My article from 2010 can be found here:

https://clintboessen.blogspot.com/2010/06/exchange-2010-role-based-access-control.html

After having not worked with RBAC for while, I found myself re-reading the principals of RBAC and re-running through what is involved to configure the security model.  Knowing this well is also essential and I'm doing my Exchange upgrade exam this Friday (MS-202 Microsoft 365 Messaging Administrator Certification Transition) so a refresher on these principals is always handy!

I had a requirement given to me by a customer which I need to spin up in my test lab, and I thought whilst I lab the requirement it might be a good idea to write a quick blog post on the process of implementing the RBAC changes.

RBAC Requirement

"A user in the business must be able to create mail enabled contacts for external workers.  This user must only be able to create mail enabled contacts and no other objects, and the contacts must be stored under a specific organisational unit only".

RBAC Security Model

With the design of RBAC, The Exchange Product Team referred to RBAC as the Triangle of Power.  This is elaborated in this blog post here:

https://techcommunity.microsoft.com/t5/Exchange-Team-Blog/RBAC-and-the-Triangle-of-Power/ba-p/597147

The security model has three arms:
  • Step 1: Where - Where are the objects stored that we want to change or what attributes or identifying factors must the object have?  They can be in a security group, organisational unit, anyone with a job title set to XYZ... the possibilities are endless.  My requirement for "Where" is contacts must be contained only under a particular organisational unit.
  • Step 2: What - What is what you want to do.  What Exchange PowerShell cmdlets do you want to give the administrator access to be able to run?  Think of "What" as the access rights you wish to delegate.  The commands we are interested in for my example are New-MailContact and Remove-MailContact.
  • Step 3: Who refers to who is able to perform the operation.  Which user do you want to delegate control to in order to run the commands under the What section.
This all glued together by a concept known as Management Role Assignments.



I will break this down using my three headers listed above.

Step 1: Where

In my lab environment we only want contact objects to be changed under the OU "Contacts".

avantlab.local/Contacts

As a result we are going to create a new "Management Scope".  By default all RBAC Management Roles have access to the entire Active Directory forest and no Management Scopes exist. 

I'm locking it down to the Organisational Unit based on a RecipientRestrictionFilter with the following query:

New-ManagementScope "AvantLab Contacts" -RecipientRestrictionFilter {DistinguishedName -Like '*,OU=Contacts,DC=Avantlab,DC=Local"}



Step 2: What

Now we are interested in assigning "What" cmdlets we want the admin to be able to run.  The "What" refers to two cmdlets:
  • New-MailContact
  • Remove-MailContact
Cmdlets are linked as "ManagementRoleEntries" to objects known as "ManagementRoles".

There are a few rules you need to understand when creating new Management Roles for the "what section":
  • It is not possible to simply create a new Management Role and add the cmdlets you wish to run under the role as Management Role Entries.  All custom (non default) Management Roles must be linked to a parent Management Role.  Parent Management Roles are ones that come by default with Microsoft Exchange Server "out of the box".  Think of it as your "cloning" the parent Management Role to create your "Custom" management role.
  • It is not possible to add additional cmdlets that were not in the parent - you can however remove cmdlets.
Base on these two rules, when your creating a new Management role your essentially taking a management role similar to what your trying to create "with too much access", and removing the additional access from the role.

So the first step I need to do is find an existing Management Role that contains the two cmdlets I'm interested in.  I did this by running the following cmdlets:

Get-ManagementRole -Cmdlet New-MailContact
Get-ManagementRole -Cmdlet Remove-MailContact


A Management Role that contains both these cmdlets is "Mail Recipient Creation" - so this is the Management Role we will use as the template for creating my new Management Role.

I'm calling my new Management Role "Contact Management" and I'm basing it on "Mail Recipient Creation".  As a result, the new Management Role was created with the following cmdlet:

New-ManagementRole -Name "Contact Management" -Parent "Mail Recipient Creation"


Now the default "Mail Recipient Creation" Management Role had more cmdlets then we want the delegated access users to have access to.  To list all the cmdlets that this Management Role can run, execute the following command:

Get-ManagementRoleEntry "Contact Management\*"


We want to remove all cmdlets not related to our Contact Management.  Most importantly we want to remove any cmdlets with "New-, Set-, Start-, Remove-, Disable-, Write-, or Remote-" as they have elevated access.  "Get-" cmdlets you cant make changes, you can only view data.

To remove the unwanted cmdlets from the Contact Management Management Role the following cmdlet was used:

Get-ManagementRoleEntry "Contact Management\*" | where {$_.name -eq "New-MailUser"} | Remove-MagementRoleEntry




And a few Get- commands I wanted removed:



This left the Contact Management role with the following cmdlets associated:

Get-ManagementRoleEntry "Contact Management\*"

This left Management Role access to run the following cmdlets.  The "Get-" commands present cannot make any changes or expose any information that I would not want the delegated user to see.  As a result I didn't clean these up but you can if you want.  Most importantly, the New-MailContact and Remove-MailContact cmdlets were left present.


Step 3: Who

The last step is the "delegation of control", which staff members will have access to be able to run the Cmdlets in the Management Role "Contact Management".

To determine the "Who" I created a new Role Group called "Contact Management Admins" defining the "Contact Management" Management Role and the "AvantLab Contacts" Management Scope.  This was done with the following command:

New-RoleGroup "Contact Management Admins" -Role "Contact Management" -CustomRecipientWriteScope "AvantLab Contacts"


This creates a new Group under Microsoft Exchange Security Groups in Active Directory for the new Role Group.



Testing the RBAC Security

I went and added a user called DelegationUser to the Contact Management Admins group.


Logging into Exchange Control Panel (now known as Exchange Admin Centre (EAC) in later revisions of the product), the webpage only renders the areas of EAC the user has access to.  RBAC and being "Cloud Friendly" are the two primary reasons the old Exchange Management Console from Exchange 2007/2010 was retired.


The user can successfully run the New-MailContact command via the webpage and place a Contact object under the Contacts OU - the Step 1: Where? section.


This is shown in the following screenshot:


However if I try and create a contact in the default Users container, I get the message:

'avantlab.local/Users/DefaultUser Container' isn't within your current write scopes.  Can't perform safe operation.


This is because Step 1 "Where"... the management scope is restricted to only the Contacts OU.  Obviously Management Scopes are ridiculously granular and you can go far beyond restrictions of something as basic as an OU.


 What about the Glue?

Stop sniffing the glue Clint, I thought there was Glue in between holding these objects together as shown in the diagram below.

When I created the Role Group I also specified the Management Role and Management Scope.  This automatically created a ManagementRoleAssignment called "Contact Management-Contact Management Admins" as shown in the screenshot below.

Creating ManagementRoleAssignments manually useful especially when you want to create associations between existing objects that already exist on the system.


If we look at the Management Role Assignment "i.e. the glue" closer, we can see that it has all three components created above are "glued" together.
  • The Management Scope... Step 1: Where (CustomRecipientWriteScope: AvantLab Contacts)
  • The Management Role... Step 2: What (Role: Contact Management)
  • The Role Group... Step 3: Who (RoleAssigneeName: Contact Management Admins)


Hopefully this step by step guide to using Role Based Access Control was useful and can be put to use within your environment!