Monday, April 23, 2018

Third Party Transport Agents delaying Emails and Crashing the Transport Databases

I had a customer experiencing extensive delays receiving email through the Exchange Transport stack both for internal and external mail relay.

When we ran the Get-Queue or Get-Message cmdlets, the following error was experienced.

Exchange can't connect to the Exchange Transport service on computer "localhost". Verify that the service is started.


Interestingly, sometimes when we ran the Get-Queue and Get-Message command, the correct output was returned so this issue was intermittent.

During the issue, I noticed that the ExchangeTransport.exe service enters a suspended state.


Looking at the Application Logs on the server, I noticed the Transport ESE Databases (the Sender Reputation and IP Filtering) were constantly being rebuilt.

  • edgetransport (34164) Sender Reputation Database: The database engine is initiating recovery steps. 
  • edgetransport (34164) Sender Reputation Database: The database engine has begun replaying logfile C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\SenderReputation\trn.log. 
  • edgetransport (34164) Sender Reputation Database: The database engine has successfully completed recovery steps. 
  • edgetransport (34164) Sender Reputation Database: The database engine started a new instance (2). (Time=0 seconds) 


Enabling verbose logging on the Exchange server for the Transport Stack we saw that the issue was being caused by third party transport agents.

Get-EventLogLevel "MSExchangeTransport" | Set-EventLogLevel -Level Expert


Log Name:      Application
Source:        MSExchangeTransport
Date:          23/04/2018 3:43:10 PM
Event ID:      10003
Task Category: PoisonMessage
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTERNAME
Description:
The transport process failed during message processing with the following call stack: System.Runtime.InteropServices.COMException (0x800706BE): Creating an instance of the COM component with CLSID {9DE1008F-1451-4E2E-8767-A1DAF9BE3560} from the IClassFactory failed due to the following error: 800706be The remote procedure call failed. (Exception from HRESULT: 0x800706BE).
   at ContentSecurity.ExchangeAgents.MailRouteItemSourceContext.processItem(String sender, ArrayList recipients, IStream pRawData, UInt32& pFlags)
   at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.ProcessMailRouteMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e, Boolean& bDeleteFromSource)
   at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.SubmittedMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e)
   at Microsoft.Exchange.Data.Transport.Routing.RoutingAgent.Invoke(String eventTopic, Object source, Object e)
   at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session)
   at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Disabling the Exchange Transport Agents for GFI and restarting the MSExchangeTransport service resolved the issue.


If anyone experiences these symptoms, please make sure you check your third party Transport Agents!

Monday, April 2, 2018

Azure AD Domain Services Overview - Removing the need for Domain Controllers in Azure IaaS clouds

Late 2017 Microsoft released some very cool technology in Azure called Azure AD Domain Services.  This service provides Azure Customers with Virtual Machines in Azure the ability to use Domain Services such as Kerberos, NTLM and Group Policy lock down without the need for deploying Domain Controllers in the cloud.

It is important to note, Azure AD Domain Services a paid service, once enabled in your Azure Tenancy, you will be billed monthly.  Azure AD Domain Services unlike other cloud services in Azure cannot be stopped or paused, it must be deleted from the Azure Tenancy to avoid further billing.  To understand how this service is charged, please see https://azure.microsoft.com/en-au/pricing/details/active-directory-ds/

What Azure AD Domain Services offers customers is the ability to remove the need for building domain controllers in the cloud and instead utilize Domain Controllers as a service.  You essentially consume these domain functionality as a service without the need to deploy, manage and patch domain controllers in the cloud.

As companies adopt cloud technologies, we want to transition our workloads into Software as a Service (SaaS) offerings.  What I tell my customers is 95% of what your IT department maintains are the applications and services provisioned within the virtual machines.  For on-premises deployments the Virtualization Layer, Servers, Storage and Network typically get updated every 5 years as part of an Infrastructure Refresh project which is generally done by an IT Integrator.  When transitioning to cloud computing, simply moving your virtual machines from your on-premises environment to an Infrastructure as a Service platform does not generally reduce your IT expenditure and for many clients is more expensive in most cases.  For majority of companies, none of what the IT team works on in a daily basis is revolves around the Virtualization, Servers, Storage and Networking layer!

We want to transition as many services as we can onto Software as a Service offerings where we consume a service but have no need to maintain the virtual machines and the applications which run on these.  Office 365 is a prime example of a Software as a Service offering, you consume Exchange, Sharepoint and Skype for Business services without the need for maintaining anything in the back end.


Well you might have guest it, Azure AD Domain Services is a Software as a Service offering for "Domain Services" removing the need for Domain Controllers.

Azure AD Domain Services utilises the user accounts, security groups and data stored in Azure Active Directory to provision domain services.  It acts as a service which communicates with Azure Active Directory in the back-end.  Servers in an Azure IaaS offering or even on-premises via Express Route or VPN Tunnel can utilise Azure Active Directory for authentication and domain services functionality.


If you have an on-premises Active Directory environment and are looking to move some virtual machine workloads up to the cloud, you can also leverage Azure AD Domain Services which I will explain.

Through utilizing Azure AD Connect (the same tool most of you are familiar with for Office 365 migrations), you can synchronize your user accounts and password hashes up to Azure AD which then can be leveraged by Azure AD Domain Services.

Important: You must synchronize password hashes to Azure AD for Azure AD Domain Services to work correctly.  Azure AD Domain Services cannot authenticate against passwords in an Active Directory Environment using Active Directory Federation Services or Passthrough Authentication.

Important: You can only create one managed domain serviced by Azure AD Domain Services for a single Azure AD directory



Setting up Azure AD Domain Services

In this article I'm going to run through the setup process of Azure AD Domain Services.  In order to be able to setup Azure AD Domain Services you must first have an Azure Tenancy and billing setup.  You will also need setup in place a Resource Group, Virtual Network for Azure AD Domain Services to operate on within Azure and ideally a Virtual Network Gateway to provide connectivity back to on-premises.

The step is to find Azure AD Domain Services from the Azure Market place.


Like creating a real Active Directory Domain in a new Forest, specify the DNS Namespace you want for the new Azure AD Domain Services domain.  Also specify the subscription you want to use, the resource group and the Azure Datacentre.

I'm using Southeast Asia (Singapore) as I get the same latency from my Office to Singapore as what I get to the Azure Sydney datacenter - and it's cheaper to put my infrastructure in Asia!


Next specify the virtual network and server subnet you want to use within Azure.  I created this earlier for other infrastructure residing in my Azure tenancy.


Within the Azure AD Domain Services there is a group called "AAD DC Administrators".  This is our traditional "Domain Admins" we are custom to in Active Directory.  Add at least one account which exists in the Azure AD tenancy we are linking to be an administrator.



Give it some time to deploy.  It took almost an hour to provision within my tenancy.


Once it finishes provisioning, you will see it in your list of resources within the Azure portal.  Go to Azure AD Domain Services and note down the DNS servers.  These will automatically be provisioned within the IP range of your virtual network.


Find the Virtual Network resource you have provisioned earlier.


Update the DNS of the Virtual Network to use the Azure AD Domain Services DNS.


After updating the Virtual Network you will need to reboot your Azure VMs on the network.  Remember the golden rule of Azure is never configure static IP addresses on your VM's.  Use the Azure Portal to configure DNS, Gateway settings and IP addressing for everything!

You should see the DNS settings referencing that of Azure AD Domain Services.


Now you can simply just join the member servers to the Azure AD Domain Services managed domain using an administrative account in the "AAD DC Administrators" group.



Administrating Azure AD Domain Services

As mentioned previously Azure AD Domain Services is Domain Services managed by Microsoft.  As a result, it is not possible to login to the AD Domain Services via Remote Desktop like you can do with Traditional Domain Controllers.  Provided you have administrative rights and are part of the "AAD DC Administrators" group in Azure AD you can however connect to the Domain Services using the Microsoft Management Console (MMC) Active Directory snap-ins which I will show you shortly.

On a member server joined to Azure AD Domain Services, simply add the Active Directory Remote Server Administration Tools.


Then you can simply Open Active Directory Users and Computers and query what is in Azure AD Domain Services.  You will be able to see all the objects which exist in Azure AD which are automatically synchronized.  As you see, my Azure AD has lots of objects! - 2 users... :)

It is important that all changes are conducted through the Azure AD portal.  This includes creating new user accounts, changing attributes, resetting passwords, creating/deleting groups and group membership management.  Any changes you make to the objects in Azure AD Domain Services will be overwritten automatically by Azure AD which is the "authoritative source".


Azure AD Domain Services supports full Group Policy functionality and provides member servers with a SYSVOL for Group Policy Objects.


If you install Group Policy Management Console on a member server you can create and manage Group Policy against your users and member computers.  All Group Policy functionality is available in Azure AD Domain Services.


There are domain controller objects in Azure AD Domain Services.  These are automatically generated and maintained by Microsoft.  You must not touch these objects within the Domain Controllers organisational unit.

Important: Encase you were wondering, you cannot add additional domain controllers to a domain running in Azure AD Domain Services.



Limitations of Azure AD Domain Services

There are a few limitations of Azure AD Domain Services that i'm aware of which I would like to share with you.

  • The Schema is administrated by Microsoft for the managed domain.  Schema extensions whilst possible are not supported by Azure AD Domain Services.  If you want to customize the Schema, create your own domain controllers using the traditional method.  This means no Microsoft Exchange, Skype for Business or other applications which change the Schema.  Use Office 365!
  • Any changes made in Azure AD take approximately 20 minutes to reflect in your managed domain.  This is a bit slow and you will need to be patient.
  • There is no way to pause or stop Azure AD Domain Services.  From a billing perspective make sure your aware of this and are fine paying the monthly fee for this service.  It is cheaper then creating your own virtual machines in Azure and configuring them as Domain Controllers.
  • All group and user changes must be conducted through Azure AD, not Azure AD Domain Services.
  • You cannot add additional domain controllers to Azure AD Domain Services.  As this service is managed by Microsoft, they will automatically provide additional servers should the number of LDAP queries be excessive or you log a support case saying its slow!

Migrating Servers to Azure AD Domain Services


If you have Azure AD Connect in place from your on-premises Active Directory Domain to Azure AD, all your user accounts and groups will be available in the cloud and can be used in Azure AD Domain Services.  This is provided your synchronizing the password hashes for the accounts and not utilizing ADFS or Passthrough authentication.

After you migrate virtual machines up to Azure, if you join them to the new Azure AD Domain Services, you will be able to continue logging into the servers using the same synchronized accounts.  Please note however, Azure AD Domain Services will provide resources with a new Security Identifier (SID) and as a result, some resource access may break.  I'm not aware of any tools at this stage that will deal with this.  In on-premises world we had the Active Directory Migration Tool (ADMT) which performed SID Translation and allowed us to utilise SID History to continue accessing old resources over a transitive forest trust during co-existance.

We can however migrate group policy objects across to Azure AD Domain Services using Group Policy with Group Policy Migration tables.  This will allow you to provide the same policies in Azure AD Domain Services as what you have on-premises without manual reconfiguration.  If you want more information on this google "Group Policy Migration Tables".

I hope this post has been helpful in giving you an understanding of Azure AD Domain Services!