Sunday, December 9, 2018

Microsoft causing issues for on-premises Exchange Customers

Microsoft is causing issues for on-premises customers running Exchange to try and push customers to move to their servers in Office 365.  After the release of 16.0.6741.2017, the Click 2 Run (C2R) version of the Outlook client for the PC is prioritising O365 for Autodiscover queries above all other Autodiscover methods (SCP, HTTPS root domain etc).

This causes problems for customers who aren't using O365 for mail service, especially if either of these conditions are true:

1. The user has a mailbox in the O365 service which is not being used. This can occur if the user has inadvertently had an Exchange license assigned.

2. The user has a personal Office subscription but has used their business email address to configure it.

The issue which users experience is they are prompted for Authentication, when the users enter their details the login fails as it's effectively requesting authentication against the O365 service.

This behavior also breaks the experience for existing profiles, not just newly created ones.

The “workaround” we have is to add a registry change to end users PC to bypass the O365 endpoints. From this article:

https://support.microsoft.com/en-gb/help/2212902/unexpected-autodiscover-behavior-when-you-have-registry-settings-under 

This property needs to be set to a DWORD value of 1: ExcludeExplicitO365Endpoint

This needs to be done on each computer running Outlook.  For managed devices on a Corporate Network we can push this out with management tools such as SCCM or Group Policy Preferences but for non-managed devices this is a huge overhead for IT staff dealing with end user support and the issues experienced.

This workaround is hard to manage, client specific, and will need to be reverted if the customer ever does in fact move to O365 so that the Direct Connect method can work again.

The process of how Autodiscover is configured on Exchange Servers is documented on the following link and now even if we have setup Autodiscover correctly, by default Outlook clients will have issues setting up Outlook profiles.

https://docs.microsoft.com/en-us/exchange/client-developer/exchange-web-services/autodiscover-for-exchange

The feedback of removing this feature for Outlook Profiles configured against an on-premises Exchange Server has been provided to Microsoft from numerous frustrated IT Professionals.  Microsoft's response to this was as follows:

We cannot fulfill this request as we will continue to optimize for the Office 365 experience. The supported implementation of Autodiscover is documented here, https://support.microsoft.com/en-us/help/3211279. Any ongoing changes and improvements will be documented in the article. We appreciate your feedback and take every request with consideration, whether we can move forward with it or not.
-Outlook Team

Here are some facts:
  • There are still more on-premises customers running Exchange then customers in Office 365.
  • Customers on-premise pay good money via various Microsoft licensing programs to utilise these products.
  • Not all customers will migrate to the cloud - there will always be customers who want to keep intellectual property on-premises instead of moving to a shared public cloud environment.
  • Making changes such as the above which will cause issues to millions and millions of on-premises customers is not acceptable.
Due to the various complaints coming in from the community, after posting the above statement, Microsoft IMMEDIATELY closed comments preventing their customers from venting further frustration.  For more on this please see:

https://outlook.uservoice.com/forums/322590-outlook-2016-for-windows/suggestions/36052099-stop-prioritizing-o365-for-autodiscover?tracking_code=bcd47b537c1d87aa7fa3220e75bd96a2

One would hope that these on-premises authentication requests hitting Microsoft's O365 servers are just authentication requests, and Microsoft are not keeping these credentials.  If they were, this would be a directory harvesting exercise to the likes we have never seen before.

As a consultant who has specialized in Microsoft technologies for a long time - i'm very disturbed and appalled by the companies actions.  Microsoft get paid good money either way (if the customer is on-premises or in the cloud) and it is up to the customer to make a decision on where they want to store their intellectual property!

Wednesday, October 24, 2018

Pressure mounting from Microsoft on Skype for Business Usage

Everyone in the IT Community knows by 2018 that Microsoft's primary vision is to no longer to make software for on-premises usage but to focus on their cloud portfolio and moving customers intellectual property (documents, email, data) who use Microsoft applications into Microsoft owned datacentre.  Many companies have made the migration to Azure and Office 365 over recent years.

It was surprising to many of us that on the 1st of October, 2018 - It was announced that all new Office 365 customers under 500 seats would no longer receive access to Skype for Business.  This news was posted here:

https://support.microsoft.com/en-au/help/4465277/microsoft-teams-now-the-primary-client-for-meetings-and-calling

The Microsoft Teams client was officially launched only 14 March 2017 and ever since the launch, Microsoft has been mounting pressure for Skype for Business to go end of life.  It has only been a year and a bit since Microsoft are now making Microsoft Teams the only client of choice for businesses under 500 seats in Office 365 removing the option.

There are still a few things that I personally do not like about this new collaboration client such as the Interface and Features.

Skype for Business gives users the familiar "MSN Messenger" style layout, slick, clean and made primarily for instant messaging, presence and voice/video calls.  Teams is a completely different layout like in my screenshot below.


Microsoft Teams also has a file section on the left side pane allowing users to easily access data in One Drive or SharePoint Sites and allows multiple users to collaborate on the documents and chat real time about works.  This however assumes that companies store all their documents in a cloud service.  Some companies still do not upload documents and data to a public cloud service provider and only want to utilise Office 365 for instant messaging and collaboration without uploading sensitive information to the cloud.  The Files collaboration feature may be something that some clients do not want as part of an Enterprise instant messaging application.

Skype for Business is also heavily utilised by companies to perform simple desktop sharing to allow users to collaborate on works.  Microsoft Teams can also do this but you need to be in a Teams Meeting for this functionality to be available.  This is something that has annoyed many users and has been a feature request which Microsoft has ignored for a while and is all over the forums.


Whilst Microsoft Teams is some cool software, one thing which I have always liked in the IT world is freedom, the ability to have the option to choose the best tool for the task.  I personally don't like being forced to use a product or solution which is the new flavour of the month.  Nor do I like how some companies are forcing companies to adopt a cloud solution when not all businesses want to store intellectual property in a public datacentre for various business reasons.  All businesses should have the freedom to adopt technology as they feel fit.

Thursday, September 13, 2018

Azure AD Application Proxy Overview, Deployment Process and Limitations

One of my customers came to me enquiring about publishing some on-premises applications through the cloud using Azure AD Application Proxy.  This feature of Azure is relatively new and there is currently limited documentation about this available online.  I put some time aside to learn the technology and get a deep understanding on how the technology works.

In this article I will be running through a high level overview of how Azure AD Application Proxy works and then go through a step by step for a basic deployment of Azure AD Application Proxy to publish Outlook Web App.

Azure AD Application Proxy Overview

Azure AD Application Proxy is essentially a reverse proxy solution which is hosted completely in the Microsoft Azure cloud that allows companies to provide secure remote access to on-premises web based applications.  Traditionally companies needed to setup client side VPN connections or use demilitarized zones to provide secure remote access to on-premises web applications - especially for some web applications which may not be built secure for direct publishing to the Internet.  There are numerous security benefits for using Azure AD Application Proxy such as leveraging rich authorization controls and security analytics in Azure, two factor authentication, DDOS protection, no inbound connections to your internal network and much more.  I'm not going to go into all the security benefits of Azure AD Application Proxy in this article but you can do some further reading on the Microsoft website about this here:


It is important to mention, Azure AD Application Proxy is regional based.  Microsoft will automatically select the Azure Datacentre to host this service for you based on the country you specify when signing up to Microsoft Azure.  When you enable Application Proxy, the Application Proxy service instances for your tenant are chosen or created in the same region as your Azure AD tenant, or the closest region to it.

I have put together a high level overview of how Azure AD Application Proxy works below which I will be going through below (click to enlarge).


Without going too deep, there are essentially four components that make Azure AD Application Proxy work:
  • Azure AD Application Proxy Apps
  • Connectors
  • Connector Groups
  • Backend Applications
I will cover each of these below.


Azure AD Application Proxy Apps

Azure AD Application Proxy Apps sit in Microsoft Azure along side all your Software as a Service (SaaS) that you have published through Azure AD.  The primary difference between Application Proxy applications and standard Web Based Cloud applications, is Proxy Apps will redirect you to the server on-premises.

In Microsoft Azure, you can see the Application Proxy Applications with all your other Enterprise Applications.  Simply login to https://portal.azure.com and select Azure Active Directory --> Enterprise Applications.


When we open the Enterprise Applications, we can see an Azure AD Application Proxy application I created which sits along with all my other SaaS applications that I have associated with this Azure AD Tenancy.


Users can access the application directly by going directly to the Application Home Page URL or by logging into their application portal where all their SaaS applications are published.  This portal is under:

http://myapps.microsoft.com

As you see my test user user1@avantlab.com.au has only been presented with a few applications:


Connectors

Connectors facilitate traffic flow from the Azure AD cloud to your on-premises applications.  They are services which are installed on servers on your internal network running Server 2012 R2 or Server 2016.

Connectors are stateless and hold no configuration data on the machine apart from settings needed to connect to the Azure AD cloud service and its authentication certificate required for authenticating itself against the cloud.  They automatically update their configuration information every few minutes from the cloud.

When installed on an on-premises server, the connector will simply show as services in the Windows MMC console.


All connectors do not require any inbound firewall or Network Address Translation (NAT) rules publishing the connectors to the Internet.  They only initiate outbound connections to Azure creating a stateless connection to the cloud which is used to receive inbound application requests.  You can move the member servers running these connectors to different datacentres or public IP addresses, and because they are creating outbound connections it will not disrupt the on-premises web applications they publish.

As the Connectors do not require any inbound ports open from the Internet, they do not need to be located within a Demilitarized Zone and can go directly on servers on your internal network.  It is recommended deploying the connectors as close to your servers as possible to minimise latency.

Connectors can service multiple Azure AD Proxy Applications on your internal network and can be made fully redundant.

For production environments, it is always recommended to have multiple (at least two) Application Proxy Connectors on an on-premises environment to provide high availability to published on-premises applications.

Microsoft Azure automatically maintains the on-premises connectors not only the configuration, but updates.  Provided the "Microsoft AAD Application Proxy Connector Updater" service is running, your connectors will be automatically updated and patched.  If you have multiple connectors on your environment associated with a connector group, updates will only target one connector at a time in the group to prevent downtime to your environment.  We will talk about connector groups next.

Lastly it is important to mention, Connectors can run on member servers that are not domain joined however if you want Single Sign-On (SSO) working for applications that use Integrated Windows Authentication, they need to be setup on domain joined servers.  I will be showing you how to deploy Windows Integrated Authentication for an Azure AAD Application Proxy connector later in this post.

Connector Groups

All Connectors are associated with a Connector Group in Azure.  If you do not first define a Connector Group, all connectors you create will get assigned to a "Default" Connector Group.

A few key things to take away from Connector Groups:
  • Every AAD Proxy App you create must be assigned to one Connector Group (one to one relationship)
  • One Connector Group can contain Many Connectors (one to many relationship)
  • One Connector Group can service many applications (one to many relationship).
Connector Groups ensure on-premises application requests are load balanced across all on-premises Connectors in the group.  No configuration is done to setup load balancing between the connectors, this is done all automatically after the connectors are installed.

For most companies, only one Connector Group will be required with a few connectors as this will look after all your internal applications.  But why would you want to create multiple Connector Groups?  There is a few reasons.
  • Multiple Datacentres
  • Network Isolation
  • IaaS Deployments
  • Multi-Forest Deployments
  • Multiple Companies in a Single Azure AD Tenant

Multiple Datacentres

If you have multiple datacentres, you will most likely want to have a separate connector group with connectors in each datacentre.  This is to ensure latency is minimised between the connector and the application to provide the best experience for the users.

Network Isolation

If you have any network isolation between applications on your internal network such as Access Control Lists or Network Segmentation, you will want to ensure separate connector groups are created for each segment.  Remember the connectors associated with the connector group need internal connectivity to the application published.

Infrastructure as a Service Deployments

If you have any member servers in an IaaS cloud running internal web applications, you will most likely want to put connectors in the IaaS tenancy to minimise latency and a separate connector group.

Multi-Forest Deployments

If you have multiple on-premise Active Directory forests, you want a connector group for each Active Directory forest so that Kerberos Constrained Delegation works correctly for each forest.

Multiple Companies for a single Azure AD Tenancy

Some IT Departments for parent companies may run the on-premises networks for many child companies which are associated against a single Azure Tenancy.  You will want a connector group for each child company as they will most likely each have their own Active Directory forest.

Backend Applications

Backend applications are the on-premises products you wish to publish to the cloud via Azure AD Application Proxy.  I displayed in my diagram Exchange or SharePoint, but you can also publish other applications from Microsoft or third party vendors - as long as the entire application is web based.

Azure AD Application proxy Access Workflow

Before I get stuck into going through the steps for configuring Azure AD Application Proxy, I want to quickly touch base on the 6 steps which occur when a user accesses an application published in Azure AD Application Proxy.  This is taken from TechNet but is important to understand.
  1. The user accesses the application through the Application Proxy service and is directed to the Azure AD sign-in page to authenticate.
  2. After a successful sign-in, a token is generated and sent to the client device.
  3. The client sends the token to the Application Proxy service, which retrieves the user principal name (UPN) and security principal name (SPN) from the token, then directs the request to the Application Proxy connector.
  4. If you have configured single sign-on, the connector performs any additional authentication required on behalf of the user.
  5. The connector sends the request to the on-premises application.
  6. The response is sent through Application Proxy service and connector to the user.


Azure AD Application Proxy Deployment How-To

I have gone through and documented the steps for publishing Exchange 2016 on-premises Outlook Web App via Azure AD Application Proxy so you get an understand what is involved in setting up this technology.  In this deployment I have a single connector which is installed on my Exchange 2016 server in a default connector group.

For a production deployment, you will want to ensure the connectors are installed on dedicated servers and that you have more then one connector server for redundancy.

In this lab deployment also is using Password Hash Synchronisation between the on-premises environment and Azure AD.  If your company does not want to store your user password hashes in Azure AD, you can also use Passthrough Authentication with the Azure AD Connect tool.

Before we start configuring the Azure Application Proxy, you need to have Azure AD Connect setup and synchronising.  You will most likely already have this in place, but if you have never done this before, I have documented the steps.  Its very straight forward!

Setting up Azure AD Connect for Password Hash Synchronisation

First of all download the Azure AD Connect tool from:

https://www.microsoft.com/en-us/download/details.aspx?id=47594

Once the tool is downloaded, run it and accept Microsoft's End User Licensing Agreement (EULA).



When you get to this screen you will be asked if you want to use Password Hash Synchronisation or Passthrough Authentication.  If your company does not want user passwords in Azure AD, you need to use Passthrough authentication.  For this lab I'm using Password Hash Synchronisation.

I have also enabled Azure Active Directory Seamless Single Sign-On (Azure AD Seamless SSO) which provides single sign-on to Azure AD services by sending the Kerberos requests directly through to Azure AD using a Computer Account AZUREADSSOACC.  If you want to understand how Azure AD Seamless SSO works, please refer to these articles here:



Enter an account with Azure AD Global Administrator rights into the wizard to connect to Azure AD.


Select the Active Directory forests you want to synchronise.  The tool can synchronise multiple forests to Azure AD.


Select the on-premises attribute you want Azure AD to use as the username.  I ususally make this the userPrincipalName of the on-premises accounts, and I make sure the UPN is the users email address.  This can be done by setting up a UPN Suffix in Active Directory Domains and Trusts and assigning the UPN on each user account to match the UPN Suffix.

Users in my test lab have the following UPN suffixes:

user1@avantlab.com.au
user2@avantlab.com.au
user3@avantlab.com.au etc.


You can synchronise sub organisational units or the entire domain.  In this lab I'm synchronising the entire domain structure.


Azure was configured to manage the source anchor (the unique identifier) between users on premises and in Azure AD.  If you have a complex network with a lot of users you will need to think about what attribute you want to use as the source anchor.


We are not filtering any users in this lab, we are synchronising everything.


No additional features were need in this lab environment for Azure AD Application Proxy.  You may need additional features for other requirements.


You need to make sure your domain is verified in the Azure Portal by adding a TXT record to your public DNS.


The wizard will show if its verified.


Enter Domain Admin credentials for an account on-premises which is required for Azure AD Seamless SSO.  You will want to use a service account for this with a super complex password.


Kick off an initial synchronisation once the wizard is finished.


You will see the wizard created the AZUREADSSOACC for Azure AD Seamless SSO.


Close the wizard.


Upgrading Azure AD

By default your Azure AD subscription will be "Azure AD Free" which can run Microsoft Office 365.  In order to use Azure AD Application Proxy you must be running at least "Azure AD Basic", "Azure AD Premium P1" or "Azure AD Premium P2".

For a comparison between the various flavours of Azure AD, please see the following Microsoft website:


If you upgrade from Azure AD Free to a higher flavour, you will need to wait 24 hours for the changes to take effect within Azure.

Setting up Azure AD Application Proxy

The following steps need to be taken to setup Azure AD Application Proxy.

Navigate to Azure AD --> Enterprise Applications.


Select New Application


Select On-premises application.


If you haven't already upgraded from Azure AD Free you will get this warning.


Click the link and select Free Trial next to Azure AD Premium for lab purposes.  If your running a business, you will need to procure Azure AD Basic or higher before proceeding.


Activate the Azure AD Premium P2 trial in the lab and wait 24 hours for the activation to complete.


Next download the Application Proxy Connector application to install on the member servers.


You will need to agree to a license agreement before download can commence.


Make sure you run the setup on a Server 2012 R2 or Server 2016 computer.  In my lab to minimise the amount of virtual machines I have dedicated to this exercise, I installed the connector directly on an Exchange 2016 server.  For production deployments you should deploy the connector on a dedicated server and have more then one for redundancy.

Click through the steps below.


Sign-in to your Azure AD Tenancy with an Azure AD Global Tenancy account when prompted.


Let the tool install and connect to Azure AD.


Click close when it was successful.


Notice on my Exchange 2016 server, it added the following two services:
  • Microsoft AAD Application Proxy Connector
  • Microsoft AAD Application Proxy Connector Updater

After the connector is installed, go back to Azure AD and add the on-premises application.
  • Give the application a name, this is what users will see in the myapps.microsoft.com portal.
  • Enter the Internal URL of the application.  This is the FQDN of the web based application on your internal network.
  • Select a DNS name for the service, you can use a Microsoft domain or lab purposes or your own domain which you will need to take care of the DNS changes yourself.  As this is a lab I just used the Microsoft msappproxy.net domain.
  • Under Pre-Authentication you can select from Azure Active Directory or Passthrough Authentication.  Azure Active Directory requires the password hashes to exist in Azure AD.
  • As we did not create a connector group it will just use the Default group.  For production deployments I recommend you create a Connector Group first and give it a descriptive name such as the datacentre/location where the connectors will exist.


After you create the application, you will need to go into the properties of it and assign a couple of users the ability to access the application through Azure AD.  This is done through Users and Groups.


Lastly we need to setup Single Sign-on.  This ensures once a user signs into Azure AD through https://myapps.microsoft.com or going directly to the external facing URL of the application, they automatically get logged into the application and don't require signing in twice.  There are many Single Sign-on methods for Azure AD Application proxy including:
  • Password-based Sign-on
  • Linked Sign-on
  • Integrated Windows Authentication
  • Header-based Sign-on
If you would like an overview of each of these, refer to the following website:


For Exchange Server, we used Integrated Windows Authentication for Outlook Web App as this is best for Exchange.  By default, Exchange is setup to use Forms based authentication for the web apps so we need to change this before configuring Azure AD.

As you see below I changed my Exchange Virtual Directories and restarted IIS so that we are using Windows Integrated Authentication for both ECP and OWA.  Forms based authentication is now disabled.


Next we need to configure Kerberos Constrained Delegation on the servers running the connectors.  Find the member server computer objects in Active Directory Users and Computers for your Connector machines and go to the Delegation Tab.  You will need to enable Advanced Features in Active Directory Users and Computers to see this.

Under Delegation, enter the computer objects for all your servers which you want to setup delegation for.  These are the servers running applications.  You can add multiple web servers here... Exchange, SharePoint and any other web servers you have internally that you want to publish.

The screenshot is off putting because the Connector and the Application are both on the same server, hence why it was delegated to itself.

For more information on Kerberos Constrained Delegation with Azure AD Application Proxy please refer to the following article:



Once Active Directory is setup, configure the Internal Application SPN of your internal server.  This is generally going to be:

http/fqdn-of-internal-app

Select the login identity we configured earlier in Azure AD Connect, which was User Principal Name.


Next navigate to the public facing URL of your published application.  In my lab its:

https://exchange-avantlab.msappproxy.net

FYI: If your reading this a month later, I properly repurposed my lab and the address no longer exists.


After signing in to the application, we see that we are automatically passed through to Exchange OWA on-premises without being prompted for credentials again.


Azure AD Application Proxy Limitations

Azure AD Application Proxy is a new feature in Azure which offers customers basic reverse proxy functionality to publish on-premises applications through the cloud.  Whilst reviewing the product and setting it up in my lab, there currently are a few limitations which I believe will act as show stoppers for some companies looking to implement.  These limitations may be addressed in the future by Microsoft and I have submitted this feedback to the Azure AD product team - so fingers crossed.

Limitation 1: On-Premises Load Balancers are Still Required!

 Azure AD Application Proxy has great load balancing between the Azure service stack in the cloud and the on-premises connectors.  Load is distributed evenly across all connectors associated with a connector group.

The connectors however have no capability of distributing load across application servers in a web farm.  You are only able to configure a single FQDN on your internal network which represents your web application address.  This is shown in the following screenshot:


Many of my customers have server farms for important applications which need to be published.  The only way you can do this is by deploying an on-premises load balancing solution and advertising a Virtual IP address which the Azure AD Application Proxy connectors connect to.

Most leading load balancers such as F5 BIG-IP, KEMP, Barracuda Networks, Citrix NetScaler and many others also provide the reverse proxy functionality and are often more powerful then the functionality offered in Azure AD Application Proxy.  Of course there are free on-premises load balancing solutions such as Network Load Balancing or Application Request Routing (Microsoft IIS feature) but these are very limited and I would not recommend for production use.

What would be a great feature addition is if the connectors could handle this and perform basic health monitoring of application service endpoints and distribute load across the web farm.

Limitation 2: No way to handle different authentication methods within a Web Application

Some web applications have different authentication methods for each virtual directory.  A good example of this is Microsoft Exchange which I published, each of the virtual directory's have different authentication settings which Azure AD Application Proxy cannot accommodate.

Below is a list of virtual directories on an Exchange 2016 frontend website.  I configured the ECP and OWA virtual directories to use Integrated Windows Authentication however if I try and establish an ActiveSync connection from a mobile phone to exchange-avantlab.msappproxy.net it will fail, as the Microsoft-Server-ActiveSync virtual directory uses basic authentication over SSL.


To get around this, it would be possible to create multiple Azure AD Application Proxy Apps for each virtual directory with separate SSO configuration, but a painful approach.  It is also possible to configure Autodiscover to return different public FQDN's on the ExternalURL setting for each of the virtual directories listed below.

The workaround will work but it is not ideal!

Limitation 3: No way to limit/restrict Application Sub Directories

Almost every reverse proxy solution I have worked with on the market have a way of restricting access to some components of the application through the proxy - but not Azure AD Application Proxy!

For example a company may want to allow all applications internally, but block EWS and PowerShell virtual directories remotely from the Internet.

[ALLOW] https://exchange-avantlab.msappproxy.net/owa
[ALLOW] https://exchange-avantlab.msappproxy.net/microsoft-active-sync
[ALLOW] https://exchange-avantlab.msappproxy.net/oab
[ALLOW] https://exchange-avantlab.msappproxy.net/ews
[ALLOW] https://exchange-avantlab.msappproxy.net/autodiscover
[BLOCK] https://exchange-avantlab.msappproxy.net/powershell
[BLOCK] https://exchange-avantlab.msappproxy.net/ecp
With Azure AD Application Proxy there is no way to do this, publishing an application through Azure AD will always publish the entire application.

Final Thoughts

I believe Azure AD Application Proxy is a fantastic low cost method for securely publishing on-premise services to the Internet.  For customers already using the Azure stack to provide users access to applications via the https://myapps.microsoft.com portal, this allows the customer to add additional on-premises applications to users through the same familiar interface.

Azure AD Application Proxy is new technology and some of the limitations which I would like to see in the product may come in the near future.

Thanks for reading and I hope this post was useful.