Showing posts with label Exchange 2007/2010. Show all posts
Showing posts with label Exchange 2007/2010. Show all posts

Monday, May 3, 2021

Resetting Broken Default Exchange RBAC

I had a customer who needed re-created the default Exchange Groups under the "Microsoft Exchange Security Groups" organisational unit.  This was done as someone had moved these groups to another location (not supported) and the support engineer was unable to move the groups back to the original location due to an error.

Moving the default groups results in you being unable to:

  • Install new Exchange Servers into the organisation
  • Perform Cumulative Updates
  • Perform Recover Server installations
After recreating the groups, all AssignedRoles were stripped off the default Role Groups.

The server is fully functional, however administrators are unable to administer Exchange.

The only way we were able to access Exchange with administration access was to add the Exchange Snap-in from an administrative PowerShell.

add-pssnapin *exch*

To re-store the default Role Based Access Control objects to factory install, use the following commands from an administrative command prompt.

Add-Pssnapin *Setup*

Install-CannedRbacRoleAssignments –InvocationMode Install –Verbose

 Install-CannedRbacRoles  -Verbose

 Install-CannedRbacRoleAssignmentsRAP  -Verbose


This will restore access so administrators can access Exchange Admin Center and Exchange Management Shell.

Monday, August 26, 2019

Preparing Exchange Topology - PrepareAD, PrepareSchema, PrepareDomain etc

I'm about to sit the Exchange Upgrade Exam (MS-202: Microsoft 365 Messaging Administrator Certification Transition) on Friday and currently doing a last minute brush up.  Having done all the certifications in Exchange Server since MCSE: Messaging 2003 all the way through to the latest Exchange 2016 exam (70-345) I know the types of Questions Microsoft has asked in the past.

One area tested that I recall from all previous exams was the process for preparing schema and domain...

Whilst the latest Exchange 2019 Exams are definitely going to be heavily focused on Exchange Online in Office 365 given this is Microsoft's primary drive, they also test on all the on-premises content.  I wouldn't be surprised if topology preparation is tested once again... even though in my professional opinion it is not an item of significant importance given that over 99% of businesses simply rely on the Cumulative Update wizard to automatically extend the schema and prepare the domains.

In Exchange 2003 we had the setup.exe /forestprep and setup.exe /domainprep.

From Exchange 2007 all the way to Exchange 2019 we have a number of commands now including:
  • Setup /PrepareAD
  • Setup /PrepareSchema
  • Setup /PrepareDomain
  • Setup PrepareAllDomains
I remember there use to be a fantastic article on TechNet which gave a breakdown of exactly what each of these commands did - however after spending a good 5 minutes on Google i came up short trying to find the article and not sure if it is still published.

I could find no "clear" breakdown of each of these commands and the descriptions given on the installer help is useless as shown below:


I did come across the book however Exchange Server 2010 Administration: Real World Skills for MCITP Certification and Beyond (Exams 70-662 and 70-663) published by Joel Stidley and Erik Gustafson that touched on these commands in more detail.

Given the lack of content covering these commands, I decided to do a quick blog post.

Setup /PrepareSchema

This command does one thing, prepares the schema (additional class objects and attributes required for Exchange Server).  It must be run in the same Active Directory site as the Schema master.

To run this command you must be a Schema Admin.

Setup /PrepareDomain

This command must be run in each domain within an Active Directory forest.  This command simply creates special domain accounts and security groups in each domain for hosting Exchange Servers.  Thing of it as creating some additional "Active Directory" objects, no schema extensions within the "Domain Partition" only.

To run this command you must be an Enterprise Admin.

Setup /PrepareAD

The PrepareAD command performs three things:
  • Prepares the Schema if not done already (same as PrepareSchema)
  • Prepares the Domain (for the forest root domain only in a multi-domain forest)
  • Creates the Global Exchange Objects in the Configuration Partition.
It is important to note, PrepareAD runs the PrepareSchema command for the forest and PrepareDomain for the forest root domain only (if not done already).

For a single forest, single domain environment - PrepareAD is the only command you need to run.

If you have multiple child domains or new tree domains in the same Forest, after you run /PrepareAD in the forest root domain, you will need to /PrepareDomain for each of the additional domains within the forest.

To run this command you must be an Enterprise Admin and a Schema Admin.

Setup /PrepareAllDomain

If you have multiple domains in an Active Directory forest and you wish to run /PrepareDomain across all domains at the same time, this is what the /PrepareAllDomains command is for.

Hopefully this post has been useful.

Exchange RBAC Example - Provide User access to manage Contacts

In June 2010, I wrote an article explaining how Exchange Role Based Access Control (RBAC) works - a new feature released with Exchange 2010.  RBAC is still heavily utilised today with Exchange 2019 and Office 365 following these principals.

My article from 2010 can be found here:

https://clintboessen.blogspot.com/2010/06/exchange-2010-role-based-access-control.html

After having not worked with RBAC for while, I found myself re-reading the principals of RBAC and re-running through what is involved to configure the security model.  Knowing this well is also essential and I'm doing my Exchange upgrade exam this Friday (MS-202 Microsoft 365 Messaging Administrator Certification Transition) so a refresher on these principals is always handy!

I had a requirement given to me by a customer which I need to spin up in my test lab, and I thought whilst I lab the requirement it might be a good idea to write a quick blog post on the process of implementing the RBAC changes.

RBAC Requirement

"A user in the business must be able to create mail enabled contacts for external workers.  This user must only be able to create mail enabled contacts and no other objects, and the contacts must be stored under a specific organisational unit only".

RBAC Security Model

With the design of RBAC, The Exchange Product Team referred to RBAC as the Triangle of Power.  This is elaborated in this blog post here:

https://techcommunity.microsoft.com/t5/Exchange-Team-Blog/RBAC-and-the-Triangle-of-Power/ba-p/597147

The security model has three arms:
  • Step 1: Where - Where are the objects stored that we want to change or what attributes or identifying factors must the object have?  They can be in a security group, organisational unit, anyone with a job title set to XYZ... the possibilities are endless.  My requirement for "Where" is contacts must be contained only under a particular organisational unit.
  • Step 2: What - What is what you want to do.  What Exchange PowerShell cmdlets do you want to give the administrator access to be able to run?  Think of "What" as the access rights you wish to delegate.  The commands we are interested in for my example are New-MailContact and Remove-MailContact.
  • Step 3: Who refers to who is able to perform the operation.  Which user do you want to delegate control to in order to run the commands under the What section.
This all glued together by a concept known as Management Role Assignments.



I will break this down using my three headers listed above.

Step 1: Where

In my lab environment we only want contact objects to be changed under the OU "Contacts".

avantlab.local/Contacts

As a result we are going to create a new "Management Scope".  By default all RBAC Management Roles have access to the entire Active Directory forest and no Management Scopes exist. 

I'm locking it down to the Organisational Unit based on a RecipientRestrictionFilter with the following query:

New-ManagementScope "AvantLab Contacts" -RecipientRestrictionFilter {DistinguishedName -Like '*,OU=Contacts,DC=Avantlab,DC=Local"}



Step 2: What

Now we are interested in assigning "What" cmdlets we want the admin to be able to run.  The "What" refers to two cmdlets:
  • New-MailContact
  • Remove-MailContact
Cmdlets are linked as "ManagementRoleEntries" to objects known as "ManagementRoles".

There are a few rules you need to understand when creating new Management Roles for the "what section":
  • It is not possible to simply create a new Management Role and add the cmdlets you wish to run under the role as Management Role Entries.  All custom (non default) Management Roles must be linked to a parent Management Role.  Parent Management Roles are ones that come by default with Microsoft Exchange Server "out of the box".  Think of it as your "cloning" the parent Management Role to create your "Custom" management role.
  • It is not possible to add additional cmdlets that were not in the parent - you can however remove cmdlets.
Base on these two rules, when your creating a new Management role your essentially taking a management role similar to what your trying to create "with too much access", and removing the additional access from the role.

So the first step I need to do is find an existing Management Role that contains the two cmdlets I'm interested in.  I did this by running the following cmdlets:

Get-ManagementRole -Cmdlet New-MailContact
Get-ManagementRole -Cmdlet Remove-MailContact


A Management Role that contains both these cmdlets is "Mail Recipient Creation" - so this is the Management Role we will use as the template for creating my new Management Role.

I'm calling my new Management Role "Contact Management" and I'm basing it on "Mail Recipient Creation".  As a result, the new Management Role was created with the following cmdlet:

New-ManagementRole -Name "Contact Management" -Parent "Mail Recipient Creation"


Now the default "Mail Recipient Creation" Management Role had more cmdlets then we want the delegated access users to have access to.  To list all the cmdlets that this Management Role can run, execute the following command:

Get-ManagementRoleEntry "Contact Management\*"


We want to remove all cmdlets not related to our Contact Management.  Most importantly we want to remove any cmdlets with "New-, Set-, Start-, Remove-, Disable-, Write-, or Remote-" as they have elevated access.  "Get-" cmdlets you cant make changes, you can only view data.

To remove the unwanted cmdlets from the Contact Management Management Role the following cmdlet was used:

Get-ManagementRoleEntry "Contact Management\*" | where {$_.name -eq "New-MailUser"} | Remove-MagementRoleEntry




And a few Get- commands I wanted removed:



This left the Contact Management role with the following cmdlets associated:

Get-ManagementRoleEntry "Contact Management\*"

This left Management Role access to run the following cmdlets.  The "Get-" commands present cannot make any changes or expose any information that I would not want the delegated user to see.  As a result I didn't clean these up but you can if you want.  Most importantly, the New-MailContact and Remove-MailContact cmdlets were left present.


Step 3: Who

The last step is the "delegation of control", which staff members will have access to be able to run the Cmdlets in the Management Role "Contact Management".

To determine the "Who" I created a new Role Group called "Contact Management Admins" defining the "Contact Management" Management Role and the "AvantLab Contacts" Management Scope.  This was done with the following command:

New-RoleGroup "Contact Management Admins" -Role "Contact Management" -CustomRecipientWriteScope "AvantLab Contacts"


This creates a new Group under Microsoft Exchange Security Groups in Active Directory for the new Role Group.



Testing the RBAC Security

I went and added a user called DelegationUser to the Contact Management Admins group.


Logging into Exchange Control Panel (now known as Exchange Admin Centre (EAC) in later revisions of the product), the webpage only renders the areas of EAC the user has access to.  RBAC and being "Cloud Friendly" are the two primary reasons the old Exchange Management Console from Exchange 2007/2010 was retired.


The user can successfully run the New-MailContact command via the webpage and place a Contact object under the Contacts OU - the Step 1: Where? section.


This is shown in the following screenshot:


However if I try and create a contact in the default Users container, I get the message:

'avantlab.local/Users/DefaultUser Container' isn't within your current write scopes.  Can't perform safe operation.


This is because Step 1 "Where"... the management scope is restricted to only the Contacts OU.  Obviously Management Scopes are ridiculously granular and you can go far beyond restrictions of something as basic as an OU.


 What about the Glue?

Stop sniffing the glue Clint, I thought there was Glue in between holding these objects together as shown in the diagram below.

When I created the Role Group I also specified the Management Role and Management Scope.  This automatically created a ManagementRoleAssignment called "Contact Management-Contact Management Admins" as shown in the screenshot below.

Creating ManagementRoleAssignments manually useful especially when you want to create associations between existing objects that already exist on the system.


If we look at the Management Role Assignment "i.e. the glue" closer, we can see that it has all three components created above are "glued" together.
  • The Management Scope... Step 1: Where (CustomRecipientWriteScope: AvantLab Contacts)
  • The Management Role... Step 2: What (Role: Contact Management)
  • The Role Group... Step 3: Who (RoleAssigneeName: Contact Management Admins)


Hopefully this step by step guide to using Role Based Access Control was useful and can be put to use within your environment!

Wednesday, May 8, 2019

Exchange 2010 and Exchange 2016 co-existance Free/Busy Issues

I have a lab environment containing Office 365, Exchange 2016 and Exchange 2010.  Free busy is not working from O365 --> 2010 or Exchange 2016 --> 2010.

The Autodiscover record points all EWS requests to Exchange 2016 Web Services Virtual Directory for the Availability Service.

The Availability Service on the Exchange 2016 server is failing to lookup requests on Exchange 2010 mailboxes.

It is also important to note, the 2016 server is the one setup in Hybrid with O365, so it is responsible for looking up all Availability requests on-premises.

After doing some research into the issue, I identified that the InternalNLBBypass URL on the Exchange 2010 server must point and resolve directly to the Exchange 2010 server.  It must not be set to $null or point to the Availability Service on Exchange 2016 (in my lab that being mail.avantlab.com.au).

Exchange 2016 Web Services Virtual Directory:


Exchange 2010 Virtual Directory:


As soon as setting the InternalNLBBypassURL to point directly at Exchange 2010, this resolved the issue.

See my lab all working:

  • Arya is in Office 365
  • Jon is on Exchange 2010
  • Bran is on Exchange 2016
And yes, they are all Game of Thrones characters :)


I blogged this one as I could not find much information online regarding this!

Sunday, December 9, 2018

Microsoft causing issues for on-premises Exchange Customers

Microsoft is causing issues for on-premises customers running Exchange to try and push customers to move to their servers in Office 365.  After the release of 16.0.6741.2017, the Click 2 Run (C2R) version of the Outlook client for the PC is prioritising O365 for Autodiscover queries above all other Autodiscover methods (SCP, HTTPS root domain etc).

This causes problems for customers who aren't using O365 for mail service, especially if either of these conditions are true:

1. The user has a mailbox in the O365 service which is not being used. This can occur if the user has inadvertently had an Exchange license assigned.

2. The user has a personal Office subscription but has used their business email address to configure it.

The issue which users experience is they are prompted for Authentication, when the users enter their details the login fails as it's effectively requesting authentication against the O365 service.

This behavior also breaks the experience for existing profiles, not just newly created ones.

The “workaround” we have is to add a registry change to end users PC to bypass the O365 endpoints. From this article:

https://support.microsoft.com/en-gb/help/2212902/unexpected-autodiscover-behavior-when-you-have-registry-settings-under 

This property needs to be set to a DWORD value of 1: ExcludeExplicitO365Endpoint

This needs to be done on each computer running Outlook.  For managed devices on a Corporate Network we can push this out with management tools such as SCCM or Group Policy Preferences but for non-managed devices this is a huge overhead for IT staff dealing with end user support and the issues experienced.

This workaround is hard to manage, client specific, and will need to be reverted if the customer ever does in fact move to O365 so that the Direct Connect method can work again.

The process of how Autodiscover is configured on Exchange Servers is documented on the following link and now even if we have setup Autodiscover correctly, by default Outlook clients will have issues setting up Outlook profiles.

https://docs.microsoft.com/en-us/exchange/client-developer/exchange-web-services/autodiscover-for-exchange

The feedback of removing this feature for Outlook Profiles configured against an on-premises Exchange Server has been provided to Microsoft from numerous frustrated IT Professionals.  Microsoft's response to this was as follows:

We cannot fulfill this request as we will continue to optimize for the Office 365 experience. The supported implementation of Autodiscover is documented here, https://support.microsoft.com/en-us/help/3211279. Any ongoing changes and improvements will be documented in the article. We appreciate your feedback and take every request with consideration, whether we can move forward with it or not.
-Outlook Team

Here are some facts:
  • There are still more on-premises customers running Exchange then customers in Office 365.
  • Customers on-premise pay good money via various Microsoft licensing programs to utilise these products.
  • Not all customers will migrate to the cloud - there will always be customers who want to keep intellectual property on-premises instead of moving to a shared public cloud environment.
  • Making changes such as the above which will cause issues to millions and millions of on-premises customers is not acceptable.
Due to the various complaints coming in from the community, after posting the above statement, Microsoft IMMEDIATELY closed comments preventing their customers from venting further frustration.  For more on this please see:

https://outlook.uservoice.com/forums/322590-outlook-2016-for-windows/suggestions/36052099-stop-prioritizing-o365-for-autodiscover?tracking_code=bcd47b537c1d87aa7fa3220e75bd96a2

One would hope that these on-premises authentication requests hitting Microsoft's O365 servers are just authentication requests, and Microsoft are not keeping these credentials.  If they were, this would be a directory harvesting exercise to the likes we have never seen before.

As a consultant who has specialized in Microsoft technologies for a long time - i'm very disturbed and appalled by the companies actions.  Microsoft get paid good money either way (if the customer is on-premises or in the cloud) and it is up to the customer to make a decision on where they want to store their intellectual property!

Monday, August 6, 2018

Forcefully Reseeding an Exchange Server on Exchange 2010

One of our customers had an Exchange 2010 DAG with databases on two servers.  One of the databases could not be mounted.  Before calling us, the customer removed one of the servers as a database copy.  So we only had one server with one copy of the Mailbox Database.

We attempted to re-add the other server as a DAG Member after the customer removed it with the following command:

Add-MailboxDatabaseCopy -Identity "MAILBOXDATABASE" -MailboxServer "EXHANGESERVER"

The DB was created, however the seed fails.  This can be caused by real-time anti-virus on Exchange servers without the appropriate exclusions in place or a corrupt transaction log / or inconsistent transaction log checkpoint file.

To fix this we forced the seed using the following procedure:


  1. We dismounted the active database.
  2. We suspended the passive copy (if active)
  3. We deleted the EDB and Log Files from the Passive Server.  (You can move these to a different location as an alternative).
  4. We checked the Active Server was in a clean shutdown state - see https://clintboessen.blogspot.com/2010/09/flush-transaction-logs-in-exchange.html
  5. Once we know the Active Server database was in a clean shutdown, we cleared all transaction logs only from the production server by moving the logs to a temp folder.
  6. We then re-mounted the Active Server database.  This database needed a force mount (Mount-Database DatabaseName -Force)
  7. Lastly we forcefully updated the passive server with the following command:
Update-MailboxDatabaseCopy -Identity "MAILBOXDATABASE\EXCHANGESERVER" -DeleteExistingFiles

The PowerShell window will show a progress bar of the reseed progress.

Hope this post has been helpful.



Saturday, May 26, 2018

432 4.3.2 STOREDRV.Deliver recipient thread limit exceeded

I had an issue with an environment with a mixture of Exchange 2010 and Exchange 2016 servers.  All mail to one of the Exchange 2010 servers was getting stuck in queue and not delivered.


This is a very weird error and I have not come across it in my many years working with Microsoft Exchange.

To fix this issue, the following two lines need to be added to the ExchangeTransport.exe.config file in the Exchange bin directory.


Then restart the Exchange Transport service.  Mail flow should begin working again.  For more information about this error please refer to the following Microsoft blog post:

https://blogs.technet.microsoft.com/exchange/2011/04/11/store-driver-fault-isolation-improvements-in-exchange-2010-sp1/

Thursday, September 7, 2017

Autodiscover: Outlook Provider - 6001 Error for one user only

We had issues with Autodiscover with only one mailbox in our environment.

When we ran the Test-OutlookWebServices against the problematic mailbox, we got an error.

Test-OutlookWebServices -Identity "journal@domain.com" -MailboxCredential (get-credential domain\journal)


Looking at the full report with the format list option "| fl" we get:


Test-OutlookWebServices -Identity "journal@domain.com" -MailboxCredential (get-credential domain\journal) | fl

RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     : mail.domain.com
Scenario            : AutoDiscoverOutlookProvider
ScenarioDescription : Autodiscover: Outlook Provider
Result              : Failure
Latency             : 16
Error               : System.Net.WebException: The remote server returned an error: (401) Unauthorized.
                         at System.Net.HttpWebRequest.GetResponse()
                         at
                      Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.InternalInvoke()
                         at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.Invoke()
Verbose             : [2017-09-08 04:38:10Z] Autodiscover connecting to
                      'https://mail.domain.com/Autodiscover/Autodiscover.xml'.
                      [2017-09-08 04:38:10Z] Test account: journal Password: ******
                      [2017-09-08 04:38:10Z] Autodiscover request:
                      User-Agent:
exchangeserver/Test-OutlookWebServices/journal@domain.com
                      Content-Type: text/xml; charset=utf-8
                      Host: mail.domain.com
                      Cookie: X-BackEndCookie=S-1-5-21-2167321796-859855631-2145623002-1367=rJqNiZqNgayprdK6p7y3vrG4utG
                      SnpGVlpKKj4yX0ZOQnJ6Tgc7GzMjGxsjGy8iBzc/OyNLPxtLPx6vPy8XLx8XOzw==
                      [2017-09-08 04:38:10Z] Autodiscover request:
                     
                      http://www.w3.org/2001/XMLSchema
"
                      xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"
                      xmlns="
http://schemas.microsoft.com/exchange/autodiscover/outlook/requestschema/2006">
                       

                          journal@domain.com
                         
http://schemas.microsoft.com/exchange/autodiscover/outlook/response
                      schema/2006a

                       

                    

                      [2017-09-08 04:38:10Z] Autodiscover response:
                      request-id: eee84b62-bc41-4363-90b6-4c47e136a08d
                      X-SOAP-Enabled: True
                      X-WSSecurity-Enabled: True
                      X-WSSecurity-For: None
                      X-OAuth-Enabled: True
                      Server: Microsoft-IIS/8.5
                      WWW-Authenticate: Negotiate,NTLM,Basic realm="mail.domain.com"
                      X-Powered-By: ASP.NET
                      X-FEServer: exchangeserver
                      Date: Fri, 08 Sep 2017 04:38:10 GMT
                      Content-Length: 0
                      [2017-09-08 04:38:10Z] Autodiscover response:
                      System.Net.WebException: The remote server returned an error: (401) Unauthorized.
                         at System.Net.HttpWebRequest.GetResponse()
                         at
                      Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.InternalInvoke()
                         at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.Invoke()
MonitoringEventId   : 6001RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     :
Scenario            : ExchangeWebServices
ScenarioDescription : Exchange Web Services
Result              : Skipped
Latency             : 0
Error               : Skipped testing Exchange Web Services because the Autodiscover step failed.
Verbose             :
MonitoringEventId   : 5002

RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     :
Scenario            : AvailabilityService
ScenarioDescription : Availability Service
Result              : Skipped
Latency             : 0
Error               : Skipped testing Availability Service because the Autodiscover step failed.
Verbose             :
MonitoringEventId   : 5003

RunspaceId          : 996337d5-8719-4dfa-b19c-84b81a2ea577
Source              : exchangeserver.domain.com
ServiceEndpoint     :
Scenario            : OfflineAddressBook
ScenarioDescription : Offline Address Book
Result              : Skipped
Latency             : 0
Error               : Skipped testing Offline Address Book because the Autodiscover step failed.
Verbose             :
MonitoringEventId   : 5004


To resolve this issue I compared all attributes from the bad mailbox "journal" against a working mailbox.  To quickly get an attribute dump from a user account in Active Directory you can use the following command:

Get-ADUser username -Properties * | Select *

To compare the attributes against a working account, I simply used the windiff tool available from http://www.grigsoft.com/download-windiff.htm

I noticed the problematic account had the protocolSettings set as shown in the screenshot below:

 
All other accounts had protocolSettings set to "RemotePowerShell§1", so I corrected this as shown in the screenshot below.
 
 
After making this change on the mailbox I tested again - it failed.  This is because Exchange caches Active Directory objects and attributes - usually for up to an hour to reduce load on Domain Controllers.  To get the web app to flush its cache, I simply did an "iisreset".
 
Running the command again and the Autodiscover test passed.
 

Friday, March 31, 2017

Unable to remove Mailbox Database or Uninstall Exchange

I had an issue removing Exchange 2010 today at a customer site.  A very generic issue occurred when attempting to perform the uninstall where the following error was generated.

Error:
Uninstall cannot continue. Database 'DEVEXCH170-01-CEO and Councillors': This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, or arbitration mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox -Database . To get a list of all mailbox plans in this database, run the command Get-MailboxPlan. To get a list of archive mailboxes in this database, run the command Get-Mailbox -Database -Archive. To get a list of all arbitration mailboxes in this database, run the command Get-Mailbox -Database -Arbitration. To disable a non-arbitration mailbox so that you can delete the mailbox database, run the command Disable-Mailbox . To disable an archive mailbox so you can delete the mailbox database, run the command Disable-Mailbox -Archive. Arbitration mailboxes should be moved to another server; to do this, run the command New-MoveRequest . If this is the last server in the organization, run the command Disable-Mailbox -Arbitration -DisableLastArbitrationMailboxAllowed to disable the arbitration mailbox. Mailbox plans should be moved to another server; to do this, run the command Set-MailboxPlan -Database .


This error is normal when you have a database that has a mailbox, archive mailbox or arbitration mailbox.  It can also happen if it is associated with a Mailbox Plan (in a multi tenant environment).

In my case, there was no mailbox in the database!


Also the verbose command is meant to show you which mailbox resides in the database in the event one is present as per Exchange MVP Tony Redmond's post here:

http://windowsitpro.com/blog/exchanges-most-annoying-and-confusing-error-message

In this environment, the mailbox causing the issue was not displayed in the verbose output!


After much troubleshooting I decided to export the entire domain partition to a text file and search for the database name.

This was done with the following command:

dsquery * domainroot -attr * -limit 0


After searching the text file for the database name in question I saw that a legacy user IORepl (which was used by the Inter-Org Replication Tool for a previous cross-forest migration) is associated with the database.


 You can do two things here:
  • Clear the HomeMDB attribute
  • Delete the object.
As this account is no longer required, I simply deleted the object.

This fixed the problem.  No idea why it didn't return the mailbox object in PowerShell!

Thursday, February 16, 2017

Kerberos Error Connecting to Exchange 2010

Using an old user account at a customer site, I had the following error when attempting to connect to Exchange Management Console (EMC).

The following error occurred while attempting to connect to the specified Exchange server 'server.domain.local:

The attempt to connect to http://server.domain.local/powershell using 'Kerberos' authentication failed: Connecting to the remote server failed with the following error message : WinRM cannot process the request. The following error occurred while using Kerberos  authentication: The network path was not found.


To resolve this issue, delete the NodeStructureSettings registry key from

HKEY_CURRENT_USER\Software\Microsoft\ExchangeServer\v14\AdminTools


For specialised IT Support in Perth, contact us.

Thursday, January 26, 2017

Displaying full values of Attributes in PowerShell

In Windows PowerShell often when you are running queries, PowerShell will only show a limited value for objects which contain large attributes.  The attribute output is cutoff with a "..." at the end of the attribute.


To configure PowerShell to display the full output of a cmdlet, enter the following into the shell window:

$FormatEnumerationLimit=-1

The shell will now push the full output onto the screen for long attributes.


Hope this post was helpful.

For IT Support in Perth, Contact Avantgarde Technologies. 

Thursday, January 19, 2017

MSExchange ActiveSync Event ID 1016

Customer with a single Exchange 2010 completely down.  The following error was spammed throughout the event log:

Log Name:      Application
Source:        MSExchange ActiveSync
Date:          18/01/2017 7:31:46 PM
Event ID:      1016
Task Category: Server
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXCHANGE2010
Description:
Exchange ActiveSync has encountered repeated failures when it tries to access data on Mailbox server [EXCHANGE2010.domain.local]. It will temporarily stop making requests to the Mailbox server for [60] seconds to reduce load on that server. This delay may occur if the Mailbox server is overloaded. If this event is logged frequently, review the Application log on this server and the Mailbox server noted above for other events that could indicate the root cause of performance problems.
Additional information:
"serverFQDN=EXCHANGE2010.domain.local
Error 0:

ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=EXCHANGE/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=User.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=140]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081 
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065 
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805 
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583 
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778 
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730 
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 1:

ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=sfso.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=133]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081 
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065 
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805 
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583 
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778 
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730 
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
Error 2:

ErrorTimeStamp:
18/01/2017 7:31:45 PM
Exception:
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException
Exception message: Cannot open mailbox /o=exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=sfso.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity)
   at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.b__10(MailboxSession mailboxSession)
   at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure)
   at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString, Boolean wantCachedConnection)
   at Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)
   at Microsoft.Exchange.AirSync.Command.OpenMailboxSession(AirSyncUser user, Boolean shouldUseBudget)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
Inner exception follows...
Exception type: Microsoft.Mapi.MapiExceptionNetworkError
Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227)
Diagnostic context:
    ......
    Lid: 13720   dwParam: 0x6D9      Msg: EEInfo: Flags: 0
    Lid: 11672   dwParam: 0x6D9      Msg: EEInfo: NumberOfParameters: 4
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp
    Lid: 8856    dwParam: 0x6D9      Msg: EEInfo: prm[1]: Unicode string: EXCHANGE2010.domain.local
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[2]: Long val: -545057711f
    Lid: 12952   dwParam: 0x6D9      Msg: EEInfo: prm[3]: Long val: 382312662
    Lid: 45169   StoreEc: 0x824    
    Lid: 44273 
    Lid: 59431   EMSMDB.EcDoConnectEx called [length=133]
    Lid: 34855   EMSMDB.EcDoConnectEx returned [ec=0x824][length=56][latency=0]
    Lid: 59505   StoreEc: 0x824    
    Lid: 25964   StoreEc: 0x824    
    Lid: 36081 
    Lid: 52465   StoreEc: 0x80040115
    Lid: 60065 
    Lid: 33777   StoreEc: 0x80040115
    Lid: 59805 
    Lid: 52209   StoreEc: 0x80040115
    Lid: 56583 
    Lid: 52487   StoreEc: 0x80040115
    Lid: 19778 
    Lid: 27970   StoreEc: 0x80040115
    Lid: 17730 
    Lid: 25922   StoreEc: 0x80040115
Exception level: 1
Exception stack trace:    at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)
   at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout)
   at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId)
   at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore)
--- Exception end ---
errorCount=3, backingOff=True".



I know there are a few causes of this error however in my instance it was due to Microsoft Exchange Replication Service not started.  The service had been misconfigured to run as a service account with invalid credentials.

Setting it back to local system resolved the issue and getting the customer back online.

The Active Sync service queries "Active Manager" which is part of the Exchange Replication Service to determine where the users active mailbox copy in a DAG resides - and still follows this model even if your not using DAG's in your environment.  Hence if Active Manager is not available, no one can locate their Active Mailbox.