Clint Boessen's Blog: May 2011

Tuesday, May 31, 2011

Continuous Replication Block Mode vs Continuous Replication File Mode

Exchange 2010 RTM only supported Continuous Replication File Mode.
Exchange 2010 SP1 now supports Continuous Replication Block Mode.

Ok so whats the difference?

Continuous Replication Block Mode

A form of continuous replication that replicates blocks of ESE transaction data between the ESE log buffers on the active mailbox database copy to replication log buffers on one or more passive mailbox databases copies. What does this mean? This means Exchange 2010 has replicate log buffers before the transaction log file is closed. This reduces the amount of transactions that are lost should a failover occur. If a transaction log is not played to the passive node in time, data may be lost during a failover. This is what AutoDatabaseMountDial is for, controls if the database will be mounted based on the based on the number of log files missing by the copy being mounted.

Continuous Replication File Mode

A form of continuous replication that replicates closed transaction log files from the active mailbox database copy to one or more passive mailbox database copies. This means the transaction logs need to be closed before they can be shipped. This is one of the reasons transaction log size as reduced from 5MB to 1MB as of Exchange 2007 - to minimize data loss during Exchange failover.

Please note if a log file does not replicate in time when a failover occurs - the hub transport server in the same site has a service a process known as transport dumpster. The transport dumpster resides on every hub transport server by default however it is not used unless a mailbox failover occurs. What the transport dumpster does is hold email that has already been delivered to the active mailbox server for a specified period of time. In the event that a DAG node fails which prevents the most recent logs from being replicated over, the transport dumpster can redeliver this email.

Transport Dumpster was first introduced in Exchange 2007 for CCR (Continuous Cluster Replication). Please see:

http://clintboessen.blogspot.com/2009/04/scc-single-copy-clusters-vs-ccr.html

So now your asking me why introduce Continuous Replication Block Mode if there is a process called Transport Dumpster that replays emails after a failover? Good question!

The answer is not all exchange mailbox transactions are caught in transport dumpster. Transport dumpster, as it resides on the hub transport will only ever capture SMTP based transactions. There are other internal transactions Exchange generates directly on the mailbox database. These transactions will not reside within Transport Dumpster. Also things like de-duplication software that replaces emails with stubs. These cause transactions to occur which are not registered in Transport Dumpster.

Can I switch between Continuous Replication File Mode and Continuous Replication Block Mode if I'm running Exchange 2010 SP1?

No - The system can automatically switch between file mode and block mode based on the passive copy's ability to keep up with continuous replication.

TechEd North America 2011

Exchange 2010 Active Sync Feature Comparison

I found an excellent article today listing all the features of Exchange Active Sync and which features are supported by what mobiles. Please check this article out here:

http://en.wikipedia.org/wiki/Comparison_of_Exchange_ActiveSync_Clients

Monday, May 30, 2011

Exchange Active Sync Logo Program

Customers have been telling Microsoft for quite some time that one of their top concerns is the increasing diversity of mobile devices that employees use to access company networks. While most mobile devices use EAS, the industry standard for mobile email, EAS policies are not consistently implemented by the mobile device manufacturers. This makes it challenging to know which EAS management features are supported by any particular device.

In the EAS Logo program, participants agree to implement a predefined set of EAS policies (or more). IT professionals can now be assured that devices that bear the logo meet a consistent set of mobile device requirements. Today, we’re launching with the following devices:

- All Windows Phone 7 and Windows Mobile 6.5 devices
- Nokia devices with the Mail for Exchange 3.0.05, including the recently launched Nokia E7
- Apple devices with iOS 4.X, including the iPhone 4, iPhone 3GS, iPad and iPad 2

Microsoft have a strong pipeline of additional device manufacturers working toward compliance. Additional participants will be announced in the coming months, including wireless carriers that will use the logo when marketing compliant devices to their customers.

The EAS Logo Program has the following requirements:
- Direct Push email, contacts & calendar
- Accept, Decline & Tentatively Accept meetings
- Rich formatted email (HTML)
- Reply/Forward state on email
- GAL Lookup
- Autodiscover
- ABQ strings (device type and device model) provided
- Remote Wipe
- Password Required
- Minimum Password Length
- Timeout without User Input
- Number of Failed Attempts

Please refer to the following blog post by Greg Smiley for more information:

http://blogs.technet.com/b/exchange/archive/2011/04/13/announcing-the-exchange-activesync-logo-program.aspx

The official page for the EAS Logo Program can be found here:

http://technet.microsoft.com/en-us/exchange/gg187968.aspx

For a full list of all the Exchange Active Sync features and an understanding of which phones support which features please see:

http://en.wikipedia.org/wiki/Comparison_of_Exchange_ActiveSync_Clients

Advanced DNS Active Directory Questions

One of my clients has 3 Active Directory forests each with multiple domains. Stub zones are in place for the forest root domain in each forest which are stored in each DomainDNSZone for each forest. When a DNS client wants to resolve a hostname for a host in a child domain in a different Active Directory forest it is using recursion referencing the root DNS server (from the Stub Zone) and obtaining the NS RR's for the child domain.

Delegation records and Conditional Forwarders need to be updated manually when DNS servers are added and removed from the environment. Currently the companies delegation records are not align with the DNS servers in each of the child domains for each Active Directory forest. The company wants to implement a solution which is automated so they will never have to worry about updating Delegates or Forwarders. Stub Zones meet the requirement.

The company is currently looking at removing all delegation between parent and child domains in each forest and use Stub Zones instead. They also want wish to use Stub Zones cross forest. To avoid having to create a Stub Zone in each domain they wish to store the Stub Zone for every domain in the ForestDNSZone partition in each forest. This means all domains in each forest will be able to utilize the Stub Zone.

I have the following questions:

Question 1
In an Active Directory Forest we want to store each domain’s DNS zone in the DomainDNSZone partition within Active Directory. We want to configure a Stub Zone for every domain within a DNS forest and store these Stub Zones in the ForestDNSZones partition within Active Directory so they replicate to all AD Domains on the given forest. Question: All domain controllers will have their domain DNS zone under the DomainDNSZones partition in Active Directory. However they will also receive the stub zone for the ForestDNSZones partition as they are a member of the same forest. What happens here with the stub zone located in ForestDNSZones?

Question 2
You can only configure forwarders or stub zones on any given DNS server. What happens if you create a forwarder on one DNS server then on another DNS server in a different AD site you create a stub zone that is AD Integrated. When it goes to replicate, it will replicate the stub zone to the DNS server which in this case both a stub zone and a forwarder exists. How does windows DNS deal with this?

Question 3
In the relationship between child and parent domains, if an organisation chooses to use stub zones instead of delegation, if they do not remove the delegation records which precedence the Stub RR's or the Delegate RR's?

To answer these questions I raised I ended up having to create a lab environment and perform testing. Below are my answers:

Question 1 Answer
If a "Stub Zone" exists in the forest DNS partition but a DNS server in the same forest has a the "Primary Zone" file stored locally or in another location within DNS, the DNS server will ignore the "Stub Zone" in the forest DNS partition. To find this out I created a lab environment with two domains within a single forest:

- contoso.com
- branch.contoso.com

Contoso.com DNS server:

Properties for the branch.contoso.com stub zone on the contoso.com DNS server:

Branch.contoso.com DNS server:

Properties for the contoso.com stub zone on the branch.contoso.com DNS server:

Question 2 Answer
If a stub zone exists on a DNS server windows will not let you create a conditional DNS forwarder matching the same name as the stub zone. If a conditional DNS forwarder exists windows will not let you create a DNS stub zone matching the same name as the conditional forwarder. If you create an AD integrated stub zone on another DNS server in a different active directory site it will replicate through standard AD replication. When it replicates to the server containing the conditional forwarder, the stub zone will take precedence over the conditional forwarder. Stub Zones always have precedence over conditional forwarders.

Question 3 Answer
Stub Zones always have precedence over delegation.

Thursday, May 26, 2011

Initializing MAPI session object failed

I had a user who upgraded from a previous version of Outlook to Outlook 2010. Whenever they open an Outlook email they recieve an error "Initializing MAPI session object failed." After clicking OK the email opens.

This is resolved by going to C:\Windows\System32 and renaming mapi32.dll to mapi32.dll.old. Then run a tool called fixmapi.exe under C:\Windows\System32. This will generate a new mapi32.dll file.

Note: Sometimes Administrators do not have rights to rename or modify mapi32.dll, only "TrustedInstaller" does. In cases like this you need to take ownership of mapi32.dll and provide Administrators access to the file.

Sunday, May 22, 2011

Pitfalls to Virtualizing Exchange 2010

Exchange performs best when it can interact with the physical components of a server directly. If you disagree with this statement that’s usually a symptom exhibited right after a vmware conference - hopefully it will go away.

Before I go into the pitfalls, let me say I love virtualization - most of my Exchange 2010 deployments are virtualized. Virtualization is fantastic for small deployments of Exchange 2010 as it provides many advantages including increasing energy efficiency and requiring less hardware with server consolidation. What us Exchange people are arguing is there are times you do not want to virtualize an Exchange server.

One statement VMware makes is "Virtualize enterprise apps, including Oracle, Exchange, SQL Server, Sharepoint and SAP, and deliver the highest SLAs and top performance.", please see: http://www.vmware.com/virtualization/. How is virtualizing your infrastructure going to achieve top performance? Top performance can only be achieved when software can interact directly with physical hardware bypassing hypervisors completely.

VMware have published publicly that Exchange 2010 DAG replication is supported - please see:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1037959

This is officially NOT supported by Microsoft. Just because VMware had it working in their lab doesn't with "simulated load generators" doesn't mean it will behave correctly in a production environment - I am talking here from experience. The Microsoft Exchange product team has clearly stated that both HyperV live migration and VMotion are not supported. You can see the link: http://technet.microsoft.com/en-us/library/aa996719.aspx. The link states that you're not supported if you use VMotion/SRM (etc) at the same time as the DAG. That's the support statement. Period.

I have seen problems directly related to virtualizing clustered Exchange 2010 mailbox servers. I have seen virtualized Exchange servers where the windows cluster had kicked the server out of being a cluster member. When you try and readd the node back into the cluster, the cluster node would join and then be evicted again. This can be resolved by evicting the node via ConfigurationOnly and removing it from Windows Clustering. The problem? The Witness! It appears the Witness was locked and the new (old) node could not access it. As it turns out the client vmotioned the server to another node and that's when all the problems began.

I have just got back to Australia from MSIDC (Microsoft Indian Development Centre). During my time at MSIDC I had an interesting debate internally with a Microsoft virtualization MVP named Susantha Silva on this subject. Virtualization MVP's ususally think everything should be virtulized, I'm sure they would virtualize the operating system on their mobile phone if they could! Present during this debate was Sheesh Dubey, Hyper V Program Manager for Microsoft. This debate was closed by Sheesh Dubey saying "If a product team states their application should not be virtualized in any particular circumstance, you should always follow the advise of the product team as they have reasons behind any statement they make."

A pitfall when Virtualizing mailbox servers that are a member of a cluster is taking snapshots in regards to transaction log replication. Go on take a snapshot of a server performing log replication. Have fun fixing it!

Another pitfall to Virtualization is low-balling the RAM because "it's a VM and we can add it whenever". Exchange 2010 was architectured to use slow disk, as a result it needs more memory to provide ample disk cache. Make sure you use the Exchange Storage calculator and calculate your RAM correctly.

Next problem we have is with a concept called Dynamic Memory. This is when a virtual machine is able to automatically extend the amount of memory it has allocated and return it to the hypervisor when its not required through a concept called Ballooning. SQL Server now supports Dynamic Memory as per http://support.microsoft.com/kb/956893, but Exchange 2010 mailbox role however does not (all other Exchange 2010 roles do). This is because the Mailbox Role in Exchange Server does not change its memory allocations on the fly. I encourage you to do a google/bing search for "dynamic memory exchange".

"Not doing the I/O numbers because it’s that magic virtual storage behind VMware". If you want the absolute best disk I/O you need to pass the physical storage directly through to the virtual machine. This provides the best possible I/O performance. If you insist on using a VHD or VMDK file make sure its fixed size! Another problem with shared storage is its very easy to overcommit a RAID Array of disks on a LUN to to many servers. I have seen so many virtual exchange servers where the disk queue lengths exceed 100 - very bad! Especially if your using the recommended Exchange 2010 TIER2 storage model, please make sure nothing else hits the disks!

Tossing a whole DAG on one virtual host. Yes I have seen it done unfortunately. We have mentioned above the reasons why we cannot add live migrate or vmotion a servers that are a member of a DAG cluster. So what is the point? Having two mailbox servers in a DAG cluster on a single host performs worse then if the server was simply a single mailbox server.

People, one virtual CPU does not represent one physical CPU. A virtual CPU represents 1 core of a physical CPU.

RDM/LUN limits per VMWare cluster (FC path limit of 1024 / 4 paths = 256 LUNs per cluster)
RDM LUN limits per Exchange VM (60 LUNs max – 4x SCSI controllers, 15 LUNs per controller)

When Exchange 2010 is virtualized I ususally see the DB/Logs on the same Disk Spindles that are used by the VMs for the operating system. Remember back to the days when all servers were physical. Did the DB and LOGs ever be on the same disks as the operating system? No they didn't.

When virtualizing your client access servers Microsoft NLB does not support Unicast Mode if Notify Switches is turned on. This means you need to configure NLB to use multicast mode and send cluster communication over the servers primary network interface card.

Virtual Sprawl. I see admins just add more and more virtual machines until the virtualization platform almost collapses and dies. They seem to have such faith in it so they don’t even monitor it, they also think that in some mystic way it will compensate for not designing and RAM, disk and CPU correct. I think this is a knowledge and maturity thing of admin.

I have a personal rule I follow, if more then 2000 users require access to a mailbox server and the users are heavy power users I ensure the Exchange server is setup physical. If the users are under 2000, require light weight access to their email and do not require DAG clustering with live migration/vmotion I set the environment up Virtual. This is pending that the disks and servers are not already over committed. On majority of networks Exchange followed by SQL requires more resources then other server. Because of this it does not always make sense to perform server consolidation through virtualization as your Exchange server may already be utilizing majority of resources on the physical host. Please note: CPU's are getting faster, this blog post is getting older. Pay close attention to the CINT Rates value of the CPU and use the sizing spreadsheet created by Ross Smith and size your CPU's appropriately.

Take advantage of cheap local storage, the whole idea of Exchange 2010 is to have multiple copies of your data in different geographical locations on extremely cheap disk/hardware. It is designed to scale out so that you CAN have servers fail. A properly designed enterprise exchange 2010 environment should be able to cope with 49% of the servers failing without taking down the environment. If you have a deep understanding on how Exchange 2010 clustering and native data protection technology you will understand where us Exchange guys are coming from in terms of virtualization.

If your a SME business with up to 1000 users I highly recommend looking at the HP e5000. It's a DAG in a box solution with 2 physical blade servers and locally attached storage. It comes bundled with the Exchange licensing setup with a plug in and go approach. It offers a fast performing lean mean highly available email system for companies that require high levels of email up time. There is loads of information about the e5000 on the internet.

Saturday, May 21, 2011

Insight into the Ignore Feature in Outlook 2010

In this blog post we will take an insight into how the Ignore feature in Outlook 2010 works. The Ignore feature excludes you from conversation threads your not interested in. If your new to the Ignore feature please read the following URL:

http://office.microsoft.com/en-us/outlook-help/ignore-all-e-mail-messages-in-a-conversation-HA010361232.aspx

Where are Ignore rules stored?

Ignore rules are stored in the inside the users mailbox within a mailbox database. They are treated like an Outlook rule and stored in the XML blob. You can view these rules using MFCMAPI.

As an end user can I view a list of which conversations I'm ignoring?

As an end user using Outlook 2010 there is currently no way of viewing which conversations your ignoring.

As an Exchange Administrator you can use MFCMAPI to view within a mailbox which conversations are currently being ignored. In MFCMAPI under the users mailbox there is a "Conversation Action Settings" hidden folder. If you right click on this folder and click "View Associated Contents table" you can see all the ignored conversations. If you look at the MAPI property “PR_SUBJECT”, you will see the conversation subject string.

Is there a limit to the number of conversations you can ignore?

This is unknown however I have spoke to a senior esculation team within the Exchange product group and me mentioned he has seen over 100,000 before.

Does the client or server process ignored conversations?

The server does it.

What happens to emails that get ignored?

Ignored emails are moved to the deleted items folder within the Exchange 2010 mailbox dumpster. Exchange 2010 uses dumpster 2.0. For more information please see:

http://clintboessen.blogspot.com/2009/10/exchange-2010-dumpster-20.html

How long do ignores last for? For example if I do an ignore to a conversation thread then 6 months later someone does a reply to all on an unrelated topic, does this mean I will not recieve the email?

Ignore Rules only last for 180 days.

Do ignore rules use the message subject of an email?

No it uses the conversationID or conversationTopic properties generated by the Exchange 2010 server.

When you ignore a conversation thread does it include "re:" and "fw:"?

Not unless you’re using conversationTopic, in which case changing the subject will sometimes cause the message to escape. However, adding "re" or "fw" (or their localized equivalents) shouldn’t have any effect.

Hope this information has been helpful.

Monday, May 2, 2011

MapiExceptionMailboxInTransit: Unable to open message store. (hr=0x80004005, ec=1292)

Problem

I am performing cross-forest migration and I am having a had an issue when attempting to perform cross-forest mailbox moves.

A "user" for each "mailbox user" in the source forest exists in the destination forest. These objects were created by Identity Lifecycle Manager. These "user" objects were then prepared to be "mail users" by using the Prepare-MoveRequest.ps1 script for each user using the following syntax:

.\Prepare-MoveRequest.Ps1 -Identity "CN=Paul,OU=User Accounts,DC=source,DC=local" -RemoteForestDomainController "sourceforest-dc1.source.local" -RemoteForestCredential $Remote -LocalForestDomainController "destinationforest-dc01.destination.local" -LocalForestCredential $Local -TargetMailUserOU "OU=FromILM,DC=destination,DC=local" –UseLocalObject

I am trying to perform the mailbox move using the following command:

New-MoveRequest -Identity "CN=Paul,OU=FromILM,DC=destination,DC=local" -RemoteLegacy -TargetDatabase "Mailbox Database 0205912051" -RemoteGlobalCatalog "sourceforest-dc1.source.local" -RemoteCredential $Remote -TargetDeliveryDomain "contoso.com"

When I create the move request, the move request status update does not change, it just stays at "InProgress".

[PS] C:\>Get-MoveRequest

DisplayName Status TargetDatabase
----------- ------ --------------
Tamara InProgress Mailbox Database 0205912051

I bumped up the EventLogLevel for MSExchangeIS\9002 System\Move Mailbox" to medium by using the following powershell command:

Set-EventLogLevel "MSExchangeIS\9002 System\Move Mailbox" -Level Medium

In the destination Exchange 2010 server the following event logs were logged under "Application Log" with the EventLogLevel turned up.

Log Name: Application
Source: MSExchange Mailbox Replication
Date: 2/05/2011 4:27:37 PM
Event ID: 1101
Task Category: Mailbox Move
Level: Warning
Keywords: Classic
User: N/A
Computer: Ex2010.destination.local
Description:
Mailbox move for 'destination.local/FromILM/Paul' (fcfdc22e-d8d8-44f1-bdfe-140fdbc0d1ca) encountered a transient failure. The operation will be retried (9 out of 60).
Error code: -2147467259
MapiExceptionMailboxInTransit: Unable to open message store. (hr=0x80004005, ec=1292)
Diagnostic context:
Lid: 18969 EcDoRpcExt2 called [length=132]
Lid: 27161 EcDoRpcExt2 returned [ec=0x0][length=132][latency=0]
Lid: 23226 --- ROP Parse Start ---
Lid: 27962 ROP: ropLogon [254]
Lid: 17082 ROP Error: 0x50C
Lid: 26937
Lid: 21921 StoreEc: 0x50C
Lid: 27962 ROP: ropExtendedError [250]
Lid: 1494 ---- Remote Context Beg ----
Lid: 26426 ROP: ropLogon [254]
Lid: 22787 Error: 0x0
Lid: 13032 StoreEc: 0x8004010F
Lid: 7588 StoreEc: 0x8004010F
Lid: 6564 StoreEc: 0x8004010F
Lid: 24316
Lid: 2199 StoreEc: 0x50C
Lid: 17097 StoreEc: 0x50C
Lid: 8620 StoreEc: 0x50C
Lid: 1750 ---- Remote Context End ----
Lid: 26849
Lid: 21817 ROP Failure: 0x50C
Lid: 26297
Lid: 16585 StoreEc: 0x50C
Lid: 32441
Lid: 1706 StoreEc: 0x50C
Lid: 24761
Lid: 20665 StoreEc: 0x50C
Lid: 25785
Lid: 29881 StoreEc: 0x50C

Log Name: Application
Source: MSExchangeIS
Date: 2/05/2011 4:27:37 PM
Event ID: 9660
Task Category: Logons
Level: Warning
Keywords: Classic
User: N/A
Computer: Ex2010.destination.local
Description:
User Paul (/o=Destination/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Paul) failed to log on because their mailbox is in the process of being moved.

In the "System Log" I also noticed the Microsoft Exchange Mailbox Replication service crashing during the move.

Log Name: System
Source: Service Control Manager
Date: 3/05/2011 9:07:13 AM
Event ID: 7031
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: Ex2010.destination.local
Description:
The Microsoft Exchange Mailbox Replication service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 5000 milliseconds: Restart the service.

My Environment

In the destination domain I have:
- A single Exchange 2010 server with CAS, MB and HT roles installed running SP1, Update Rollout 4.
- Public folders

In the source domain I have:
- A single Exchange 2003 server running Service Pack 2
- An Exchange 2010 SP1 Server running Update Rollout 4 only running the CAS role. This is for MRSProxy.

Resolution

The problem occured because MRSProxy was not enabled on the Exchange 2010 Client Access Server in the source forest. When mailboxes are moved from one Exchange 2010 forest to another Exchange 2010 forest, the process is handled through Exchange 2010 Client Access Servers using the MRSProxy service. The only port required to be open between the forests for MRSProxy to use HTTPS traffic is port 443. This works even if the source mailboxes are on 2003 or 2007 MBX servers as long as an Exchange 2010 CAS server exists in both organizations.

MRSProxy is not enabled on client access servers by default. You need to enable this on the source forest client access server. To enable this navigate to the following directory on the Exchange 2010 CAS server in the source forest.

Exchange Installation Path\V14\ClientAccess\ExchWeb\EWS\web.config

Open the web.config in a text editor and find where it says under MRSProxyConfiguration

IsEnabled="false"
MaxMRSConnections="100"
DataImportTimeout="00:01:00

Change IsEnabled to true:

I then rebooted the Exchange 2010 CAS server in the source forest. Upon reboot the cross-forest mailbox moves worked.

The Exchange 2010 CAS Server in the source domain will proxy the move request from the Ex2003 mailbox server.

I also found a TechNet article that explains how to enable this:

http://technet.microsoft.com/en-us/library/ee732395.aspx

NotifyChangeDirectory throws INSUFFICIENT RESOURCES

We had an application called Kentico CMS which we deployed on Windows Server 2008 R2 with IIS 7.5.

Looking in the event log for the app, it's getting "File change notification errors" and tracking (causing the restarts) it with ProcMon we see:

NotifyChangeDirectory calls being made - for a while they succeed and then they start throwing INSUFFICIENT RESOURCES in the status.

The following errors were experienced in event viewer under the application logs:

EventID: 9145
EventType: W
EventTime: 2/11/2011 12:25:04 AM
Source: Application_End
EventCode: ENDAPP

EventDescription: Message: File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_groups
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_groups
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_medialibrary
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_messageboards
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_newsletter
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_polls
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_reporting
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_webanalytics
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_tools
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_groups
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_medialibrary
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_messageboards
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_newsletter
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_polls
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_reporting
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\app_themes\default\images\cmsmodules\cms_webanalytics
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\cmspages
CONFIG change
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\cmspages
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\CMSPages
File Change Notification Error in \\files\domains$\k\kwinana.wa.gov.au\public_html\cmspages
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown<br />
Shutdown stack: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
at System.Web.Hosting.HostingEnvironment.InitiateShutdown()
at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace)
at System.Web.Configuration.HttpConfigurationSystem.OnConfigurationChanged(Object sender, InternalConfigEventArgs e)
at System.Configuration.Internal.InternalConfigRoot.OnConfigChanged(InternalConfigEventArgs e)
at System.Configuration.BaseConfigurationRecord.OnStreamChanged(String streamname)
at System.Web.Configuration.WebConfigurationHostFileChange.OnFileChanged(Object sender, FileChangeEvent e)
at System.Web.DirectoryMonitor.FireNotifications()
at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state)
at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)<br />
Call stack: at Global.LogApplicationEnd()
at Global.Application_End(Object sender, EventArgs e)
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Web.HttpApplication.ProcessSpecialRequest(HttpContext context, MethodInfo method, Int32 paramCount, Object eventSource, EventArgs eventArgs, HttpSessionState session)
at System.Web.HttpApplicationFactory.FireApplicationOnEnd()
at System.Web.HttpApplicationFactory.Dispose()
at System.Web.HttpRuntime.Dispose()
at System.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain(Object state)
at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)

And...

EventMachineName: GIR1
EventID: 289983
EventType: E
EventTime: 2/11/2011 4:24:23 PM
Source: Application_End
EventCode: EXCEPTION
EventDescription: Message: File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
Directory rename change notification for '\\files\domains$\o\omsintl.com\public_html'.
public_html dir change or directory rename
Change Notification for critical directories.
bin dir change or directory rename
Change Notification for critical directories.
App_Browsers dir change or directory rename
CONFIG change
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html\cmsdesk
CONFIG change
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html\cmsdesk
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html\cmspages
CONFIG change
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
File Change Notification Error in \\files\domains$\o\omsintl.com\public_html\cmspages<br />
Shutdown stack: at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
at System.Web.Hosting.HostingEnvironment.InitiateShutdown()
at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace)
at System.Web.Configuration.HttpConfigurationSystem.OnConfigurationChanged(Object sender, InternalConfigEventArgs e)
at System.Configuration.Internal.InternalConfigRoot.OnConfigChanged(InternalConfigEventArgs e)
at System.Configuration.BaseConfigurationRecord.OnStreamChanged(String streamname)
at System.Web.Configuration.WebConfigurationHostFileChange.OnFileChanged(Object sender, FileChangeEvent e)
at System.Web.DirectoryMonitor.FireNotifications()
at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback)
at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state)
at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)<br />
Call stack: at Global.LogApplicationEnd()
at Global.Application_End(Object sender, EventArgs e)
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Web.HttpApplication.ProcessSpecialRequest(HttpContext context, MethodInfo method, Int32 paramCount, Object eventSource, EventArgs eventArgs, HttpSessionState session)
at System.Web.HttpApplicationFactory.FireApplicationOnEnd()
at System.Web.HttpApplicationFactory.Dispose()
at System.Web.HttpRuntime.Dispose()
at System.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain(Object state)
at System.Threading._ThreadPoolWaitCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)

The following INSUFFICIENT RESOURCES errors were experienced in process monitor:

To resolve the problem, there are a few settings that can be configured in IIS and ASP.NET To optimize this behavior.

To resolve the issue we set the DWORD HKLM\Software\Microsoft\ASP.NET\FCNMode is a Decimal value of "2" as per http://support.microsoft.com/kb/911272.

If your running a 32bit process on a 64bit server then set this key instead:

HKLM\SOFTWARE\Wow6432Node\Microsoft\ASP.NET\FCNMode

Clint Boessen's Blog

Avantgarde Technologies