I was creating a new active directory domain in a forest that had a single labelled DNS suffix for the forest root domain. I had the "AllowSingleLabelDnsDomain" registry DWORD with a value of "1" on all domain controllers in the forest root domain and the tree new domain I was adding to allow the domain controllers to join the single labelled domain suffix as per:
I also had set UpdateTopLevelDomainZones to enabled on the forest root level single labelled domain on the Default Domain Controllers Policy as per KB300684:
Computer Configuration\Administrative Templates\Network\DNS Client\Update Top Level Domain Zones with a value of "1".
This group policy change adds UpdateTopLevelDomainZones with a value of 1 in the registry of each domain controller.
However KB300684 does not say it needs to be done in every domain in the forest, just the root domain controllers in the forest root domain (please note Microsoft are updating this document correcting this so if you are reading this after it has been already changed my appologies.) Because this was not done in my other domain replication was failing from "domain.local" to "domain" but succeeding from "domain" to "domain.local" as in the following diagram:
Also the KCC was not able to generate replication connection objects from domain.local to domain.
Running repadmin /showrepl on a domain controller in the "domain" domain showed the following errors:
Last error: 8524 (0x214c):
The DSA operation is unable to proceed because of a DNS lookup failure.
******* WARNING: KCC could not add this REPLICA LINK due to error.
After a support call to Claude Wang from the Microsoft AD Team he told me to make the "Update Top Level Domain Zones" group policy change on the default domain controllers policy on every domain. This resolved the problem.
What was actually causing the the issue from a more technical point was the forest root domain has a list of CNAME records under the _msdcs container in DNS that reference the guid of each domain controller in the forest. Only the forest root domain has a list of these CNAME records, no other domains have this information. For single labelled forest root domains other domain controllers in child/tree domains are not able to write their CNAME record to this location without the UpdateTopLevelDomainZones value set... this issue only occurs with single labelled forest root domains.
When the issue was occuring we only had 2 CNAME records for the "domain" domain, not the "domain.local" domain. Now we have all 4:
Also I'd like to point out that the "domain.local" domain does not contain any of these CNAME records as it is not the forest root domain!