.. Reminder for header structure:        
  Parts (H1)          : #################### with overline
  Chapters (H2)       : ******************** with overline
  Sections (H3)       : ====================
  Subsections (H4)    : --------------------
  Subsubsections (H5) : ^^^^^^^^^^^^^^^^^^^^
  Paragraphs (H6)     : """""""""""""""""""""

.. |date| date::

.. meta::
  :description: Working with replicated Active Directory
  :keywords: Security, Samba-AD, documentation, replication, tombstoneLifetime,
    USN, whenChanged

.. _samba_replication:

########################################
Working with replicated Active Directory
########################################

********************
General presentation
********************

Active Directory provides a set of authentication servers that replicate to allow:

* Redundancy (protection against failure).

* Better load management (several ADs to distribute requests).

* Or to provide a local authentication server for remote sites.

.. note::

  Bandwidth consumption for Active Directory replication is generally negligible on modern networks.
  Indeed, the Active Directory architecture was imagined in the 90’s when an inter-site ISDN 64kbps connection was already an envied luxury.

Replication is one of the most complicated and least understood topics in Active Directory operation.
Even if it is not necessary to understand these concepts (and most sysadmin, even certified ones, don’t really master them), it is still important to understand the basics to facilitate diagnosis in case of problems.

Putting DCs on remote sites is rarely a choice related to a bandwidth issue, but rather a desire to ensure continuity of service even if the :abbr:`WAN (Wide Area Network)` network has fallen down.

.. note::

  Samba-AD is a mature product that can manage domains with several dozen domain controllers in replication without any problem.
  The largest domain that Tranquil IT co-administers has more than 140 domain controllers using replication.

*********************
How replication works
*********************

The operation of Active Directory replication is very different from the replication mode of :program:`OpenLDAP` :command:`Syncrepl` or other replication systems:

* Active Directory replication works in *Pull* mode (the server pulls modifications from other servers) and not in *Push* mode (the server sends its modified data).

* The state of the replications is contained in the AD tree itself, there is no log file that records all the modifications.

* Although AD servers replicate together, they are not exactly identical. 
  Indeed there are objects and attributes that are not replicated, such as the state of the replications.
  If we have to compare two ADs we must therefore exclude certain attributes to avoid unnecessary *noise*, and thus facilitate a post-incident replication search for example.

************************
Notification vs. polling
************************

Active Directory servers on the same Active Directory site send each other notifications when changes are made.

Example: when *srvads1* has a change, it sends a notification to *srvads2*, *srvads2* receives the notification from *srvads1* and connects to *srvads1* to ask for changes that have taken place since its last replication.

This ensures that changes are spread quickly.
Indeed, since user workstations and servers can connect indifferently to the domain controllers of the same site, it is important that consistency between ADs to be kept.

On the other hand if the AD servers are on different sites, the remote server *srvads3* will regularly ask its replication partner *srvads1* if there have been any changes since the last replication.
This mode of operation is designed to save bandwidth.

*************
USN Operation
*************

:abbr:`USN (Unique sequence Number)` is a monotonous counter (which only increases) which is specific to each AD server (it is not replicated between ADs in a domain).
The USN counter is incremented by 1 each time a change is made to an object, and the value is assigned to the ``uSNChanged`` field of the changed object.

Thus the object with the highest USN is the last object changed in the AD.
Each AD object has a unique USN on the same AD. Since USNs are not replicated, the same object has a different ``uSNChanged`` value on each Active Directory.
Since there is no relationship between the ``uSNChanged`` attributes between two DCs, the same value can be set on two DCs (but never on the same domain controller).

When an AD replicates with another AD, it stores the ``USN max`` value of its last replication with the remote AD.
Each AD stores a value of the ``USN max`` of the replicating ADs at the root of the main partition (DC=mydomain,DC=lan) in the ``repFrom`` and ``repTo`` attributes (these are binary data not directly readable) which are used to display the result of the **samba-tool drs showrepl** command.

*********************
whenChanged attribute
*********************

The ``whenChanged`` attribute is not replicated from one AD to another.
The value of this attribute corresponds to the moment when the object was modified, either by an external call (LDAP modification made by an administrator, a user station, an application, etc.) or by a replication.

That is to say that on the AD where the modification was made, the attribute ``whenChanged`` will have the *timestamp* of the moment of the modification, while the other ADs will have the timestamp corresponding to the moment when it was replicated.

.. _samba_tombstone_lifetime:

*************************************************
Deleting tombstoneLifetime objects and attributes
*************************************************

As mentioned above, the replication status is stored in the AD tree itself.
The case of deleted objects then arises. If we definitively delete an object from the LDAP tree, it will have no more USN and therefore we will not be able to replicate the deletion.

To solve this problem the AD developers have used the concept of a *trash can* where objects are moved when they are deleted: the container ``CN=Deleted Obects,DC=mydomain,DC=lan``.
When an LDAP entry is deleted several things happen:

* The attribute ``ISDELETED`` is added with the value 1.

* Attributes that are not necessary for deletion replication are removed (``members``, ``memberOf``, ``address``, ``mail``, etc).

* The object is moved into the ``CN=Deleted Objects`` container.

* The DN of the object, and thus its CN for users and groups, is renamed by suffixing it with *\0ADEL:* and its *objectGUID*. For example, when deleting the object ``CN=dcardon`` would have as new DN:

  .. code-block:: bash

    DN: CN=dcardon\0ADEL:<object_guid>,CN=Deleted Objects,DC=mydomain,DC=lan

.. note::

  With Active Directory 2003 Microsoft has introduced a new concept to help restore a deleted object: *the Active Directory Recycle Bin*.
  The recycle bin allows to recover a deleted object and to put it back in the normal tree structure.

  This feature is not yet supported in Samba-AD.

******************************
Managing replication conflicts
******************************

When the same object has been modified on two domain controllers during replication, this will cause reconciliation problems.
Active Directory has different methods for reconciling conflicting entries.
For example, if a ``mail`` attribute was changed on two ADs at the same time, the entry with the most recent timestamp will be kept.

.. note::

  For replications, it is not the *timestamps* that are used to make comparisons between the different DAs but the USNs.
  The timestamps are only used here for reconciliation when changes are in conflict during replications.

If the DC is unable or unwilling to make a decision on which value to keep, e.g. the same account created on two DCs at the same time, the entry with the highest timestamp will be renamed according to the following standard:

.. code-block:: ini

   CN=dcardon\0ACNF:<object_guid>,OU=company_users,DC=testing,DC=lan

This prevents :abbr:`DN (Distinguished Name)` from conflicting and allows to finish replication (you can’t have two objects with the same DN in an AD tree).
On the other hand it does not guarantee the uniqueness of the ``samAccountName``.
Indeed to manage this case, Microsoft engineers did not put a unique index on the attribute ``samAccountName``.

Renaming ``samAccountName`` to avoid duplicates is the role of another recurring task that validates the uniqueness of this attribute, and renames the conflicting attributes.
This recurring task is not currently implemented in Samba-AD. However, this case occurs very rarely.

Conflicting objects can appear in different partitions of the AD, especially in the *DNSDomainZones* and *ForestDNSZones* partitions.
Entries renamed *OACNF* in the DNS console can normally be deleted without any impact.

********************
Replication topology
********************

Managing replication connections, role of the KCC
=================================================

In order for replications to go well in a domain, it is important that each controller is connected to every other controller through one or more *hops*.
If a change is made to *srvads1*, it can be replicated to *srvads2* which will replicate it to *srvads3*.
Even if *srvads1* and *srvads3* do not replicate directly, the modification will be propagated by transitivity.

The role of interconnecting ADs is that of the :abbr:`KCC (Knowledge Consistancy Checker)`, the verification of the consistency of the knowledge.
It is a name that reflects its role well.
The KCC uses the information configured by the administrator in the Site and Services Active Directory console to choose its AD interconnection strategy.
However, if the configuration set by the administrator does not allow the KCC to have a fully connected network, the KCC may make decisions that may be in disagreement with what the administrator had intended.

By default the KCC will make at least one replication connection with another AD at the same site, and with at least one AD at a remote site to which it must make replications.

In practice we advise to define star replication topologies with a replication site in the center and replication sites around it that replicate only with the replication site.
This simple method will avoid loops (see paragraph on the subject).

The notion of *bridgehead* can be defined on a remote site for the KCC to choose this default *bridgehead* domain controller to manage replications with the remote site.

Depending on the number of controllers in the domain, it can be interesting to have a site dedicated to replication.

Replication loop
================

The replication topology can cause replication loops.
That is to say, for example, that server A replicates with B and C, and that B and C replicate with each other.
We can therefore have a modification on A which is propagated to B which is propagated again to C which propagates it again to server A.

In this case the replication protocol has a mechanism to recognize this duplicate replication information and to ignore it.

Replicating the SYSVOL directory
================================

The :file:`SYSVOL` directory of an Active Directory contains the :abbr:`GPO (Group Policy Object)` definition files and the *NetLogon* login scripts for the domain.
The :file:`SYSVOL` directory must be available on all AD servers of a domain with the same content.

Replication of the LDAP DRS (Directory Replication Service) tree is a completely different protocol from that used for replication of the :file:`SYSVOL` directory.

To ensure the replication of the :file:`SYSVOL` directory, Microsoft has implemented two protocols:

* :abbr:`FRS (File Replication Service)`: protocol integrated in AD from AD2k to AD2k16 (except last version October 2019).

* :abbr:`DFS-R (Distributed FileSystem - Replication)`: protocol integrated in AD since AD2k8.

Currently Samba-AD does not support FRS or DFS-R. There is no official Samba methodology for replicating the :file:`SYSVOL` directory.
There are different methods.
You can use :ref:`the Tranquil IT method <samba_tis_sysvolsync>`.

The DFS-R protocol used for the :file:`SYSVOL` directory is the same protocol used to replicate file shares between Microsoft file servers.
It is an extension of the DFS protocol.
Samba supports the DFS protocol, but not the DFS-R protocol.

************************************
Case of read-only domain controllers
************************************

Starting with Active Directory 2008, it is possible to have read-only Active Directory servers :abbr:`RODC (Read Only Domain Controller)`.
As the name implies, these domain controllers do not allow local data modification.
Changes are redirected to a normal domain controller (which will be called in this documentation by abuse of language :abbr:`RWDC (Read-Write Domain Controller)`).

However, as stated earlier in this documentation, the replication status data itself is stored in the AD.
This implies that the AD is not entirely read-only.
Indeed some attributes are still read/write. The ``instanceType`` attribute defines whether the attribute is read/write (*value 4*) or read-only (*value 2*).
