Hunting With Active Directory Replication Metadata

With the recent release of BloodHound’s ACL Attack Path Update as well as the work on Active Directory DACL backdooring by @_wald0 and myself (whitepaper here), I started to investigate ACL-based attack paths from a defensive perspective. Sean Metcalf has done some great work concerning Active Directory threat hunting (see his 2017 BSides Charm “Detecting the Elusive: Active Directory Threat Hunting” presentation) and I wanted to show how replication metadata can help in detecting this type of malicious activity.

Also, after this post had been drafted, Grégory LUCAND pointed out to me the extensive article (in French) he authored on the same subject area titled “Metadata de réplication et analyse Forensic Active Directory (fr-FR)”. He walks through detecting changes to an OU, as well as an excellent deep dive (deeper than this article) into how some of the replication components work such as linked value replication. I highly recommend you check his post out, even if you have to use Google Translate as I did :)

I’ll dive into some background concerning domain replication metadata and then will break down each ACL attack primitive and how you can hunt for these modifications. Unfortunately, replication metadata can be a bit limited, but it can at least help us narrow down the modification event that took place as well as the domain controller the event occurred on.

Note: all examples here use my test domain which runs at a Windows 2012 R2 domain functional level. Other functional domain versions will vary. Also, all examples were done in a lab context, so exact behavior in a real network will vary as well.

Active Directory Replication Metadata

When a change is made to a domain object on a domain controller in Active Directory, those changes are replicated to other domain controllers in the same domain (see the “Directory Replication“ section here). As part of the replication process, metadata about the replication is preserved in two constructed attributes, that is, attributes where the end value is calculated from other attributes. These two properties are msDS-ReplAttributeMetaData and msDS-ReplValueMetaData.

Sidenote: previous work I found on replication metadata includes this article on tracking UPN modification as well as this great series of articles on different use cases for this data. These articles show how to use both REPADMIN /showobjmeta as well as the Active Directory cmdlets to enumerate and parse the XML formatted data returned. A few months ago, I pushed a PowerView commit that simplifies this enumeration process, and I’ll demonstrate these new functions throughout this post.

msDS-ReplAttributeMetaData

First off, how do we know which attributes are replicated? Object attributes are themselves represented in the forest schema and include a systemFlags attribute that contains various meta-settings. This includes the FLAG_ATTR_NOT_REPLICATED flag, which indicates that the given attribute should not be replicated. We can use PowerView to quickly enumerate all of these non-replicated attributes using a bitwise LDAP filter to check for this flag:

Get-DomainObject -SearchBase 'ldap://CN=schema,CN=configuration,DC=testlab,DC=local' -LDAPFilter '(&(objectClass=attributeSchema)(systemFlags:1.2.840.113556.1.4.803:=1))' | Select-Object -Expand ldapdisplayname

If we want attributes that ARE replicated, we can just negate the bitwise filter:

Get-DomainObject -SearchBase 'ldap://CN=schema,CN=configuration,DC=testlab,DC=local' -LDAPFilter '(&(objectClass=attributeSchema)(!systemFlags:1.2.840.113556.1.4.803:=1))' | Select-Object -Expand ldapdisplayname

So changes to any of the attributes in the above set on an object are replicated to other domain controllers, and, therefore, have replication metadata information in msDS-ReplAttributeMetaData (except for linked attributes, more on that shortly). Since this is a constructed attribute, we have to specify that the property be calculated during our LDAP search. Luckily, you can already do this with PowerView by specifying -Properties msDS-ReplAttributeMetaData for any of the Get-Domain* functions:

You can see that we get an array of XML text blobs that describes the modification events. PowerView’s brand new Get-DomainObjectAttributeHistory function will automatically query msDS-ReplAttributeMetaData for one or more objects and parse out the XML blobs into custom PSObjects:

Breaking down each result, we have the distinguished name of the object itself, the name of the replicated attribute, the last time the attribute was changed (LastOriginatingChange), the number of times the attribute has changed (Version), and the directory service agent distinguished name the change originated from (LastOriginatingDsaDN). The “Sidenote: Resolving LastOriginatingDsaDN” section at the end of this post shows how to resolve this distinguished name to the appropriate domain controller object itself. Unfortunately, we don’t get who made the change, or what the previous attribute value was; however, there are still a few interesting things things we can do with this data which I’ll show in a bit.

msDS-ReplValueMetaData

In order to understand msDS-ReplValueMetaData and why it’s separate from msDS-ReplAttributeMetaData, you need to understand linked attributes in Active Directory. Introduced Windows Server 2003 domain functional levels, linked value replication “allows individual values of a multivalued attribute to be replicated separately.” In English: attributes that are constructed/depend on other attributes were broken out in such a way that bits of the whole could be replicated one by one, instead of the entire grouping all at once. This was introduced in order to cut down on replication traffic in modern domain environments.

With linked attributes, Active Directory calculates the value of a given attribute, referred to as the back link, from the value of another attribute, referred to as the forward link. The best example of this is member / memberof for group memberships: the member property of a group is the forward link while the memberof property of a user is the backward link. When you enumerate the memberof property for a user, the backlinks are crawled to produce the final membership set.

There are two additional caveats about forward/backwards links you should be aware of. First, forward links are writable, while backlinks are not, so when a forward-linked attribute is changed the value of the associated backlink property is updated automatically. Second, because of this, only forward-linked attributes are replicated between domains, which then automatically calculate the backlinks. For more information, check out this great post on the subject.

A huge advantage for us is that because forward-linked attributes are replicated in this way, the previous values of these attributes are stored in replication metadata. This is exactly what the msDS-ReplValueMetaData constructed attribute stores, again in XML format. The new Get-DomainObjectLinkedAttributeHistory PowerView function wraps this all up for you:

We now know that member/memberof is a linked set, hence the modification results to member above.

In order to enumerate all forward-linked attributes, we can again examine the forest schema. Linked properties have a Link and LinkID in the schema – forward links have an even/nonzero value while back links have an odd/nonzero value. We can grab the current schema with  [DirectoryServices.ActiveDirectory.ActiveDirectorySchema]::GetCurrentSchema() and can then use the FindAllClasses() method to enumerate all the current schema classes. If we filter by class properties that are even, we can find all linked properties that therefore have their previous values replicated in Active Directory metadata.

https://gist.github.com/HarmJ0y/74556b87c4040638f1599699a26e7b0a

There are a lot of results here, but the main ones we likely care about are member/memberOf and manager/directReports, unfortunately. So member and manager are the only interesting properties for an object we can track previous modification values on. However, like with msDS-ReplAttributeMetaData, we unfortunately can’t see who actually initiated the change.

Hunting With Replication Metadata

Alright, so we have a bunch of this seemingly random replication metadata, how the hell do we actually use this to “find bad?” Metadata won’t magically tell you an entire story, but I believe it can start to point you in the right direction, with the added bonus of being pre-existing functionality already present in your domain. I’ll break down the process for hunting for each ACL attack primitive that @_wald0 and myself covered, but for most situations the process will be:

1. Use Active Directory replication metadata to detect changes to object properties that might indicate malicious behavior.
2. Collect detailed event logs from the domain controller linked to the change (as indicated by the metadata) in order to track down who performed the modification and what the value was changed to.

There’s one small exception to this process:

Group Membership Modification

This one is the easiest. The control relationship for this is the right to add members to a group (WriteProperty to Self-Membership) and the attack primitive through PowerView is Add-DomainGroupMember. Let’s see what the information from Get-DomainObjectLinkedAttributeHistory can tell us:

In the first entry, we see that ‘EvilUser’ was originally added (TimeCreated) at 21:13 and is still present (TimeDeleted == the epoch). Version being 3 means that the EvilUser was originally added at TimeCreated, deleted at some point, and then readded at 17:53 (LastOriginatingChange). Big note: these timestamps are in UTC!

In the second example, TestOUUser was added to the group at 21:12 (TimeCreated) and removed at 21:19 (TimeDeleted). The Version being even, as well as the non-epoch TimeDeleted value, means that this user is no longer present in the group and was removed at the indicated time. PowerView’s last new function, Get-DomainGroupMemberDeleted, will return just metadata components indicating deleted users:

If we want more details, we have the Directory System Agent (DSA) where the change originated, meaning the domain controller in this environment that handled the modification (PRIMARY here). Since we have the group that was modified (TestGroup) and the approximate time the change occurred (21:44 UTC), we can go to the domain controller that initiated the change (PRIMARY) to pull more event log detail (see the “Sidenote: Resolving LastOriginatingDsaDN” section for more detail on this process).

The auditing we really want isn’t on by default, but can be enabled with “Local Computer Policy -> Computer Configuration -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> Account Management -> Audit Security Group Management”:

This will result in event log IDs of 4735/4737/4755 for modifications to domain local, global, and universally scoped security groups:

We can see in the event detail that TESTLAB\dfm.a is the principal who initiated the change, with correlates with the deletion event we observed in the replication metadata.

User Service Principal Name Modification

This is also another interesting case. The vast majority of users will never have a service principal name (SPN) set unless the account is registered to… run a service. SPN modification is an attack primitive that I’ve spoken about before, and grants us a great opportunity to take advantage of the “Version” field of the metadata, i.e. the number of times a property has been modified.

If we set and then unset a SPN on a user, the Version associated with the attribute metadata will be even, indicating there used to be a value set:

If we enable the “Audit User Account Management” and “Audit Computer Account Management” settings, we can grab more detailed information about the changes:

The event ID will be 4738, but the event log detail unfortunately does not break out the value of servicePrincipalName on change. However, we do again get the principal who initiated the change:

Note the logged timestamp of the event matches the LastOriginatingChange of the replication metadata. If we wanted to do a mass enumeration of EVERY user account that had a SPN set and then deleted, we can use -LDAPFilter ‘(samAccountType=805306368)’ -Properties servicePrincipalName, and filtering out anything with an odd Version:

Object Owner/DACL Modification

I originally thought this scenario would be tough as well, as I had guessed that whenever delegation is changed on an OU those new rights were reflected in the ntSecurityDescriptor of any user objects down the inheritance chain. However, I was mistaken- any delegation changes are in the ntSecurityDescriptor of the OU/container, and I believe those inherited rights are calculated on LDAP enumeration by the server. In other words, the ntSecurityDescriptor of user/group/computer objects should only change when the owner is explicitly changed, or a new ACE is manually added to that object.

Since an object’s DACL and owner are both stored in ntSecurityDescriptor, and the event log data doesn’t provide details on the previous/changed value, we have no way of knowing if it was a DACL or owner based changed. However, we can still figure out who initiated the change again using event 4738:

Just like with SPNs, we can also sweep for any users (or other objects) that had their DACL or owner changed (i.e. Version > 1):

If we periodically enumerate all of this data for all users/other objects, we can start to timeline and calculate change deltas, but that’s for another post :)

User Password Reset

Unfortunately, this is probably the hardest scenario. Since password changes/resets are a fairly common occurrence, it’s difficult to reliably pull a pattern out of the data based solely on the password last set time. Luckily however, enabling the “Audit User Account Management” policy also produces event 4723 (a user changed their own password) and event 4724 (a password reset was initiated):

And we get the time of the reset, the user that was force-reset, and the principal that initiated it!

Group Policy Object Editing

If you’re able to track down a malicious GPO edit, and want to know the systems/users affected, I’ve talked about that process as well. However, this section will focus on trying to identify what file was edited and by whom.

Every time a GPO is modified, the versionNumber property is increased. So if we pull the attribute metadata concerning the last time versionNumber was modified, and correlate this time (as a range) with edits to all files and folders in the SYSVOL path we can identify the files that were likely modified by the last edit to the GPO. Here’s how we might accomplish that:

https://gist.github.com/HarmJ0y/43f59722565de09670be6b541935562b

You can see above that the Groups.xml group policy preferences file was likely the file edited. To identify what user made the changes, we need to tweak “Local Computer Policy -> Computer Configuration -> Windows Settings -> Security Settings -> Advanced Audit Policy Configuration -> DS Access -> Audit Active Directory Service Changes”:

We can then comb for event IDs of 5136 to and use the alert data to narrow down the event that caused the versionNumber modification:

We see the distinguishedName of the GPO object being modified, as well as who initiated the change. There is some more information here in case you’re interested.

Sidenote: Resolving LastOriginatingDsaDN

As I previously mentioned, the LastOriginatingDsaDN property indicates the the last directory service agent that the given change originated from. For us to make the most use of this information, we want to map this particular DSA record back to the domain controller it’s running on. This is unfortunately a multi-step process, but I’ll walk you through it below using PowerView.

Say the change we want to track back is the following deleted Domain Admin member:

We see that the DSA distinguished name exists in the CN=Configuration container of the associated domain. We can retrieve the full object this references by using PowerView’s Get-DomainObject with the -SearchBase set to “ldap://CN=Configuration,DC=testlab,DC=local”:

We see above that this has a NTDS-DSA object category, and we see a serverreferencebl (backlink) property that points us in the right direction. If we resolve this new object DN, we get the following:

Now we see the actual domain controller distinguished name linked in the msdfsr-computerreference property of this new result, and the serverreference matches the LastOriginatingDsaDN from our initial result. This means we can skip that middle step and query for this ms-DFSR-Member object directory, linked by the serverreference attribute, by way of a custom LDAP filter. To finish, we can extract the msdfsr-computerreference property and resolve it to the actual domain controller object:

https://gist.github.com/HarmJ0y/ab7f72748b2d4c62786ba622a80a5b18

Success! \m/

Wrapup

Hopefully this causes at least a few people to think about the hunting/forensic possibilities from the Active Directory side. There’s a wealth of opportunity here to detect our ACL based attack components, as well as a myriad of other Active Directory “bad”. Also, observant readers may have noticed that I ignored an entire defensive component here, system access control lists (SACLs), which provide that chance to implement additional auditing. I’ll cover SACLs in a future post, showing how to utilize BloodHound to identify “key terrain” to place very specific SACL auditing rules.

Until then, have fun!