Increasing Granularity of API Output

Intended Audience: Technical staff with basic knowledge of Python

Purpose: To provide a walkthrough in the use of include_details and how it may be used to obtain a granular view of common and specific fields to solve business problems.

 

Introduction

In complex multi-domain environments, it becomes increasingly difficult to determine the lineage of information originating from various sources. With multiple connections associated with each adapter, there becomes an increasingly higher convergence of the same type of information from multiple sources. As the number of sources of information increase, it becomes increasingly difficult to determine where information has originated from and thus increasingly difficult to act upon that information. Identifying the source of information is critical when remediating stale data, correcting user password policies across domains, and identifying sources of data conflicts.

In this article we will explore the means within which we can associate field values to their source and explore a use case where this is useful. This article is intended for organizations where the number of adapters and connections are large and complex, and where users currently utilize the Axonius Python API Client. For further information on the API client, please see the documentation section at the end of the article.

 

Use Case: Password Policy Management

When trying to enforce password policies, we need to identify where the data originated from in order to quickly remediate the accounts which are not following appropriate standards. To effectively enforce password policies in an environment where we have multiple domains (for this example, we will use Active Directory), we run into the issue of having multiple accounts per user, with only 1 requiring changes. 

The current behavior of the UI and API is to return all information (and thus all accounts) associated to any user that may have an account that has been identified as having an issue. Put simply, query output is presented at the User level, not by account or domain, thus increasing the amount of noise when attempting to conduct account level queries.

In the Axonius Python API Client, we can use the include_details option to provide a granular view of each user record. include_details fills a separate _details field for each field in the query based on its originating connection. The _details fields are a list of all information from each connection and are index correlated, meaning that all information on each individual position of each list correlates to the information in other _details fields with the same index.

By obtaining this granular view of each field, users are now provided a means of accomplishing an account or domain level view for each user, and reducing noise originating from other out-of-scope domains.

 

What is Index Correlation?

Index correlation is the ordering of two or more lists so that the values in each list correspond with each other. This means that the ordering of each list is identical and is tied to a separate field (in this case the connection) which is the common attribute that links the lists.

The process of how the _details fields are populated at a high level includes the following:

  1. The first connection is picked up
  2. Populate all fields that the connection has values for (Index 0)
  3. Any fields that are not populated in step 2, are given a NULL value instead (Index 0)
  4. The Second connection is picked up
  5. Populate all fields that the connection has values for (Index 1)
  6. Any fields that are not populated in step 5, are given a NULL value instead (Index 1)
  7. And so on…

As an illustration lets look a few Mockup Pseudo-Axonius records (Note: Field Names are not exact)

Given the above input, we would expect the API Output to appear as:


Adapters: [ “Active Directory”, “Active Directory”, “Active Directory”,”Qualys”],
Connection: [“Connection1”,”Connection2”,”Connection3”,”Connection4”],
Domain: [“Domain1@example.com”,”Domain2@example.com”,”Domain3@example.com”]
Domain_Details: [“Domain1@example.com”,”Domain2@example.com”,”Domain3@example.com”, ”Domain1@example.com”]
AD_User: “User123”
AD_User_Details: [“User123”,“User123”,“User123”,NULL]
Password_Flag: [“True”,”False”]
Password_Flag_Details:  [“True”,”False”,”False”,NULL]
}

Using the example, we can see that the first entry for each list corresponds to the first connection in the table, the second entry in each list to the second connection in the table and so on. Additionally we can see where, in cases where we do not have the information (such as connection 4 which is a Qualys entry), the _details fields will contain NULL where information was not available.

Due to how these fields are populated, we are able to process filters at a connection level using the contents of each _details field, knowing that the information contained is all from the same source.

 

Pulling Connection Level Data

To illustrate the solution to this use case, assume that we wish to create a table which presents each username, its domain and whether or not the password policy allows the password to expire.'

This report may appear in the following format:

To source the underlying information in an unformatted way, the following command may be used:

user = client.users.get(fields=[
"adapters_data.active_directory_adapter.ad_sAMAccountName",
"adapters_data.active_directory_adapter.ad_uac_dont_expire_password",
"adapters_data.active_directory_adapter.domain",
"adapters_data.active_directory_adapter.last_seen",
"specific_data.data.fetch_time"],
include_details=True) 

This will retrieve all of the User asset information in the typical format that is provided from the Axonius Python API Client. However, we have made 2 notable changes to the standard users.get() command:

  • Option: Fields
    • This specifies what fields we want to include in addition to the standard information. 
    • Here you can see we supplied several fields:
      • The Account Name
        • "adapters_data.active_directory_adapter.ad_sAMAccountName"
      • The Password expires flag
        • "Adapters_data.active_directory_adapter.ad_uac_dont_expire_password”
      • The Domain
        • "Adapters_data.active_directory_adapter.domain"
      • The time this user was last seen in this domain
        • "Adapters_data.active_directory_adapter.last_seen"
      • The time this information was last pulled from AD
        • "Adapters_data.active_directory_adapter.fetch_time"
  • Option: Include_details
    • This flags to the API whether you wish to include the granular information that makes up each field (providing a _details field for each field returned).
    • This will result in a list being provided for each field per each connection to each adapter, ordered consistently between fields
      • E.g. user[“ad_sAMAccountName”][0] will correspond to the information supplied for user[“domain”][0] and so on.

 

With these two options we are including all fields which we will  examine and have also provided an index correlated list of data for each field, describing its value for each connection/domain.

 

Leveraging Detail fields

Using the include_details and fields additions to our users.get() call, the query now contains the information laid out in the requirements (domain, username, password does not expire flag). The next step is to format the information and apply a secondary filter (password does not expire == True) in order to provide the desired output. In order to format this information into the table format we specified, we can now use a for loop to parse it (see below as an example printing out to a csv file) and additionally use the same for loop to apply the secondary filter logic.

Example code:

import csv

#Create CSV
with open('accounts with passwords that do not expire.csv', 'w', newline='') as csvfile:
csvwriter = csv.writer(csvfile, delimiter=',',quotechar='”', quoting=csv.QUOTE_MINIMAL)

#Iterate through the user object list and create/write each row to CSV
list_id = 0

while list_id < len(user["specific_data.data.domain_details"]):

row = []
row.append(user["adapters_data.active_directory_adapter.ad_sAMAccountName_details"][list_id])
row.append(user["adapters_data.active_directory_adapter.ad_uac_dont_expire_password_details"] [list_id])
row.append(user["adapters_data.active_directory_adapter.domain_details"][list_id])
row.append(user["adapters_data.active_directory_adapter.last_seen_details"][list_id])
row.append(user["adapters_data.active_directory_adapter.fetch_time_details"][list_id])
list_id += 1

if user["Adapters_data.active_directory_adapter.ad_uac_dont_expire_password_details"] [list_id] == True:
csvwriter.writerow(row)

Having now formatted and filtered down the output, the remaining data now contains the information needed to carry out password remediation, highlighting every account where the password never expires. 

 

Wrapping up

By using additional operations such as fields and include_details, users are able to greatly expand the information they wish to consume, in addition to providing increase granularity of that information. These operations are useful in improving visibility into the source of issues as well as tracing data quality issues back to their respective sources.

For the full working code of this example, please refer to the snippet below.

Full Code:

import axonius_api_client as axonapi
import csv

# setup connection as per ax.env 
# (remember to specify your own env variables for URL, Key and Secret
client_args = axonapi.get_env_connect(ax_env="./ax.env")


# connect and start the axon client api
client = axonapi.Connect(**client_args)
client.start()


# Pull all users alongside their sAMAccountName, PasswordNeverExpires Flag and Domain
user = client.users.get(fields=["adapters_data.active_directory_adapter.ad_sAMAccountName","adapters_data.active_directory_adapter.ad_uac_dont_expire_password","adapters_data.active_directory_adapter.domain","adapters_data.active_directory_adapter.last_seen","specific_data.data.fetch_time"],include_details=True)

#Create CSV
with open('accounts with passwords that do not expire.csv', 'w', newline='') as csvfile:
    csvwriter = csv.writer(csvfile, delimiter=',',quotechar='”', quoting=csv.QUOTE_MINIMAL)

#Iterate through the user object list and create/write each row to CSV

list_id = 0

while list_id < len(user["specific_data.data.domain_details"]):

row = []
row.append(user["adapters_data.active_directory_adapter.ad_sAMAccountName_details"][list_id])
row.append(user["adapters_data.active_directory_adapter.ad_uac_dont_expire_password_details"] [list_id])
row.append(user["adapters_data.active_directory_adapter.domain_details"][list_id])
row.append(user["adapters_data.active_directory_adapter.last_seen_details"][list_id])
row.append(user["adapters_data.active_directory_adapter.fetch_time_details"][list_id])
list_id += 1

if user["Adapters_data.active_directory_adapter.ad_uac_dont_expire_password_details"] [list_id] == True:
csvwriter.writerow(row)

#End of File



Further Documentation

Axonius API Client GH Site: https://github.com/Axonius/axonius_api_client 

Axonius API Doc Site: https://axonius-api-client.readthedocs.io/en/latest/index.html

CLI Doc Link: https://axonius-api-client.readthedocs.io/en/latest/main/usage_cli/usage_cli.html 

Py Doc Link: https://axonius-api-client.readthedocs.io/en/latest/main/usage_api/usage_api.html

0

Comments

0 comments

Please sign in to leave a comment.

Didn't find what you were looking for?

New post