The Problem With Point-in-Time Evidence

Audits of security controls often ask whether a control operated for an entire period. Evidence for identity controls usually arrives as screenshots of an admin console: a password policy page, an MFA configuration, captured close to the date of the request. A screenshot records one moment. The settings behind it can change the next day, and nothing in the screenshot process notices.

Policies state intent in plain terms. Passwords are long and complex. Multi-factor authentication protects sensitive systems. Devices must be trusted before they reach sensitive systems. Continuous monitoring turns each statement into a daily question: does the system still behave that way, and where is the proof of the answer?

This post describes one way to answer that question with a scheduled job, a Git repository, and a chat channel. The example system is Okta. The pattern applies to any system with a read API. I work in compliance, not engineering, and I built this pattern with Claude as the engineering partner. The design decisions, the approvals, and the review of every output stayed with me.

Build or Buy

Compliance platforms sell continuous monitoring, and buying avoids building and maintaining the integrations. A platform typically offers prebuilt connectors, a library of tests mapped to common frameworks, evidence collection, dashboards, and an auditor who already knows the format. It also absorbs the work of keeping connectors current when vendors change their APIs.

Building offers different things. The checks map to the exact wording of your own policies, including unusual choices. The logic stays readable, so an auditor can see what each check compares. Cost scales with the number of checks. The price is engineering and maintenance effort, because someone owns the code, the credentials, and the schedule.

Three questions help decide. How closely do your policies match the platform’s built-in tests? How many systems need coverage, and does the platform support them? Who will maintain the job a year from now? The pattern below also works as a complement. A platform covers breadth across many integrations, and a small custom monitor covers the few settings where the policy wording is specific and a change carries real consequences.

The Pattern in Five Steps

A scheduled job runs five steps each day. A GitHub Actions workflow on a cron schedule is enough to host it.

Scheduled job, daily
  1. Collect    read-only API calls -> snapshot (JSON)
  2. Normalize  nested rules -> short readable lines
  3. Compare    snapshot vs approved baselines in Git -> run record
  4. Store      snapshot, hash, run record, baselines -> locked evidence bucket
  5. Notify     chat message; the run fails on drift or error

The collector uses a read-only credential and makes GET requests only, so the job cannot change the system it watches. For Okta, the Policies API returns password policies with GET /api/v1/policies?type=PASSWORD, and each policy’s rules come from its own rules endpoint. The collector saves the responses as one JSON snapshot.

Approved Settings Live in Git

Approved settings live in Git as baselines. Each baseline is a YAML file that lists the settings, ties each one to the policy statement and the control it supports, and names the person who approved it. A baseline changes only through a pull request, so the change history doubles as change management evidence.

A baseline setting names a path into the snapshot, an operator, and an expected value. Three operators cover most needs. A gte check treats the value as a minimum, so a stricter change passes. An eq check suits deliberate design choices, and any change flags. A set_equals check compares the full list of items that exist, which catches a new policy appearing.

baseline: okta-password
approved_by: <approver name>
approved_on: <date>
settings:
  - id: min-length
    description: Minimum password length
    policy: Password Policy, section 4.1
    control: ACCESS-01
    path: 'password_policies[name=Workforce Password Policy].settings.password.complexity.minLength'
    operator: gte
    expected: 14

  - id: active-password-policies
    description: The set of active password policies. A new one shows as drift.
    path: 'password_policies[status=ACTIVE][].name'
    operator: set_equals
    expected:
      - Default Policy
      - Workforce Password Policy

A password baseline can cover strength, rotation and lockout choices, which group the policy applies to, whether weaker recovery paths stay off, and the set and priority order of active password policies. The set and priority order of active policies need a check of their own. Okta applies the highest-priority policy that covers a user, so a weaker policy added above the main one lowers the bar for the people it covers. A set check on active policies detects that addition.

Why Not Terraform

Terraform makes a system match a declaration, which means deploying changes and holding write access. The goal here is detection and proof, so a read-only check fits the job. Enforcement is a separate decision. Some teams move parts of an identity configuration into Terraform on their own timeline. A read-only monitor gives that work an independent verifier: one system makes the change and a second system confirms it.

Reading Rules in Priority Order

MFA and device trust need a different shape. Okta evaluates sign-in rules in priority order and applies the first rule that matches. A rule that requires a managed device, followed by a looser rule for everyone, sends anyone who fails the device check to the looser rule. No individual setting shows that gap. Reviewing it means reading every rule in order.

Each rule is a deeply nested object, so a normalizer reduces it to one line. The readable part serves the reviewer. The trailing hash covers the whole rule, so a change that the readable text omits still changes the line.

#1 Managed devices [ACTIVE] ALLOW 1FA | who: Managed devices group | device: managed=yes registered=yes | factor: possession(hw=REQUIRED,phish=REQUIRED) | reauth: 5y | platform: MACOS/DESKTOP | ~3fa9c2d1

A draft generator turns a live snapshot into a proposed baseline, and approval means reviewing the generated output in a pull request. The generator marks every draft as pending review, and the compare engine refuses to run any baseline that has no approver.

Evidence an Auditor Can Repeat

Each run writes a folder to a cloud storage bucket. The folder holds the snapshot, its hash, a plain-language report, the check summary, one run record per baseline, and a copy of the baselines used. A run record holds the result, every finding, the hashes of the exact inputs, the commit of the code that ran, and a summary of how the collector produced the snapshot. CI logs expire on the platform’s schedule, while the evidence follows your audit retention period, so the record carries its own provenance.

The bucket has a locked retention policy, so nobody can delete or overwrite an object until it expires, including the bucket’s administrators. The account that writes evidence can only create new objects. It authenticates through GitHub’s OIDC identity, so no cloud keys sit in the repository, and it accepts only workflows that run on the main branch. A separate group holds read access. Whoever changes a baseline cannot edit the evidence of a past run.

An auditor can repeat any run. Download the folder, check the snapshot against its hash, check out the repository at the commit named in the run record, and run the same command against the stored snapshot and the stored baselines. The result should match the record. Nothing in that process contacts Okta.

Alerts and the Daily All-Clear

A run that finds drift or hits an error fails in GitHub and posts to a chat channel. The message lists each finding. For set checks it lists the items added, the items removed, and the rules edited in place.

Okta baseline check: DRIFT
* rules-production-apps in okta-sign-in-rules:
    changed: #6 Fallback [ACTIVE] ALLOW 1FA | who: ALL USERS | ...  ->  #6 Fallback [ACTIVE] ALLOW 2FA | who: ALL USERS | ...
    added: #9 Temporary access [ACTIVE] ALLOW 1FA | who: ALL USERS | ...
Next: ask the owner of the change why it was made, then log the outcome.

A clean run posts a single line, such as "Okta baseline check: clean. 20 settings checked." That line has a purpose. A run that never starts produces no failure, because no run exists to fail. GitHub can delay or skip a scheduled run, and someone can disable the schedule. Without the daily line, silence reads as good news. With it, a missing line signals a problem.

The webhook address lives in a repository secret that the script validates before it posts: the address must belong to the chat service, and nothing else receives the message. Name the secret for the specific repository. Organization-level secrets flow down to repositories automatically, and a generic name such as SLACK_WEBHOOK_URL can collide with one that serves another purpose.

After an alert, a person asks the owner of the change why it happened. The outcome is one of two things: the tenant returns to the baseline, or the baseline changes through a reviewed pull request. A ticket records the event, the owner, and the date.

Four Rules for the Design

A script decides whether a result is clean, drifted, or an error, and the same inputs always produce the same answer. AI makes no part of that call. A setting that the script cannot find counts as drift. An incomplete collection never reads as clean. A baseline that nobody approved refuses to run.

Working With an AI Assistant

Claude wrote the collector, the compare engine, the chat notifier, and the tests. I reviewed behavior: the run summary, the alert text, the evidence folder, and what happened when I broke things on purpose. Three practices made that workable.

Every check was tested against deliberately broken copies of real data: a rule edited in place, a rule added, a device condition removed, SMS switched on, a new enrollment policy, a weakened password. Each break produced a specific, readable finding. A content fingerprint compared every file in the repository with the tested version before merge, so the code that ran matched the code I reviewed. Testing also found defects before production did. A malformed path in one draft baseline surfaced during a test run, and the engine reported it as an error.

Limits of the Approach

The check covers what the baselines cover. A setting outside the baselines changes without notice, and so does a new policy type that no baseline anticipates. A minimum check passes a stricter change by design. The job runs on a schedule, so a change can sit undetected until the next run. A snapshot that holds no user records cannot report who has enrolled in MFA. Vendor APIs also differ in what they expose, and some settings have no read endpoint. Each limit has a design response, and listing the limits in the monitoring documentation keeps the coverage claim honest.

How to Start

Start with one sentence from a policy. Find the API endpoint that holds the matching setting. Create a read-only credential and collect one snapshot by hand. Write three to five baseline settings that express the sentence, and get them approved in a pull request. Run the comparison locally, then schedule it. Add the evidence bucket and the chat message after the comparison runs cleanly. Extend to the next policy sentence once the first one runs without attention.

Lessons

The policy sets the ceiling for the automation. Where a policy leaves a term undefined, the baseline can record what exists but cannot say whether it is right. Writing the automation exposes the terms that need definitions. Device trust, for example, needs a policy definition of which systems count as sensitive before anyone can test it.

Read the way the vendor evaluates rules before baselining them. First-match behavior explains why a strong rule followed by a weaker catch-all is a gap, and no individual setting shows that.

Design for the person who audits the result. The folder an auditor opens should contain everything needed to repeat the check, and nothing should depend on logs that expire or on trusting the person who ran it.

Make the absence of a signal visible. The daily all-clear line costs one chat message and covers the case where nothing runs and nothing reports.