Compliance Evidence Integration Platform

Problem

Compliance evidence is scattered across cloud provider APIs — Security Hub findings, GuardDuty alerts, IAM state, CloudTrail history, GCP container scans, GitHub approval chains — and every compliance platform wants it in its own shape. The obvious approach is a bespoke script per source-destination pair, which means N×M scripts, N×M auth implementations, and N×M things that fail silently at 2am.

Solution

One pipeline with the varying parts pulled out. A source adapter normalizes each provider's API response into a common evidence record, and an output-target factory decides where that record lands. Adding an integration means writing one adapter, not a new pipeline; adding a destination means writing one sink. The whole thing deploys as a Terraform module with a zip bundle and no build step at apply time.

Impact

• →Built 23 integrations on the pattern across AWS (Security Hub, GuardDuty, IAM, RDS, CloudTrail, EKS, S3, KMS, and more), GCP Container Analysis, and GitHub pull-request data.
• →Runs in production on a daily schedule, syncing evidence into a SaaS client's Vanta tenant.
• →Supports three auth models without touching pipeline code: Secrets Manager OAuth, GitHub App, and keyless GCP Workload Identity Federation.
• →Covered by 50 Lambda unit tests plus 35 CLI integration tests across the adapters, API clients, and error paths.
• →Ships as a reusable Terraform module clients apply with one secret ARN and zero external dependencies at apply time.

Architecture

1. 01An EventBridge rule triggers the Lambda on a daily schedule (14:00 UTC by default).
2. 02The handler loads config from env vars and resolves credentials through whichever auth model the integration declares.
3. 03A source adapter paginates the provider API and maps each record into the common evidence schema — for Security Hub findings, 15+ typed custom properties and a console deep-link.
4. 04An output-target factory selects the sink. GitHub appears on both axes: a source for pull-request approval evidence, and a destination for other integrations.
5. 05A full-state PUT pushes the transformed evidence over a bearer token, with an empty-payload guard in front of it.
6. 06A Terraform module (lambda_pipeline) provisions the log group, function, EventBridge rule, and IAM role per integration; GitHub Actions builds and releases the bundle on tag via OIDC.

Capabilities

• ·Source adapters normalize 23+ provider APIs into a common evidence record — one adapter per integration, no pipeline changes.
• ·An output-target factory routes evidence to Vanta or GitHub, selected by configuration rather than by forking the code.
• ·Three auth models behind one interface: Secrets Manager OAuth with in-memory token caching and a 60-second expiry buffer, GitHub App, and keyless GCP Workload Identity Federation.
• ·Paginated extraction with optional compliance-status, record-state, and severity filters.
• ·Full-state PUT to the destination that rejects empty payloads, so a failed upstream fetch cannot wipe existing evidence.
• ·Least-privilege IAM per integration, scoped to a single secret ARN and the one read action that integration needs.
• ·Structured run output (records fetched, records pushed, status) to CloudWatch with configurable retention.