Case Study: Splunk Log Aggregation

Background

When migrating their workloads into AWS, a federal law enforcement agency had developed security and audit requirements to capture activity in all accounts residing in their GovCloud environment. Simple Technology Solutions (STS) was tasked to architect a solution to satisfy these needs in a secure and efficient manner.

Analysis

STS proposed and developed a solution that leveraged cloud-native AWS services, as well as third party security logging tools, ultimately using Splunk. Guided by our automation ethos, all resources were to be created using infrastructure as code and deployed through a pipeline. All AWS GovCloud accounts were targeted for complete coverage.

Solution

STS implemented and configured the following AWS services:

• CloudTrail: Enabled logging and delivery to a secure/lockdown S3 bucket in a separate security account.
• CloudWatch Logs: Enabled and manipulated with Lambda, then written to S3.
• Config: Enabled centralized aggregation from all accounts and delivered logs to S3 for ingestion.
• GuardDuty: Enabled in each account and sent logs to S3 for ingestion.
• Virtual Private Cloud (VPC) Flow Logs: Enabled on all VPCs and delivery to a secure/locked down S3 bucket in a separated security account.
• IAM: Created role, policy and instance profile for the Splunk EC2 log collectors in order to access log buckets and AWS services themselves.
• S3: Used multiple buckets for log storage and ingestion.
• Lambda: Developed formatting functions to ingest specific AWS service logs and write to S3. Logs would then be picked up by Splunk.

Benefit

All originally targeted services were successfully configured to stream thousands of KBs of logs per day, to the target bucket(s), and Splunk was successful in collecting/parsing logs from each of the five AWS services mentioned above. Information assurance was able to successfully monitor and identify activity within the federal law enforcement agency’s GovCloud AWS environment. STS helped the client improve their centralized logging maturity by following the guidance of the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-92, Guide to Computer Security Log Management.