AWS CloudTrail#

Integration Details

Ingester

S3 Ingester
SQS Ingester

Kit

CloudTrail

CloudTrail Configuration#

There are two primary methods to get CloudTrail data into Gravwell - S3 Bucket Ingest, and SQS-backed ingest.

  • S3 Bucket Ingester (Simplest): This workflow uses the S3 Ingester to pull logs directly from the CloudTrail S3 bucket. It requires an identity with read access to the S3 bucket. The polling interval may introduce a 5-15 minute delay before logs appear in Gravwell.

  • SQS-S3-Listener Ingester (Near Real-Time): For faster delivery SQS can be configured to notify the ingester as events are generated.

Note

Regardless of the ingestion method, CloudTrail inherently has its own internal delay between an action occurring and the event being logged by AWS.

[Option 1] CloudTrail Configuration for: S3 Bucket Ingest#

Note

Create a dedicated IAM user with read-only S3 access for the Gravwell ingester. Do not use privileged credentials for dedicated applications like data ingestion.

No other CloudTrail-side configuration is required. For additional configuration options, see the S3 Ingester documentation.

[Option 2] CloudTrail Configuration for: SQS-S3-Listener Ingest#

  • Configure CloudTrail with an SNS topic

    • Set up SQS in AWS

    • Configure IAM policies

    • User that can query SQS, and pull from S3

    • CRITICAL: Configure long polling for SQS. By default, SQS queues don’t specify a delay between received messages. The ingester will query SQS endlessly resulting in ~12,000 requests per minute, which inflates SQS costs.

      • If you specify a limited range (1-20 seconds), that significantly reduces the amount of requests that SQS will respond to.

      • To configure long polling, access the SQS queue in AWS Console. Click Edit in the top right, and set Receive message wait time. 20 seconds usually results in an imperceptible delay in the time between polling for new logs.

    • Note: If you have KMS encryption enabled (default setting) for SQS, you will need additional permissions for kms:Decrypt.

Sample JSON policy for a user that can query CloudTrail utilizing SQS

Note

Replace ACCOUNT_ID:key/YOUR_KMS_KEY_ID with your KMS Resource

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SQSConsume",
            "Effect": "Allow",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:ChangeMessageVisibility"
            ],
            "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:YOUR_QUEUE_NAME"
        },
        {
            "Sid": "S3Read",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_CLOUDTRAIL_S3_BUCKET",
                "arn:aws:s3:::YOUR_CLOUDTRAIL_S3_BUCKET/*"
            ]
        },
        {
            "Sid": "KMSDecryptForSQS",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "arn:aws:kms:REGION:ACCOUNT_ID:key/YOUR_KMS_KEY_ID"
        }
    ]
}

Gravwell Configuration#

Gravwell Storage Well Configuration#

Setup the well configuration in your Gravwell indexers.

Sample well config: Create or edit: /opt/gravwell/etc/gravwell.conf.d/cloudtrail-well.conf

[Storage-Well "cloudtrail"]
    Location=/opt/gravwell/storage/cloudtrail
    Tags=aws-cloudtrail*

Gravwell Ingester Configuration#

[Option 1] Gravwell S3 Ingester Configuration#

The simplest workflow is to ingest logs from CloudTrail’s S3 bucket using the S3 Ingester. The polling required may introduce a 5-15 minute delay on new logs landing in Gravwell once they hit CloudTrail, but only requires an identity that can query the CloudTrail S3 bucket.

Sample S3 ingester config:
Create or edit: /opt/gravwell/etc/s3.conf.d/cloudtrail.conf

[Bucket "default"]
    Region="us-east-1"
    ID="AKI..."
    Secret="SuperSecretKey..."
    Bucket-ARN = "arn:aws:s3:::aws-cloudtrail-logs-123456-7890"
    Tag-Name="aws-cloudtrail"
    Reader=cloudtrail
    File-Filters=**/*.json.gz

[Option 2] Gravwell SQS Ingester Configuration#

Configure the S3 ingester to use SQS. Replace the [Bucket] declaration with the following snippet.

Sample SQS config snippet
Create or edit: /opt/gravwell/etc/s3.conf.d/cloudtrail.conf

[SQS-S3-Listener "CloudTrail_Queue-Placeholder"] ## REPLACE PLACEHOLDER WITH YOUR ACTUAL SQS QUEUE NAME
    Tag-Name="aws-cloudtrail"
    Reader=cloudtrail
    ID="AKI..."
    Secret="SuperSecretKey..."
    Region="us-east-1"
    Queue-URL="https://sqs.us-east-1.amazonaws.com/..."
    File-Filters=**/*.json.gz

The S3 ingester reads messages from the SQS queue, retrieves the S3 objects they reference, and sends the contents to the Gravwell indexer.

Note

Remember to restart the service to apply the new config: sudo systemctl restart gravwell_s3_ingest.service