Amazon SQS Ingester

The Amazon SQS Ingester (sqsIngester) is a simple ingester that can subscribe to both standard and FIFO SQS queues for ingest. Amazon SQS is a high volume message queue service that supports message delivery guarantees, “soft” ordering of messages, and “at-least-once” delivery of messages.

For Gravwell, “at-least-once” delivery is an important caveat - The SQS ingester may receive duplicate messages with identical timestamps (depending on your configuration). It’s also possible that the SQS ingester doesn’t see some messages, depending on how your SQS workflow is deployed with other connected services. See Amazon SQS for more information.

Installation

To install the Debian package, make sure the Gravwell Debian repository is configured as described in the quickstart. Then run the following command as root:

apt update && apt install gravwell-sqs

To install the Redhat package, make sure the Gravwell Redhat repository is configured as described in the quickstart. Then run the following command as root:

yum install gravwell-sqs

To install via the standalone shell installer, download the installer from the downloads page, then run the following command as root, replacing X.X.X with the appropriate version:

bash gravwell_sqs_ingest_installer_X.X.X.sh

You may be prompted for additional configuration during the installation.

There is currently no Docker image for this ingester

Basic Configuration

The SQS ingester uses the unified global configuration block described in the ingester section. Like most other Gravwell ingesters, SQS supports multiple upstream indexers, TLS, cleartext, and named pipe connections, a local cache, and local logging.

The configuration file is at /opt/gravwell/etc/sqs.conf. The ingester will also read configuration snippets from its configuration overlay directory (/opt/gravwell/etc/sqs.conf.d).

Queue Examples

[Queue "default"]
	Region="us-east-2"
	Queue-URL="https://us-east-2.amazon..."
	Tag-Name="sqs"
	AKID="AKID..."
	Secret="..."
	Assume-Local-Timezone=false #Default for assume localtime is false
	Source-Override="DEAD::BEEF" #override the source for just this Queue 
	Credentials-Type="static"

[Queue "default"]
	Region="us-west-1"
	Queue-URL="https://us-west-1.amazon..."
	Tag-Name="sqs"
	AKID="AKID..."
	Secret="..."

Credentials-Type Authentication Options

Both listener types (Bucket and SQS-S3-Listener) support multiple authentication methods. By default, the “static” method is used, which requires that you set the ID and Secret fields for the listener. The following additional methods are supported, and can be used by setting the Credentials-Type field:

Credential Type	Description
static	Default credential type. Uses the AKID and Secret fields set in the listener.
environment	Uses the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables to authenticate.
ec2role	Uses the host-based EC2 role authentication method. See Amazon EC2 Documentation for more information.

Installation

If you’re using the Gravwell Debian repository, installation is just a single apt command:

apt-get install gravwell-sqs

Otherwise, download the installer from the Downloads page. To install the Netflow ingester, simply run the installer as root (the actual file name will typically include a version number):

root@gravserver ~ # bash gravwell_sqs.sh

If there is no Gravwell indexer on the local machine, the installer will prompt for an Ingest-Secret value and an IP address for an indexer (or a Federator). Otherwise, it will pull the appropriate values from the existing Gravwell configuration. In any case, review the configuration file in /opt/gravwell/etc/sqs.conf after installation. A typical configuration will look like:

[Global]
Ingest-Secret = IngestSecrets
Connection-Timeout = 0
Insecure-Skip-TLS-Verify=false
Pipe-Backend-Target=/opt/gravwell/comms/pipe 
Log-Level=INFO
Log-File=/opt/gravwell/log/sqs.log

# A Queue pulls from a specific SQS queue with a given AKID and Secret. See
# https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys
# for information about obtaining an AKID/Secret for your user.
[Queue "default"]
	Region="us-east-2"
	Queue-URL="https://us-east-2.amazon..."
	Tag-Name="sqs"
	AKID="AKID..."
	Secret="..."

Note that this configuration sends entries to a local indexer via /opt/gravwell/comms/pipe. Entries are tagged ‘sqs’.

You can configure any number of Queue entries, one for each SQS queue, and provide unique authentication, tag names, etc., for each one.