Amazon SQS Ingester#
The Amazon SQS Ingester (sqsIngester) is a simple ingester that can subscribe to both standard and FIFO SQS queues for ingest. Amazon SQS is a high volume message queue service that supports message delivery guarantees, “soft” ordering of messages, and “at-least-once” delivery of messages.
For Gravwell, “at-least-once” delivery is an important caveat - The SQS ingester may receive duplicate messages with identical timestamps (depending on your configuration). It’s also possible that the SQS ingester doesn’t see some messages, depending on how your SQS workflow is deployed with other connected services. See Amazon SQS for more information.
To install the Debian package, make sure the Gravwell Debian repository is configured as described in the quickstart. Then run the following command as root:
apt update && apt install gravwell-sqs
To install the Redhat package, make sure the Gravwell Redhat repository is configured as described in the quickstart. Then run the following command as root:
yum install gravwell-sqs
To install via the standalone shell installer, download the installer from the downloads page, then run the following command as root, replacing X.X.X with the appropriate version:
You may be prompted for additional configuration during the installation.
There is currently no Docker image for this ingester
The SQS ingester uses the unified global configuration block described in the ingester section. Like most other Gravwell ingesters, SQS supports multiple upstream indexers, TLS, cleartext, and named pipe connections, a local cache, and local logging.
The configuration file is at
/opt/gravwell/etc/sqs.conf. The ingester will also read configuration snippets from its configuration overlay directory (
[Queue "default"] Region="us-east-2" Queue-URL="https://us-east-2.amazon..." Tag-Name="sqs" AKID="AKID..." Secret="..." Assume-Local-Timezone=false #Default for assume localtime is false Source-Override="DEAD::BEEF" #override the source for just this Queue [Queue "default"] Region="us-west-1" Queue-URL="https://us-west-1.amazon..." Tag-Name="sqs" AKID="AKID..." Secret="..."
If you’re using the Gravwell Debian repository, installation is just a single apt command:
apt-get install gravwell-sqs
Otherwise, download the installer from the Downloads page. To install the Netflow ingester, simply run the installer as root (the actual file name will typically include a version number):
root@gravserver ~ # bash gravwell_sqs.sh
If there is no Gravwell indexer on the local machine, the installer will prompt for an Ingest-Secret value and an IP address for an indexer (or a Federator). Otherwise, it will pull the appropriate values from the existing Gravwell configuration. In any case, review the configuration file in
/opt/gravwell/etc/sqs.conf after installation. A typical configuration will look like:
[Global] Ingest-Secret = IngestSecrets Connection-Timeout = 0 Insecure-Skip-TLS-Verify=false Pipe-Backend-Target=/opt/gravwell/comms/pipe Log-Level=INFO Log-File=/opt/gravwell/log/sqs.log # A Queue pulls from a specific SQS queue with a given AKID and Secret. See # https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys # for information about obtaining an AKID/Secret for your user. [Queue "default"] Region="us-east-2" Queue-URL="https://us-east-2.amazon..." Tag-Name="sqs" AKID="AKID..." Secret="..."
Note that this configuration sends entries to a local indexer via
/opt/gravwell/comms/pipe. Entries are tagged ‘sqs’.
You can configure any number of
Queue entries, one for each SQS queue, and provide unique authentication, tag names, etc., for each one.