GCP PubSub Ingester#

Gravwell provides an ingester capable of fetching entries from Google Compute Platform’s PubSub stream service. The ingester can process multiple PubSub streams within a single GCP project. The process of setting up a PubSub stream is outside the scope of this document, but in order to configure the PubSub ingester for an existing stream you will need:

  • The Google Project ID

  • A file containing GCP service account credentials (see the Creating a service account documentation)

  • The name of a PubSub topic

Once the stream is configured, each record in the PubSub stream topic will be stored as a single entry in Gravwell.

Basic Configuration#

The PubSub ingester uses the unified global configuration block described in the ingester section. Like most other Gravwell ingesters, PubSub supports multiple upstream indexers, TLS, cleartext, and named pipe connections, a local cache, and local logging.

PubSub Examples#

[PubSub "gravwell"]
	Topic-Name=mytopic	# the pubsub topic you want to ingest

[PubSub "my_other_topic"]
	Topic-Name=foo # the pubsub topic you want to ingest

Installation and configuration#

First, download the installer from the Downloads page, then install the ingester:

root@gravserver ~# bash gravwell_pubsub_ingest_installer.sh

If the Gravwell services are present on the same machine, the installation script should automatically extract and configure the Ingest-Auth parameter and set it appropriately. You will now need to open the /opt/gravwell/etc/pubsub_ingest.conf configuration file and set it up for your PubSub topic. Once you have modified the configuration as described below, start the service with the command systemctl start gravwell_pubsub_ingest.service

The example below shows a sample configuration which connects to an indexer on the local machine (note the Pipe-Backend-target setting) and feeds it from a single PubSub topic named “mytopic”, which is part of the “myproject-127400” GCP project.

Ingest-Secret = IngestSecrets
Connection-Timeout = 0
Insecure-Skip-TLS-Verify = false
Pipe-Backend-target=/opt/gravwell/comms/pipe #a named pipe connection, this should be used when ingester is on the same machine as a backend
Log-Level=ERROR #options are OFF INFO WARN ERROR

# The GCP project ID to use

[PubSub "gravwell"]
	Topic-Name=mytopic	# the pubsub topic you want to ingest

Note the following essential fields:

  • Project-ID - the Project ID string for a GCP project

  • Google-Credentials-Path - the path to a file containing GCP service account credentials in JSON format

  • Topic-Name - the name of a PubSub topic within the specified GCP project

You can configure multiple PubSub sections to support multiple different PubSub topics within a single GCP project.

You can test the config by running /opt/gravwell/bin/gravwell_pubsub_ingester -v by hand; if it does not print out errors, the configuration is probably acceptable.

The PubSub ingester does not provide the Ignore-Timestamps option found in many other ingesters. PubSub messages include an arrival timestamp; by default, the ingester will use that as the Gravwell timestamp. If Parse-Time=true is specified in the data consumer definition, the ingester will instead attempt to extract a timestamp from the message body.