Packet Fleet Ingester#
The Packet Fleet Ingester provides a mechanism to query Google Stenographer instances and have results ingested per-packet into Gravwell.
Each Stenographer ingester listens on a given port (
Listen-Address) and accepts Stenographer queries (see query syntax below) as an HTTP POST. On receiving a query, the ingester returns an integer job ID, and asynchronously queries the Stenographer instance and begins to ingest the returned PCAP. Multiple in-flight queries can be ran concurrently. Job status can be viewed by issuing an HTTP GET on “/status”, which returns a JSON-encoded array of in-flight job IDs.
A simple web interface to submit and view job status is also available by browsing to the specified ingester port.
To install the Debian package, make sure the Gravwell Debian repository is configured as described in the quickstart. Then run the following command as root:
apt update && apt install gravwell-packet-fleet
To install the Redhat package, make sure the Gravwell Redhat repository is configured as described in the quickstart. Then run the following command as root:
yum install gravwell-packet-fleet
To install via the standalone shell installer, download the installer from the downloads page, then run the following command as root, replacing X.X.X with the appropriate version:
You may be prompted for additional configuration during the installation.
There is currently no Docker image for this ingester
Packet Fleet uses the unified global configuration block described in the ingester section. Like most other Gravwell ingesters, Packet Fleet supports multiple upstream indexers, TLS, cleartext, and named pipe connections, a local cache, and local logging.
The configuration file is at
/opt/gravwell/etc/packet_fleet.conf. The ingester will also read configuration snippets from its configuration overlay directory (
[Stenographer "Region 1"] URL="https://127.0.0.1:9001" CA-Cert="ca_cert.pem" Client-Cert="client_cert.pem" Client-Key="client_key.pem" Tag-Name=steno Assume-Local-Timezone=false #Default for assume localtime is false Source-Override="DEAD::BEEF" #override the source for just this Queue [Stenographer "Region 2"] URL="https://my.url:1234" CA-Cert="ca_cert.pem" Client-Cert="client_cert.pem" Client-Key="client_key.pem" Tag-Name=steno
Packet Fleet requires several Global and per-stenographer configuration options. Global settings include setting up TLS (if applicable) and the listen address for the web interface, as shown below:
Use-TLS=true Listen-Address=":9002" Server-Cert="server.cert" Server-Key="server.key"
For each Stenographer instance, the following stanza is required. The example name
Region 1 here is used by the web interface to list Stenographer instances.
[Stenographer "Region 1"] URL="https://127.0.0.1:9001" CA-Cert="ca_cert.pem" Client-Cert="client_cert.pem" Client-Key="client_key.pem" Tag-Name=steno #Assume-Local-Timezone=false #Default for assume localtime is false #Source-Override="DEAD::BEEF" #override the source for just this Queue
A user requests packets from stenographer by specifying them with a very simple query language. This language is a simple subset of BPF, and includes the primitives:
host 220.127.116.11 # Single IP address (hostnames not allowed) net 18.104.22.168/8 # Network with CIDR net 22.214.171.124 mask 255.255.255.0 # Network with mask port 80 # Port number (UDP or TCP) ip proto 6 # IP protocol number 6 icmp # equivalent to 'ip proto 1' tcp # equivalent to 'ip proto 6' udp # equivalent to 'ip proto 17' # Stenographer-specific time additions: before 2012-11-03T11:05:00Z # Packets before a specific time (UTC) after 2012-11-03T11:05:00-07:00 # Packets after a specific time (with TZ) before 45m ago # Packets before a relative time before 3h ago # Packets after a relative time
Relative times must be measured in integer values of hours or minutes as demonstrated above.
Primitives can be combined with and/&& and with or/||, which have equal precedence and evaluate left-to-right. Parens can also be used to group.
(udp and port 514) or (tcp and port 8080)
This section sourced from Google Stenographer