HTTP#
The HTTP ingester sets up HTTP listeners on one or more paths. If an HTTP request is sent to one of those paths, the request’s Body will be ingested as a single entry.
This is an extremely convenient method for scriptable data ingest, since the curl
command makes it easy to do a POST request using standard input as the body.
Installation#
To install the Debian package, make sure the Gravwell Debian repository is configured as described in the quickstart. Then run the following command as root:
apt update && apt install gravwell-http-ingester
To install the Redhat package, make sure the Gravwell Redhat repository is configured as described in the quickstart. Then run the following command as root:
yum install gravwell-http-ingester
To install via the standalone shell installer, download the installer from the downloads page, then run the following command as root, replacing X.X.X with the appropriate version:
bash gravwell_http_ingester_installer_X.X.X.sh
You may be prompted for additional configuration during the installation.
The Docker image is available on Dockerhub.
Basic Configuration#
The HTTP ingester uses the unified global configuration block described in the ingester section. Like most other Gravwell ingesters, the HTTP ingester supports multiple upstream indexers, TLS, cleartext, and named pipe connections, a local cache, and local logging.
Listener Examples#
In addition to the universal configuration parameters used by all ingesters, the HTTP POST ingester has two additional global configuration parameters that control the behavior of the embedded webserver. The first configuration parameter is the Bind
option, which specifies the interface and port that the webserver listens on. The second is the Max-Body
parameter, which controls how large of a POST the webserver will allow. The Max-Body parameter is a good safety net to prevent rogue processes from attempting to upload very large files into your Gravwell instance as a single entry. Gravwell can support up to 1GB as a single entry, but we wouldn’t recommend it.
Multiple “Listener” definitions can be defined allowing specific URLs to send entries to specific tags. In the example configuration we define two listeners which accept data from a weather IOT device and a smart thermostat.
Example using basic authentication
[Listener "basicAuthExample"]
URL="/basic"
Tag-Name=basic
AuthType=basic
Username=user1
Password=pass1
[Listener "jwtAuthExample"]
URL="/jwt"
Tag-Name=jwt
AuthType=jwt
LoginURL="/jwt/login"
Username=user1
Password=pass1
Method=PUT # alternate method, data is still expected in the body of the request
[Listener "cookieAuthExample"]
URL="/cookie"
Tag-Name=cookie
AuthType=cookie
LoginURL="/cookie/login"
Username=user1
Password=pass1
Method=PUT # alternate method, data is still expected in the body of the request
[Listener "presharedTokenAuthExample"]
URL="/preshared/token"
Tag-Name=pretoken
AuthType="preshared-token"
TokenName=Gravwell
TokenValue=Secret
[Listener "presharedTokenAuthExample"]
URL="/preshared/param"
Tag-Name=preparam
AuthType="preshared-parameter"
TokenName=Gravwell
TokenValue=Secret
Installation#
If you’re using the Gravwell Debian repository, installation is just a single apt command:
apt-get install gravwell-http-ingester
Otherwise, download the installer from the Downloads page. Using a terminal on the Gravwell server, issue the following command as a superuser (e.g. via the sudo
command) to install the ingester:
root@gravserver ~ # bash gravwell_http_ingester_installer_3.0.0.sh
If the Gravwell services are present on the same machine, the installation script will automatically extract and configure the Ingest-Auth
parameter and set it appropriately. However, if your ingester is not resident on the same machine as a pre-existing Gravwell backend, the installer will prompt for the authentication token and the IP address of the Gravwell indexer. You can set these values during installation or leave them blank and modify the configuration file in /opt/gravwell/etc/gravwell_http_ingester.conf
manually.
Configuring HTTPS#
By default the HTTP Ingester runs a cleartext HTTP server, but it can be configured to run an HTTPS server using x509 TLS certificates. To configure the HTTP Ingester as an HTTPS server provide a certificate and key PEM files in the Global configuration space using the TLS-Certificate-File
and TLS-Key-File
parameters.
An example global configuration with HTTPS enabled might look like the following:
[Global]
TLS-Certificate-File=/opt/gravwell/etc/cert.pem
TLS-Key-File=/opt/gravwell/etc/key.pem
Listener Configuration Options#
Listener configuration blocks support the following configuration parameters:
Parameter |
Type |
Required |
Default Value |
Description |
---|---|---|---|---|
URL |
string |
YES |
Endpoint URL to handle requests. |
|
Method |
string |
NO |
`POST`` |
HTTP method for requests. |
Tag-Name |
string |
YES |
Tag assigned to entries received by the endpoint. |
|
Multiline |
boolean |
NO |
false |
Treat request body as a multiline file and process each line as an individual entry. |
Ignore-Timestamps |
boolean |
NO |
false |
Do not extract or process timestamps, use current time. |
Assume-Local-Timzeone |
boolean |
NO |
false |
Assume local timezone on timestamps without a specified timezone. |
Timzeone-Override |
string |
NO |
Specify a specific timezone to attach to entries if the derived timestamp does not contain a timezone. |
|
Timestamp-Format-Override |
string |
NO |
Force timestamp processing to look for a specific timestamp format. |
|
AuthType |
string |
NO |
none |
Specify authentication type (basic, jwt, cookie, etc…). |
Username |
string |
NO |
Username credential, required when using cookie, basic, or JWT based authentication methods. |
|
Password |
string |
NO |
Password credential, required when using cookie, basic, or JWT based authentication methods. |
|
LoginURL |
string |
NO |
Specify login URL when performing cookie or JWT based authentication. Required when using login based authentication methods. |
|
TokenName |
string |
NO |
Authorization token name, required when using preshared-token or preshared-parameter authentication method. |
|
TokenValue |
string |
NO |
Authorization token value, required when using preshared-token or preshared-parameter authentication method. |
|
Preprocessor |
string array |
NO |
Set of preprocessors to apply to entries. |
Listener Authentication#
Each HTTP Ingester listener can be configured to enforce authentication. The supported authentication methods are:
none
basic
jwt
cookie
preshared-token
preshared-parameter
When specifying an authentication system other than none credentials must be provided. The jwt
and cookie
and cookie authentication systems require a username and password while the preshared-token
and preshared-parameter
must provide a token value and optional token name.
Warning
Like any other webpage, authentication is NOT SECURE over cleartext connections and attackers that can sniff traffic can capture tokens and cookies.
No Authentication#
The default authentication method is none, allowing anyone that can reach the ingester to push entries. The basic
authentication mechanism uses HTTP Basic authentication, where a username and password is base64 encoded and sent with every request.
Here is an example listener using the basic authentication system:
[Listener "basicauth"]
URL="/basic/data"
Tag-Name=stuff
AuthType=basic
Username=secretuser
Password=secretpassword
An example curl command to send an entry with basic authentication might look like:
curl -d "only i can say hi" --user secretuser:secretpassword -X POST http://10.0.0.1:8080/basic/data
JWT Authentication#
The JWT authentication system uses a cryptographically signed token for authentication. When using jwt authentication you must specify an Login URL where clients will authenticate and receive a token which must then be sent with each request. The jwt tokens expire after 48 hours. Authentication is performed by sending a POST
request to the login URL with the username
and password
form fields populated.
Authenticating with the HTTP ingester using jwt authentication is a two step process and requires an additional configuration parameter. Here is an example configuration:
[Listener "jwtauth"]
URL="/jwt/data"
LoginURL="/jwt/login"
Tag-Name=stuff
AuthType=basic
Username=secretuser
Password=secretpassword
Sending entries requires that endpoints first authenticate to obtain a token, the token can then be reused for up to 48 hours. If a request receives a 401 response, clients should re-authenticate. Here is an example using curl to authenticate and then push data.
x=$(curl -X POST -d "username=user1&password=pass1" http://127.0.0.1:8080/jwt/login) #grab the token and stuff it into a variable
curl -X POST -H "Authorization: Bearer $x" -d "this is a test using JWT auth" http://127.0.0.1:8080/jwt/data #send the request with the token
Listener Methods#
The HTTP Ingester can be configured to use virtually any method, but data is always expected to be in the body of the request.
For example, here is a Listener configuration that expects the PUT method:
[Listener "test"]
URL="/data"
Method=PUT
Tag-Name=stuff
The corresponding curl command would be:
curl -X PUT -d "this is a test 2 using basic auth" http://127.0.0.1:8080/data
The HTTP Ingester can go out of spec on methods, accepting almost any ASCII string that does not contain special characters.
[Listener "test"]
URL="/data"
Method=SUPER_SECRET_METHOD
Tag-Name=stuff
curl -X SUPER_SECRET_METHOD -d "this is a test 2 using basic auth" http://127.0.0.1:8080/data
Splunk HEC Compatibility#
The HTTP ingester supports a listener block that is API compatible with the Splunk HTTP Event Collector. This special listener block enables a simplified configuration so that any endpoint that can send data to the Splunk HEC can also send to the Gravwell HTTP Ingester. The HEC compatible configuration block looks like so:
[HEC-Compatible-Listener "testing"]
URL="/services/collector/event"
TokenValue="thisisyourtoken"
Tag-Name=HECStuff
The HEC-Compatible-Listener
block requires the TokenValue
and Tag-Name
configuration items. If the URL
configuration item is omitted, it will default to /services/collector/event
.
Both Listener
and HEC-Compatible-Listener
configuration blocks can be specified on the same HTTP ingester.
The HEC-Compatible-Listener
supports the following configuration parameters:
Parameter |
Type |
Required |
Default Value |
Description |
---|---|---|---|---|
URL |
string |
NO |
|
Endpoint URL for Splunk events. |
TokenValue |
string |
YES |
Authentication Token. |
|
Tag-Name |
string |
YES |
Tag assigned to entries received by the endpoint. |
|
Ignore-Timestamps |
boolean |
NO |
false |
Do not extract or process timestamps, use current time. |
Ack |
boolean |
NO |
false |
Acknowledge receipt and respond with entry IDs. |
Max-Size |
unsigned int |
NO |
524288 (512k) |
Maximum size for each decoded entry. |
Tag-Match |
string array |
NO |
Sourcetype value to tag mapping, multiple can be specified. |
|
Preprocessor |
string array |
NO |
Set of preprocessors to apply to entries. |
Health Checks#
Some systems (such as AWS load-balancers) require an unauthenticated URL that can be probed and interpreted as “proof of life”. The HTTP ingester can be configured to provide a URL which will always return a 200 OK when accessed with any method, body, and/or query parameters. To enable this health check endpoint, add the Health-Check-URL
stanza to the Global configuration block.
Here is a minimal example configuration snippet with the health check URL /logbot/are/you/alive
:
[Global]
Ingest-Secret = IngestSecrets
Connection-Timeout = 0
Pipe-Backend-Target=/opt/gravwell/comms/pipe #a named pipe connection, this should be used when ingester is on the same machine as a backend
Log-Level=INFO #options are OFF INFO WARN ERROR
Bind=":8080"
Max-Body=4096000 #about 4MB
Log-File="/opt/gravwell/log/http_ingester.log"
Health-Check-URL="/logbot/are/you/alive"