The forwarding preprocessor is used to split a log stream and forward it to another endpoint. This can be extremely useful when standing up additional logging tools or when feeding data to external archive processors. The Forwarding preprocessor is a forking preprocessor. This means that it will not alter the data stream; it only forwards a copy of the data on to additional endpoints.
By default the forwarding preprocessor is blocking, which means that if you specify a forwarding endpoint using a stateful connection like TCP or TLS and the endpoint is not up or is unable to accept data it will block ingestion. This behavior can be altered using the
Non-Blocking flag or by using the UDP protocol.
The forwarding preprocessor also supports several filter mechanisms to cut down or specify exactly which data streams are forwarded. Streams can be filtered using entry tags, sources, or regular expressions which operate on the actual log data. Each filter specification can be specified multiple times to create an OR pattern.
Multiple forwarding preprocessors can be specified, allowing for a specific log stream to be forwarded to multiple endpoints.
The Forwarding preprocessor Type is
Target(string, required): Endpoint for forwarded data. Should be a host:port pair unless using the
Protocol(string, required): Protocol to use when forwarding data. Options are
Delimiter(string, optional): Optional delimiter to use when sending data via the
Format(string, optional): Output format to send data. Options are
Tag(string, optional, multiple allowed): Tags used to filter events. Multiple specifications implies an OR.
Regex(string, optional, multiple allowed): Regular expressions used to filter events. Multiple specifications implies an OR.
Source(string, optional, multiple allowed): IP or CIDR specifications used to filter events. Multiple specifications implies an OR.
Timeout(unsigned int, optional, specified in seconds): Timeout on connection and write attempts for the forwarder.
Buffer(unsigned int, optional): Specifies the number of events that the forwarder can buffer when attempting to send data.
Non-Blocking(boolean, optional): True specifies that the forwarder will make a best effort to forward data, but will not block ingestion.
Insecure-Skip-TLS-Verify(boolean, optional): Specifies that TLS-based connections can ignore invalid TLS certificates.
Example: Forwarding syslog from a specific set of hosts#
For this example we are using the SimpleRelay ingester to ingest syslog messages and forward them in their raw form to another endpoint. We are using the
Source filter in the
forward preprocessor to only forward logs that have the source field from either the
192.168.0.1 IP or within the
192.168.1.0/24 subnet. The logs are forwarded on in their original format with a newline between each:
[Listener "default"] Bind-String="0.0.0.0:601" Reader-Type=rfc5424 Tag-Name=syslog Preprocessor=forwardprivnet [Preprocessor "forwardprivnet"] Type=Forwarder Protocol=tcp Target="172.17.0.3:601" Format="raw" Delimiter="\n" Buffer=128 Source=192.168.0.1 Source=192.168.1.0/24 Non-Blocking=false
Example: Forwarding Specific Windows Event Logs#
For this example we are using the Federator to forward data streams from potentially many downstream ingesters. We are using both the
Regex filters to capture a specific set of entries and forward them on. Note that we are using the
syslog format, which means that we will send data to the endpoint with an RFC5424 header and the data in the body of the syslog message. Forwarded data using the syslog format uses
gravwell as the Hostname and the entry TAG as the Appname.
Tag filters specify that we only want to forward entries that are using the
Regex filters are used so that we only get event data from specific Channel/EventID combinations, namely login events from the security provider and execution events from the sysmon provider.
[IngestListener "enclaveB"] Ingest-Secret = IngestSuperSecrets Cleartext-Bind = 0.0.0.0:4123 Tags=win* Preprocessor=forwardloginsysmon [Preprocessor "forwardloginsysmon"] Type=Forwarder Protocol=tcp Target="172.17.0.3:601" Format="syslog" Buffer=128 Tag=windows Tag=sysmon Regex="Microsoft-Windows-Sysmon.+>(1|22)</EventID>" Regex="Microsoft-Windows-Security-Auditing.+>(4624|4625|4626)</EventID>" Non-Blocking=false
Example: Forwarding logs to multiple hosts#
For this example we are using the Gravwell Federator to forward subsets of logs to different endpoints using different formats. Because the forwarder preprocessor can be stacked the same way as any other preprocessor, we can specify multiple forwarding preprocessors with their own filters, endpoints, and formats.
[IngestListener "enclaveA"] Ingest-Secret = IngestSuperSecrets Cleartext-Bind = 0.0.0.0:4023 Tags=win* Preprocessor=forwardloginsysmon Preprocessor=forwardprivnet [Preprocessor "forwardloginsysmon"] Type=Forwarder Protocol=tcp Target="172.17.0.3:601" Format="syslog" Buffer=128 Tag=windows Tag=sysmon Regex="Microsoft-Windows-Sysmon.+>(1|22)</EventID>" Regex="Microsoft-Windows-Security-Auditing.+>(4624|4625|4626)</EventID>" Non-Blocking=false [Preprocessor "forwardsyslog"] Type=Forwarder Protocol=tcp Target="172.17.0.3:601" Format="raw" Delimiter="\n" Buffer=128 Tag=syslog Source=192.168.0.1 Source=192.168.1.0/24 Non-Blocking=false