Flows provide a no-code method for developing advanced automations in Gravwell. By wiring together nodes in a drag-and-drop user interface, you can:
Generate PDF reports
Fire off Slack and MS Teams messages
This document will describe what makes a flow, the flow editor, and how to debug & deploy your own flows.
Basic flow concepts#
Flows are automations, meaning they are normally executed on a user-specified schedule by the search agent. You can also run them manually through the user interface. The basic process of flow development is:
Create a new flow
Instantiate nodes in the flow and connect them together
Test the flow with debug runs
Deploy the flow by setting a schedule & enabling scheduled execution
A flow is a collection of nodes, linked together to define an order of execution. Each node does a single task, such as running a query or sending an email. In the example below, the leftmost node runs a Gravwell query, then the middle node formats the results of that query into a PDF document, and finally the rightmost node sends that PDF document as an email attachment.
All nodes have a single output socket. Most have only a single input socket, but some nodes which merge payloads (see below) have multiple input sockets.
One node’s output socket may be connected to the inputs of multiple other nodes, but each input socket can only take one connection.
Payloads are collections of data passed from node to node, representing the state of execution. For instance, the “Run a Query” node will insert an item named “search” into the payload, containing things like the query results and metadata about the search. The PDF node can read that “search” item, format it into a nice PDF document, and insert the PDF file back into the payload with a name like “gravwell.pdf”. Then the Email node can be configured to attach “gravwell.pdf” to the outgoing email.
The node receives an incoming payload through its input socket, then passes its outgoing payload via the output socket. In most cases, the outgoing payload will be a modified version of the incoming payload.
Nodes are always executed one at a time. A node can be executed if all nodes upstream of it (its dependencies) have executed. If multiple nodes are ready to execute, one will be chosen at random. In the example below, both the “Run a Query” node and the “HTTP” node are candidates to run first. After the Query node finishes, the If node can execute; when it is done, the Slack Message node may run. We say that the If node is downstream of the Query node, and the Slack node is downstream of both the If and Query nodes.
Note that some nodes may block execution of downstream nodes. The If node is configured with a boolean logic expression; if that expression evaluates to false, none of the If node’s downstream nodes are executed. Nodes which can block downstream execution will always have a note to that effect in the online documentation.
Flows are created using the flow editor. Please refer to the flow editor documentation for a detailed description of the editor, instructions on how to use it, and information about debugging & scheduling flows.
Common Flow Patterns#
Because flows are quite flexible, it can be a bit daunting trying to figure out how to tackle a given task. We have collected some common flow patterns to help with this.
Background/Save Search: Save or background a Gravwell query.
Email: send email.
Flow Storage Read: read items from a persistent storage.
Flow Storage Write: write items into a persistent storage.
Gravwell Notification: set Gravwell notifications.
Go Scripting: execute Go code in the flow.
HTML Format: format variables as HTML.
HTTP: do HTTP requests.
If: perform logical operations.
Indexer Info: get information about Gravwell indexers.
Ingest: ingest data into Gravwell.
Ingester Info: get information about Gravwell ingesters.
JSON Encode/Decode: encode and decode JSON.
Mattermost Message: send a Mattermost message.
Nest Merge: join multiple input payloads into one.
PDF: create PDF documents.
Query Log Ingest: convert search results to alert entries & ingest.
Read Macros: read Gravwell macros.
Read Resources: read Gravwell resources.
Rename: rename variables in the payload.
Run a Query: run a Gravwell query.
Run Query - Advanced: run a Gravwell query with more flexible options.
Set Variables: inject variables into the payload.
Slack File: upload a file to a Slack channel.
Slack Message: send a message to a Slack channel.
Sleep: pause flow execution for a given period of time.
Splunk Query: run a Splunk query.
Stack Merge: join multiple input payloads into one.
Teams Message: send a Microsoft Teams message.
Text Template: format text.
Throttle: limit execution frequency of certain nodes within a flow.
Update Resources: create or update Gravwell resources.
The following nodes tend to be needed only in particular advanced cases: