Changelog for version 4.0.0#

Released July 28 2020#

Web UI Changes#

  • Implemented Gravwell Kits user interface: browse, install, and update kits containing pre-built analysis tools such as dashboards, actionables, and templates

  • Implemented actionables, which help pivot from search results–just click or highlight text in the results for a menu of possible actions

  • Implemented search templates, stored queries containing variables (such as an IP address) to set at search time

  • Re-organized main menu makes navigation less cluttered

  • Added experimental search studio feature

  • Major improvements to scheduled script development and testing workflow

  • Dashboard tiles can now be temporarily “maximized” to view data more comfortably

  • Map renderers now include tables listing locations and the metadata associated with them

  • Favorites and kits are now easily accessible from the home page

  • Faster-loading dashboards with grid customization

Backend Changes#

  • Implemented Gravwell Kits

  • Added wildcard support to well tag specifications

  • Added transparent compression to ingest streams

  • Implemented more efficient B-tree indexing system (more effective for extremely large indexes)

  • Added maclookup module to look up MAC address vendors

  • Added getMacro function for anko scripts

  • Exposed github.com/RackSec/srslog to anko scripts, allowing scripts to send syslog messages

  • Exposed encoding/base64, encoding/xml, and encoding/hex packages to anko scripts

  • Added duration filtering and extraction for netflow and ipfix modules

  • Improved autoextractor behavior in multi-tag queries with disjoint field sets

  • Improved webserver→indexer ingest functionality (entries are now ingested in parallel rather than blocking the control port)

  • Improved recovery for corrupted shards & indexes

  • Added Max-Log-Files configuration option to limit the number of log files in /opt/gravwell/log

  • stats module now properly handles TAG, e.g. tag=* stats unique_count(TAG) as foo | gauge foo

  • Improved handling of search deletion (admin users were unable to delete other users’ ACTIVE/SAVED searches)

  • syslog module will no longer drop entries which it fails to parse as syslog entries (matching the behavior of other extraction modules)

  • Fixed bug where entries could come back with entry DATA fields during a replication hot-failover scenario

  • Fixed bug where a query containing a macro immediately followed by a newline would fail to parse

  • Fixed error where an improperly shut-down indexer could cause replication failures

  • Fixed issue where an error in the renderer module would be marked as originating from a different module instead

  • Fixed incorrect time reversal behavior of sort module in certain cases

  • Fixed off-by-one error in query tag handling which could cause crashes

  • Fixed bug in macro creation in which groups were not properly preserved

  • Fixed SSO bug where user removal from a group in Active Directory was not mirrored in Gravwell

  • Fixed incorrect nil pointer handling in scheduled search httpPost function

  • Fixed potential webserver crash during shutdown due to lingering websockets

  • Fixed bug where gauge module could not parse multiple (magnitude label) specifications

  • Fixed bug in gauge module where enumerated values containing underscores were not parsed correctly

Ingesters & Ingest Library Changes#

  • Implemented new, more efficient cache system, including new cache mode which allows ingestion without any active indexer connections

  • Added preprocessor to handle Amazon VPC flow logs

  • Fixed bug where cached entries could fail tag translation on ingester restart

  • Kinesis ingester now reports metrics via gravwell tag

  • Kafka Federator now supports TLS-encrypted Kafka transports

  • Fixed line number reporting on ingester log messages

  • Improved logging in early ingester startup

  • Added code to handle Zeek’s tendency to leak PIDs

  • Added transparent compression to ingest library

  • Implemented new gravwellforward preprocessor which can duplicate entries to other Gravwell indexers

Kits#

  • Network Enrichment: Provides databases for enriching network data (geoip, ASN, IP protocols, etc.)

  • CoreDNS: Analyze entries from the Gravwell CoreDNS plugin

  • Netflow v5: Analyze Netflow v5 flow records

  • IPFIX: Analyze IPFIX flow records

  • Grok: Pre-built database of extraction patterns for use with the grok module

  • Weather: Track weather conditions over time

  • Coming soon: Zeek, Windows, Sysmon, and syslog kits

General/Miscellaneous#

  • Added Debian and RPM packages for datastore and loadbalancer components

  • Migrated some personally-maintained code forks into github.com/gravwell umbrella account

  • Transitioned most Gravwell open-source code to a monorepo: github.com/gravwell/gravwell

  • Community Edition licenses will no longer allow the creation of more than 2 users (any existing users will be preserved)