Legacy Eval#

The legacy eval module uses the Anko scripting language to provide generic scriptability within the pipeline.

The eval module has several important restrictions:

  • Only a single statement may be defined: (x==y || j < 2) is acceptable, as is if SrcPort==80 { setEnum("http", true) }, but setEnum(foo, "1"); setEnum(bar, "2") is two statements and will not work. See the following section for more details

  • Functions cannot be defined or imported

  • Loops are not allowed

  • No access to the resource system

Note

To make the structure of your eval expression more clear, hit Ctrl-Enter while typing the query to insert newlines if needed.

See the generic description of the scripting languages used in the Anko scripting language documentation for more details about the language itself.

Filtering: Expressions vs. Statements#

The eval module will filter entries when the argument is an expression; it will not filter when the argument is a statement. Consider the following example:

tag=reddit json Body | eval len(Body) < 20 | table Body

len(Body) < 20 is an expression, so only entries which match the expression are allowed to continue down the pipeline. In contrast, consider the following:

tag=reddit json Body | eval if len(Body) <= 10 { setEnum("postlen", "short"); setEnum(“anotherEnum”, “foo”) } | table Body

The if form is a statement; all entries continue down the pipeline regardless of the outcome of the if statement.

An expression is something which evaluates to a value, like the string “foo” or the number 1.5. This also includes function calls, like DoStuff(15), and boolean expressions such as myVariable == 42. Basically, anything which you could assign to a variable or pass as an argument to a function can be considered an expression.

A statement controls the flow and structure of the script, like if and switch statements, variable creation/assignment, a return, etc. A statement might itself contain multiple sub-statements; for example, an if statement contains an expression and several lists of statements. If the expression evaluates to true, one list of statements is executed. If the expression evaluates to false, a different list is executed.

Enumerated Values#

Within an eval statement, existing enumerated values may be referred to as if they were regular variables:

tag=reddit json Body | eval len(Body) < 20 | table Body

However, to set an enumerated value, a more explicit statement is required. The setEnum function takes a name and a value as arguments. It creates or updates an enumerated value with the given name, inferring the correct enumerated value type for the given value:

tag=reddit json Body | eval if len(Body) <= 10 { setEnum("postlen", "short") } else if len(Body) > 10 && len(Body) < 300 { setEnum("postlen", "medium") } else { setEnum("postlen", "long") } | count by postlen | table postlen count

The delEnum function will delete the specified enumerated value if desired, although this is rarely needed.

Utility Functions#

Eval provides built-in utility functions, listed below in the format functionName(<functionArgs>) <returnValues>:

  • delEnum(key) deletes an enumerated value named key.

  • hasEnum(key) bool returns a boolean indicating whether the entry has the enumerated value.

  • setEnum(key, value) creates an enumerated value named key containing value, which can be any valid enumerated value type.

  • delPersistentMap(mapname, key) value deletes the value associated with the given key from the named persistent map.

  • getPersistentMap(mapname, key) value returns the value associated with the given key from the named persistent map.

  • setPersistentMap(mapname, key, value) stores a key-value pair in a map which will persist for the entire search.

  • len(val) int returns the length of val, which can be a string, slice, etc.

  • sprintf(val) string implements the Golang Sprintf function.

  • toBool(val) bool attempts to convert val to a boolean. Returns false if no conversion is possible. Non-zero numbers and the strings “y”, “yes”, and “true” will return true.

  • toChar(val) char converts val to a char.

  • toDuration(val) time.Duration converts val to a time.Duration.

  • toFloat(val) float64 converts val to a floating point number if possible. Returns 0.0 if no conversion is possible.

  • toIP(string) IP converts string to an IP, suitable for comparing against IPs generated by e.g. the packet module.

  • toInt(val) int64 converts val to an integer if possible. Returns 0 if no conversion is possible.

  • toMAC(string) MAC converts string to a MAC address.

  • toRune(val) rune converts val to a rune.

  • toString(val) string converts val to a string.

  • toHumanCount(val) string attempts to convert val into an integer, then represent it as a human-friendly number, e.g. toHumanCount(15127) will be converted to “15.13 K”.

  • toHumanSize(val) string attempts to convert val into an integer, then represent it as a human-readable byte count, e.g. toHumanSize(15127) will be converted to “14.77 KB”.

  • toBoolSlice(val) []bool converts val to a []bool.

  • toByteSlice(val) []byte converts val to a []byte.

  • toFloatSlice(val) []float64 converts val to a []float64.

  • toIntSlice(val) []int64 converts val to a []int64.

  • toRuneSlice(val) []rune converts val to a []rune.

  • toStringSlice(val) []string converts val to a []string.

  • typeOf(val) type returns the type of val as a string, e.g. “string”, “bool”.

The conversion functions are particularly important because eval can’t always do implicit conversion the way you want. For example, the packet module extracts IPs in a special type, not just a string. In order to properly compare against an enumerated value containing an IP, you must use the IP function:

tag=pcap packet ipv4.SrcIP | eval SrcIP != toIP("192.168.0.1") | count by SrcIP | table SrcIP count

You can check the type of an enumerated value by using the typeOf function, in case you’re getting unexpected results:

tag=pcap packet ipv4.SrcIP | eval setEnum("type", typeOf(SrcIP)) | table type

In this case, the results show that SrcIP is a net.IP.

Persistent Maps#

Eval provides a method to store data in a map, meaning data from one entry can be stored for later comparison against another entry. The SetPersistentMap(mapname, key, value) function creates or updates an entry in a map (specified by mapname parameter) mapping the given string key to the value, which can be anything. The GetPersistentMap(mapname, key) function can then be used to retrieve that information.

To give an example of the persistent map functionality, consider the following json-formatted Hacker News comment (not a real comment):

{"body": "The eval function has persistent map capabilities", "author": "Gravwell", "article-id": 1234000, "parent-id": 1234111, "article-title": "Gravwell for data analysis", "date-string": "0 minutes ago", "type": "comment", "id": 1234222}

Note that the comment contains an ID for the current comment (1234222), the author’s name (“Gravwell”), and the ID for the parent comment (12341111), but not the name of the parent comment’s author.

The following command attempts to match parent IDs to author names. For every comment it sees, it stores a mapping of the comment ID to the author name in a map named “id_to_name”. Then, it checks if the parent ID has an entry in that map; if so, it sets the parentauthor enumerated value to that name, otherwise it sets the enumerated value to “unknown”.

tag=hackernews json author id "parent-id" as parentid | sort asc | eval if true { setPersistentMap("id_to_name", id, author);  name = getPersistentMap("id_to_name", parentid); if name != nil { setEnum("parentauthor", name) } else { setEnum("parentauthor", "unknown") } } | table author id parentid parentauthor

One limitation of this approach is that the oldest comments, processed first, will be replying to even older comments which will not appear in the search; thus, the first results will all have a parent author of “unknown”, with more and more comments finding valid parent authors as you scroll through the results. A better solution would be to do a search over the span of perhaps a week, pulling out the author name and comment ID for each comment and saving the results in a lookup table. Then that lookup table could be stored as a resource for use with the lookup module in a search of shorter duration.

This example also shows another limitation of the eval module in its current state: it only executes one statement or expression, so we have to wrap the logic inside an if true {} statement.