IPFIX and Netflow V9#

The ipfix processor is designed to extract and filter raw IPFIX and Netflow V9 data frames, allowing you to quickly identify network flows, filter on ports, or generally monitor the behavior of aggregate flows. Gravwell has a native IPFIX + Netflow ingester which is open source and available at https://github.com/gravwell/ingesters or as an installer in the quickstart section.

A Note About Templates#

Gravwell is flexible with regards to entry order. You can run queries from oldest entry to newest, from newest to oldest, or you can sort entries based on some other criteria entirely.

Unfortunately, this flexibility does not interact well with IPFIX and Netflow V9. These protocols define templates, which describe the format of data records. Templates are transmitted infrequently; any given IPFIX message is unlikely to contain a template record, but it will have data records referring to previously-defined templates. Without access to the appropriate template, a data record is meaningless. This means that parsing IPFIX messages from newest to oldest (or sorted in some other fashion) cannot work properly unless additional steps are taken.

We determined that the most straight-forward course of action was to have the ingester repack every incoming IPFIX/Netflow V9 message with the appropriate templates for its data records. This increases the size of each message stored, but because messages typically include tens of data records, the increase is not large. It also means that every single message (entry) stands completely on its own and does not depend on any previous messages for parsing.

Supported Options#

  • -e <name>: The “-e” option specifies that the ipfix module should operate on an enumerated value. Operating on enumerated values can be useful when you have extracted an ipfix frame using upstream modules.

  • -s: The “-s” option puts the ipfix module in strict mode. In strict mode, any specified extractions must succeed, or the entry will be dropped; thus, ipfix protocolIdentifier==1 srcPort would drop all entries, because ICMP messages (IP protocol 1) will never contain a source port field.

  • -f: The “-f” flag tells the module to extract a list of available fields for each IPFIX data record it sees and drop them, in human-readable format, into an enumerated value called “FIELDS”. This is mutually exclusive with field extraction; if you specify -f, you cannot specify any fields to extract.

Processing Operators#

Each IPFIX field supports a set of operators that can act as fast filters. The filters supported by each operator are determined by the data type of the field. Numeric values support everything except the subset operators, while IP addresses support only the subset operators.

Operator

Name

Description

==

Equal

Field must be equal

!=

Not equal

Field must not be equal

<

Less than

Field must be less than

>

Greater than

Field must be greater than

<=

Less than or equal

Field must be less than or equal to

>=

Greater than or equal

Field must be greater than or equal to

~

Subset

Field must be a member of

!~

Not subset

Field must not be a member of

Data Items#

The ipfix search module is designed to process raw IPFIX & Netflow v9 message. A single IPFIX (or NFv9) message consists of a header and N data records. One essential difference between IPFIX/NFv9 and Netflow v5 is that all fields in Netflow v5 are pre-defined, while IPFIX/NFv9 data records conform to templates specified by the generating device and transmitted in previous messages. Thus, one IPFIX/NFv9 generator might send source and destination IP & ports for flows, while a switch just sends records containing packet counts.

All elements of the IPFIX/NFv9 message header can be used for filtering, as can the elements in the data records themselves. When filtering on header data items, the filter applies to all records in the message. Header data items are processed first, and only if the header filters do not drop the frame are the individual records processed.

The ipfix processor is an expanding module; expanding modules break input entries into multiple output entries. The module takes in individual entries corresponding to whole IPFIX messages. It then processes each data record within the message and emits a new entry for each record; thus, a single input entry might turn into 30 output entries.

Note

If you specify only header items to extract (e.g. ipfix Version Sequence), entries are not expanded, because there is only one header per IPFIX message. Each incoming entry will result in no more than one outgoing entry. Specifying a mix of header and data items will trigger expansion.

IPFIX Header Data Items#

Field

Description

Supported Operators

Example

Version

The Netflow version in use. 10 means IPFIX, 9 means Netflow v9.

> < <= >= == !=

Version != 0xa

Length

IPFIX: the total length of this IPFIX message. Netflow v9: the number of records contained in this message.

> < <= >= == !=

Length > 1000

Sec

Current Unix timestamp of the sensing device

> < <= >= == !=

Sec == 1526511023

Uptime

Number of milliseconds the sensing device has been online (Netflow v9 only)

> < <= >= == !=

Sec == 3516293216

Sequence

Sequence counter of total messages sent by the sensing device

> < <= >= == !=

Sequence == 234

Domain

The observation domain from which the record originated

> < <= >= == !=

Domain == 0x1A

Data Record Items#

IPFIX and Netflow v9 define fields which may make up a data record. Netflow defines several dozens of fields which can be found in RFC3954. A Netflow v9 field is defined by a field ID (a positive integer), a data type (e.g. 16-bit unsigned integer, etc.), and a name (e.g. L4_SRC_PORT).

IANA has defined a similar set of fields for IPFIX, which can be found here. An IPFIX fields are similar to Netflow v9 fields (they have a field ID, a type, and a name), but IPFIX introduces the concept of the enterprise ID in addition to the field IDs used by Netflow v9. The enterprise ID allows users to define their own set of fields in addition to the predefined ones, which all have an enterprise ID of 0.

Attention

Because IPFIX and Netflow v9 are template-based, any given data record may or may not contain the fields described below. If you attempt to extract e.g. “sourceIPv4PrefixLength” but get empty results, it is possible that your IPFIX records do not contain that field.

For convenient, we list some of the most common IPFIX and Netflow v9 fields below; refer to the documents linked above for a complete list of supported names.

| IPFIX Name | Netflow v9 Name | Description | Supported Operators | Example | |——-|————————–|———————|———| | octetDeltaCount | IN_BYTES | The number of octets since the previous report (if any) in incoming packets for this Flow at the Observation Point. The number of octets includes IP header(s) and IP payload | > < <= >= == != | octetDeltaCount == 80 | packetDeltaCount | IN_PKTS | The number of incoming packets since the previous report (if any) for this Flow at the Observation Point. | > < <= >= == != | packetDeltaCount == 80 | deltaFlowCount | FLOWS | The conservative count of Original Flows contributing to this Aggregated Flow; may be distributed via any of the methods expressed by the valueDistributionMethod Information Element. | > < <= >= == != | packetDeltaCount == 80 | protocolIdentifier | PROTOCOL | Protocol number of the flow (TCP = 6, UDP = 17 | > < <= >= == != | protocolIdentifier == 17 | ipClassOfService | TOS | For IPv4 packets, this is the value of the TOS field in the IPv4 packet header. For IPv6 packets, this is the value of the Traffic Class field in the IPv6 packet header | > < <= >= == != | ipClassOfService != 0 | tcpControlBits | TCP_FLAGS | TCP control bits observed for the packets of this Flow. | > < <= >= == != | tcpControlBits != 0x0004 | sourceTransportPort | L4_SRC_PORT | Source port of the flow. If the protocol does not have a port the value is zero | > < <= >= == != | sourceTransportPort != 0 | sourceIPv4Address | IPV4_SRC_ADDR| IPv4 source address of the flow. | ~ !~ == != | sourceIPv4Address ~ 10.0.0.0/24 | sourceIPv4PrefixLength | SRC_MASK | IPv4 source address prefix length. | > < <= >= == != | sourceIPv4PrefixLength < 24 | sourceIPv6Address | IPV6_SRC_ADDR | IPv6 source address of the flow. | ~ !~ == != | sourceIPv6Address == ::1 | sourceIPv6PrefixLength | IPV6_SRC_MASK | IPv6 source address prefix length. | > < <= >= == != | sourceIPv6PrefixLength < 64 | destinationTransportPort | L4_DST_PORT | Destination port of the flow. If the protocol does not have a port the value is zero | > < <= >= == != | destinationTransportPort != 0 | destinationIPv4Address | IPV4_DST_ADDR | IPv4 destination address of the flow. | ~ !~ == != | destinationIPv4Address ~ 10.0.0.0/24 | destinationIPv4PrefixLength | DST_MASK | IPv4 destination address prefix length. | > < <= >= == != | destinationIPv4PrefixLength < 24 | destinationIPv6Address | IPV6_DST_ADDR | IPv6 destination address of the flow. | ~ !~ == != | destinationIPv6Address == ::1 | destinationIPv6PrefixLength | IPV6_DST_MASK | IPv6 destination address prefix length. | > < <= >= == != | destinationIPv6PrefixLength < 64

Note

In general, you can specify Netflow v9 field names when extracting IPFIX messages and vice versa. However, because data types vary slightly between the two protocols, it is always safest to use IPFIX field names when processing IPFIX and Netflow names when processing Netflow.

The module also provides a handful of “shortcuts” for convenience:

Field

Description

Supported Operators

Example

src

The source address for this flow (IPv4 or IPv6)

~ !~ == !=

src == ::1

dst

The destination address for this flow (IPv4 or IPv6)

~ !~ == !=

dst !~ PRIVATE

srcPort

The source port for this flow.

> < <= >= == !=

srcPort == 80

dstPort

The destination port for this flow.

> < <= >= == !=

dstPort == 80

ip

Extract the first IP that matches a filter. If no filter is specified the Src is used

~ !~ == !=

ip ~ 10.0.0.0/24

port

Extract the first port that matches the filter. If no filter is specified the lower value is used

> < <= >= == !=

port == 80

vlan

Extracts the first VLAN that matches a filter, where the VLAN may be drawn from either the vlanId or dot1qVlanId field.

> < <= >= == !=

vlan == 100

srcMac

Extracts the source MAC address for the flow

== !=

srcMac==01:23:45:67:89:00

dstMac

Extracts the destination MAC address for the flow

== !=

dstMac==01:23:45:67:89:00

bytes

Total bytes seen, this adds the octetDeltaCount and postOctetDeltaCount values together

> < <= >= == !=

bytes <= 10000

packets

Total packets seen, this adds the packetDeltaCount and postPacketDeltaCount values together

> < <= >= == !=

packets > 0xffffff

flowStart

The start timestamp of the flow, this shorthand uses any of the following fields: flowStartSeconds, flowStartMilliseconds, flowStartMicroseconds, flowStartNanoseconds and outputs a proper timestamp

flowStart

flowEnd

The end timestamp of the flow, this shorthand uses any of the following fields: flowEndSeconds, flowEndMilliseconds, flowSEndMicroseconds, flowEndNanoseconds and outputs a proper timetamp

flowStart

flowDuration

The duration value calculated from flowEnd and flowStart.

== != < > <= >=

flowDuration > 1m

Filtering on other fields#

You can also specify fields by giving an enterprise ID and field ID separated by a colon, e.g. “0x1ad7:0x15”. We recommend extracting this to a more convenient name: ipfix 0x1ad7:0x15 as foo. If you wish to extract Netflow v9 fields in this fashion, just assume an enterprise ID of 0, e.g. ipfix 0:7 as srcport to extract the source port.

Examples#

Number of HTTPS flows by Source IP over time#

tag=ipfix ipfix destinationIPv4Address as Dst destinationTransportPort==443 | count by Dst | chart count by Dst

Number of flows by ip

Find out which IPs are using port 80#

tag=ipfix ipfix port==80 ip ~ PRIVATE | unique ip | table ip

Find the most common protocol in Netflow v9 records#

This example filters down to only Netflow v9 records, then extracts the protocol and counts how many flows appeared for each.

tag=v9 ipfix Version==9 PROTOCOL | count by PROTOCOL | table PROTOCOL count

Extract a list of fields in each data record#

tag=ipfix ipfix -f | table FIELDS