OpenTelemetry#
Integration Details |
|
Ingester |
OpenTelemetry Configuration#
OpenTelemetry uses Collectors to forward data from your application or infrastructure directly into Gravwell. While this guide demonstrates using Kubernetes, it could be adapted to route logs from any OpenTelemetry-compatible service.
Example: Simple OpenTelemetry Configuration#
To send data to Gravwell, add an otlphttp exporter that points to your Gravwell HTTP Ingester to the exporters section of your OpenTelemetry Collector configuration file.
exporters:
otlphttp:
endpoint: http://path.to.gravwell:port # Change this line to point to your Gravwell instance
encoding: json
tls:
insecure: true
Next, update your active data pipelines under the service section to route your data through the new otlphttp exporter:
service:
pipelines:
metrics:
receivers: [otlp, prometheus]
processors: [batch]
exporters: [otlphttp]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlphttp]
Note
The Gravwell HTTP Ingester currently has to listener to receive trace pipelines.
Note
You can validate your Collector config with:
otelcol validate --config=customconfig.yaml
Example: Kubernetes Manifest File#
This example provides a native Kubernetes manifest file that sets up an OpenTelemetry Collector to automatically ingest logs and performance metrics from your Kubernetes cluster. To route this data to your environment, you must modify the following line in the manifest to point to your Gravwell HTTP Ingester:
endpoint: http://path.to.gravwell:port
This manifest utilizes Custom Resource Definitions from the OpenTelemetry Operator. The following commands provide an example that could be run on your master node to install the operator via Helm to deploy the collector.
# 1. Add the OpenTelemetry Helm repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
# 2. Install the OpenTelemetry Operator chart
helm install my-otel-operator open-telemetry/opentelemetry-operator --set admissionWebhooks.certManager.enabled=false
# 3. Deploy the custom Collector configuration
kubectl apply -f otel_gravwell_collector.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: otelcol
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: otelcol
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/stats
- nodes/proxy
- pods
- namespaces
- services
- endpoints
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: otelcol
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: otelcol
subjects:
- kind: ServiceAccount
name: otelcol
namespace: default
---
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: default
spec:
image: otel/opentelemetry-collector-contrib:0.112.0
mode: daemonset
serviceAccount: otelcol
# hostNetwork is required for the `from: connection` pod_association in
# k8sattributes to resolve the real client pod IP. Without it, all incoming
# OTLP traffic looks like it came from the CNI's masqueraded address.
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
securityContext:
runAsUser: 0
env:
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumes:
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
volumeMounts:
- name: varlogpods
mountPath: /var/log/pods
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
kubeletstats:
collection_interval: 20s
auth_type: serviceAccount
# Explicit https:// scheme is required by recent versions
endpoint: https://${env:K8S_NODE_NAME}:10250
insecure_skip_verify: true
metric_groups: [node, pod, container]
filelog:
include: [/var/log/pods/*/*/*.log]
# Exclude this collector's own pods to avoid a feedback loop.
# The operator names DaemonSet pods `<cr-name>-collector-<hash>`,
# so for a CR named `otel` in namespace `default` the pod dirs are:
# /var/log/pods/default_otel-collector-<hash>_<uid>/...
exclude:
- /var/log/pods/default_otel-collector-*/*/*.log
include_file_path: true
start_at: end
operators:
- type: container
id: container-parser
processors:
batch: {}
k8sattributes:
auth_type: serviceAccount
passthrough: false
extract:
metadata:
- k8s.pod.name
- k8s.pod.uid
- k8s.namespace.name
- k8s.node.name
- k8s.deployment.name
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.uid
- sources:
- from: connection
exporters:
otlphttp:
endpoint: http://path.to.gravwell:port # Change this line to point to your Gravwell instance
encoding: json
tls:
insecure: true
debug:
verbosity: basic
service:
pipelines:
metrics:
receivers: [otlp, kubeletstats]
processors:
- k8sattributes
- batch
exporters: [otlphttp]
logs:
receivers: [otlp, filelog]
processors: [k8sattributes, batch]
# NOTE: do NOT add `debug` here unless your filelog `exclude`
# pattern definitely matches this collector's pod path, or you
# will create a feedback loop.
exporters: [otlphttp]
Gravwell Configuration#
Gravwell Storage Well Configuration#
Setup the well configuration in your Gravwell indexers.
Sample well config:
Create or edit: /opt/gravwell/etc/gravwell.conf.d/opentelemetry-well.conf
[Storage-Well "opentelemetry"]
Location=/opt/gravwell/storage/opentelemetry
Tags=otel-*
Gravwell Ingester Configuration: HTTP Ingester#
This guide assumes the HTTP Ingester is already installed and configured as described in HTTP Ingester Setup. Add the following to a new overlay file; you do not need a separate HTTP Ingester for this integration.
Sample OpenTelemetry config:
Create or edit: /opt/gravwell/etc/gravwell_http_ingester.conf.d/opentelemetry.conf
[OpenTelemetry-Logs-Listener "kubelogs"]
URL="/v1/logs"
Tag-Name=otel-k8s-logs
[OpenTelemetry-Metrics-Listener "kubemetrics"]
URL="/v1/metrics"
Tag-Name=otel-k8s-metrics
These listeners use the default OpenTelemetry paths. If other integrations also send OpenTelemetry data to the same HTTP Ingester, give each one its own path prefix, as described in HTTP Ingester Setup.
Note
Remember to restart the service to apply the new config:
sudo systemctl restart gravwell_http_ingester.service