Monitoring Dgraph with Elastic's Metricbeat, Prometheus Module

Users Dgraph
1

Our team leverages the Elastic stack for our system observability needs over the prometheus monitoring product, so this post is aimed at asking some questions about integration there and documenting our experience thus far.

We’ve been able to successfully able to connect to the prometheus_metrics endpoint for output with curl:

> curl http://alpha1:8080/debug/prometheus_metrics
# HELP badger_blocked_puts_total Total number of blocked puts
# TYPE badger_blocked_puts_total untyped
badger_blocked_puts_total 0

How to set this up with Elasticsearch and Kibana
Having had metricbeat already setup, we needed to add the corresponding prometheus module to the metricbeat configuration.

/etc/metricbeat/modules.d/prometheus.yml:

- module: prometheus
  period: 10s
  # metricsets: ["collector"]
  hosts:
    - 10.100.1.1:18080
    - 10.100.1.2:18081
    - 10.100.1.3:18082
    - 10.100.1.4:18083
    - 10.100.1.5:18084
    - 10.100.1.6:18085
    - 10.100.1.1:6080
    - 10.100.1.3:6081
    - 10.100.1.5:6082
 metrics_path: /debug/prometheus_metrics
 query:
   'match[]': '{__name__=~""}'
 use_types: true
 rate_counters: true

Question
The docs for prometheus module in elastic note that there are three metrics sets that can be gathered with prometheus based metric endpoints: collector, query, and remote_write. However, to receive the full set of data being produced by the /debug/prometheus_metrics endpoint, we needed to use the query with match above.

When any of these metricsets were configured, the module doesn’t gather any metrics at all, instead the line needs commented out and the additional query with match added.

Why is this?

Helpful Dgraph Links:

Helpful Elastic Links: