At this year’s [FOSDEM] conference I did a 20 minutes presentation on how to implement
tail -f in [Go]. The video is available below.
Abstract: As part of a log file monitoring tool, I implemented a file tailer that keeps reading new lines from log files. This turned out to be much more challenging than I thought, especially because it should run on multiple operating systems and it should be robust against logrotate. In this 20 Minutes talk I will present the lessons learned, the pitfalls and dead-ends I ran into.
[Prometheus] is an open source monitoring tool, which is conceptually based on Google’s internal Borgmon monitoring system. Unlike traditional tools like Nagios, Prometheus implements a white-box monitoring approach: Applications actively provide metrics, these metrics are stored in a time-series database, the time-series data is used as a source for generating alerts. Prometheus comes with a powerful query language allowing for statistical evaluation of metrics.
[PromCon 2016] was the first conference around the [Prometheus] monitoring system. It took place from August 25 - 26 2016 at Google Berlin as a single-track event with space for 80 attendants.
We took the opportunity and did a lightning talk introducing [grok_exporter], which is a tool for extracting Prometheus metrics from application logs.
Counting the number of error messages in log files and providing the counters to [Prometheus] is one of the main uses of [grok_exporter], a tool that we introduced in the [previous post].
The counters are collected by the [Prometheus] server, and are evaluated using Prometheus’ query language. The query results can be visualized in [Grafana] dashboards, and they are the basis for defining [alerts].
We found that evaluating error counters in Prometheus has some unexpected pitfalls, especially because Prometheus’ [increase()] function is somewhat counterintuitive for that purpose. This post describes our lessons learned when using [increase()] for evaluating error counters in Prometheus.
[Prometheus] is an open-source systems monitoring and alerting toolkit. At its core, Prometheus uses time-series data, and provides a powerful query language to analyze that data. Most Prometheus deployments integrate [Grafana] dashboards and an [alert manager].
Prometheus is mainly intended for white box monitoring: Applications either provide Prometheus metrics natively, or they are instrumented with an [exporter] to make application-specific metrics available.
For some applications, parsing log files is the only way to acquire metrics. The [grok_exporter] is a generic Prometheus exporter extracting metrics from arbitrary unstructured log data.
This post shows how to use [grok_exporter] to extract metrics from log files and make them available to the Prometheus monitoring toolkit.