At this year’s [FOSDEM] conference I did a 30 minutes presentation on [Monitoring Legacy Java Applications with Prometheus]. The talk gives an overview of some of the options you have for monitoring Java applications with [Prometheus] when you cannot modify the application’s source code:

• Logfile monitoring ([grok_exporter]), and how it differs from [Elastic stack]
• Blackbox monitoring ([blackbox_exporter])
• JMX ([jmx_exporter])
• Write your own Java agent ([promagent.io])

The video is available below.

more...

At this year’s [FOSDEM] conference I did a 20 minutes presentation on how to implement tail -f in [Go]. The video is available below.

Abstract: As part of a log file monitoring tool, I implemented a file tailer that keeps reading new lines from log files. This turned out to be much more challenging than I thought, especially because it should run on multiple operating systems and it should be robust against logrotate. In this 20 Minutes talk I will present the lessons learned, the pitfalls and dead-ends I ran into.

more...

[Prometheus] is an open source monitoring tool, which is conceptually based on Google’s internal Borgmon monitoring system. Unlike traditional tools like Nagios, Prometheus implements a white-box monitoring approach: Applications actively provide metrics, these metrics are stored in a time-series database, the time-series data is used as a source for generating alerts. Prometheus comes with a powerful query language allowing for statistical evaluation of metrics.

more...

[PromCon 2016] was the first conference around the [Prometheus] monitoring system. It took place from August 25 - 26 2016 at Google Berlin as a single-track event with space for 80 attendants.

We took the opportunity and did a lightning talk introducing [grok_exporter], which is a tool for extracting Prometheus metrics from application logs.

more...

Counting the number of error messages in log files and providing the counters to [Prometheus] is one of the main uses of [grok_exporter], a tool that we introduced in the [previous post].

The counters are collected by the [Prometheus] server, and are evaluated using Prometheus’ query language. The query results can be visualized in [Grafana] dashboards, and they are the basis for defining [alerts].

We found that evaluating error counters in Prometheus has some unexpected pitfalls, especially because Prometheus’ [increase()] function is somewhat counterintuitive for that purpose. This post describes our lessons learned when using [increase()] for evaluating error counters in Prometheus.

more...

[Prometheus] is an open-source systems monitoring and alerting toolkit. At its core, Prometheus uses time-series data, and provides a powerful query language to analyze that data. Most Prometheus deployments integrate [Grafana] dashboards and an [alert manager].

Prometheus is mainly intended for white box monitoring: Applications either provide Prometheus metrics natively, or they are instrumented with an [exporter] to make application-specific metrics available.

For some applications, parsing log files is the only way to acquire metrics. The [grok_exporter] is a generic Prometheus exporter extracting metrics from arbitrary unstructured log data.

This post shows how to use [grok_exporter] to extract metrics from log files and make them available to the Prometheus monitoring toolkit.

more...