Basic Monitoring Setup with Promethues and Grafana

Introduction

If you are a developer, sysadmin or a dev-ops guy, you always need to have an eye on your applications and servers which are running somewhere around the world. There are so many tools that let you monitor them and get alerts whenever something is going wrong. Prometheus & Grafana are a couple which let you measure different aspects of your application and also have nice charts that you could understand easily. Here I’m going to have an introduction for a simple setup using docker. Here I’m not covering any details and just trying to set it up and running.

You could take a look at their documentations

Overview

Grafana documentation

How does it work?

Prometheus is a tool which collects the data from different sources and have a nice query format which lets you see what was happening at any time you want.

Grafana is one of the best applications for creating charts, alerts, etc. It could also use different data sources, and prometheus is one of them.

Here is a simple docker-compose file which lets you bring up both of them and then connect them together.

version: "3"
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - /home/apps/system_monitoring/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - /home/apps/system_monitoring/prometheus/data:/prometheus
    ports:
      - '127.0.0.1:9090:9090'
  grafana:
    image: grafana/grafana
    environment:
      GF_DATABASE_TYPE: postgres
      GF_DATABASE_HOST: 'postgres'
      GF_DATABASE_NAME: 'MY_DB_NAME'
      GF_DATABASE_USER: 'MY_DB_PASSWORD'
      GF_DATABASE_PASSWORD: 'MY_DB_PASSWORD'
    ports:
      - '127.0.0.1:3100:3000'
    links:
      - prometheus
      - postgres
    user: "USER_ID"
    volumes:
      - /home/apps/system_monitoring/grafana/data:/var/lib/grafana
      - /var/log/grafana:/var/log/grafana
    depends_on:
      - postgres
  postgres:
    image: postgres:latest
    environment:
      POSTGRES_USER: MY_DB_USER
      POSTGRES_PASSWORD: MY_DB_PASSWORD
      POSTGRES_DB: MY_DB_NAME
    volumes:
      - /home/apps/system_monitoring/postgres:/var/lib/postgresql/data
      - /var/log/postgres:/var/log/postgresql

Before going into any details, take a look at Docker Overview if you are not familiar with docker and also you could read my article, Tips on Using Docker, which covers the important parts for using docker.

So now let’s take a deeper look into this and also add the missing mounted files like prometheus configuration.

Prometheus Configuration

Here is a simple configuration file from their documentation.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` 
  # to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets
    # from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

With this configurations, the prometheus reads data from localhost:9090 every 5 seconds and store them in a timeseries format in data directory. But what do we have on that port in our localhost? We are going to cover that later by describing the Node Exporter.

Grafana Configuration

Grafana is one of the most powerful tools for displaying data. Here we are running it with postgres database. You could also take a look at the docker installation here for more information.

Remember that the mounted data directory needs to have the right permission on host machine, so be careful and change the ownership to the right user before firing up the container.

Node Exporter

Inside prometheus configurations, it was listing to port 9090 on localhost for collecting data, but what do we need to have there?

The node exporter is an official data collector for hardware and OS metrics. You just need to install and run it, it will export so many metrics on port 9090 by default and prometheus will collect them.

Here you could see a list of available exporters for other services as well, and also you could write your own exporter.

Final Step

Now if you fire up the containers, you would be able to see grafana login page at localhost:3100, then you could login into it with default credentials.

Username: admin
Password: admin

Now you need to go to Data Sources under Configuration menu, click on Add data source and fill it like below.

After adding the data source successfully, we need to have some dashboards to show us the collected metrics. There is a full dashboard for node exporter here which you could import easily, just click on importing dashboard and paste the link, then you are able to see a full dashboard like this.

It was completely a basic setup without going into any details. There are so many things that you could add or customize for each one of them which will be covered later in separated posts. Hope you’ll find this useful and handy.