How to setup Prometheus and Grafana using Docker

Prometheus and Grafana

This is the first in a series of posts about monitoring using Prometheus and Grafana, and how they can be used in a variety of scenarios.

The setups in these post make use of Docker to run the Grafana, Prometheus and Prometheus exporter applications - but key to this is the setup of Prometheus and Grafana which does not differ significantly between each scenario.

The components and their roles are as follows:

• Docker - hosts the containers.
• Prometheus - scrapes the metric data from endpoints and stores it.
• Prometheus Exporters - interface with various technologies, allowing Prometheus to collect and store metrics about them.
• Grafana - for the graphical front end.

Note - the commands and details listed below can also be found in this House of Logic github repo.

Docker Setup

The easiest way to configure Grafana and Prometheus is to use Docker containers. You can set these up directly on a host if you prefer, but the instructions here make use of Docker running on an Ubuntu host. Specifically, the examples were set up on Proxmox (which we also monitor in a later post), but this is not a requirement. Docker can be installed on either a VM or LXC (see guide here.)

Note: The username of “user” is used in the examples - update the files and paths to reflect your username.

To install Docker on Ubuntu, run the following:

apt-get update

apt-get install docker.io

Exporter setup

You may wish to install any environment specific Prometheus Exporters and configure these before progressing to the Prometheus setup.

If you do not, you will need to configure these afterwards, include their details in the prometheus.yml file and restart the container. This will also be true if you need to correct or add to the Prometheus configuration.

Prometheus setup

With your desired exporters running, you can now configure Prometheus and connect them up to it.

Create a config file directory and editing the prometheus.yml config file.

cd ..

mkdir prometheus

nano prometheus.yml

Edit the prometheus.yml file as desired, eg:

scrape_configs:
  - job_name: "prometheus"

    # metrics_path defaults to 'metrics'
    # scheme defaults to 'http'

    static_configs:
      - targets: ["localhost:9090"]

  # enter additional jobs using job_name for each in the scrape configs section

You should also create a docker volume for your prometheus data to be stored in, eg:

sudo docker volume create prometheus-data

You should now start the Prometheus container, mapping in the config file and data volume, again running as a daemon.

sudo docker run \
    -p 9090:9090 \
    --name=prometheus\
    -d \
    -v /home/user/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
    -v prometheus-data:/prometheus \
    prom/prometheus

Check that the container is running using:

sudo docker ps

curl http://localhost:9090/metrics

Alternatively, you can browse to the docker host’s IP address and prometheus port, eg:

http://192.168.2.11:9090

After a minute or two any exporter metrics should also be visible.

Grafana setup

With Prometheus and any additional exporters configured, the final stage of the setup is to configure Grafana for visualisations.

Grafana does not need a config directory or file to be edited, but these can be configured as follows.

cd ..

mkdir grafana

cd grafana

Additional options can be configured in the grafana.ini file - this may be desirable if you want to set the username and password, or other options, before starting Grafana. See https://grafana.com/docs/grafana/latest/administration/provisioning/ for details.

nano grafana.yaml

Note - if you do use the configuration file, add the command “-v /home/user/grafana/grafana.ini:/etc/grafana/grafana.ini " to the docker run command below.

To create a storage volume for Grafana, run the following:

sudo docker volume create grafana-storage

You can then start the Grafana container as a service using the following command:

sudo docker run -d -p 3000:3000 --name=grafana \
  --volume grafana-storage:/var/lib/grafana \
  grafana/grafana-enterprise

Once again, you should be able to check that the grafana container is running in Docker and the service responds on the network using the following:

sudo docker ps

curl http://localhost:3000

You should now be able to login to Grafana using a web browser to connect to the Docker host IP and Grafana port (3000 by default), eg:

http://192.168.2.11:3000

The default credentials for Grafana are as follows:

username = admin

password = admin 

You will be prompted to change the default password on first login, at which point the new password will persist.

Datasources and Dashboards

When you are logged into Grafana, you should be able to add Prometheus as a data source, and use it to build dashboards of your metrics.

Note: When adding the data source, make sure to use the IP address of the host rather than “localhost”, which will Grafana will interpret as the address of its own container, rather than the host or Prometheus container which it needs to connect to.

Dashboards can be built by adding a visualization, selecting the prometheus data source and then selecting a metric.

Demonstration

A demonstration of the setup can be found here, which includes the Proxmox PVE exporter - see this post for more details on Promxmox exporter specifically.

Useful links

Installing Prometheus using Docker

Installing Grafana using Docker