How to Setup and Configure Prometheus on Linux Host - Crunchify Tips

What is Prometheus?

Prometheus is world class Monitoring System comes with Time Series Database as default. It’s an open-source systems originally built in year 2012.

World’s top 500 companies have been using Prometheus for collecting metrics for their infrastructure and application. Prometheus supports multi-dimensional data model with time series data which mainly identified as Metrics.

There is no special data storage requirement as Prometheus uses Time Series DB (TSDB) part of it’s own installation.

In this tutorial we will install Prometheus on Linux Ubuntu and CentOS with super simple steps.

Let’s get started:

Step-1 Verify Linux OS version

Command: cat /etc/os-release

root@localhost:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="19.04 (Disco Dingo)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 19.04"
VERSION_ID="19.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=disco
UBUNTU_CODENAME=disco

Below steps apply to Ubuntu Linux OS.

Step-2 Create Prometheus Data and Config Folder

Commands:

root@localhost:~# sudo mkdir /crunchify/prometheus/conf

root@localhost:~# sudo mkdir /crunchify/prometheus/data

Let’s verify both folders:

root@localhost:/crunchify/prometheus# pwd
/crunchify/prometheus

root@localhost:/crunchify/prometheus# ls -ltra
total 16
drwxr-xr-x 2 root root 4096 Aug 25 23:37 conf
drwxr-xr-x 3 root root 4096 Aug 25 23:37 ..
drwxr-xr-x 2 root root 4096 Aug 25 23:37 data
drwxr-xr-x 4 root root 4096 Aug 25 23:37 .

Step-3 Change ownership of Data folder

This is required to avoid permission issue in the later steps.

root@localhost:/crunchify/prometheus# sudo chown 65534:65534 /crunchify/prometheus/data/

Step-4 Create prometheus.yml file

Go to folder /crunchify/prometheus/conf and create file prometheus.yml.

root@localhost:/# cd /crunchify/prometheus/conf

root@localhost:/crunchify/prometheus/conf# vi prometheus.yml

prometheus.yml file content:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any time series scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['65.19.71.11:9090']

Note: In above file – replace IP with your host/VM IP.

Here is a Directory structure for this tutorial:

Prometheus Setup on Linux - directory tree strucutre

Step-5 Make sure you have installed Docker on your VM.

How to check if Docker is installed or not?

root@localhost:~# docker

Command 'docker' not found

That means docker is not installed on your host/VM. Please follow Docker installation tutorial completely to install Docker on your host/VM.

Step-6 Install Docker Compose

Pre-requisite is to install Docker first before executing below steps. (Step-5)

Execute below three commands:

root@localhost:~# sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0   2448      0 --:--:-- --:--:-- --:--:--  2448
100 15.4M  100 15.4M    0     0  11.2M      0  0:00:01  0:00:01 --:--:-- 15.7M

root@localhost:~# sudo chmod +x /usr/local/bin/docker-compose

root@localhost:~# sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Step-7 Verify Docker Compose Version

root@localhost:~# docker-compose --version 

docker-compose version 1.24.1, build 4667896b

Congrats – docker compose is successfully installed on your VM.

Step-8 Create docker-compose.yml file

Go to folder /crunchify/prometheus.

root@localhost:/crunchify# cd /crunchify/prometheus/

root@localhost:/crunchify/prometheus# vi docker-compose.yml

docker-compose.yml file content:

version: "3"

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - /crunchify/prometheus/conf:/etc/prometheus
      - /crunchify/prometheus/data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"

Step-9 Install Prometheus Start/Stop as part of system control.

Create file vi /etc/systemd/system/prometheus.service

prometheus.service file content:

[Unit]
Description=Prometheus monitoring docker container
After=docker.service
BindsTo=docker.service

[Service]
Restart=always
WorkingDirectory=/crunchify/prometheus/
# Ubuntu
ExecStart=/usr/bin/docker-compose up
ExecStop=/usr/bin/docker-compose down
# CentOS
#ExecStart=/usr/bin/docker-compose up
#ExecStop=/usr/bin/docker-compose down

[Install]
WantedBy=multi-user.target

Step-10 Start Prometheus service

root@localhost:/crunchify/prometheus/config# sudo systemctl daemon-reload

root@localhost:/crunchify/prometheus/config# sudo systemctl enable prometheus
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.

root@localhost:/crunchify/prometheus/config# sudo systemctl start prometheus

Prometheus Server Started on Linux:

Prometheus Server Started on Linux Ubuntu
root@localhost:/crunchify/prometheus# sudo systemctl start prometheus
Creating network "prometheus_default" with the default driver
Pulling prometheus (prom/prometheus:latest)...
latest: Pulling from prom/prometheus
8e674ad76dce: Pull complete
e77d2419d1c2: Pull complete
d7c0a2a2ca7e: Pull complete
bd46de67c79d: Pull complete
35f6078df7e9: Pull complete
8b3f04e26151: Pull complete
9d234d9c8a77: Pull complete
1a8a2a4f314d: Pull complete
62e28fd716ec: Pull complete
Digest: sha256:cd93b8711bb92eb9c437d74217311519e0a93bc55779aa664325dc83cd13cb32
Status: Downloaded newer image for prom/prometheus:latest
Creating prometheus ... done
Attaching to prometheus
prometheus    | level=info ts=2019-08-26T02:57:48.796Z caller=main.go:293 msg="no time or size retention was set so using the default time retention" duration=15d
prometheus    | level=info ts=2019-08-26T02:57:48.797Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.12.0, branch=HEAD, revision=43acd0e2e93f9f70c49b2267efa0124f1e759e86)"
prometheus    | level=info ts=2019-08-26T02:57:48.798Z caller=main.go:330 build_context="(go=go1.12.8, user=root@7a9dbdbe0cc7, date=20190818-13:53:16)"
prometheus    | level=info ts=2019-08-26T02:57:48.798Z caller=main.go:331 host_details="(Linux 5.0.0-13-generic #14-Ubuntu SMP Mon Apr 15 14:59:14 UTC 2019 x86_64 d870070a8ec7 (none))"
prometheus    | level=info ts=2019-08-26T02:57:48.799Z caller=main.go:332 fd_limits="(soft=1048576, hard=1048576)"
prometheus    | level=info ts=2019-08-26T02:57:48.799Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)"
prometheus    | level=info ts=2019-08-26T02:57:48.803Z caller=main.go:654 msg="Starting TSDB ..."
prometheus    | level=info ts=2019-08-26T02:57:48.813Z caller=head.go:509 component=tsdb msg="replaying WAL, this may take awhile"
prometheus    | level=info ts=2019-08-26T02:57:48.815Z caller=head.go:557 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
prometheus    | level=info ts=2019-08-26T02:57:48.818Z caller=main.go:669 fs_type=EXT4_SUPER_MAGIC
prometheus    | level=info ts=2019-08-26T02:57:48.819Z caller=main.go:670 msg="TSDB started"
prometheus    | level=info ts=2019-08-26T02:57:48.819Z caller=main.go:740 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
prometheus    | level=info ts=2019-08-26T02:57:48.823Z caller=main.go:768 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
prometheus    | level=info ts=2019-08-26T02:57:48.823Z caller=main.go:623 msg="Server is ready to receive web requests."
prometheus    | level=info ts=2019-08-26T02:57:48.824Z caller=web.go:448 component=web msg="Start listening for connections" address=0.0.0.0:9090

Congratulations!! You have successfully started Prometheus on your host.

Let’s play with Prometheus Dashboard and UI 🙂


How to verify if Prometheus is running correctly or not?

URL: http://65.19.71.11:9090/metrics
Prometheus Metrics Endpoint - Crunchify Tips

How to verify Host or VM’s UP status using Prometheus?

URL: http://65.19.71.11:9090/targets
How to verify Host or VM's UP status using Prometheus?

How to check Prometheus Configuration on Browser?

How to check Prometheus Configuration on Browser

Prometheus Runtime Information

URL: http://65.19.71.11:9090/status
Uptime 2019-08-26 03:07:24.244852166 +0000 UTC
Working Directory /prometheus
Configuration reload Successful
Last successful configuration reload 2019-08-26 03:07:24 +0000 UTC
Head chunks 533
Head time series 533
WAL corruptions 0
Goroutines 36
GOMAXPROCS 1
GOGC  
GODEBUG  
Storage Retention 15d

Prometheus Build Information

Version 2.12.0
Revision 43acd0e2e93f9f70c49b2267efa0124f1e759e86
Branch HEAD
BuildUser root@7a9dbdbe0cc7
BuildDate 20190818-13:53:16
GoVersion go1.12.8

Prometheus Command-Line Flags:

URL: http://65.19.71.11:9090/flags
alertmanager.notification-queue-capacity 10000
alertmanager.timeout 10s
config.file /etc/prometheus/prometheus.yml
log.format logfmt
log.level info
query.lookback-delta 5m
query.max-concurrency 20
query.max-samples 50000000
query.timeout 2m
rules.alert.for-grace-period 10m
rules.alert.for-outage-tolerance 1h
rules.alert.resend-delay 1m
storage.remote.flush-deadline 1m
storage.remote.read-concurrent-limit 10
storage.remote.read-sample-limit 50000000
storage.tsdb.allow-overlapping-blocks false
storage.tsdb.max-block-duration 36h
storage.tsdb.min-block-duration 2h
storage.tsdb.no-lockfile false
storage.tsdb.path /prometheus
storage.tsdb.retention 0s
storage.tsdb.retention.size 0B
storage.tsdb.retention.time 0s
storage.tsdb.wal-compression false
storage.tsdb.wal-segment-size 0B
web.console.libraries console_libraries
web.console.templates consoles
web.cors.origin .*
web.enable-admin-api false
web.enable-lifecycle false
web.external-url  
web.listen-address 0.0.0.0:9090
web.max-connections 512
web.page-title Prometheus Time Series Collection and Processing Server
web.read-timeout 5m
web.route-prefix /

And you are all set. You have successfully installed Prometheus and verified all of it’s features.

The post How to Install and Configure Prometheus on Linux? (Ubuntu and CentOS) appeared first on Crunchify.