Ansible: How to automatically monitor newly created Amazon EC2 instances using Prometheus?

Ansible Get list of IPs and Update Prometheus Config Dynamically

Do you have any of below questions?

Use file-based service discovery to discover scrape targets
How to Automatically discover the use of multiple IPs?
Automatically monitoring EC2 Instances
Monitoring with Prometheus
Using Prometheus to Monitor newly created Amazon EC2 instances?
How to dynamically update Prometheus Targets IP List file after creating Amazon EC2 VMs?

Ansible is no doubt the best operations Automation tool out there in industry. On Crunchify, we have published more than dozen Ansible articles on numerous topics.

In this tutorial we will go over how to update Prometheus Targets IP List file after creating Amazon EC2 VMs Dynamically.

Consider this scenario:

You are running Prometheus for monitoring your production infrastructure.
- Follow Prometheus setup tutorial in details.
Using ansible spawn 5 new Amazon EC2 VMs
- Follow spawning new Amazon EC2 instance tutorial completely.
Get 5 IPs and update [crunchify] group at runtime
- Follow ansible hosts file update group tutorial completely.
Update crunchify_prometheus.txt file with updated IP
Push file to new host where you have prometheus running
Prometheus will read this new updated IP file dynamically
You will have all new hosts monitored automatically

For steps 4 to 7 we will go over all steps here.

How to update Prometheus Targets IP List file after creating Amazon EC2 VMs Dynamically

Detailed steps:

Get list of all IPs from ansible hosts file under group [Crunchify].
Add all IPs to file crunchify.txt just for testing purpose. We are not going to use this file in next steps.
Delete existing crunchify_prometheus.json file.
Create new crunchify_prometheus.json file with sample new prometheus IP list file.
Replace iplist from file crunchify_prometheus.json with list of IPs.
Replace u' with '.
Replace ' with ".
Delete existing crunchify_prometheus.json file from remote host where Prometheus process is running.
Copy file crunchify_prometheus.json from local host to remote host so prometheus can consume it.

crunchify_prometheus_file_update.yml file.

---
- name: Update Prometheus Targets IP List file after creating Amazon EC2 VMs Dynamically
  hosts: local
  connection: local
  gather_facts: True

  tasks:
    - name: Get list of all IPs from group Crunchify
      set_fact: nodelist=""

    - name: Add all IPs to file crunchify.txt just for testing purpose. We are not going to use this file in next steps.
      lineinfile:
         path: "crunchify.txt"
         line: ""

    - name: Delete existing crunchify_prometheus.json file
      ignore_errors: yes
      shell: rm crunchify_prometheus.json

    - name: Create new crunchify_prometheus.json file with sample new prometheus IP list file.
      copy:
        dest: "crunchify_prometheus.json"
        content: [{"targets":iplist,"labels":{"job":"crunchify_prometheus_job"}}]

    - name: Replace iplist from file crunchify_prometheus.json with list of IPs
      replace:
         path: "crunchify_prometheus.json"
         regexp: '"iplist"'
         replace: ""

    - name: Replace u' with '
      replace:
        path: "crunchify_prometheus.json"
        regexp: 'u''
        replace: '''

    - name: Replace ' with "
      replace:
        path: "crunchify_prometheus.json"
        regexp: "'"
        replace: "\""

- hosts: linode
  become: yes

  tasks:
    - name: Delete existing crunchify_prometheus.json file from remote host where Prometheus process is running.
      ignore_errors: yes
      command: rm //cdn.crunchify.com/root/crunchify/crunchify_prometheus.json
          
    - name: Copy file crunchify_prometheus.json from local host to remote host so prometheus can consume it.
      copy: src= dest=
      with_items:
        - { src: '//cdn.crunchify.com/Users/crunchify/Documents/ansible/crunchify_prometheus.json', dest: '/root/crunchify/' }

Execute Ansible Playbook:

ansible-playbook -v -i ./hosts crunchify_prometheus_file_update.yml

Ansible Playbook result:

bash Crunchify$ ansible-playbook -v -i ./hosts crunchify_prometheus_file_update.yml

WARNING: Executing a script that is loading libcrypto in an unsafe way. This will fail in a future version of macOS. Set the LIBRESSL_REDIRECT_STUB_ABORT=1 in the environment to force this into an error.
No config file found; using defaults
/Users/crunchify/Documents/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/Users/crunchify/Documents/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected

PLAY [Update Prometheus Targets IP List file after creating Amazon EC2 VMs Dynamically] ************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************
ok: [localhost]

TASK [Get list of all IPs from group Crunchify] *******************************************************************************************************************
ok: [localhost] => {"ansible_facts": {"nodelist": ["5.17.14.87", "8.17.24.26", "96.17.67.11", "62.17.64.87"]}, "changed": false}

TASK [Add all IPs to file crunchify.txt just for testing purpose. We are not going to use this file in next steps.] ********************************************
ok: [localhost] => {"backup": "", "changed": false, "msg": ""}

TASK [Delete existing crunchify_prometheus.json file] **********************************************************************************************************
 [WARNING]: Consider using the file module with state=absent rather than running 'rm'.  If you need to use command because file is insufficient you can add
'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.

changed: [localhost] => {"changed": true, "cmd": "rm crunchify_prometheus.json", "delta": "0:00:00.004614", "end": "2019-08-31 19:05:28.214368", "rc": 0, "start": "2019-08-31 19:05:28.209754", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Create new crunchify_prometheus.json file with sample new prometheus IP list file.] **********************************************************************
changed: [localhost] => {"changed": true, "checksum": "0f561a2aedb23d9b6e76a93af377cf0128c9f4c6", "dest": "./crunchify_prometheus.json", "gid": 20, "group": "staff", "md5sum": "6a8d4a45c6b38bd6e664ce1cdc7f001a", "mode": "0644", "owner": "Crunchify", "size": 70, "src": "/Users/crunchify/.ansible/tmp/ansible-tmp-1567296328.29-229442409470865/source", "state": "file", "uid": 502}

TASK [Replace iplist from file crunchify_prometheus.json with list of IPs] *************************************************************************************
changed: [localhost] => {"changed": true, "msg": "1 replacements made"}

TASK [Replace u' with '] ***************************************************************************************************************************************
ok: [localhost] => {"changed": false, "msg": ""}

TASK [Replace ' with "] ****************************************************************************************************************************************
changed: [localhost] => {"changed": true, "msg": "8 replacements made"}

PLAY [linode] **************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************
ok: [11.33.64.98]

TASK [Delete existing crunchify_prometheus.json file from remote host where Prometheus process is running.] ****************************************************
changed: [11.33.64.98] => {"changed": true, "cmd": ["rm", "//cdn.crunchify.com/root/crunchify/crunchify_prometheus.json"], "delta": "0:00:00.002704", "end": "2019-09-01 00:05:32.687400", "rc": 0, "start": "2019-09-01 00:05:32.684696", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [Copy file crunchify_prometheus.json from local host to remote host so prometheus can consume it.] ********************************************************
changed: [11.33.64.98] => (item={u'dest': u'/root/crunchify/', u'src': u'//cdn.crunchify.com/Users/crunchify/Documents/ansible/crunchify_prometheus.json'}) => {"changed": true, "checksum": "ecea3eafaf9ba3e66698e0851d2ee2513d9fcd5f", "dest": "//cdn.crunchify.com/root/crunchify/crunchify_prometheus.json", "gid": 0, "group": "root", "item": {"dest": "/root/crunchify/", "src": "//cdn.crunchify.com/Users/crunchify/Documents/ansible/crunchify_prometheus.json"}, "md5sum": "8b8f073e30b180c63341cd3fd4c47974", "mode": "0644", "owner": "root", "size": 122, "src": "/root/.ansible/tmp/ansible-tmp-1567296332.67-196855844989682/source", "state": "file", "uid": 0}

PLAY RECAP *****************************************************************************************************************************************************
11.33.64.98              : ok=3    changed=2    unreachable=0    failed=0   
localhost                  : ok=8    changed=4    unreachable=0    failed=0

That’s it. Congratulations. You have successfully copied new updated file to new host and Prometheus is now automatically monitoring your newly created Amazon EC2 instances.