How to collect Metrics Data using Metricbeat in Elastic Stack

Here in this article we will see how we can setup elasticstack using Ansible playbook and collect the metric data from the linux system using the metricbeat module. Metricbeat module helps in collecting the relevant information related to system like cpu, memory, disk and network statistics wherein its installed. We can further index this data using elasticsearch and visualize using the kibana dashboard portal.

Test Environment

Fedora 36 server

You can find the complete playbooks and roles related to elasticstack in my github repository for your reference. Currently it still being refactored so there might be changes to the playbooks and roles in future.

If you are interested in watching the video. Here is the YouTube video on the same step by step procedure outlined below.

Procedure

Step1: Install and Configure Elasticsearch service using Ansible

Here in this step we are going to prepare an ansible role to install and configure the elasticsearch service. We will be using elasticsearch to index the metrics data that we are going to collect in the future steps.

Here is my linux_install_elasticsearch_server.yml and the corresponding roles – linux_install_elasticsearch_server directory structure which we will use to install the elasticsearch service.

[admin@fedser elasticstack]$ ls -ltr linux_install_elasticsearch_server.yml 
-rw-r--r--. 1 admin admin 146 Jul 22 14:08 linux_install_elasticsearch_server.yml
[admin@fedser elasticstack]$ tree roles/linux_install_elasticsearch_server/
roles/linux_install_elasticsearch_server/
├── files
│   └── elasticsearch.repo
├── tasks
│   └── main.yml
├── templates
│   └── elasticsearch.yml.j2
└── vars
    └── main.yml 

This is the playbook from which we are going to call our role named – linux_install_elasticsearch_server.

[admin@fedser elasticstack]$ cat linux_install_elasticsearch_server.yml 
---
- name: elasticsearch management
  hosts: sandbox
  become: true
  become_user: root
  roles:
  - {role: 'linux_install_elasticsearch_server'}

We have kept the elasticsearch repository details from where we can download the package in the files section of the role as shown below.

[admin@fedser elasticstack]$ cat roles/linux_install_elasticsearch_server/files/elasticsearch.repo 
[elasticsearch]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

We will using the below reference variables in our tasks. Please update the hostname and port as per your requirements on your server.

[admin@fedser elasticstack]$ cat roles/linux_install_elasticsearch_server/vars/main.yml 
---
elasticsearch_gpg_key: "https://artifacts.elastic.co/GPG-KEY-elasticsearch"
elasticsearch_port: 9200
elasticsearch_host: fedser36.stack.com
#elasticsearch_ca_cert: /etc/elasticsearch/certs/http_ca.crt
elasticsearch_url: https://fedser36.stack.com:9200

Below is the configuration file elasticsearch.yml which is the core file for elasticsearch service. Here we are updating the network.host to FQDN so that we can access the elasticsearch service using FQDN rather than the default localhost.

[admin@fedser elasticstack]$ cat roles/linux_install_elasticsearch_server/templates/elasticsearch.yml.j2 | grep -v ^#
...
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: "{{ elasticsearch_host }}"


xpack.security.enabled: true

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["fedser36.stack.com"]

http.host: 0.0.0.0
...

Here is the main.yml for our elasticsearch installation which is going to import gpg keys, setup repostory, install package, update the elasticsearch configuration, update firewall to allow elasticsearch port access, reload daemon and start the elastic search service. We are also validating the elasticsearch service url using the basic authentication for which we need to capture the default password when we install elasticsearch for the first time.

[admin@fedser elasticstack]$ cat roles/linux_install_elasticsearch_server/tasks/main.yml 
- name: import elasticsearch gpg key
  shell: rpm --import "{{ elasticsearch_gpg_key }}"

- name: configure elasticsearch repository
  copy:
    src: "elasticsearch.repo"
    dest: /etc/yum/repos.d/

- name: enable elasticsearch repository
  shell: dnf config-manager --set-enabled elasticsearch

- name: ensure elasticsearch installed
  dnf:
    name: elasticsearch
    state: present

- name: copy the elasticsearch config
  template:
    src: "elasticsearch.yml.j2"
    dest: "/etc/elasticsearch/elasticsearch.yml"
    owner: root
    group: elasticsearch
    mode: '0660'

- name: reload systemd daemon
  systemd:
    daemon_reload: yes

- name: ensure elasticsearch service is running
  service:
    name: elasticsearch
    state: started

- name: allow elasticsearch port
  firewalld:
    port: "{{ elasticsearch_port }}/tcp"
    permanent: yes
    state: enabled
    immediate: true

- name: validate elasticsearch service
  uri:
    url: "{{ elasticsearch_url }}"
    user: elastic
    password: ifcyGR*4EKlz9*XWj*Fv
    method: GET
    #ca_path: "{{ elasticsearch_ca_cert }}"
    validate_certs: no
  register: escheck_output

- name: print elasticsearch check output
  debug: msg="{{ escheck_output.json }}"

Here is the output of the linux_install_elasticsearch_server.yml playbook execution.

[admin@fedser elasticstack]$ ansible-playbook linux_install_elasticsearch_server.yml -K
BECOME password: 

PLAY [elasticsearch management] **************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : import elasticsearch gpg key] *********************************************************************************
[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'.  If you need to use command because yum, dnf or zypper is insufficient
you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
changed: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : configure elasticsearch repository] ***************************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : enable elasticsearch repository] ******************************************************************************
[WARNING]: Consider using the dnf module rather than running 'dnf'.  If you need to use command because dnf is insufficient you can add 'warn: false' to
this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
changed: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : ensure elasticsearch installed] *******************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : copy the elasticsearch config] ********************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : reload systemd daemon] ****************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : ensure elasticsearch service is running] **********************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : allow elasticsearch port] *************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : validate elasticsearch service] *******************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_elasticsearch_server : print elasticsearch check output] *****************************************************************************
ok: [fedser36.stack.com] => {
    "msg": {
        "cluster_name": "elasticsearch",
        "cluster_uuid": "kVXWWF4WQtGRhljA7EVYaQ",
        "name": "fedser36.stack.com",
        "tagline": "You Know, for Search",
        "version": {
            "build_date": "2022-07-06T15:15:15.901688194Z",
            "build_hash": "8b0b1f23fbebecc3c88e4464319dea8989f374fd",
            "build_snapshot": false,
            "build_type": "rpm",
            "lucene_version": "9.2.0",
            "minimum_index_compatibility_version": "7.0.0",
            "minimum_wire_compatibility_version": "7.17.0",
            "number": "8.3.2"
        }
    }
}

PLAY RECAP ***********************************************************************************************************************************************
fedser36.stack.com         : ok=11   changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Step2: Install and Configure Kibana service using Ansible

Here in this step we are going to prepare an ansible role to install and configure the kibana service. We will be using kibana to discover the indexed metrics data and visualize it using the kibana dashboard.

Here is the playbook and the corresponding role directory structure for kibana installation.

[admin@fedser elasticstack]$ ls -ltr linux_install_kibana_server.yml 
-rw-r--r--. 1 admin admin 132 Jul 22 14:06 linux_install_kibana_server.yml

[admin@fedser elasticstack]$ tree roles/linux_install_kibana_server/
roles/linux_install_kibana_server/
├── files
│   └── kibana.repo
├── tasks
│   └── main.yml
├── templates
│   └── kibana.yml.j2
└── vars
    └── main.yml

We will the following playbook – linux_install_kibana_server which will execute the role – linux_install_kibana_server as shown below.

[admin@fedser elasticstack]$ cat linux_install_kibana_server.yml 
---
- name: kibana management
  hosts: sandbox
  become: true
  become_user: root
  roles:
  - {role: 'linux_install_kibana_server'}

We are setting up the kibana.repo file with the details from where we are going to download and install the kibana package under the files section of the role.

[admin@fedser elasticstack]$ cat roles/linux_install_kibana_server/files/kibana.repo 
[kibana-8.x]
name=Kibana repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

In vars section of the role we define some custom variables which we will be using in the tasks section of the role.

[admin@fedser elasticstack]$ cat roles/linux_install_kibana_server/vars/main.yml 
---
elasticsearch_gpg_key: "https://artifacts.elastic.co/GPG-KEY-elasticsearch"
kibana_port: 5601
kibana_host: fedser36.stack.com

Here is the kibana.yml configuration file which is the core file for the kibana service. In this we are going to update just the server.host with the FQDN of the server so that we can access it from the FQDN.

[admin@fedser elasticstack]$ cat roles/linux_install_kibana_server/templates/kibana.yml.j2 | grep -v ^#
...
server.host: "{{ kibana_host }}"
...

Here is our main.yml for the task where we do the actual installation of the kibana service by setting up the repository, updating the configuration, updating the firewall settings and reloading the daemon and starting the kibana service.

[admin@fedser elasticstack]$ cat roles/linux_install_kibana_server/tasks/main.yml 
- name: import elasticsearch gpg key
  shell: rpm --import "{{ elasticsearch_gpg_key }}"

- name: configure kibana repository
  copy:
    src: "kibana.repo"
    dest: /etc/yum/repos.d/

- name: enable elasticsearch repository
  shell: dnf config-manager --set-enabled kibana-8.x

- name: ensure kibana installed
  dnf:
    name: kibana
    state: present
  register: kibana_output

- name: ensure elasticsearch service is running
  service:
    name: elasticsearch
    state: started

- name: generate enrollment token for kibana
  shell: /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
  register: kibana_token

- name: print kibana enrollment token
  debug: msg="{{ kibana_token.stdout }}"

- name: enroll kibana with elasticsearch
  shell: /usr/share/kibana/bin/kibana-setup --enrollment-token {{ kibana_token.stdout }}
  register: kibana_enrollment

- name: print kibana enrollment output
  debug: msg="{{ kibana_enrollment }}"

- name: reload systemd daemon
  systemd:
    daemon_reload: yes

- name: copy the kibana config
  template:
    src: "kibana.yml.j2"
    dest: "/etc/kibana/kibana.yml"
    owner: root
    group: kibana
    mode: '0660'

- name: ensure kibana service is running
  service:
    name: kibana
    state: started

- name: print kibana installation output
  debug: msg="{{ kibana_output }}"

- name: allow kibana port
  firewalld:
    port: "{{ kibana_port }}/tcp"
    permanent: yes
    state: enabled
    immediate: true

Here is the output of the execution of playbook linux_install_kibana_server as shown below.

[admin@fedser elasticstack]$ ansible-playbook linux_install_kibana_server.yml -K
BECOME password: 

PLAY [kibana management] *********************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_kibana_server : import elasticsearch gpg key] ****************************************************************************************
[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'.  If you need to use command because yum, dnf or zypper is insufficient
you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : configure kibana repository] *****************************************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : enable elasticsearch repository] *************************************************************************************
[WARNING]: Consider using the dnf module rather than running 'dnf'.  If you need to use command because dnf is insufficient you can add 'warn: false' to
this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : ensure kibana installed] *********************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_kibana_server : ensure elasticsearch service is running] *****************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_kibana_server : generate enrollment token for kibana] ********************************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : print kibana enrollment token] ***************************************************************************************
ok: [fedser36.stack.com] => {
    "msg": "eyJ2ZXIiOiI4LjMuMiIsImFkciI6WyIxOTIuMTY4LjEyMi41MDo5MjAwIl0sImZnciI6IjkwMTk4MWJmNDE0MjI3NGU0MmNmN2JhMzNiZTUwMmQyNDRjNmY0ZDE0NTVkMmYxNDFiMjE1ZDBmMjI5ZDAzMDkiLCJrZXkiOiJrVnlaS1lJQl80SFdVUm9hdmx5QjpjMGxORGtOb1R6T1U2bThjblg5SjR3In0="
}

TASK [linux_install_kibana_server : enroll kibana with elasticsearch] ************************************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : print kibana enrollment output] **************************************************************************************
ok: [fedser36.stack.com] => {
    "msg": {
        "changed": true,
        "cmd": "/usr/share/kibana/bin/kibana-setup --enrollment-token eyJ2ZXIiOiI4LjMuMiIsImFkciI6WyIxOTIuMTY4LjEyMi41MDo5MjAwIl0sImZnciI6IjkwMTk4MWJmNDE0MjI3NGU0MmNmN2JhMzNiZTUwMmQyNDRjNmY0ZDE0NTVkMmYxNDFiMjE1ZDBmMjI5ZDAzMDkiLCJrZXkiOiJrVnlaS1lJQl80SFdVUm9hdmx5QjpjMGxORGtOb1R6T1U2bThjblg5SjR3In0=",
        "delta": "0:00:03.652761",
        "end": "2022-07-23 11:18:42.637122",
        "failed": false,
        "rc": 0,
        "start": "2022-07-23 11:18:38.984361",
        "stderr": "- Configuring Kibana...\n✔ Kibana configured successfully.",
        "stderr_lines": [
            "- Configuring Kibana...",
            "✔ Kibana configured successfully."
        ],
        "stdout": "\n\nTo start Kibana run:\n  bin/kibana",
        "stdout_lines": [
            "",
            "",
            "To start Kibana run:",
            "  bin/kibana"
        ]
    }
}

TASK [linux_install_kibana_server : reload systemd daemon] ***********************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_kibana_server : copy the kibana config] **********************************************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : ensure kibana service is running] ************************************************************************************
changed: [fedser36.stack.com]

TASK [linux_install_kibana_server : print kibana installation output] ************************************************************************************
ok: [fedser36.stack.com] => {
    "msg": {
        "changed": false,
        "failed": false,
        "msg": "Nothing to do",
        "rc": 0,
        "results": []
    }
}

TASK [linux_install_kibana_server : allow kibana port] ***************************************************************************************************
ok: [fedser36.stack.com]

PLAY RECAP ***********************************************************************************************************************************************
fedser36.stack.com         : ok=15   changed=7    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Step3: Install and Configure Metricbeat service using Ansible

Here in this step we are going to install Metricbeat service using the ansible playbook and the corresponding role as shown below.

[admin@fedser elasticstack]$ cat linux_install_metric_beat.yml 
---
- name: metricbeat management
  hosts: sandbox
  become: true
  become_user: root
  roles:
  - {role: 'linux_install_metric_beat'}

[admin@fedser elasticstack]$ cat roles/linux_install_metric_beat/tasks/main.yml 
- name: ensure metricbeat installed
  dnf:
    name: https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.3.2-x86_64.rpm
    state: present

This playbook makes sure we have the metricbeat module installed on the server as shown below.

[admin@fedser elasticstack]$ ansible-playbook linux_install_metric_beat.yml -K
BECOME password: 

PLAY [metricbeat management] *****************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_install_metric_beat : ensure metricbeat installed] *******************************************************************************************
ok: [fedser36.stack.com]

PLAY RECAP ***********************************************************************************************************************************************
fedser36.stack.com         : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Once the metricbeat module is installed we need to update the metricbeat configuration with the elasticsearch and kibana service details which we are going to carry out using this playbook –
linux_metricbeat_config_update.

[admin@fedser elasticstack]$ cat linux_metricbeat_config_update.yml 
---
- name: metricbeat management
  hosts: sandbox
  become: true
  become_user: root
  roles:
#  - {role: 'linux_metricbeat_config_backup'}
#  - {role: 'linux_elasticsearch_start_server'}
#  - {role: 'linux_kibana_start_server'}
  - {role: 'linux_metricbeat_config_update'}
  - {role: 'linux_metricbeat_start_server'}

We will update the metricbeat.yml configuration file and enable the linux metrics module as shown below. We need to ensure that elasticsearch and kibana service are up and running before we go ahead and start the metricbeat service.

[admin@fedser elasticstack]$ cat roles/linux_metricbeat_config_update/tasks/main.yml 
- name: configure elasticsearch
  template:
    src: "metricbeat.yml.j2"
    dest: /etc/metricbeat/metricbeat.yml
    owner: root
    group: root
    mode: '0600'

- name: enable linux metrics module
  shell: metricbeat modules enable linux

- name: ensure kibana is running
  service:
    name: kibana
    state: started

- name: setup metribeat assets
  shell: metricbeat setup -e

Retrieve the SHA256 fingerprint for the elasticsearch CA certificate as shown below. This is required if you have the elasticsearch configured with a self signed certificate. We will need this details to configure the metricbeat elasticsearch configuration as shown below.

[admin@fedser36 ~]$ sudo openssl x509 -fingerprint -sha256 -in /etc/elasticsearch/certs/http_ca.crt | grep "sha256" | awk -F"=" '{print $2}' | sed -e 's/://g'
901981BF4142274E42CF7BA33BE502D244C6F4D1455D2F141B215D0F229D0309

[admin@fedser elasticstack]$ cat roles/linux_metricbeat_config_update/templates/metricbeat.yml.j2 | grep -v ^#
...
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"
  host: "fedser36.stack.com:5601" 
  username: "elastic"  
  password: "ifcyGR*4EKlz9*XWj*Fv"
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:
...
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["fedser36.stack.com:9200"]

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "ifcyGR*4EKlz9*XWj*Fv"
  ssl:
    ca_trusted_fingerprint: "901981BF4142274E42CF7BA33BE502D244C6F4D1455D2F141B215D0F229D0309"

Now, we are going to use the following playbook linux_metricbeat_start_server to start the metricbeat service.

[admin@fedser elasticstack]$ cat linux_metricbeat_start_server.yml 
---
- name: metricbeat management
  hosts: sandbox
  become: true
  become_user: root
  roles:
  - {role: 'linux_metricbeat_start_server'}

[admin@fedser elasticstack]$ cat roles/linux_metricbeat_start_server/tasks/main.yml 
- name: ensure metricbeat service is running
  service:
    name: metricbeat
    state: started

Here is the output from our linux_metricbeat_config_update.yml playbook which updates the configuration, load the metrics module and carries metricbeat assets setup and start up the metricbeat service.

[admin@fedser elasticstack]$ ansible-playbook linux_metricbeat_config_update.yml -K
BECOME password: 
...
TASK [linux_metricbeat_config_update : configure elasticsearch] ******************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_metricbeat_config_update : enable linux metrics module] **************************************************************************************
changed: [fedser36.stack.com]

TASK [linux_metricbeat_config_update : ensure kibana is running] *****************************************************************************************
ok: [fedser36.stack.com]

TASK [linux_metricbeat_config_update : setup metribeat assets] *******************************************************************************************
changed: [fedser36.stack.com]

TASK [linux_metricbeat_start_server : ensure metricbeat service is running] ******************************************************************************
changed: [fedser36.stack.com]

PLAY RECAP ***********************************************************************************************************************************************
fedser36.stack.com         : ok=13   changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Now you can discover you data in kibana portal under the metricbeat-* index as shown below.

Also you can prepare some graphical representation of your data using the kibana dashboard as shown below.

Hope you enjoyed reading this article. Thank you..