How to configure fluent-bit with collectd as metrics input source

Here in this article we will try to use collectd which is a daemon used to collect the system and application performance metrics as an input source to fluent-bit and display the metrics through console output using. We will be carrying out this setup and configuration using ansible.

Table of Contents

Test Environment

Fedora 39 workstation
Ansible 2.16.2

Architecture

If you are interested in watching the video. Here is the YouTube video on the same step by step procedure outlined below.

Procedure

Step1: Ensure collectd service running

As a first step we need to ensure that we have our input source up and running. Here we are going to use collectd to capture the system level metrics like cpu, memory, load and network interface statistics and expose it through network port. Please follow “How to install and configure collectd with network plugin using Ansible” to carry out the collectd setup.

Step2: Ensure fluent-bit service running

Once our input source is ready, we need to setup and start our fluent-bit service. Please follow “How to install and configure Fluent Bit using Ansible” to carry of the default setup of fluent-bit.

Step3: Create fluent-bit config

Here in this step we are going to create a new template file “collectd_stdout.conf” for fluent-bit configuration. Here the input source is collectd service and the output destination is console. As per below the configuration is relaxed to listen on any interface on port 25826 for collectd metrics but it can restricted to a specific ip address from where the collectd metrics data is being received.

The output plugin that we are using the stdout which basically prints out all the matching event records (ie. metrics data) to standard output. You can look at the metrics data collected using the journalctl utility.

admin@fedser:fluentbit$ cat roles/linux_configure_fluentbit/templates/collectd_stdout.conf 
[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    parsers_file parsers.conf
    plugins_file plugins.conf
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on

[INPUT]
    name collectd
    listen 0.0.0.0
    port 25826
    typesDB      /usr/share/collectd/types.db

[OUTPUT]
    name  stdout
    match *

Step4: Update fluent-bit config

Once our fluent-bit config is ready with the required input and output plugin configured. We are now good to update our configuration as shown below. As you can see from the playbook command we are passing an extra-vars as an input to pass our customized config file location to update the fluent-bit configuration.

admin@fedser:fluentbit$ ansible-playbook linux_setup_fluentbit.yml -i inventory/hosts --extra-vars "fluentbit_config_template=collectd_stdout.conf" --tags "linux_configure_fluentbit" -v
Using /etc/ansible/ansible.cfg as config file
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

PLAY [fluent-bit] ***********************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [fedres.stack.com]

TASK [linux_configure_fluentbit : upload fluent-bit config] *****************************************************************************************************************
ok: [fedres.stack.com] => {"changed": false, "checksum": "e98bd8d5f53dc4b1683f0cf0fa33d7d05ea976f2", "dest": "/etc/fluent-bit/fluent-bit.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/fluent-bit/fluent-bit.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 369, "state": "file", "uid": 0}

PLAY RECAP ******************************************************************************************************************************************************************
fedres.stack.com           : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Step5: Start fluent-bit service

Now, its time to start our fluent-bit service with the updated configuration.

admin@fedser:fluentbit$ ansible-playbook linux_setup_fluentbit.yml -i inventory/hosts --tags "linux_start_fluentbit" -v

Step6: Validate results

Now you can login to your remote system wherein fluent-bit service is running and try to check the fluent-bit service journal logs. You should be able to see the system level metrics being posted into the journal logs as shown below for cpu, memory, load and network interface.

admin@fedres:~$ journalctl -fu fluent-bit
Journal file /var/log/journal/7a739d8ccf224d768c349563fc2f723f/system@00060d3673e28e40-307d596b210db788.journal~ is truncated, ignoring file.
Dec 24 00:27:32 fedres.stack.com fluent-bit[2126]: [56] collectd.0: [1703357851.334347380, {"type"=>"cpu", "type_instance"=>"steal", "time"=>1703357851.333174, "interval"=>10.000000, "plugin"=>"cpu", "plugin_instance"=>"0", "host"=>"fedres.stack.com", "value"=>28}]
Dec 24 00:27:32 fedres.stack.com fluent-bit[2126]: [57] collectd.0: [1703357851.334357009, {"type"=>"cpu", "type_instance"=>"steal", "time"=>1703357851.333175, "interval"=>10.000000, "plugin"=>"cpu", "plugin_instance"=>"1", "host"=>"fedres.stack.com", "value"=>19}]
Dec 24 00:27:32 fedres.stack.com fluent-bit[2126]: [58] collectd.0: [1703357851.334415195, {"type"=>"cpu", "type_instance"=>"steal", "time"=>1703357851.333176, "interval"=>10.000000, "plugin"=>"cpu", "plugin_instance"=>"2", "host"=>"fedres.stack.com", "value"=>16}]
...
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [6] collectd.0: [1703357861.333648328, {"type"=>"load", "type_instance"=>"", "time"=>1703357851.333570, "interval"=>10.000000, "plugin"=>"load", "plugin_instance"=>"", "host"=>"fedres.stack.com", "shortterm"=>0.002441, "midterm"=>0.026367, "longterm"=>0.001953}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [7] collectd.0: [1703357861.333658538, {"type"=>"if_packets", "type_instance"=>"", "time"=>1703357851.333626, "interval"=>10.000000, "plugin"=>"interface", "plugin_instance"=>"lo", "host"=>"fedres.stack.com", "rx"=>97, "tx"=>97}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [8] collectd.0: [1703357861.333667826, {"type"=>"if_octets", "type_instance"=>"", "time"=>1703357851.333629, "interval"=>10.000000, "plugin"=>"interface", "plugin_instance"=>"lo", "host"=>"fedres.stack.com", "rx"=>101045, "tx"=>101045}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [9] collectd.0: [1703357861.333677638, {"type"=>"if_errors", "type_instance"=>"", "time"=>1703357851.333630, "interval"=>10.000000, "plugin"=>"interface", "plugin_instance"=>"lo", "host"=>"fedres.stack.com", "rx"=>0, "tx"=>0}]
...
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [18] collectd.0: [1703357861.333764675, {"type"=>"memory", "type_instance"=>"used", "time"=>1703357851.334208, "interval"=>10.000000, "plugin"=>"memory", "plugin_instance"=>"", "host"=>"fedres.stack.com", "value"=>208609280.000000}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [19] collectd.0: [1703357861.333772584, {"type"=>"memory", "type_instance"=>"buffered", "time"=>1703357851.334208, "interval"=>10.000000, "plugin"=>"memory", "plugin_instance"=>"", "host"=>"fedres.stack.com", "value"=>3383296.000000}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [20] collectd.0: [1703357861.333779300, {"type"=>"memory", "type_instance"=>"free", "time"=>1703357851.334208, "interval"=>10.000000, "plugin"=>"memory", "plugin_instance"=>"", "host"=>"fedres.stack.com", "value"=>3563884544.000000}]
...

Hope you enjoyed reading this article. Thank you..

How to configure fluent-bit with collectd as metrics input source