How to configure fluent-bit with collectd as metrics input source
Here in this article we will try to use collectd which is a daemon used to collect the system and application performance metrics as an input source to fluent-bit and display the metrics through console output using. We will be carrying out this setup and configuration using ansible.
Test Environment
Fedora 39 workstation
Ansible 2.16.2
Architecture
If you are interested in watching the video. Here is the YouTube video on the same step by step procedure outlined below.
Procedure
Step1: Ensure collectd service running
As a first step we need to ensure that we have our input source up and running. Here we are going to use collectd to capture the system level metrics like cpu, memory, load and network interface statistics and expose it through network port. Please follow “How to install and configure collectd with network plugin using Ansible” to carry out the collectd setup.
Step2: Ensure fluent-bit service running
Once our input source is ready, we need to setup and start our fluent-bit service. Please follow “How to install and configure Fluent Bit using Ansible” to carry of the default setup of fluent-bit.
Step3: Create fluent-bit config
Here in this step we are going to create a new template file “collectd_stdout.conf” for fluent-bit configuration. Here the input source is collectd service and the output destination is console. As per below the configuration is relaxed to listen on any interface on port 25826 for collectd metrics but it can restricted to a specific ip address from where the collectd metrics data is being received.
The output plugin that we are using the stdout which basically prints out all the matching event records (ie. metrics data) to standard output. You can look at the metrics data collected using the journalctl utility.
admin@fedser:fluentbit$ cat roles/linux_configure_fluentbit/templates/collectd_stdout.conf
[SERVICE]
flush 1
daemon Off
log_level info
parsers_file parsers.conf
plugins_file plugins.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on
[INPUT]
name collectd
listen 0.0.0.0
port 25826
typesDB /usr/share/collectd/types.db
[OUTPUT]
name stdout
match *
Step4: Update fluent-bit config
Once our fluent-bit config is ready with the required input and output plugin configured. We are now good to update our configuration as shown below. As you can see from the playbook command we are passing an extra-vars as an input to pass our customized config file location to update the fluent-bit configuration.
admin@fedser:fluentbit$ ansible-playbook linux_setup_fluentbit.yml -i inventory/hosts --extra-vars "fluentbit_config_template=collectd_stdout.conf" --tags "linux_configure_fluentbit" -v
Using /etc/ansible/ansible.cfg as config file
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
PLAY [fluent-bit] ***********************************************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [fedres.stack.com]
TASK [linux_configure_fluentbit : upload fluent-bit config] *****************************************************************************************************************
ok: [fedres.stack.com] => {"changed": false, "checksum": "e98bd8d5f53dc4b1683f0cf0fa33d7d05ea976f2", "dest": "/etc/fluent-bit/fluent-bit.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/fluent-bit/fluent-bit.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 369, "state": "file", "uid": 0}
PLAY RECAP ******************************************************************************************************************************************************************
fedres.stack.com : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Step5: Start fluent-bit service
Now, its time to start our fluent-bit service with the updated configuration.
admin@fedser:fluentbit$ ansible-playbook linux_setup_fluentbit.yml -i inventory/hosts --tags "linux_start_fluentbit" -v
Step6: Validate results
Now you can login to your remote system wherein fluent-bit service is running and try to check the fluent-bit service journal logs. You should be able to see the system level metrics being posted into the journal logs as shown below for cpu, memory, load and network interface.
admin@fedres:~$ journalctl -fu fluent-bit
Journal file /var/log/journal/7a739d8ccf224d768c349563fc2f723f/system@00060d3673e28e40-307d596b210db788.journal~ is truncated, ignoring file.
Dec 24 00:27:32 fedres.stack.com fluent-bit[2126]: [56] collectd.0: [1703357851.334347380, {"type"=>"cpu", "type_instance"=>"steal", "time"=>1703357851.333174, "interval"=>10.000000, "plugin"=>"cpu", "plugin_instance"=>"0", "host"=>"fedres.stack.com", "value"=>28}]
Dec 24 00:27:32 fedres.stack.com fluent-bit[2126]: [57] collectd.0: [1703357851.334357009, {"type"=>"cpu", "type_instance"=>"steal", "time"=>1703357851.333175, "interval"=>10.000000, "plugin"=>"cpu", "plugin_instance"=>"1", "host"=>"fedres.stack.com", "value"=>19}]
Dec 24 00:27:32 fedres.stack.com fluent-bit[2126]: [58] collectd.0: [1703357851.334415195, {"type"=>"cpu", "type_instance"=>"steal", "time"=>1703357851.333176, "interval"=>10.000000, "plugin"=>"cpu", "plugin_instance"=>"2", "host"=>"fedres.stack.com", "value"=>16}]
...
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [6] collectd.0: [1703357861.333648328, {"type"=>"load", "type_instance"=>"", "time"=>1703357851.333570, "interval"=>10.000000, "plugin"=>"load", "plugin_instance"=>"", "host"=>"fedres.stack.com", "shortterm"=>0.002441, "midterm"=>0.026367, "longterm"=>0.001953}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [7] collectd.0: [1703357861.333658538, {"type"=>"if_packets", "type_instance"=>"", "time"=>1703357851.333626, "interval"=>10.000000, "plugin"=>"interface", "plugin_instance"=>"lo", "host"=>"fedres.stack.com", "rx"=>97, "tx"=>97}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [8] collectd.0: [1703357861.333667826, {"type"=>"if_octets", "type_instance"=>"", "time"=>1703357851.333629, "interval"=>10.000000, "plugin"=>"interface", "plugin_instance"=>"lo", "host"=>"fedres.stack.com", "rx"=>101045, "tx"=>101045}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [9] collectd.0: [1703357861.333677638, {"type"=>"if_errors", "type_instance"=>"", "time"=>1703357851.333630, "interval"=>10.000000, "plugin"=>"interface", "plugin_instance"=>"lo", "host"=>"fedres.stack.com", "rx"=>0, "tx"=>0}]
...
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [18] collectd.0: [1703357861.333764675, {"type"=>"memory", "type_instance"=>"used", "time"=>1703357851.334208, "interval"=>10.000000, "plugin"=>"memory", "plugin_instance"=>"", "host"=>"fedres.stack.com", "value"=>208609280.000000}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [19] collectd.0: [1703357861.333772584, {"type"=>"memory", "type_instance"=>"buffered", "time"=>1703357851.334208, "interval"=>10.000000, "plugin"=>"memory", "plugin_instance"=>"", "host"=>"fedres.stack.com", "value"=>3383296.000000}]
Dec 24 00:27:42 fedres.stack.com fluent-bit[2126]: [20] collectd.0: [1703357861.333779300, {"type"=>"memory", "type_instance"=>"free", "time"=>1703357851.334208, "interval"=>10.000000, "plugin"=>"memory", "plugin_instance"=>"", "host"=>"fedres.stack.com", "value"=>3563884544.000000}]
...
Hope you enjoyed reading this article. Thank you..
Leave a Reply
You must be logged in to post a comment.