Skip to content

How to use Fluentd

This guide describes how to send user definitive logs that uses in Analytics v2 via fluentd.

Fluentd install

Note

Installing td-agent

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
  • Edit the td-agent settings file
$ vim /etc/td-agent/td-agent.conf

Fluentd default settings

  • Log file: The default path of log file is /var/log/td-agent, and the td-agent.log file is located the sub folder.
  • Event: Event in fluentd means record and log.
  • source: enables to receive various input and many plugins are able to use by using @ type.
  • match: It handles the input event that sent via .

Fluentd plugin

  • Check and install the plugin: Install the plugin you need after checking the list of plugins. (The plugins such as tail, forward and stdout are provided in default, but others like forest need to be install separately.)
$ td-agent-gem list  #Check the list of plugin
# Available to install a plugin as the formatted td-agent-gem install [plugin you want]
$ td-agent-gem install fluent-plugin-forest          # Install the fluent-plugin-forest plugin
  • tag: Tag is a value that consists of Event, and the <match> section handles the value matched with tag.
# Usage example for tag
<source>
@type tail
tag ha2union.game.com2us        # Attached this tag to an event that sent using in_tail
path /var/log/sample.log
</source>
# An input event transferred through sources is sent to a match tag mapped with an event tag.
<match ha2union.game.com2us>
@type stdout
<match>

[Input Plugin]

  • in_tail: The in_tail plugin reads data by using the tail command.
<source>
  @type tail
  path /var/log/com2us.log                           # Reading path such as files
  pos_file /var/log/fluent/com2us_posfile.log.pos    # Records the last reading part
  tag ha2union.game.com2us                           # Event tag. Attached the tag to the sending event.
</source>
# pos_file: When re-executing fluentd, handle the file from the last reading part.
  • in_forward: Generally, it is used for receiving the data from other fluentd nodes.
<source>
  @type forward
  port 24224        # Port to send ( default : 24224 )
  bind 0.0.0.0      # Bind address to send ( default : 0.0.0.0(all of address))
</source>

[Output Plugin]

Note

The output plugin defined in the section defines the settings for buffering and flushing as the sub section of the section.

  • output\_stdout: Exports event logs with stdout and mainly uses for debugging.
<match <i></i>>
  @type stdout
</match>
  • output\_forward: Uses when sending event logs to other fluentd nodes.
<match <i></i>>
  @type forward
  <server>
    name com2us.fluentd1
    host [Server1's IP or domain for receiving]
    port 24224
    weight 50                    # Set up the load balancing weight in the <server> section
  </server>
  <server>
    name com2us.fluentd2
    host [Server2's IP or domain for receiving]
    port 24224
    weight 50                   # Set up the load balancing weight in the <server> section
  </server>
</match>
  • fluent-plugin-forest: Enables to save the log files using a tag name. (By setting as below, logs can be saved in local.)
<store>
  @type forest
  subtype file
  <template>
    time_slice_format %Y%m%d%H
    time_slice_wait 10s
    path /com2us/log/td-agent/data/${tag}/${tag}.*.log
    compress gz
    format json
  </template>
</store>

Usage examples of fluentd

Using fluentd enables to send logs with the methods as below. For more details, click the links below.

Edit settings

  • Check the td-agent.conf context
$ td-agent -c /etc/td-agent/td-agent.conf
  • Re-start td-agent
$ service td-agent restart