Fluentd
1.0
1.0
  • Introduction
  • Overview
    • Life of a Fluentd event
    • Support
    • FAQ
    • Logo
    • fluent-package v5 vs td-agent v4
  • Installation
    • Before Installation
    • Install fluent-package
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install calyptia-fluentd
      • RPM Package (Red Hat Linux)
      • DEB Package (Debian/Ubuntu)
      • .dmg Package (macOS)
      • .msi Installer (Windows)
    • Install by Ruby Gem
    • Install from Source
    • Post Installation Guide
    • Obsolete Installation
      • Treasure Agent v4 (EOL) Installation
        • Install by RPM Package v4 (Red Hat Linux)
        • Install by DEB Package v4 (Debian/Ubuntu)
        • Install by .dmg Package v4 (macOS)
        • Install by .msi Installer v4 (Windows)
      • Treasure Agent v3 (EOL) Installation
        • Install by RPM Package v3 (Red Hat Linux)
        • Install by DEB Package v3 (Debian/Ubuntu)
        • Install by .dmg Package v3 (macOS)
        • Install by .msi Installer v3 (Windows)
  • Configuration
    • Config File Syntax
    • Config File Syntax (YAML)
    • Routing Examples
    • Config: Common Parameters
    • Config: Parse Section
    • Config: Buffer Section
    • Config: Format Section
    • Config: Extract Section
    • Config: Inject Section
    • Config: Transport Section
    • Config: Storage Section
    • Config: Service Discovery Section
  • Deployment
    • System Configuration
    • Logging
    • Signals
    • RPC
    • High Availability Config
    • Performance Tuning
    • Multi Process Workers
    • Failure Scenarios
    • Plugin Management
    • Trouble Shooting
    • Fluentd UI
    • Linux Capability
    • Command Line Option
    • Source Only Mode
    • Zero-downtime restart
  • Container Deployment
    • Docker Image
    • Docker Logging Driver
    • Docker Compose
    • Kubernetes
  • Monitoring Fluentd
    • Overview
    • Monitoring by Prometheus
    • Monitoring by REST API
  • Input Plugins
    • tail
    • forward
    • udp
    • tcp
    • unix
    • http
    • syslog
    • exec
    • sample
    • monitor_agent
    • windows_eventlog
  • Output Plugins
    • file
    • forward
    • http
    • exec
    • exec_filter
    • secondary_file
    • copy
    • relabel
    • roundrobin
    • stdout
    • null
    • s3
    • kafka
    • elasticsearch
    • opensearch
    • mongo
    • mongo_replset
    • rewrite_tag_filter
    • webhdfs
    • buffer
  • Filter Plugins
    • record_transformer
    • grep
    • parser
    • geoip
    • stdout
  • Parser Plugins
    • regexp
    • apache2
    • apache_error
    • nginx
    • syslog
    • ltsv
    • csv
    • tsv
    • json
    • msgpack
    • multiline
    • none
  • Formatter Plugins
    • out_file
    • json
    • ltsv
    • csv
    • msgpack
    • hash
    • single_value
    • stdout
    • tsv
  • Buffer Plugins
    • memory
    • file
    • file_single
  • Storage Plugins
    • local
  • Service Discovery Plugins
    • static
    • file
    • srv
  • Metrics Plugins
    • local
  • How-to Guides
    • Stream Analytics with Materialize
    • Send Apache Logs to S3
    • Send Apache Logs to Minio
    • Send Apache Logs to Mongodb
    • Send Syslog Data to Graylog
    • Send Syslog Data to InfluxDB
    • Send Syslog Data to Sematext
    • Data Analytics with Treasure Data
    • Data Collection with Hadoop (HDFS)
    • Simple Stream Processing with Fluentd
    • Stream Processing with Norikra
    • Stream Processing with Kinesis
    • Free Alternative To Splunk
    • Email Alerting like Splunk
    • How to Parse Syslog Messages
    • Cloud Data Logging with Raspberry Pi
  • Language Bindings
    • Java
    • Ruby
    • Python
    • Perl
    • PHP
    • Nodejs
    • Scala
  • Plugin Development
    • How to Write Input Plugin
    • How to Write Base Plugin
    • How to Write Buffer Plugin
    • How to Write Filter Plugin
    • How to Write Formatter Plugin
    • How to Write Output Plugin
    • How to Write Parser Plugin
    • How to Write Storage Plugin
    • How to Write Service Discovery Plugin
    • How to Write Tests for Plugin
    • Configuration Parameter Types
    • Upgrade Plugin from v0.12
  • Plugin Helper API
    • Plugin Helper: Child Process
    • Plugin Helper: Compat Parameters
    • Plugin Helper: Event Emitter
    • Plugin Helper: Event Loop
    • Plugin Helper: Extract
    • Plugin Helper: Formatter
    • Plugin Helper: Inject
    • Plugin Helper: Parser
    • Plugin Helper: Record Accessor
    • Plugin Helper: Server
    • Plugin Helper: Socket
    • Plugin Helper: Storage
    • Plugin Helper: Thread
    • Plugin Helper: Timer
    • Plugin Helper: Http Server
    • Plugin Helper: Service Discovery
  • Troubleshooting Guide
  • Appendix
    • Update from v0.12 to v1
    • td-agent v2 vs v3 vs v4
Powered by GitBook
On this page
  • Prerequisites
  • Set Up Sematext
  • Install Elasticsearch Plugin
  • Set Up rsyslogd
  • Store and Search Logs
  • Debugging
  • Conclusion
  • Learn More

Was this helpful?

  1. How-to Guides

Send Syslog Data to Sematext

PreviousSend Syslog Data to InfluxDBNextData Analytics with Treasure Data

Last updated 3 months ago

Was this helpful?

is a tool for managing logs, and considered an alternative to Splunk, but with cheaper and more flexible . In this article, we present an alternative to Splunk by combining Fluentd with the Sematext open Elasticsearch API.

is an open source search engine known for its ease of use. runs and manages Elasticsearch in the cloud. You also have the option to use alongside the dashboards in the Sematext UI.

By combining Fluentd and Sematext's managed Elasticsearch + Kibana you get a scalable, flexible, easy to use log management tool and search engine with an intuitive native web UI. You also get Kibana, if you want to use it. This provides a managed Splunk alternative, for a fraction of the cost.

In this guide, we'll cover the installation, setup, and basic use of this log management solution. This setup was tested on Ubuntu 18.04. If you're not familiar with Fluentd, please learn more about Fluentd first.

Prerequisites

The following software/services are required to be set up correctly:

You can install Fluentd via major packaging systems.

Set Up Sematext

Install Elasticsearch Plugin

Now you'll configure the fluent-package (Fluentd) to interface properly with Elasticsearch. Please edit /etc/fluent/fluentd.conf as shown below:

# Switch to debug if you need to debug
<system>
  log_level debug
</system>

# get logs from syslog
<source>
  @type syslog
  port 42185
  tag syslog
</source>

# get logs from fluent-logger, fluent-cat or other Fluentd instances
<source>
  @type forward
</source>

<match syslog.**>
  @type elasticsearch
  host logsene-receiver.sematext.com
  # for EU
  # host logsene-receiver.eu.sematext.com
  port 443
  scheme https
  index_name <LOGS_TOKEN>
  <buffer>
    @type file
    path /tmp/fluent/es-buffer/es.all.*.buffer
    chunk_limit_size 250k
    flush_interval 50s
    flush_thread_count 1
    retry_limit 5
    retry_wait 60
  </buffer>
</match>

Once everything has been set up and configured, start fluentd:

$ sudo systemctl start fluentd

Set Up rsyslogd

Finally, configure forwarding logs from your rsyslogd to Fluentd. Create /etc/rsyslog.d/90-fluentd.conf, and restart rsyslog. This will forward your local syslog to Fluentd, and Fluentd will forward the logs to Sematext:

*.* @127.0.0.1:42185

Please restart the rsyslog service once the modification is complete:

$ sudo systemctl restart rsyslog

Store and Search Logs

Once Fluentd receives logs from rsyslog and ships them to Sematext, you can view, search and visualize the log data using prebuilt Dashboards, by creating custom Dashboards, or with Kibana.

First of all, open up the Sematext UI and access your App. You'll see prebuilt dashboards with full-text search, filters, and alerts out-of-the-box.

Sematext will automatically figure out hosts, idents, pids, timestamps, and the origin of the logs. In this case the origin is Fluentd.

After you start receiving logs, you can create custom charts, reports, and alerts to fine-tune your own personal use-case.

Debugging

To manually send logs to Sematext, please use the logger command:

$ logger -t test foobar
<filter **>
  @type stdout
</filter>

<match **>
  @type elasticsearch
  host logsene-receiver.sematext.com
  # for EU
  # host logsene-receiver.eu.sematext.com
  port 443
  scheme https
  index_name <LOGS_TOKEN>
  <buffer>
    flush_interval 5s # for testing
  </buffer>
</match>

Conclusion

This how-to guide introduced an alternative SaaS tool to use instead of Splunk. The combination of Fluentd and Sematext, with an open Elasticsearch API and Kibana, gives you tooling you are used to, with the added benefit of not having to manage an Elasticsearch cluster.

You'll get access to storing and searching logs from infrastructure, apps, and software. The example provided in this article has been tested for the current production environments of Sematext.

Learn More

You need to and create an App. Read more in the docs .

If (fluent-plugin-elasticsearch) is not installed yet, please install it manually.

See section how to install fluent-plugin-elasticsearch on your environment.

If you are used to Kibana, you can still use it as well. For more details, read .

When debugging your fluent-package configuration, using will be useful. All the logs including errors can be found at /etc/fluent/fluentd.log.

If this article is incorrect or outdated, or omits critical information, please . is an open-source project under . All components are available under the Apache 2 License.

Fluentd
Sematext
Installation
sign up
here
out_elasticsearch
Plugin Management
Kibana's official manual
filter_stdout
Fluentd Architecture
Fluentd Get Started
Downloading Fluentd
Set up Fluentd with Sematext
let us know
Fluentd
Cloud Native Computing Foundation (CNCF)
Sematext
pricing
Elasticsearch
Sematext
Kibana
Sematext Dashboard
Sematext: Configure Logs
Sematext: Logs Overview