Windows Event Collection

In this article, we explain how to get started with collecting data from Windows machines (This setup has been tested on a 64-bit Windows 8 machine).

As of v10, Fluentd does NOT support Windows. However, there are times when you must collect data streams from Windows machines. For example:

1. Tailing log files on Windows: collect and analyze log data from

   a Windows application.

2. Collecting Windows Event Logs: collect event logs from your

   Windows servers for system analysis, compliance checking, etc.

If you're not familiar with Fluentd, please learn more about Fluentd first.

What is Fluentd?

Prerequisites

1. nxlog, an open source log management tool that

   runs on Windows.

2. A Linux server (we assume Ubuntu 12 for this article)

Setup

Set up a Linux server with rsyslogd and Fluentd

1. Get hold of a Linux server. In this example, we assume it is Ubuntu.

2. **Make sure it has ports open for TCP. In the following example, we

   assume port 5140 is open.**

3. Install td-agent. (See

   here for various ways to install

   Fluentd/Treasure Agent)

4. Edit td-agent's configuration file located at /etc/td-agent/td-agent.conf and add the following lines

    <source>
      @type tcp
      format none
      port 5140
      tag windowslog
    </source>    
    <match windowslog>
      @type stdout
    </match>
   
    The above code listens to port 5140 (UDP) and outputs the data to stdout (which is piped to `/var/log/td-agent/td-agent.log`)

5. Start td-agent by running sudo service td-agent start

Set up nxlog on Windows

1. Follow this link and download a copy of

   nxlog onto the Windows machine you want to collect log data from.

   Open the downloaded installer and follow the instructions. By

   default, it should be installed in C:\Program Files (x86)\nxlog

2. Create an nxlog config file as follows and save it as nxlog.conf:

    #define ROOT C:\Program Files\nxlog
    define ROOT C:\Program Files (x86)\nxlog
   
    Moduledir %ROOT%\modules
    CacheDir %ROOT%\data
    Pidfile %ROOT%\data\nxlog.pid
    SpoolDir %ROOT%\data
    LogFile %ROOT%\data\nxlog.log
   
    <Input in>
      Module im_file
      File 'C:\Users\SomeUser\Desktop\nxlog_test.log' #Put the file to be tailed here.
      SavePos TRUE
      InputType LineBased
    </Input>
   
    <Output out>
      Module om_tcp
      Host LINUX_MACHINE_RUNNING_FLUENTD
      Port 5140
    </Output> 
   
    <Route r>
      Path in => out
    </Route>

   This configuration will send each line of the log file (see the File parameter inside \<Input in>...\</Input>) as a syslog message to a remote Fluentd/Treasure Agent instance.

Test

1. Go to nxlog's directory (in Powershell or Command Prompt) and run the following command:

    \nxlog.exe -f -c  <path to nxlog.conf>

   The "-f" option runs nxlog in the foreground (this is for testing). If this is for production, you would want to turn it into a Windows Service.

2. Once nxlog is running, add a new line "Windows is awesome" into the tailed file like this:

    echo Windows is awesome >> 'C:\Users\SomeUser\Desktop\nxlog_test.log'

3. Now, go to the Linux server and run

    $ sudo tail -f /var/log/td-agent/td-agent.log
    ...
    ...
    ...
    2014-12-20 02:19:36 +0000 windowslog: {"message":"Windows is awesome \r"}

4. You successfully sent data from a Windows machine to a remote Fluentd instance running on Linux.

Parsing JSON Logs

If you are sending JSON logs on Windows to Fluentd, Fluentd can parse them as they come in. To do, simply change Fluentd's configuration as follows. Note the change from format none to format json. (See this article for more details about the parser plugins)

<source>
  @type tcp
  format json
  port 5140
  tag windowslog
</source>    
<match windowslog>
  @type stdout
</match>

Then, if you add a new line to the file on your windows machine like this:

echo {"name":"Sadayuki", "age":27} >> 'C:\Users\SomeUser\Desktop\nxlog_test.log'

On the Linux machine running Fluentd, you see the following line:

2014-12-20 02:22:44 +0000 windowslog: {"name":"Sadayuki","age":27}

Next Step

This example showed that we can collect data from a Windows machine and send it to a remote Fluentd instance. However, the data is not terribly useful because each line of data is placed into the "message" field as unstructured text. For production purposes, you would probably want to write a plugin/extend the syslog plugin so that you can parse the "message" field in the event.

Learn More

• Fluentd Architecture

• Fluentd Get Started

If this article is incorrect or outdated, or omits critical information, please let us know. Fluentd is a open source project under Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.