Collectors

Skip to end of metadata
Go to start of metadata

Introduction

LOGalyze uses Collectors to receive or gather data from external sources. Active Collectors fetch data from the source, while Passive Collectors are listening and waiting for pushed data.

Architecture of Collectors

A Collector consists of two parts:

  • DTP: Data Transport Protocol. This part is responsible for getting the data in from the external source. For example: socket DTP can receive data through a network connection.
  • DF: Data Format. This part is responsible to handle the data after it is received by DTP. For example: JSON DF can parse a JSON object.

A DTP and a DF together form a Collector instance. For example:

  • to receive syslog over the network we have to create a Collector by adding a socket DTP and a syslog DF
  • to read a syslog file from the file system we have to create a Collector by adding a file DTP and a syslog DF

Available Collector types

Available DTPs:

  • Socket: handles TCP/UDP network connections
  • File: reads data from local file or pipe
  • JDBC: connects to a database server and fetches data
  • Syslog: for structured syslog (RFC 5424)

Available DFs:

  • Identity: does nothing with data
  • Syslog: parses BSD syslog
  • CSV: parses CSV or delimited text data
  • JSON: parses JSON objects
  • NCSA: parses NCSA format (common, combined)
  • Auditd: parses Linux Audit Subsystem logs
DTP/DF
Identity
Syslog
CSV
JSON
NCSA
Auditd
Socket
Lines of simple text
BSD Syslog
CSV or Delimited data over the network
JSON over the network
-
LAuS logs
File
General text file
Syslog file (messages)
CSV or Delimited file
File with JSON data
Apache and similar access log
 
JDBC
Data from DB
-
Data from DB
-
-
-
Syslog
Structured Syslog
-
-
-
-
-

Configuration

Configuration of Collectors are in the conf/collectors.xml file. See details about configuration files in General Configuration section.

Collector Options

Field
XPath
Description
Default Value
ID
collector@id
UUID of collector. If you create the Collector manually, you have to generate a unique UUID.
 
Name
collector/name
A descriptive Name of collector.
 
Weight
collector/weight
Numeric value. Number of messages the log processor fetches from the collector in every processing cycle.
1
Startup Mode
collector/startup@mode
Collector startup mode. Available values:
  • automatic: collector will start when engine starts
  • manual: collector can be started manually within Admin or API call
  • scheduled: collector will be scheduled
  • disabled: collector is disabled, will not start and can not be started manually
automatic
Tag
collector/tag
Multiple value property. Defined tags (alphanumeric string) will added to the tag field of log message what handled by this collector.
 
DTP type
collector/dtp/type
Type of Data Transport Protocol. Available values:
  • socket
  • file
  • jdbc
 
DTP Paramaters
collector/dtp/parameter
See parameters in the Reference.
 
DF type
collector/df/type
Type of Data Format. Available values:
  • identity
  • syslog
  • csv
  • kv
 
DF Parameters
collector/df/parameter
See parameters in the Reference.
 

Default Configuration

By default, LOGalyze has two collectors. They are started automatically when the system starts and are ready to receive BSD syslog data on ports UDP/TCP 1670. For quick starting you can configure your syslog client to send data to this port.

Collector Name
DTP
DF
Proto
Interface
Port
Syslog UDP
socket
syslog
udp
0.0.0.0
1670
Syslog TCP
socket
syslog
tcp
0.0.0.0
1670

Default configuration in XML format (conf/collectors.xml):

<collector id="eade5440-921b-457b-8c97-4160a48ff82d">
	<name>Syslog UDP</name>
	<weight>1</weight>
	<startup mode="automatic" />
	<dtp>
		<type>socket</type>
		<parameter key="proto" value="udp"/>
		<parameter key="interface" value="0.0.0.0"/>
		<parameter key="port" value="1670"/>
	</dtp>
	<df>
		<type>syslog</type>
	</df>
	<tag>syslog</tag>
</collector>
<collector id="6d054c93-0c5c-43c5-b175-e014ec9e7e7c">
	<name>Syslog TCP</name>
	<weight>1</weight>
	<startup mode="automatic" />
	<dtp>
		<type>socket</type>
		<parameter key="proto" value="tcp"/>
		<parameter key="interface" value="0.0.0.0"/>
		<parameter key="port" value="1670"/>
		<parameter key="max_tcp_connections" value="100"/>
		<parameter key="backlog" value="50"/>
	</dtp>
	<df>
		<type>syslog</type>
	</df>
	<tag>syslog</tag>
</collector>
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.