easiest way to query the logs, find out when a client connected or disconnected,
or view the top ten clients logging into the system over the last six hours?
-The logstash/elasticsearch/kibana stack is designed and built to do just that.
-elasticsearch is a search engine; logstash is commonly used to feed data in,
-and kibana the web interface to query the logs in near real time.
+The elastic stack is designed and built to do just that. elasticsearch is a
+search engine; logstash is commonly used to feed data in, and kibana the web
+interface to query the logs in near real time.
-Installing the ELK stack is beyond the scope of this document, but can be done
+Installing the elastic stack is beyond the scope of this document, but can be done
in a short amount of time by any competent sysadmin. Then comes getting the
logs in.
tab-delimited key-value pairs out. Some additional data is then extracted
from certain key attributes.
- The file will need to be edited at least to set the input method: for
- experimentation the given input (stdin) may be used. If logstash is running on
- the RADIUS server then 'file' input may be appropriate, otherwise a different
- input such as log-courier or logstash-forwarder may be better to get the data
- over the network to the logstash server.
+ The logstash config will need to be edited at least to set the input method:
+ for experimentation the given input (file) may be used. If logstash is running
+ on the RADIUS server itself then this example input may be appropriate,
+ otherwise a different input such as log-courier or filebeat may be better to
+ get the data over the network to logstash.
It would be best to use an input method that can join the multiple lines of
the detail file together and feed them to logstash as a single entry, rather
- than using the logstash multiline filter.
+ than using the logstash multiline codec.
log-courier.conf
kibana4-dashboard.json
- Basic RADIUS dashboard for Kibana4.
+ Basic RADIUS dashboard for Kibana 4 and Kibana 5.
To import the dashboard first create a new index called "radius-*" in
Settings/Indices. Then go to Kibana's Settings page, "Objects" and "Import".
-------------
Install mapping (only needs to be done once):
-$ ./radius-mapping.sh
-Feed a detail file in:
-$ /path/to/logstash -f logstash-radius.conf < acct-detail
+ $ ./radius-mapping.sh
+
+Edit logstash-radius.conf to point to the correct file, then feed a detail file
+in:
+
+ # /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f logstash-radius.conf
See also
following software versions (note that elasticsearch 2.x may not yet
work with this config).
- elasticsearch 1.7.4
- logstash 1.4.5
- kibana 4.1.2
- kibana 3.1.0
+ elasticsearch 5.1.2
+ logstash 5.1.2
+ kibana 5.1.2
+ kibana 4.1.11
Matthew Newton
-January 2016
+January 2017
# logstash configuration to process RADIUS detail files
#
# Matthew Newton
-# January 2016
+# January 2017
+#
+# This config has been tested with logstash version 5.1.2.
#
# RADIUS "detail" files are textual representations of the RADIUS
# packets, and are written to disk by e.g. FreeRADIUS. They look
-# Example input - read data from a file. This can be useful for
-# testing, but usually not so much for live service. For example,
-# to read in a detail file with this input you could use:
+# Example input - read data from a file. For example, to read in a
+# detail file with this input you could use:
+#
+# # /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f logstash-radius.conf
#
-# /opt/logstash/bin/logstash -v -f logstash-radius.conf < detailfile
input {
- stdin {
+ file {
+ path => "/path/to/radius/detail/file"
+
+ # Note when testing that logstash will remember where
+ # it got to and continue from there.
+ start_position => "beginning"
+
+ # Set the type, for below.
type => radiusdetail
+
+ # It is preferable to use a log feeder that can join
+ # multiple lines together, rather than using multiline
+ # here. For an example, see the log-courier
+ # configuration in this directory.
+
+ # If you didn't read the above, go back and read it again.
+
+ # If that is not possible you may be able to use the
+ # following section. Note that if you are using the
+ # "stdin" input, the file is chunked into 16k blobs,
+ # so every 16k a detail record is likely to be chopped
+ # in half. If you are using the "file" input (as in this
+ # example), the blank links between records are not
+ # passed through so the regex here has to be aware of
+ # that. Basically, do multiline as early as possible
+ # in your log feeder client not here and you'll avoid
+ # most issues that are likely to come up.
+
+ codec => multiline {
+ pattern => "^\t"
+ negate => false
+ what => "previous"
+ }
+
+ # If you really want to use the "stdin" input, this
+ # will work better, but be aware of the comments
+ # above.
+
+ #codec => multiline {
+ # pattern => "^[A-Z\t]"
+ # negate => false
+ # what => "next"
+ #}
}
}
# Moving into production will likely need something more reliable.
# There are many input methods, an example here using log-courier
# (which supports client-site multiline processing and does not
-# lose log events if logstash is restarted).
+# lose log events if logstash is restarted). You could also
+# investigate e.g. filebeat from Elastic.
# input {
# courier {
# port => 5140
# transport => "tcp"
+#
+# # Don't set the type here, as it's set in the
+# # log-courier config instead.
+# #type => radiusdetail
# }
# }
if [type] == "radiusdetail" {
- # If you are using a log feeder that can join
- # multiple lines together then that is preferrable
- # to using multiline here, because this can not be
- # used with threaded logstash (i.e. -w<n> at
- # startup).
-
- # In that case you should comment out the following
- # section. For example, see the log-courier
- # configuration configuration in this directory.
-
- multiline {
- pattern => "^[A-Z\t]"
- negate => false
- what => "next"
- }
-
# Pull off the timestamp at the start of the
# detail record. Note there may be additional data
# after it that has been added by the local admin,
# This is the bulk of processing that adds all of
# the RADIUS attributes as elasticsearch fields.
+ # Note issue https://github.com/logstash-plugins/logstash-filter-kv/issues/10
+ # currently means that all spaces will be stripped
+ # from all fields. If this is a problem, adjust the
+ # trim setting.
+
kv {
field_split => "\n"
source => "message"
# possible to make sure all MAC addresses look the
# same, which has obvious benefits.
#
- # https://github.com/mcnewton/elk/blob/master/logstash-filters/sanitize_mac.rb
+ # https://github.com/mcnewton/logstash-filter-sanitize_mac
# sanitize_mac {
# match => {
# "Called-Station-Id_mac" => "Called-Station-Id_mac"
# "Calling-Station-Id_mac" => "Calling-Station-Id_mac"
# }
- # separator => ":"
+ # separator => "-"
# fixcase => "lower"
# }
if ([Acct-Input-Octets]) {
ruby {
- code => "event['Acct-Input-Octets_long'] =
- event['Acct-Input-Octets'].to_i + ( event['Acct-Input-Gigawords'] ? (event['Acct-Input-Gigawords'].to_i * (2**32)) : 0)"
+ code => "event.set('Acct-Input-Octets_long', event.get('Acct-Input-Octets').to_i +
+ (event.get('Acct-Input-Gigawords') ? (event.get('Acct-Input-Gigawords').to_i * (2**32)) : 0))"
}
}
if ([Acct-Output-Octets]) {
ruby {
- code => "event['Acct-Output-Octets_long'] =
- event['Acct-Output-Octets'].to_i + ( event['Acct-Output-Gigawords'] ? (event['Acct-Output-Gigawords'].to_i * (2**32)) : 0)"
+ code => "event.set('Acct-Output-Octets_long', event.get('Acct-Output-Octets').to_i +
+ (event.get('Acct-Output-Gigawords') ? (event.get('Acct-Output-Gigawords').to_i * (2**32)) : 0))"
}
}
-# Output data to the local elasticsearch cluster (called
-# "elasticsearch") using type "detail" in index "radius-DATE".
+# Output data to the local elasticsearch cluster
+# using type "detail" in index "radius-DATE".
output {
if [type] == "radiusdetail" {
elasticsearch {
- host => localhost
- protocol => http
- cluster => elasticsearch
- index_type => "detail"
+ document_type => "detail"
index => "radius-%{+YYYY.MM.dd}"
flush_size => 1000
}
#! /bin/sh
-# Create a template mapping for RADIUS data
+# Create an elasticsearch template mapping for RADIUS data
# Matthew Newton
# April 2015
# This should be run on an elasticsearch node. Alternatively,
# adjust the curl URI below.
+# This version has been tested on elasticsearch 5.1.2
+
# The template will be called "radius", and will apply to all
# indices prefixed with "radius-" that contain data type "detail".
# As not all RADIUS attributes are known to begin with it has the
#
# Acct-Input- or Acct-Output- attributes are numbers;
# Acct-Session-Time is a number;
-# Everything else is a string.
+# Everything else is a keyword, which is a non-analysed string.
# Additionally, the supplied logstash config will try and extract
# MAC addresses, IP addresses and ports from the data. These are
"detail":{
"properties": {
- "@timestamp": { "format": "dateOptionalTime", "type": "date" },
- "@version": { "type" : "string" },
- "message": { "type" : "string" },
- "Acct-Session-Time": { "type" : "long", "doc_values": true },
- "offset": { "type" : "long", "doc_values": true }
+ "@timestamp": { "format" : "date_optional_time", "type" : "date" },
+ "@version": { "type" : "keyword" },
+ "message": { "type" : "text" },
+ "Acct-Session-Time": { "type" : "long" },
+ "offset": { "type" : "long" }
},
"dynamic_templates": [
"match_pattern": "regex",
"match": "^Acct-(Input|Output)-.*$",
"mapping": {
- "type": "long",
- "doc_values": true
+ "type": "long"
}
}
},
{ "ipv4_address": {
"path_match": "*_ip",
"mapping": {
- "type": "ip",
- "doc_values": true
+ "type": "ip"
}
}
},
{ "network_port": {
"path_match": "*_port",
"mapping": {
- "type": "integer",
- "doc_values": true
+ "type": "integer"
}
}
},
{ "long_number": {
"path_match": "*_long",
"mapping": {
- "type": "integer",
- "doc_values": true
+ "type": "long"
}
}
},
{ "no_analyze_strings": {
"match": "*",
"mapping": {
- "type": "string",
- "index": "not_analyzed",
- "doc_values": true
+ "type": "keyword"
}
}
}
projects / freeradius.git / commitdiff