ElasticSearch Alerting

Think about a solution that keeps track incoming documents at Elasticsearch and when certain context appear system itself detect that and letting people aware such occurrence. If that’s what you need, I can tell, you have visited the correct blog.

First take your attention on prerequisites.

• Elasticsearch environment
• Email sending application – I would prefer on Postfix but you can select your own preference
• python2.7

Please consider to read our previous Articles about Install and Configure Elesticsearch 

This whole goal is being achieved by a program called “ElastAlert” – which is a python based program. To tell you what the program does, it simply executes elasticsearch own DSL queries within a configured interval and if the search result is True, send alerts based on email addresses or other supported medium.

Getting Started:

01. Installing python prerequisites

yum install python-devel libevent-devel python-pip git

02. Clone Elastalert remote git repository and install the python dependacies via ‘pip’ – a python based package manager

git clone https://github.com/Yelp/elastalert.git
cd /elastalert
pip install -r requirements.txt

NOTE that you might find a situation that the program throws exceptional cases while starting up. Don’t worry, its probably a part of unmet dependencies that python is unable to find in your system.  Going through the error will help to find what packages are missing. Once you do, install them as well via “pip install <packageName>”, that will resolve the issue that you unable to start up the program.

03. As far as configuration goes, there are two aspects.

• Main configuration file => a location where we can define our Elasticsearch URL and other basics.

# vim /opt/elastalert/config.yaml

# The Elasticsearch hostname for metadata writeback
es_host: 192.168.0.1

# The Elasticsearch port
es_port: 9200

# When rule match a certain condition, where should the alert go in order to find the SMTP sender
smtp_host: "localhost"
smtp_port: 25
from_addr: '[email protected]'

• Rules configuration files => Inside the download directory you will find a sub directory call “example_rules”. This is where you can find some basics example rules which are in yaml format. For now, I won’t discuss them but at the end of the article, I will share different types of configuration whose got different methodologies to detect events.

04. Starting up a rule. (assume Elastalert parent directory locates in /opt/elastalert/)

python2.7 /opt/elastalert/elastalert.py --verbose --rule/opt/elastalert/exmaple_rules/rule1.yaml

That will start the process of capturing the given occurrence against what condition we mentioned in the rules1.yaml

I already mentioned that “rules” are what define how these alerts are being captured. Further, they have different types of mechanism to detect different methodologies. Below examples are basic types that people often required to detect such incidents.

01. Frequency Type: This type of rules match when there are at least certain number of events with in a given time frame.

• If “http-404” text appears five times within the message flow in last minute..

name: rune-1
type: frequency
index: testindex-*
num_events: 5
timeframe:
  minutes: 1
filter:
- term:
    name: "http-404"
alert:
- "email"
email:
- "[email protected]"

02. Change Type: Monitor a certain field (target-field) within a message flow and it will be a match if that field changes within the subsequent messages. Important thing here is that in-between two message blocks target-field must change with respect to another field – can call “reference-field” – which of course should not change across the target field.

• John user (reference-field) logs-in www.osradar.com in two different countries (target-filed) with in a day.

name: Abnormal behavior about john user
type: change
index: testindex-*
compare_key: countryName
ignore_null: true
query_key: userName
timeframe:
  minutes: 1
filter:
- query:
    query_string:
        query: "siteURL:\"www.osradar.com\""
alert:
- "email"
email:
- "[email protected]"

Now If you note down, both the rules types have email address associated and I know you already guessed that it would be the alert recipient mail id. Yes absolutely. However, one thing is missing in our setup and that would be SMTP agent which suppose to send mails. If you go back and find the main Elastalert configuration file, you even can see there is a line “smtp_host“. This is what instruct Elastalert to use send mails to. Setting up mail server is another task, but don’t worry visit link here where you find my other post which guide you to get the job done.

Now, it time to make your won Alerts by building python ElastAlert. Good Luck!