Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconauto-discovery
file-type-iconbeats-iam-policies
file-type-iconelastalert-iam-policies
file-type-iconelastalert-security-group-rules
file-type-iconelastalert
file-type-iconelasticsearch-cluster-backup
file-type-iconelasticsearch-cluster-restore
file-type-iconelasticsearch-cluster
file-type-iconelasticsearch-iam-policies
file-type-iconelasticsearch-security-group-rules
file-type-iconinstall-collectd
file-type-iconinstall-elastalert
file-type-iconinstall-elasticsearch
file-type-iconinstall-filebeat
file-type-iconinstall-kibana
file-type-iconinstall-logstash
file-type-iconkibana-cluster
file-type-iconkibana-security-group-rules
file-type-iconload-balancer-alb-target-group
file-type-iconlogstash-cluster
file-type-iconlogstash-iam-policies
file-type-iconlogstash-security-group-rules
file-type-iconrun-collectd
file-type-iconrun-elastalert
file-type-iconbin
file-type-iconREADME.md
file-type-iconinstall.sh
file-type-iconrun-elasticsearch
file-type-iconrun-filebeat
file-type-iconrun-kibana
file-type-iconrun-logstash
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md
file-type-iconterraform-cloud-enterprise-private-module-...

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconauto-discovery
file-type-iconbeats-iam-policies
file-type-iconelastalert-iam-policies
file-type-iconelastalert-security-group-rules
file-type-iconelastalert
file-type-iconelasticsearch-cluster-backup
file-type-iconelasticsearch-cluster-restore
file-type-iconelasticsearch-cluster
file-type-iconelasticsearch-iam-policies
file-type-iconelasticsearch-security-group-rules
file-type-iconinstall-collectd
file-type-iconinstall-elastalert
file-type-iconinstall-elasticsearch
file-type-iconinstall-filebeat
file-type-iconinstall-kibana
file-type-iconinstall-logstash
file-type-iconkibana-cluster
file-type-iconkibana-security-group-rules
file-type-iconload-balancer-alb-target-group
file-type-iconlogstash-cluster
file-type-iconlogstash-iam-policies
file-type-iconlogstash-security-group-rules
file-type-iconrun-collectd
file-type-iconrun-elastalert
file-type-iconbin
file-type-iconREADME.md
file-type-iconinstall.sh
file-type-iconrun-elasticsearch
file-type-iconrun-filebeat
file-type-iconrun-kibana
file-type-iconrun-logstash
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md
file-type-iconterraform-cloud-enterprise-private-module-...
Elasticsearch (self-hosted)

Elasticsearch (self-hosted)

Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.

Code Preview

Preview the Code

mobile file icon

README.md

down

Run Elastalert Script

This folder contains a script for configuring and running ElastAlert on an AWS server. This script has been tested on the following operating systems:

  • Ubuntu 16.04
  • Ubuntu 18.04
  • Amazon Linux 2
  • CentOS 7

There is a good chance it will work on other flavors of Debian, CentOS, and RHEL as well.

What is ElastAlert?

ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. Note that it's not an Elasticsearch plugin, just a Python service that connects to the ElasticSearch API. However, there is a separate ElastAlert Kibana Plugin that exposes ElastAlert functionality in Kibana.

The key idea is that you define an Elasticsearch query as part of a rule, which is of some rule type that specifies when that query should trigger an alert, which is of some alert type.

For example, you might use the frequency rule type ("Match where there are at least X events in Y time") to define a rule that triggers an alert of type Slack so that a Slack message appears when the rule is triggered.

Quick start

This script depends on bash-commons, so you must install that project first as documented in its README.

This script also assumes you installed ElastAlert using the install-elastalert module, and are already running a live Elasticsearch cluster (most likely from using the run-elasticsearch module).

  1. Review the config.yaml file in /opt/elastalert/config/config.yaml and make sure the rules_folder property specifies the folder that contains all the ElastAlert rule definitions.

  2. Review the example rules. As a quick start, use the sns alert type by making sure the example rule contains the following and no other alert definitions:

alert:
- "sns"

aws_region: "fill-this-in"
sns_topic_arn: "fill-this-in"
  1. Now start ElastAlert by executing the run-elastalsert script as follows:

    /opt/elastalert/bin/run-elastalert --config /opt/elastalert/config/config.yaml
    

    This will begin querying data in Elasticsearch starting from right this moment.

  2. Finally, send some data into the Elasticsearch cluster and confirm that you've received an email (TODO: How best to send data to cluster?)

Command line Arguments

Run run-elastalsert --help to see all available arguments, or just check out the run-elastalert source code.

Alternatives

Elastalert vs. X-Pack

X-Pack is a full-featured, commercial plugin maintained by elastic.co that includes the following features:

  • Security. Lock down elasticsearch nodes to certain IPs or require authentication via LDAP, Active Directory, or basic authentication.
  • Monitoring. Monitor the components of the Elasticsearch cluster: Elasticsearch, Logstash, and Kibana.
  • Alerting and Notifications. Define a query and a condition under which its results should trigger a notification.
  • Reports. You can download Kibana reports as PDFs optimized for printing.
  • Query Relationships Between Data. Explore how different attributes of a table connecto to each other with visual graph renderings.
  • Machine Learning. Perform "Time Series Anomaly Detection" to automatically alert on unusual changes.

By contrast, Elastalert is limited to alerting and notifications. We chose Elastalert over X-Package because, as an alerting plugin, Elastalert compares favorably to X-Pack, and it's open source. However, if you require other features of X-Pack, you may prefer to use X-Pack for alerting and notifications as well.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?