Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconauto-discovery
file-type-iconbeats-iam-policies
file-type-iconelastalert-iam-policies
file-type-iconelastalert-security-group-rules
file-type-iconelastalert
file-type-iconelasticsearch-cluster-backup
file-type-iconelasticsearch-cluster-restore
file-type-iconelasticsearch-cluster
file-type-iconelasticsearch-iam-policies
file-type-iconelasticsearch-security-group-rules
file-type-iconinstall-collectd
file-type-iconinstall-elastalert
file-type-iconinstall-elasticsearch
file-type-iconinstall-filebeat
file-type-iconinstall-kibana
file-type-iconinstall-logstash
file-type-iconkibana-cluster
file-type-iconkibana-security-group-rules
file-type-iconload-balancer-alb-target-group
file-type-iconlogstash-cluster
file-type-iconlogstash-iam-policies
file-type-iconlogstash-security-group-rules
file-type-iconrun-collectd
file-type-iconrun-elastalert
file-type-iconrun-elasticsearch
file-type-iconrun-filebeat
file-type-iconbin
file-type-iconREADME.md
file-type-iconinstall.sh
file-type-iconrun-kibana
file-type-iconrun-logstash
file-type-icontest
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconauto-discovery
file-type-iconbeats-iam-policies
file-type-iconelastalert-iam-policies
file-type-iconelastalert-security-group-rules
file-type-iconelastalert
file-type-iconelasticsearch-cluster-backup
file-type-iconelasticsearch-cluster-restore
file-type-iconelasticsearch-cluster
file-type-iconelasticsearch-iam-policies
file-type-iconelasticsearch-security-group-rules
file-type-iconinstall-collectd
file-type-iconinstall-elastalert
file-type-iconinstall-elasticsearch
file-type-iconinstall-filebeat
file-type-iconinstall-kibana
file-type-iconinstall-logstash
file-type-iconkibana-cluster
file-type-iconkibana-security-group-rules
file-type-iconload-balancer-alb-target-group
file-type-iconlogstash-cluster
file-type-iconlogstash-iam-policies
file-type-iconlogstash-security-group-rules
file-type-iconrun-collectd
file-type-iconrun-elastalert
file-type-iconrun-elasticsearch
file-type-iconrun-filebeat
file-type-iconbin
file-type-iconREADME.md
file-type-iconinstall.sh
file-type-iconrun-kibana
file-type-iconrun-logstash
file-type-icontest
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md
Elasticsearch (self-hosted)

Elasticsearch (self-hosted)

Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.

Code Preview

Preview the Code

mobile file icon

README.md

down

Filebeat Run Script

This folder contains a script for configuring and running Filebeat on an AWS EC2 instance. This script has been tested on the following operating systems:

  • Ubuntu 16.04
  • Ubuntu 18.04
  • CentOS 7
  • Amazon Linux 2

Quick start

This script assumes that you already have Filebeat installed. If you don't, we recommend using the install-filebeat module. As part of the installation process, we recommend that you create a Filebeat YAML config file and install it using the --config-file option of the install-filebeat script.

You may want some of the configs, such as the IPs of the Logstash/Elasticsearch servers, to be filled in dynamically, when the server is booting up. You can do this using the run-filebeat script! Simply leave placeholders in your Filebeat config file like this (see the filebeat-ami folder for a full example):

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - <__PATH__> # /var/log/source.log

#----------------------------- File output --------------------------------
# This is purely for testing purposes, real world usage will most likely send logs to Logstash or Elasticsearch
output.file:
  path: "/var/log"
  filename: destination.log

Now you can fill in those placeholders and start Filebeat by executing the run-filebeat script as follows:

run-filebeat --auto-fill "<__PATH__>=/path/to/source.log"

This will:

  1. Replace all instances of the text <__PATH__> in the Filebeat config file with the path to the log file that Filebeat will read from
  2. Start Filebeat on the local node.

We recommend using the run-filebeat command as part of User Data, so that it executes when the EC2 Instance is first booting.

See the examples folder for fully working sample code.

Command line Arguments

Run run-filebeat --help to see all available arguments.

Usage: run-filebeat [options]

This script can be used to configure and run Filebeat. This script has been tested with Ubuntu 16.04 + 18.04, CentOS 7 and Amazon Linux 2.

Options:

  --config-file				      The path to a YAML config file for Filebeat. Default: /etc/filebeat/filebeat.yml.
  --skip-auto-discovery			Specifies whether this script should skip checking for Logstash nodes. Default: false.
  --auto-fill KEY=VALUE			Search the Filebeat config file for KEY and replace it with VALUE. May be repeated.
  --help				            Show this help text and exit.

Required arguments only if auto discovery is enabled

  --tag					The tag key and value on the instances.
  --port				The port on which the Logstash nodes listen for beats connections. Default: 5044
  --aws-region				The AWS region where the ASG is deployed. Default: us-east-1
  --use-public-ips			Sets whether to get the public or private IPs from the ASG. Default: true
  --schedule				A cron expression specifying the schedule this script will run with. Default: */5 * * * *


Example:

  run-filebeat --auto-fill '<__PATH__>=/var/log/*.log'

Auto Discovery

The run-filebeat script automatically discovers Logstash nodes by using the auto-discovery module. You can disable this feature by setting the --skip-auto-discovery to true which is useful if you have your own in-house auto discovery mechanism.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?