Shantanu's Blog

1) Install packetbeat
deb:
sudo apt-get install libpcap0.8
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-6.2.2-amd64.deb
sudo dpkg -i packetbeat-6.2.2-amd64.deb

rpm:
sudo yum install libpcap
curl -L -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-6.2.2-x86_64.rpm
sudo rpm -vi packetbeat-6.2.2-x86_64.rpm

2) Make sure that "query" property in "mysql" section is "text" and not "keyword".

[root@localhost packetbeat]# vi packetbeat.template-es6x.json

        "mysql": {
          "properties": {
            "affected_rows": {
              "type": "long"
            },
             "query": {
              "type": "text"
            }
          }
        },
        "nfs": {
          "properties": {
            "minor_version": {


3) Change the host, protocol and password in elasticsearch output secion of config file. Enable template overwriting and make sure version 6x will be loaded.

[root@localhost packetbeat]# vi packetbeat.yml

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://944fe807b7525eaf163f502e08a412c5.us-east-1.aws.found.io:9243"]
  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "xxx"

 # Set to false to disable template loading.
  template.enabled: true

  # Template name. By default the template name is packetbeat.
  template.name: "packetbeat"

  # Path to template file
  template.path: "${path.config}/packetbeat.template.json"

  # Overwrite existing template
  template.overwrite: true

  # If set to true, packetbeat checks the Elasticsearch version at connect time, and if it
  # is 2.x, it loads the file specified by the template.versions.2x.path setting. The
  # default is true.
  template.versions.2x.enabled: false

  # If set to true, packetbeat checks the Elasticsearch version at connect time, and if it
  # is 6.x, it loads the file specified by the template.versions.6x.path setting. The
  # default is true.
  template.versions.6x.enabled: true

  # Path to the Elasticsearch 6.x version of the template file.
  template.versions.6x.path: "${path.config}/packetbeat.template-es6x.json"


4) Check the logs that everything is being loaded correctly.

[root@localhost packetbeat]# cat /var/log/packetbeat/packetbeat| more
2018-02-25T11:53:30+05:30 INFO Metrics logging every 30s
2018-02-25T11:53:30+05:30 INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/packetbeat/packetbeat.template-es6x.json
2018-02-25T11:53:30+05:30 INFO Elasticsearch url: https://944fe807b7525eaf163f502e08a412c.us-east-1.aws.found.io:9243
2018-02-25T11:53:30+05:30 INFO Activated elasticsearch as output plugin.
2018-02-25T11:53:30+05:30 INFO Publisher name: localhost.localdomain
2018-02-25T11:53:30+05:30 INFO Flush Interval set to: 1s
2018-02-25T11:53:30+05:30 INFO Max Bulk Size set to: 50
2018-02-25T11:53:30+05:30 INFO Process matching disabled
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: amqp
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: mongodb
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: mysql
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: nfs
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: pgsql
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: thrift
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: cassandra
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: dns
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: http
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: memcache
2018-02-25T11:53:30+05:30 INFO registered protocol plugin: redis
2018-02-25T11:53:30+05:30 INFO packetbeat start running.
2018-02-25T11:53:32+05:30 INFO Connected to Elasticsearch version 6.2.2
2018-02-25T11:53:32+05:30 INFO Trying to load template for client: https://944fe807b7525eaf163f502e08a412c.us-east-1.aws.found.io:9243
2018-02-25T11:53:32+05:30 INFO Existing template will be overwritten, as overwrite is enabled.
2018-02-25T11:53:32+05:30 INFO Detected Elasticsearch 6.x. Automatically selecting the 6.x version of the template
2018-02-25T11:53:33+05:30 INFO Elasticsearch template with name 'packetbeat' loaded

_____

Or use docker image:

[root@localhost ~]# docker run --cap-add=NET_ADMIN --network=host -e HOST="https://944fe807b7525eaf163f502e08a412c5.us-east-1.aws.found.io:9243" -e PASS="rzmYYJUdHVaglRejr8XqjIX7" shantanuo/packetbeat-agent

Here are the 5 steps to take backup of elastic index

1) Install s3 plugin
2) Set access and secret key
3) create repo
4) Take backup
5) Test if backup was successful

cd /home/elasticsearch/elasticsearch/bin/

# sh elasticsearch-plugin install repository-s3

# sh elasticsearch-keystore create  s3.client.default.access_key
Created elasticsearch keystore in /home/elasticsearch/elasticsearch/config

# sh elasticsearch-keystore add  s3.client.default.access_key
Enter value for s3.client.default.access_key:

# sh elasticsearch-keystore add s3.client.default.secret_key
Enter value for s3.client.default.secret_key:

# curl -XPUT "http://localhost:9200/_snapshot/my_s3_repository1" -H'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "todel162"
  }
}'

# curl -XPUT "http://localhost:9200/_snapshot/my_s3_repository1/snap2" -H'Content-Type: application/json' -d'
{
   "indices": "products, index_1, index_2", 
   "ignore_unavailable": true,
   "include_global_state": false
}'

{"accepted":true}

#  curl http://localhost:9200/_cat/snapshots/my_s3_repository?v

id     status start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
snap1 SUCCESS 1518343272  10:01:12   1518343272 10:01:12     52ms       0                 0             0            0
snap2 SUCCESS 1518343818  10:10:18   1518343818 10:10:18     64ms       0                 0             0            0