Logstash - Configure logstash-indexer the right way to avoid high CPU usage

In the previous blog post about setting logstash with ElasticSearch (here), I was output like this:

input {
        redis {
                host => "127.0.0.1"
                data_type => "list"
                key => "logstash"
                codec => json
        }
}
output {
        elasticsearch {
                embedded => true
        }
}

As you can see, what logstash does is only move logs from redis to elasticsearch. There is no way it needs a lot of cpu power as I pointed out in the previous blog post.

What's wrong? After a while searching around Google and digging the logstash documentation, I found out the reason. It is the "embbedded" option in my configuration.


According to the logstash documentation about storing logs to elasticsearch:

embedded

Value type is boolean
Default value is false

Run the elasticsearch server embedded in this process. This option is useful if you want to run a single logstash process that handles log processing and indexing; it saves you from needing to run a separate elasticsearch process.

I'd already run a ElasticSearch instance in the indexer server. So, It caused conflict. The CPU usage of logstash process raised to 100%, and a lot of exceptions in logstash output.


To fix that, I just remove the "embedded" and add the "host" option to elasticsearch output setting:

input {
        redis {
                host => "localhost"
                type => "redis"
                data_type => "list"
                key => "logstash"
                codec => json
        }
}
output {
        stdout {
                debug => true
                debug_format => "json"
        }
        elasticsearch {
             # embedded => true
                host => "localhost"
        }
}


Restart the logstash service in the indexer server and you will see the awesomeness.

References:
[0] http://logstash.net/docs/1.2.2/outputs/elasticsearch#embedded
[1] http://logstash.net/docs/1.2.2/tutorials/getting-started-centralized
[2] https://github.com/goncalopereira/Logstash/blob/master/output.conf