Mark Reddy - Software EngineerMark Reddy - Software Engineer

Elasticsearch and New Relic

Mark — Mon, 06 May 2013 16:07:26 +0000

The installation of the Elasticsearch-NewRelic plugin is quite straightforward. First we need to download and install New Relic's Java Agent, this agent is responsible for sending back all the statistical data to New Relic's servers, for storage, charting, etc. To obtain the latest Java Agent from New Relic we must:

Log in to New Relic.
Go to the Account Settings section.
Download the Java agent from the list of 'latest agents'.

Now that we have the agent, it is just a matter of installing it:

In your app server's root directory, create a new directory named newrelic.
Unpack the downloaded file into the newrelic directory.

To have New Relic start up when Elasticsearch starts we can edit the elasticsearch.in.sh script located in the bin directory and add this line: [code] JAVA_OPTS="$JAVA_OPTS -javaagent:/$ES_HOME/newrelic/newrelic.jar" [/code] Log into your NR account and within a few minutes you will see statistical data on the OS and JVM being populated for your ES instance. Now that we have New Relic running, we can install the Elasticsearch-NewRelic plugin to allow us to monitor ES itself. The plugin can be obtained from here. We need to build the plugin using gradle. I work on a Mac (OSX), so I can run the following command to install gradle: [code] brew install gradle [/code] If you do not have brew installed, please check out my post on how to do so: Installing Homebrew on Mac OSX. To build the plugins Jar via gradle, navigate to the plugins root directory and run: [code] gradle jar [/code] This will build a jar file called elasticsearch-newrelic-master-0.0.1.jar located in [build -> libs] To install the plugin, like all ES plugins we can run: [code] bin/plugin -url file://path/to/elasticsearch-newrelic-master-0.0.1.jar -install newrelic [/code] That should be it for the setup, with the Java agent and plugin installed you can now monitor your ES instance with ease. Now to actually display the metrics collected by the plugin you will need to create a custom dashboard and from there you can create charts and tables. The plugin collects six main categories: indices, pool, network, transport, http and filesystem. The metrics are created using newrelic hierarchical approach, so for example: To see the report on the total number of searches executed you can type the following: indices/search/total or to group all metrics under an individual catagory you can do: indices/search/* For my own ease of use I like to create a custom dashboard for each individual catagory, as so: And that is it, you now have a historical data monitor for your Elasticsearch cluster.

Monitoring Elasticsearch

Mark — Mon, 06 May 2013 15:07:33 +0000

Two of my favourite applications at the moment are Elasticsearch and New Relic. Elasticsearch is a distributed, RESTful, free/open source search server based on Apache Lucene, while New Relic is a software as a service (SaaS) based application performance management (APM) tool that can monitor a wide range of applications, written various different languages. I have been using New Relic in a production environment for some time now, on the other hand I have only recently deployed ES into the same. As ES is the new kid on the block in the search space, there is no real standard/optimal configuration for different scenarios, it really is on each developer to benchmark their own use case and tinker with the config until they are satisfied. Concerned that my configuration of ES may have not been optimal for my use case, I went in search for some monitoring tools to help me evaluate the state of my cluster under production load and diagnose any problems as they arise. Initially I used the likes of Head, BigDesk and Paramedic during the proof of concept (POC) phase of development. These tools are fantastic for real time analysis of an ES cluster, during testing I was able to hop on an monitor every aspect of my cluster. This was great, but I needed a tool that would store historical data spanning weeks/months, something like the functionality that New Relic provides, I bet you know where I am going with this....After digging around I found this Elasticsearch-NewRelic Plugin. This plugin gathers all the metrics I need and sends them back to New Relic via New Relic's java agent. Here is just a small sample for what it can report on: To install New Relic's java agent and the elasticsearch-newrelic plugin, please read this follow up post: Ealsticsearch and New Relic

Running two Elasticsearch instances on one machine

Mark — Sun, 05 May 2013 22:44:38 +0000

Elasticsearch by default runs assuming a one machine, one node setup. However for local dev I needed to boot up more than one node on my machine, to test certain aspects of a cluster and for my own personal exposure. The quickest way I though of was to specify paramaters via the cli when starting a new instance, for example: [code] $ bin/elasticsearch -Des.node.master=true -Des.node.name=Node1 ... $ bin/elasticsearch -Des.node.master=false -Des.node.name=Node2 ... $ bin/elasticsearch -Des.node.master=false -Des.node.name=Node3 ... [/code] However this can get tedious, passing in params for each node started, every time you want to start one. So the most convient answer is to create multiple elasticsearch.yml files (elasticsearch.1.yml, elasticsearch.2.yml etc) and then start each instance from the command line referencing the new config files. [code] $ bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.1.yml $ bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.2.yml $ bin/elasticsearch -Des.config=$ES_HOME/config/elasticsearch.testNoData.yml [/code]

Installing Homebrew on Mac OSX

Mark — Sat, 04 May 2013 23:44:56 +0000

Homebrew is a package management system written by Max Howell that simplifies the installation of software on the Mac OS X operating system. It is a free/open source software project to simplify installation of other free/open source software packages. It is similar in aim and function to MacPorts and Fink. The installation is very straightforward, simply run the

Unix Command: man

Mark — Wed, 01 May 2013 22:23:25 +0000

Recently I have been working more and more frequently in a *nix environment (thats not to say I haven't spent my fair share of time in the terminal). While running a few commands that I 'just know' how to run, I dug into them a bit deeper to find out their true functionality and get more acquainted with them. As I was doing this I thought "This would make for an interesting category for my blog - Unix Commands Expanded". However, then I thought anything I write up, the 'man' command would do a better job than I ever could. So this article will introduce you to the 'man' command and from there navigating a *nix terminal should be a breeze. To use the man command, at the Unix prompt, enter: [code lang="bash"]man topic[/code] Replace topic with the name of the manual item about which you want more information. For example, to find out more about the FTP command, at the Unix prompt, enter: [code lang="bash"]man ftp[/code] If you are unsure which manual item you want to read, you can do a keyword search. At the Unix prompt, enter: [code lang="bash"]man -k keyword | more[/code] On some systems, you need to replace man -k with apropos. For example, at the Unix prompt, enter: [code lang="bash"]apropos keyword | more[/code] In both of the examples above, replace keyword with a specific topic (e.g., ftp, mail). For more information about the man command, at the Unix prompt, enter: [code lang="bash"]man man[/code]

Software Architecture “ilities”

Mark — Sun, 28 Apr 2013 22:26:56 +0000

I recently posted another ‘Define’ article (this is where I expand on an acronym) based on NFR’s in which I mentioned the idea of software ‘ilities’. In software projects we must take into consideration ‘ilities’. Prioritising them is necessary because the client will inevitably demand that the project adheres to all of them (within reason).

Evaluating the Status of a FOSS Project

Mark — Sat, 27 Apr 2013 12:06:33 +0000

Throughout my career I have worked with more open source projects than I care to remember. Some were good, some were great, some were poorly documented, and some were plain fun to use and some required stringent security analysis due to the application leveraging them. However the biggest concern of any developer incorporating a FOSS project into their codebase is the status of the project; is it too immature, buggy but on the road to stability with a thriving ecosystem of developers backing it or dead in the water. How does one evaluate this? Here are a few pointers that might indicate a project is not worth the risk.

A project appears to be obviously inactive, having had no updates for years.
A project that is obviously not going to be used in any new deployments because the standard language, library, or platform now has the capability built in
A project that is rapidly losing developers to some more-trendy alternative project
A project whose status is unclear, with some releases and statements in the forums that they are 'definitely alive,' but which seem to have lost direction or momentum.
A project that has had no updates but is highly stable and does what is necessary, but is risky because they may not interoperate with future upgrades to other components.

I would like to expand on the last point a little, which is where the beauty of FOSS comes to light. Stable project but not longer maintained, this is not the ideal situation. However if it is crucial to your project, you have the choice to undertake the maintenance yourself. As your project changes and matures you will need to take the FOSS project along with it and see it more as something you extend, develop upon, tear apart and bandage up, rather than just another 3rd party library. The beauty of FOSS is that the possibilities are endless.

Define: FOSS

Mark — Sat, 20 Apr 2013 22:28:14 +0000

Free and open source software (F/OSS, FOSS) or free/libre/open source software (FLOSS) is software that is both free software and open source. It is liberally licensed to grant users the right to use, copy, study, change, and improve its design through the availability of its source code. &nbsp

Language Agnostic: Unit Test Principles

Mark — Thu, 07 Feb 2013 13:11:00 +0000

After spending a few days churning out unit tests for a recent project of mine, I took a step back and asked myself: What are the properties of good unit tests? Tests are seen as something of a burden that devs have to do to keep business happy, but in reality they can save countless hours of debugging and ensure the behaviour of a system is not changed (for the worst) when new code is introduced. So here are my, principles / properties of a good test: Automatic: Invoking of tests as well as checking results for PASS/FAIL should be automatic. Thorough: As in Coverage; Although bugs tend to cluster around certain regions in the code, ensure that you test all key paths and scenarios, even the most innocent looking code can turn sour if the system does not feed it the right input. There are several tools that analyse untested lines of code so leverage them to find the gaps. Repeatable: Tests should produce the same results every time they are run, they should not depend on uncontrollable params. Independent: Tests should be isolated, not relying on data in DBs or external modules, think mocking frameworks here. Within the suite itself, no assumptions about order of test execution should be made, you should be able to pick and choose which test to run regardless. This clean slate approach to each test ensures that when a test fails, it should pinpoint the exact location of the problem. Professional: In the long run you'll have as much test code as production (if not more), therefore follow the same standard of good-design for your test code. Well factored methods/classes with intention revealing names, no needless duplication, etc. Tests are documentation, they point out the corner cases that can cripple a system, therefore should be highly readable. Speed: Good tests should run Fast. Any test that takes over half a second to run, needs to be worked upon. The longer the test suite takes for a run, the less frequently a developer will choose to run it. Some other more obvious guidelines, that may cut out some of the bloat:

Don't test code that you don't own (e.g. third-party libs).
Don't go about testing getters and setters.
Keep an eye on cost-to-benefit ratio or defect probability.
Write your tests in an Arrange/Act/Assert style.

Define: NFR

Mark — Wed, 06 Feb 2013 21:57:02 +0000

NFR stands for Non-Functional requirements. These are often associated with the state of the system and not with the functionality that the system has to offer. NFR aren't something that your system does but rather something that your system is. General 'ilities' of the system such as scalability, interoperability, maintainability, portability, performance and security fall under the category of NFR.