It was nice to see Lisp, created in 1962, referenced as being the first
language that featured a JIT compiler as well as garbage collection. It was
also good to see Smalltalk referenced as pioneering class libraries, visual GUI
driven IDEs and bytecode.

As such Java essentially stands on the shoulders of giants. Now dynamic
language writers can themselves use the JVM to boost their productivity by
profiting from the VM’s memory management, JIT optimisations, native threading.
The result could be a smaller code base and more time to concentrate on
interesting language features (of course another result would be that the JVM
becomes interesting not only for Java developers but also to people who want to
use dynamic languages instead).

The projects invoke dynamic as well as the DaVinci machine are both interesting
areas for people to follow who are interested in running dynamic languages on
the JVM.

When dealing with multiple systems you deal with correlating values across
machines and systems, providing meaningful visualisations to draw the correct
decisions.

When thinking of your log architecture you might want to consider storing not
only log messages. In addition facts like release numbers should be tracked
somewhere - ready to join in when needed to correlate behaviour with release
version. To do that also track events like rolling out a release to production.
Launching in a new market, switching traffic to a new system could be other
events. Introduce not only pure log messages but also provide aggregated
metrics and counters. All of these pieces should be stored and tracked
automatically to free operations for more important work.

Have you ever thought about documenting not only your software, it’s interfaces
and input/output format? What about documenting the logged information as well?
What about the fields contained in each log message? Are they documented or do
people have to infer their meaning from the content? What about valid ranges
for values - are they noted down somewhere? Did you store whether a specific
field can only contain integers or whether some day it also could contain
letters? What about the number format - is it decimal, hexadecimal?

For a nice architecture documentation of the BBC checkout

Winning the metrics battle by the BBC dev blog.

There’s an abundance of tools out there to help you with all sorts of logging
related topics:

• For visualisation and transport: Datadog, kibana, logstash, statsd,
  graphite, syslog-ng
  


• For providing the values: JMX, metrics, Jolokia
  


• For collection: collecd, statsd, graphite, newrelic, datadog
  


• For storage: typical RRD tools including RRD4j, MongoDB, OpenTSDB based
  on HBase, Hadoop
  


• For charting: Munin, Cacti, Nagios, Graphit, Ganglia, New Relic, Datadog
  


• For Profiling: Dynatrace, New Relic, Boundary
  


• For events: Zabbix, Icinga, OMD, OpenNMS, HypericHQ, Nagios,JbossRHQ
  


• For logging: splunk, Graylog2, Kibana, logstash
  

Make sure to provide metrics consistently and be able to add them with minimal
effort. Self adaption and automation are useful for this. Make sure developers,
operations and product owners are able to use the same system so there is no
information gap on either side. Your logging pipeline should be tailored to
provide easy and fast feedback on the implementation and features of the
product.

To reach a decent level of automation a set of tools is needed for:

• Configuration management (where to store passwords, urls or ips, log
  levels etc.). Typical names here include Zookeeper,but also CFEngine, Puppet
  and Chef.
  


• Deployment management. Typical names here are UC4, udeploy, glu, etsy
  deployment.
  


• Server orchestration (e.g. what is started when during boot). Typical
  names include UC4, Nolio, Marionette Collective, rundeck.
  


• Automated provisioning (think “how long does it take from server failure
  to bringing that service back up online?”). Typical names include kickstart,
  vagrant, or typical cloud environments.
  


• Test driven/ behaviour driven environments (think about adjusting not
  only your application but also firewall configurations). Typical tools that
  come to mind here include Server spec, rspec, cucumber, c-puppet, chef.
  


• When it comes to defining the points of communication for the whole
  pipeline there is no tool you can use that is better than traditional pen and
  paper, socially getting both development and operations into one room.
  

The tooling to support this process goes from simple self-written bash scripts
in the startup model to frameworks that support the flow partially, up to
process based suites that help you. No matter which path you choose the goal
should always be to end up with a well documented, reproducable step into
production. When introducing such systems problems in your organisation may
become apparent. Sometimes it helps to just create facts: It’s easier to ask for
forgiveness than permission.

From his consultancy and fire fighter background he has a good overview of what
logging in the average corporate environment looks like: No logging plan, no
rules, dozens of logging frameworks in active use, output in many different
languages, no structured log events but a myriad of different quoting,
formatting and bracketing standards instead.

So what should the ideal log line contain? First of all it should really be a
log line instead of a multi line something that cannot be reconstructed when
interleaved with other messages. The line should not only contain the class
name that logged the information (actually that is the least important piece of
information), it should contain the thread id, server name, a (standardised and
always consistently formatted) timestamp in a decent resolution (hint: one new
timestamp per second is not helpful when facing several hundred requests per
second). Make sure to have timing aligned across machines if timestamps are
needed for correlating logs. Ideally there should be context in the form of
request id, flow id, session id.

When thinking about logs, do not think too much about human readability - think
more in terms of machine readability and parsability. Treat your logging system
as the db in your data center that has to deal with most traffic. It is what
holds user interactions and system metrics that can be used as business
metrics, for debugging performance problems, for digging up functional issues.
Most likely you will want to turn free text that provides lots of flexibility
for screwing up into a more structured format like json, or even some binary
format that is storage efficient (think protocol buffers, thrift, avro).

In terms of log levels, make sure to log development traces on trace, provide
detailed problem analysis stuff on debug, put normal behaviour onto info. In
case of degraded functionality, log to warn. In case of things you cannot
easily recovered from put them on error. When it comes to logging hierarchies -
do not only think in class hierarchies but also in terms of use cases: Just
because your http connector is used in two modules doesn’t mean that there
should be no way to turn logging on just for one of the modules alone.

When designing your logging make sure to talk to all stakeholders to get clear
requirements. Make sure you can find out how the system is being used in the
wild, be able to quantify the number of exceptions; max, min and average
duration of a request and similar metrics.

Tools you could look at for help include but are not limited to splunk, jmx,
jconsole, syslog, logstash, statd, redis for log collection and queuing.

As a parting exercise: Look at all of your own logfiles and count the different
formats used for storing time.

It was interesting to learn how many different aspects of both software and
hardware contribute to code performance. Micro benchmarks are considered
dangerous for a reason - creating a well controlled environment that matches
what the code will encounter in production is influenced by things like just in
time compilation, cpu throttling, etc.

Some myths that Arno proved wrong include final making code faster (in case of
method parameters it doesn’t make a difference up to bytecode being identical
with and without), inheritance being always expensive (even with an abstract
class between the interface and the implementation Java 6 and 7 can still
inline the method in question). Another one was on often wrongly scoped Java
vs. C comparisons. One myth resolved around the creation of temporary objects -
since Java 6 and 7 in simple cases even these can be optimised away.

When it comes to (un-)boxing and reflection there is a performance penalty. For
the latter mostly for method lookup, not so much for calling the method. What we
are talking about however are penalties in the range of about 1000 compute
cycles. Compared to doing any remote calls this is still dwarfed. Reflection on
fields is even cheaper.

One of the more wide spread myths resolved around string concatenation being
expensive - doing a “A” + “B” in code will be turned into “AB” in
bytecode. Even doing the same with a variable will be turned into the use of
StringBuilder ever since -XX:OptimizeStringConcat was turned on by default.

The main message here is to stop trusting your intuition when reasoning about a
system’s performance and performance bottlenecks. Instead the goal should be to
go and measure what is really going on. Those are simple examples where your
average Java intuition goes wrong. Make sure to stay on top with what the JVM
turns your code into and how that is than executed on the hardware you have
rolled out if you really want to get the last bit of speed out of your
application.

When looking at current Java optimisations there is quite a bit going on to
support better parallelisation: Work is being done to provide for improving
lock contention situations, the GC adaptive sizing policy has been improved to
a usable state, there is added support for parallel arrays and lampbda’s
splitable interface.

When it comes to better locking optimisations what is most notable is work
towards coarsening locks at compile and JIT time (essentially moving locks from
the inside of a loop to the outside); eliminating locks if objects are being
used in a local, non-threaded context anyway; and support for biased locking
(that is forcing locks only when a second thread is trying to access an
object). All three taken together can lead to performance improvements that
will almost render StringBuffer and StringBuilder to exhibit equal performance
in a single threaded context.

For pieces of code that suffer from false sharing (two variables used in
separate threads independently that end up in the same CPU cacheline and as a
result are both flushed on update) there is a new annotation: Adding the
“@contended” annotation can help the compiler for which pieces of code to add
cacheline padding (or re-arrange entirely) to avoid that false sharing from
happening. One other way to avoid false sharing seems to be to look for class
cohesion - coherent classes where methods and variables are closely related
tend to suffer less from false sharing. If you would like to view the resulting
layout use the “-XX:PrintFieldLayout” option.

Java 8 will bring a few more notable improvements including changes to the
adaptive sizing GC policy, the introduction of parallel arrays that allow for
parallel execution of predicates on array entries, changes to the concurrency
libraries, internalised iterators.

Simply put performance issues usually arise due to a difference in either data
volumn, concurrency levels or resource usage between the dev, qa and production
environments. The tooling to uncover and explain them is pretty well known:
Staring with looking at logfiles, ARM tools, using aspects, bytecode
instrumentalisation, sampling, watching JMX statistics, and PMI tools.

All of theses tools have their own unique advantages and disadvantages. With
logs you get the most freedom, however you have to know what to log at
development time. In addition logging is i/o heavy, so doing too much can slow
the application down itself. In a common distributed system logs need to be
aggregated somehow. As a simple example of what can go wrong are cascading
exceptions spilled to disk that cause machines to run out of disk space one
after the other. When relying on logging make sure to keep transaction
contexts, in particular transaction ids across machines and services to
correlate outages. In terms of tool support, look at scribe, splunk and flume.

A tool often used for tracking down performance issues in development is the
well known profiler. Usually it creates lots of very detailed data. However it
is most valuable in development - in production profiling a complete server
stack produces way too much load and data to be feasable. In addition there’s
usually no transaction context available for correlation again.

A third way of watching applications do their work is to watch via JMX. This
capability is built in for any Java application, in particular for servlet
containers. Again there is not transaction context. Unless you take care of it
there won’t be any historic data.

When it comes to diagnosing problems, you are essentially left with fixing
either the “it does not work” case or the “it is slow case”.

For the “it is slow case” there are a few incarnations:

• It was always slow, we got used to it.
  


• It gets slow over time.
  


• It gets slower exponentially.
  


• It suddenly gets slow.
  


• There is a spontanous crash.
  

In the case of “it does not work” you are left with the following observations:

• Sudden outages.
  


• Always flaky.
  


• Sporadic error messages.
  


• Silent death.
  


• Increasing error rates.
  


• Misleading error messages.
  

In the end you will always be spinning in a Look at symptoms, Elimnate
non-causes, Identifiy suspects, Confirm and Eliminate comparing to normal. If
not done with that, leather, rinse, repeat. When it comes to causes for errors
and slowness you will usually will run into one of the following causes: In
many cases bad coding practices are a problem, too much load, missing backends,
resource conflicts, memory and resource leakage as well as hardware/networking
issues are causes.

Some symptoms you may observe include foreseeable lock ups (it’s always slow
after four hours, so we just reboot automatically before that), consistent
slowness, sporadic errors (it always happens after a certain request came in),
getting slow and slower (most likely leaking resources), sudden chaos (e.g.
someone pulling the plug or someone removing a hard disk), and high utilisation
of resources.

Linear memory leak

In case of a linear memory leak, the application usually runs into an OOM
eventually, getting ever slower before that due to GC pressure. Reasons could
be linear structures being filled but never emptied. What you observe are
growing heap utilisation and growing GC times. In order to find such leakage
make sure to turn on verbose GC logging, do heapdumps to find leaks. One
challenge though: It may be hard to find the leakage if the problem is not one
large object, but many, many small ones that lead to a death by 1000 cuts
bleeding the application to death.

In development and testing you will do heap comparisons. Keep in mind that
taking a heap dump causes the JVM to stop. You can use common profilers to look
at the heap dump. There are variants that help with automatic leak detection.

A variant is the pig in a python issue where sudden unusually large objects
cause the application to be overloaded.

Resource leaks and conflicts

In case of conflicting resource usage you usually face code that was developed
with overly cautious locking and data integrity constraints. The way to go are
threaddumps to uncover threads in block and wait states.

Bad coding practices

When it comes to locking you might run into dead-/live-lock problems. There
could be chokepoints (resources that all threads need for each processing
chain). In a thread dump you will typically see lots of wait instead of block
time.

In addition there could be internal and external bottlenecks. In particular
keep those in mind when dealing with databases.

The goal should be to find an optimum for your application between too many too
small requests that waste resources getting dispatched, and one huge request
that everyone else is waiting for.

Norman started his talk clarifying what he means by high scale: Anything above
1000 concurrent connections in his talk are considered high scale, anything
below 100 concurrent connections is fine to be handled with threads and blocking
IO. Before tuning anything, make sure to measure if you have any problem at
all: Readability should always go before optimisation.

He gave a few pointers as to where to look for optimisations: Get started by
studying the socket options - TCP-NO-DELAY as well as the send and receive
buffer sizes are most interesting. When under GC pressure (check the GC locks
to figure out if you are) make sure to minimise allocation and deallocation of
objects. In order to do that consider making objects static and final where
possible. Make sure to use CMS or G1 for garbage collection in order to
maximise throughput. Size areas in the JVM heap according to your access
patterns. The goal should always be to minimise the chance of running into a
stop the world garbage collection.

When it comes to using buffers you have the choice of using direct or heap
buffers. While the former are expensive to create, the latter come with the
cost of being zero’ed out. Often people start buffer pooling, potentially
initialising the pool in a lazy manner. In order to avoid memory fragmentation
in the Java heap, it can be a good idea to create the buffer at startup time
and re-use it later on.

In particular when parsing structured messages like they are common in
protocols it usually makes sense to use gathering writes and scattering reads
to minimise the number of system calls for reading and writing. Also try to
buffer more if you want to minimise system calls. Use slice and duplicate to
create views on your buffers to avoid mem copies. Use a file channel when
copying files without modifications.

Make sure you do not block - think of DNS servers being unavailable or slow as
an example.

As a parting note, make sure to define and document your threading model. It
may ease development to know that some objects will always only be used in a
single threaded context. It usually helps to reduce context switches as well as
may ease development to know that some objects will always only be used in a
single threaded context. It usually helps to reduce context switches as well as
keeping data in the same thread to avoid having to use synchronisation and the
use of volatile.

Also make a conscious decision about which protocol you would like to use for
transport - in addition to tcp there’s also udp, udt, sctp. Use pipelining in
order to parallelise.

After breakfast was over the first day started with a talk by Bernd on the
Hadoop ecosystem. He did a good job selecting the most important and
interesting projects related to storing data in HDFS and processing it with Map
Reduce. After the usual “what is Hadoop”, “what does the general architecture
look like”, “what will change with YARN” Bernd gave a nice overview of which
publications each of the relevant projects rely on:

• HDFS is mainly based on the paper on GFS.
  


• Map Reduce comes with it’s own publication.
  


• The big table paper mainly inspired Cassandra (to some extend), HBase,
  Accumulo and Hypertable.
  


• Protocol Buffers inspired Avro and Thrift, and is available as free
  software itself.
  


• Dremel (the storage side of things) inspired Parquet.
  


• The query language side of Dremel inspired Drill and Impala.
  


• Power Drill might inspire Drill.
  


• Pregel (a graph database) inspired Giraph.
  


• Percolator provided some inspiration to HBase.
  


• Dynamo by Amazon kicked of Cassandra and others.
  


• Chubby inspired Zookeeper, both are based on Paxos.
  


• On top of Map Reduce today there are tons of higher level languages,
  starting with Sawzall inside of Google, continuing with Pig and Hive at Apache
  we are now left with added languages like Cascading, Cascalog, Scalding and
  many more.
  


• There are many other interesting publications (Megastore, Spanner, F1 to
  name just a few) for which there is no free implementation yet. In addition
  with Storm, Hana and Haystack there are implementations lacking canonical
  publications.
  

After this really broad clarification of names and terms used, Bernd went into
some more detail on how Zookeeper is being used for defining the namenode in
Hadoop 2, how high availablility and federation works for namenodes. In
addition he gave a clear explanation of how block reports work on cluster
bootup. The remainder of the talk was reserved for giving an intro to HBase,
Giraph and Drill.

Together with Uwe Schindler I had published a series of articles on Apache
Lucene at Software and Support Media’s Java Mag several years ago. Earlier this
year S&S kindly invited my to their BigDataCon - co-located with JAX to give a
talk of my choosing that at least touches upon Lucene.

Thinking back and forth about what topic to cover what came to my mind was to
give a talk on how easy it is to do text classification with Mahout when
relying on Apache Lucene for text analysis, tokenisation and token filtering.
All classes essentially are in place to integrate Lucene Analyzers with Mahout
vector generation - needed e.g. as a pre-processing step for classification or
text clustering.

Feel free to check out some of my sandbox code over at <a
href=“http://github.org/MaineC/sofia”>github</a>.

After attending the conference I can only recommend everyone interested in Java
programming and able to understand German to buy a ticket for the conference.
It’s really well executed, great selection of talks (though the sponsored
keynotes usually aren’t particularly interesting), tasty meals, interesting
people to chat with.

About a month ago I attended the first European Hadoop Summit, organised by
Hortonworks in Amsterdam. The two day conference brought together both vendors
and users of Apache Hadoop for talks, exhibition and after conference beer
drinking.

Russel Jurney kindly asked me to chair the Hadoop applied track during
Apache Con EU. As a result I had a good excuse to attend the event. Overall
there were at least three times as many submissions than could reasonably be
accepted. Accordingly accepting proposals was pretty hard.

Though some of the Apache community aspect was missing at Hadoop summit it was
interesting nevertheless to see who is active in this space both as users as
well as vendors.

If you check out the talks on Youtube make sure to not miss the two sessions by
Ted Dunning as well as the talk on handling logging data by Twitter.