Inductive Bias

For slides see also: Speakerdeck: High performance networking on the JVM

Norman started his talk clarifying what he means by high scale: Anything above
1000 concurrent connections in his talk are considered high scale, anything
below 100 concurrent connections is fine to be handled with threads and blocking
IO. Before tuning anything, make sure to measure if you have any problem at
all: Readability should always go before optimisation.

He gave a few pointers as to where to look for optimisations: Get started by
studying the socket options - TCP-NO-DELAY as well as the send and receive
buffer sizes are most interesting. When under GC pressure (check the GC locks
to figure out if you are) make sure to minimise allocation and deallocation of
objects. In order to do that consider making objects static and final where
possible. Make sure to use CMS or G1 for garbage collection in order to
maximise throughput. Size areas in the JVM heap according to your access
patterns. The goal should always be to minimise the chance of running into a
stop the world garbage collection.

When it comes to using buffers you have the choice of using direct or heap
buffers. While the former are expensive to create, the latter come with the
cost of being zero’ed out. Often people start buffer pooling, potentially
initialising the pool in a lazy manner. In order to avoid memory fragmentation
in the Java heap, it can be a good idea to create the buffer at startup time
and re-use it later on.

In particular when parsing structured messages like they are common in
protocols it usually makes sense to use gathering writes and scattering reads
to minimise the number of system calls for reading and writing. Also try to
buffer more if you want to minimise system calls. Use slice and duplicate to
create views on your buffers to avoid mem copies. Use a file channel when
copying files without modifications.

Make sure you do not block - think of DNS servers being unavailable or slow as
an example.

As a parting note, make sure to define and document your threading model. It
may ease development to know that some objects will always only be used in a
single threaded context. It usually helps to reduce context switches as well as
may ease development to know that some objects will always only be used in a
single threaded context. It usually helps to reduce context switches as well as
keeping data in the same thread to avoid having to use synchronisation and the
use of volatile.

Also make a conscious decision about which protocol you would like to use for
transport - in addition to tcp there’s also udp, udt, sctp. Use pipelining in
order to parallelise.

mainec Event JAX, netty, performance

After breakfast was over the first day started with a talk by Bernd on the
Hadoop ecosystem. He did a good job selecting the most important and
interesting projects related to storing data in HDFS and processing it with Map
Reduce. After the usual “what is Hadoop”, “what does the general architecture
look like”, “what will change with YARN” Bernd gave a nice overview of which
publications each of the relevant projects rely on:

• HDFS is mainly based on the paper on GFS.
  


• Map Reduce comes with it’s own publication.
  


• The big table paper mainly inspired Cassandra (to some extend), HBase,
  Accumulo and Hypertable.
  


• Protocol Buffers inspired Avro and Thrift, and is available as free
  software itself.
  


• Dremel (the storage side of things) inspired Parquet.
  


• The query language side of Dremel inspired Drill and Impala.
  


• Power Drill might inspire Drill.
  


• Pregel (a graph database) inspired Giraph.
  


• Percolator provided some inspiration to HBase.
  


• Dynamo by Amazon kicked of Cassandra and others.
  


• Chubby inspired Zookeeper, both are based on Paxos.
  


• On top of Map Reduce today there are tons of higher level languages,
  starting with Sawzall inside of Google, continuing with Pig and Hive at Apache
  we are now left with added languages like Cascading, Cascalog, Scalding and
  many more.
  


• There are many other interesting publications (Megastore, Spanner, F1 to
  name just a few) for which there is no free implementation yet. In addition
  with Storm, Hana and Haystack there are implementations lacking canonical
  publications.
  

After this really broad clarification of names and terms used, Bernd went into
some more detail on how Zookeeper is being used for defining the namenode in
Hadoop 2, how high availablility and federation works for namenodes. In
addition he gave a clear explanation of how block reports work on cluster
bootup. The remainder of the talk was reserved for giving an intro to HBase,
Giraph and Drill.

mainec Event BigDataCon, Hadoop, JAX

Together with Uwe Schindler I had published a series of articles on Apache
Lucene at Software and Support Media’s Java Mag several years ago. Earlier this
year S&S kindly invited my to their BigDataCon - co-located with JAX to give a
talk of my choosing that at least touches upon Lucene.

Thinking back and forth about what topic to cover what came to my mind was to
give a talk on how easy it is to do text classification with Mahout when
relying on Apache Lucene for text analysis, tokenisation and token filtering.
All classes essentially are in place to integrate Lucene Analyzers with Mahout
vector generation - needed e.g. as a pre-processing step for classification or
text clustering.

Feel free to check out some of my sandbox code over at <a
href=“http://github.org/MaineC/sofia”>github</a>.

After attending the conference I can only recommend everyone interested in Java
programming and able to understand German to buy a ticket for the conference.
It’s really well executed, great selection of talks (though the sponsored
keynotes usually aren’t particularly interesting), tasty meals, interesting
people to chat with.

mainec Event BigDataCon, JAX

About a month ago I attended the first European Hadoop Summit, organised by
Hortonworks in Amsterdam. The two day conference brought together both vendors
and users of Apache Hadoop for talks, exhibition and after conference beer
drinking.

Russel Jurney kindly asked me to chair the Hadoop applied track during
Apache Con EU. As a result I had a good excuse to attend the event. Overall
there were at least three times as many submissions than could reasonably be
accepted. Accordingly accepting proposals was pretty hard.

Though some of the Apache community aspect was missing at Hadoop summit it was
interesting nevertheless to see who is active in this space both as users as
well as vendors.

If you check out the talks on Youtube make sure to not miss the two sessions by
Ted Dunning as well as the talk on handling logging data by Twitter.

mainec Event amsterdam, Hadoop, hadoopsummit

In his talk on Spdy Mathew Steele explained how he implemented the spdy protocol
as an Apache httpd module - working around most of the safety measures and
design decisions in the current httpd version. Essentially to get httpd to
support the protocol all you need now is mod_spdy plus a modified version of
mod_ssl.

The keynote on the last day was given by the Puppet founder. Some interesting
points to take away from that:

• Though hard in the beginning (and half way through, and after years) it
  is important to learn giving up control: It usually is much more productive and
  leads to better results to encourage people to do something than to be
  restrictive about it. A single developer only has so much bandwidth - by
  farming tasks out to others - and giving them full control - you substantially
  increase your throughput without having to put in more energy.
  


• Be transparent - it’s ok to have commercial goals with your project. Just
  make sure that the community knows about it and is not surprised to learn about
  it.
  


• Be nice - not many succeed at this, not many are truely able to ignore
  religion (vi vs. emacs). This also means to be welcoming to newbies, to hustle
  at conferences, to engage the community as opposed to announcing changes.
  

Overall good advise for those starting or working on an OSS project and seeking
to increase visibility and reach.

mainec Apache Con ApacheCon, ApacheConNA, Httpd, spdy

Have you ever measured the general behaviour of your Hadoop jobs? Have you
sized your cluster accordingly? Do you know whether your work load really is IO
bound or CPU bound? Legend has it noone expecpt Allen Wittenauer over at
Linked.In, formerly Y! ever did this analysis for his clusters.

Steve Watt gave a pitch for actually going out into your datacenter measuring
what is going on there and adjusting the deployment accordingly: In small
clusters it may make sense to rely on raided disks instead of additional
storage nodes to guarantee “replication levels”. When going out to vendors to
buy hardware don’t rely on paper calculations only: Standard servers in Hadoop
clusters are 1 or 2u. This is quite unlike beefy boxes being sold otherwise.

Figure out what reference architecture is being used by partners, run your
standard workloads, adjust the configuration. If you want to run the 10TB
Terrasort to benchmark your hardware and system configuration. Make sure to
capture data during all your runs - have Ganglia or SAR, watch out for
intersting behaviour in io rates, cpu utilisation, network traffic. The goal is
to get the cpu busy, not wait for network or disk.

After the instrumentation and trial run look for over- and underprovisionings,
adjust, leather, rinse, repeat.

Also make sure to talk to the datacenter people: There are floor space, power
and cooling constraints to keep in mind. Don’t let the whole datacenter go down
because your cpu intensive job is drawing more power than the DC was designed
for. Ther are also power constraints per floor tile due to cooling issues -
those should dictate the design.

Take a close look at the disks you deploy: SATA vs. SAS can make a 40%
performance difference at a 20% cost difference. Also the number of cores per
machines dictates the number of disks to spread the likelyhood of random read
access. As a rule of thumb - in a 2U machine today there should be at least
twelve large form factor disks.

When it comes to controllers he goal should be to get a dedicated lane to disc,
safe one controller if price is an issue. Trade off compute power against power
consumption.

Designing your network keep in mind that one switch going down means that one
rack will be gone. This may be a non-issue in a Y! size cluster, in your
smaller scale world it might be worth the money investing in a second switch
though: Having 20 nodes go black isn’t a lot of fun if you cannot farm out the
work and re-replication to other nodes and racks. Also make sure to have enough
ports in rack switches for the machines you are planning to provision.

Avoid playing the ops whake-a-mole game by having one large cluster in the
organisation than many different ones where possible. Multi-tenancy in Hadoop is
still pre-mature though.

If you want to play with future deployments - watch out for HP currently
packing 270 servers where today are just two via system on a chip designs.

mainec Apache Con ApacheCon, ApacheConNA, Hadoop, sizing

Monitoring - a task generally neglected - or over done - during development.
But still vital enough to wake up people from well earned sleep at night when
done wrong. Rainer Jung provided some valuable insights on how to monitor Apache httpd and Tomcat.

Of course failure detection, alarms and notifications are all part of good
monitoring. However so is avoidance of false positives and metric collection,
visualisation, and collection in advance to help with capacity planning and
uncover irregular behaviour.

In general the standard pieces being monitored are load, cache utilisation,
memory, garbage collection and response times. What we do not see from all that
are times spent waiting for the backend, looping in code, blocked threads.

When it comes to monitoring Java - JMX is pretty much the standard choice. Data
is grouped in management beans (MBeans). Each Java process has default beans,
on top there are beans provided by Tomcat, on top there may be application
specific ones.

For remote access, there are Java clients that know the protocol - the server
must be configured though to accept their connection. Keep in mind to open the
firewall in between as well if there is any. Well known clients include
JVisualVM (nice for interactive inspection), jmxterm as a command line client.

The only issue: Most MBeans encode source code structure, where what you really
need is change rates. In general those are easy to infer though.

On the server side for Tomcat there is the JMXProxy in Tomcat manager that
exposes MBeans. In addition there is Jolohia (including JSon serialisation) or
the option to roll your own.

So what kind of information is in MBeans:

• OS - load, process cpu time, physical memory, global OS level
  stats. As an example: Here deviding cpu time by time geves you the average cpu
  concurrency.
  



• Runtime MBean gives uptime.
  


• Threading MBean gives information on count, max available threads etc
  


• Class Loading MBean should get stable unless you are using dynamic
  languaes or have enabled class unloading for jsps in Tomcat.
  


• Compliation contains HotSpot compiler information.
  


• Memory contains information on all regions thrown in one pot. If you need
  more fine grained information look out for the Memory Pool and GC MBeans.
  

As for Tomcat specific things:

• Threadpool (for each connector) has information on size, number of busy
  threads.
  


• GlobalRequestProc has request counts, processing times, max time bytes
  received/sent, error count (those that Tomcat notices that is).
  


• RequestProcessor exists once per thread, it shows if a request is
  currently running and for how long. Nice to see if there are long running
  requests.
  


• DataSource provides information on Tomcat provided database connections.
  

Per Webapp there are a couple of more MBeans:

• ManagerMBean has information on session management - e.g. session
  counter since start, login rate, active sessions, expired sessions, max active
  sinse restart sessions (here a restart is possible), number of rejected
  sessions, average alive time, processing time it took to clean up sessions,
  create and required rate for last 100 sessions
  


• ServletMBean contains request count, accumulated processing time.
  


• JspMBean (together with activated loading/unloading policy) has
  information on unload and reload stats and provides the max number of loaded
  jsps.
  

For httpd the goals with monitoring are pretty similar. The only difference is
the protocol used - in this case provided by the status module. As an
alternative use the scoreboard connections.

You will find information on

• restart time, uptime
  


• serverload
  


• total number of accesses and traffic
  


• idle workers and number of requests currently processed
  


• cpu usage - though that is only accurate when all children are stopped
  which in production isn’t particularly likely.
  

Lines that indicate what threads do contain waitinng, request read, send reply
- more information is documented online.

When monitoring make sure to monitor not only production but also your stress
tests to make meaningful comparisons.

mainec Apache Con ApacheCon, ApacheConNA, Httpd, tomcat

During the security talk at Apache Con a topic commonly glossed over by
developers was covered in quite some detail: With software being developed that
is being deployed rather widely online (over 50% of all websites are powered
by the Apache webserver) natually security issues are of large concern.

Currently there are eight trustworthy people on the foundation-wide security
response team, subscribed to security@apache.org. The team was started by
William A. Rowe when he found a volnarability in httpd. The general work mode -
as opposed to the otherwise “all things open” way of doing things at Apache -
is to keep the issues found private until fixed and publicise widely
afterwards.

So when running Apache software on your servers - how do you learn about
security issues? There is no such thing as a priority list for specific
vendors. The only way to get an inside scoop is to join the respective
project’s PMC list - that is to get active yourself.

So what is being found? 90% of all security issues are found be security
researches. The remaining 10% are usually published accidentially - e.g. by
users submitting the issue through the regular public bug tracker of the
respective project.

In Tomcat currently no issues was disclosed w/o letting the project know. httpd
still is the prime target - even of security researchers who are in favour of
a full disclosure policy - the PMC cannot do a lot here other than fix issues
quickly (usually within 24 hours).

As a general rule of thumb: Keep your average release cycle time in mind - how
long will it take to get fixes into people’s hands? Communicate transparently
which version will get security fixes - and which won’t.

As for static analysis tools - many of those are written for web apps and as
such not very helpful for a container. What is highly dangerous in a web app
may just be the thing the container has to do to provide features to web apps.
As for Tomcat, they have made good experiences with Findbugs - most others have
too many false positives.

When dealing with a volnarability yourself, try to get guidance from the
security team on what is actually a security volnarability - though the final
decision is with the project.

Dealing with the tradeoff of working in private vs. exposing users affected by
the volnarability to attacks is up to the PMC. Some work in public but call the
actual fix a refactoring or cleanup. Given enough coding skills on the attacker
side this of course will not help too much as sort of reverse engineering what
is being fixed by the patches is still possible. On the other hand doing
everything in private on a separate branch isn’t public development anymore.

After this general introduction Mark gave a good overview of the good, the bad
and the ugly way of handling security issues in Tomcat. For his slides
(including an anecdote of what according to the timing and topic looks like it
was highly related to the 2011 Java Hash Collision talk at Chaos Communication
Congress).

mainec Apache Con ApacheCon, ApacheConNA, community, security

In her talk on documentation on OSS Noirin gave a great wrap up of the topic of
what documentation to create for a project and how to go about that task.

One way to think about documentation is to keep in mind that it fulfills
different tasks: There is conceptual, procedural and task-reference
documentation. When starting to analyse your docs you may first want to debug
the way it fails to help its users: “I can’t read my mail” really could mean
“My computer is under water”.

A good way to find awesome documentation can be to check out Stackoverflow
questions on your project, blog posts and training articles. Users today really
are searching instead of browsing docs. So where to find documentation actually
is starting to matter less. What does matter though is that those pages with
relevant information are written in a way that makes it easy to find them
through search engines: Provide a decent title, stable URLs, reasonable tags
and descriptions. By the way, both infra and docs people are happy to talk to
*good* SEO guys.

In terms of where to keep documentation:

For conceptual docs that need regular review it’s probably best to keep them in
version control. For task documentation steps should be easy to upgrade once
they fail for users. Make sure to accept bug reports in any form - be it on
Facebook, Twitter or in your issue tracker.

When writing good documentation always keep your audience in mind: If you don’t
have a specific one, pitch one. Don’t try to cater for everyone - if your docs
are too simplistic or too complex for others, link out to further material.
Understand their level of understanding. Understand what they will do after
reading the docs.

On a general level always include an about section, a system overview, a
description of when to read the doc, how to achieve the goal, provide
examples, provide a trouble shooting section and provide further information
links. Write breadth first - details are hard to fill in without a bigger
picture. Complete the overview section last. Call out context and
pre-requesites explicitly, don’t make your audience do more than they really
need to do. Reserve the details for a later document.

In general the most important and most general stuff as well as the basics
should come first. Mention the number of steps to be taken early. When it comes
to providing details: The more you provide, the more important the reader will
deem that part.

mainec Apache Con ApacheCon, ApacheConNA, community

In her talk on delegation Deb Nicholson touched upon a really important topic in
OSS: Your project may live longer than you are willing to support it yourself.

The first important point about delegation is to delegate - and to not wait
until you have to do it. Soon you will realise that mentoring and delegation
actually is a way to multiply your resources.

In order to delegate people to delegate to are needed. To find those it can be
helpful to understand what motivates people to work in general as well as on
open source in particular: Sure, fixing a given problem and working on great
software projects may be part of it. As important though is recognition
individually and in groups of people.

Keeping that in mind, “Thanking” is actually a license to print free money in
the open source world. Do it in a verbose manner to be believable, do it in
public and in a way that makes your contributors feel a little bit of glory.

Another way to lead people in is to help out socially: Facilitate connections,
suggest connections, introduce people. Based on the diversity of the project
you are working on you may be in a way larger network and have access to much
more corporations and communities than any peer who is not active. Use that
potential.

Also when leading OSS projects keep in eye on people being rude: Your project
should be accessible to facilitate participation.

In case of questions treat them as a welcome opportunity to pull a new
community member in: Answer quickly, answer on your list, delegate to middle
seniors to pull them in. Have training missions for people who want to get
started and don’t know your tooling yet. Have prepared documents to provide
links to in case questions occur.

In Apache we tend to argue people should not fall victim of volunteeritis.
Another way to put that is to make sure to avoid the licked cookie syndrom:
When people volunteer to do a task and never re-appear that task is tainted
until explicitly marked as “not taken” later on. One way to automate that is
to have a fixed deadline after which tasks are automatically marked as free to
take and tackle by anyone.

When it comes to the question of When to write documentation: There really is
no point in time that should stop you from contributing docs - all the way from
just above getting started level (writing the getting started docs for those
following you) up to the “I’m an awesome super-hacker” mode for those trying
to hack on similar areas.

Especially when delegating to newbies make sure to set the right expectations:
How long is it going to take to fix an issue, what is the task complexity, tell
them who is going to be involved, who is there to help out in case of road
blocks.

In general make sure to be a role model for the behaviour you want in your
project: Ask questions yourself, step back when your have taken on too much,
appreciate people stepping back.

Understand the motivation of your new comers - try to talk to them one on one
to understand their motivation and help to align work on the project with their
life goal. When starting to delegate, start with tasks that seem to small to
delegate at all to get new people familiar with the process - and to get
yourself familiar with the feeling of giving up control. Usually you will need
to pull tasks apart that before were done by one person. Don’t look for a
person replacement - instead look for separate tasks and how people can best
perform these.

Make visible and clear what you need: Is it code or reviews? Documentation or
translations, UX helpers? Incentivise what you really need - have code sprints,
gamify the process of creating better docs, put the logo creation under a
challenge.

All of this is great if you have only people who all contribute in a very
positive way. What if there is someone who’s contributions are actually
detrimental to the project? How to deal with bad people? They may not even do
so intentionally… One option is to find a task that better suits their
skills. Another might be to find another project for them that better fits
their way of communicating. Talk to the person in question, address head on
what is going on. Talking around or avoiding that conversation usually only
delays and enlarges your problem. One simple but effective strategy can be to
tell people what you would like them to do in order to help them find out that
this is not what they want to do - that they are not the right people for you
and should find a better place.

More on this can be found in material like “How assholes are killing your
project” as well as the “Poisonous people talk” and the book “Producing
open source software”.

On the how of dealing with bad people make sure to criticise privately first,
chack in a backchannel of other committers for their opinion - otherwise you
might be lonely very quickly. Keep to criticising the bahaviour instead of the
person itself. Most people really do not want to be a jerk.

mainec Apache Con ApacheCon, ApacheConNA, community