Inductive Bias

In his talk on Spdy Mathew Steele explained how he implemented the spdy protocol
as an Apache httpd module - working around most of the safety measures and
design decisions in the current httpd version. Essentially to get httpd to
support the protocol all you need now is mod_spdy plus a modified version of
mod_ssl.

The keynote on the last day was given by the Puppet founder. Some interesting
points to take away from that:

• Though hard in the beginning (and half way through, and after years) it
  is important to learn giving up control: It usually is much more productive and
  leads to better results to encourage people to do something than to be
  restrictive about it. A single developer only has so much bandwidth - by
  farming tasks out to others - and giving them full control - you substantially
  increase your throughput without having to put in more energy.
  


• Be transparent - it's ok to have commercial goals with your project. Just
  make sure that the community knows about it and is not surprised to learn about
  it.
  


• Be nice - not many succeed at this, not many are truely able to ignore
  religion (vi vs. emacs). This also means to be welcoming to newbies, to hustle
  at conferences, to engage the community as opposed to announcing changes.
  

Overall good advise for those starting or working on an OSS project and seeking
to increase visibility and reach.

If you want to learn more on what other talks were given at ApacheCon NA or want to follow up in more detail on some of the talks described here check out the slides archive online.

Monitoring - a task generally neglected - or over done - during development.
But still vital enough to wake up people from well earned sleep at night when
done wrong. Rainer Jung provided some valuable insights on how to monitor Apache httpd and Tomcat.

Of course failure detection, alarms and notifications are all part of good
monitoring. However so is avoidance of false positives and metric collection,
visualisation, and collection in advance to help with capacity planning and
uncover irregular behaviour.

In general the standard pieces being monitored are load, cache utilisation,
memory, garbage collection and response times. What we do not see from all that
are times spent waiting for the backend, looping in code, blocked threads.

When it comes to monitoring Java - JMX is pretty much the standard choice. Data
is grouped in management beans (MBeans). Each Java process has default beans,
on top there are beans provided by Tomcat, on top there may be application
specific ones.

For remote access, there are Java clients that know the protocol - the server
must be configured though to accept their connection. Keep in mind to open the
firewall in between as well if there is any. Well known clients include
JVisualVM (nice for interactive inspection), jmxterm as a command line client.

The only issue: Most MBeans encode source code structure, where what you really
need is change rates. In general those are easy to infer though.

On the server side for Tomcat there is the JMXProxy in Tomcat manager that
exposes MBeans. In addition there is Jolohia (including JSon serialisation) or
the option to roll your own.

So what kind of information is in MBeans:

• OS - load, process cpu time, physical memory, global OS level
  stats. As an example: Here deviding cpu time by time geves you the average cpu
  concurrency.
  



• Runtime MBean gives uptime.
  


• Threading MBean gives information on count, max available threads etc
  


• Class Loading MBean should get stable unless you are using dynamic
  languaes or have enabled class unloading for jsps in Tomcat.
  


• Compliation contains HotSpot compiler information.
  


• Memory contains information on all regions thrown in one pot. If you need
  more fine grained information look out for the Memory Pool and GC MBeans.
  

As for Tomcat specific things:

• Threadpool (for each connector) has information on size, number of busy
  threads.
  


• GlobalRequestProc has request counts, processing times, max time bytes
  received/sent, error count (those that Tomcat notices that is).
  


• RequestProcessor exists once per thread, it shows if a request is
  currently running and for how long. Nice to see if there are long running
  requests.
  


• DataSource provides information on Tomcat provided database connections.
  

Per Webapp there are a couple of more MBeans:

• ManagerMBean has information on session management - e.g. session
  counter since start, login rate, active sessions, expired sessions, max active
  sinse restart sessions (here a restart is possible), number of rejected
  sessions, average alive time, processing time it took to clean up sessions,
  create and required rate for last 100 sessions
  


• ServletMBean contains request count, accumulated processing time.
  


• JspMBean (together with activated loading/unloading policy) has
  information on unload and reload stats and provides the max number of loaded
  jsps.
  

For httpd the goals with monitoring are pretty similar. The only difference is
the protocol used - in this case provided by the status module. As an
alternative use the scoreboard connections.

You will find information on

• restart time, uptime
  


• serverload
  


• total number of accesses and traffic
  


• idle workers and number of requests currently processed
  


• cpu usage - though that is only accurate when all children are stopped
  which in production isn't particularly likely.
  

Lines that indicate what threads do contain waitinng, request read, send reply
- more information is documented online.

When monitoring make sure to monitor not only production but also your stress
tests to make meaningful comparisons.

Tuesday started early with a plenary - run by the sponsor, not too many news there, except for the very last slide that raised a question that is being discussed often also within the ASF - namely how to define oneself compared to non-ASF projects. What is the real benefit for our users - and what is the benefit for people to go with the ASF. The speaker concentrated on pointing out the difference to github. Yes tooling changes are turning into a real game changer - but that is nothing that the foundation could not adopt over time. What I personally find interesting is not so much what makes us different from others but more what can be learnt from other projects - not only on github but also in a broader scope from the KDE, Python, Gnome, Debian, Open/Libre-Office communities, from people working on smaller but non-the-less successful projects as well as the larger foundations, maybe even the corporate driven projects. Over time lots of wisdom has accumulated within and outside of the foundation on how to run successful open source projects - now the question is how to transfer that knowledge to relatively young projects and whether that can help given the huge amount of commercial interest in open source - not only in using it but also in driving individual projects including all the benefits (more people) and friction around it.

The first talk I went to was an introduction to was Rainer Jung’s presentation on the new Apache httpd release. Most remarkably the event mpm available as “experimental” feature is now marked as default for the Apache distribution - though it is not being used for ssl connections. In addition there is support for async write completion, better support for sizing and monitoring. In particular when sizing the event mpm the new scoreboard comes in handy. When using it, keep in mind to adjust the number of allowed open file handles as well.

In order to better support server-to-client communication there is html5 web socket standardisation on it’s way. If you are interested in that check out the hybi standardisation list. Also taking a look at the Google SPDY could be interesting.

Since 2.4 dynamic loadable modules are supported and easy to switch. When it comes to logging there is now support for sub second timestamp precision, per module log levels. Process and thread ids are kept in order to be able to untwist concurrent connection handling. There are unique message tokens in the error log to track requests. Also the error log format is configurable - including trace levels one to eight, configurable per directory, location and module.
They’ve added lots of trace messages to core, correlation ids between error and access log entries (format entry %L). In addition there is a mod_log_debug module to help you log exactly what you want when you want.

Speaking of modules - in order to upgrade them from 2.2 to 2.4 it’s in general sufficient to re-compile. With the new version though not all modules are going to be loaded as default anymore. New features include dynamic configurations based on mod_lua. AAA was changed again, there are filters to rewrite content before it’s sent out to clients (mod_substitute, mode_sed, mod_proxy_html). mod_remoteip helps to keep the original ip in your logs instead of the procy ip.

When it comes to documentation better check the English documentation - or better yet provide patches to it. mod_rewrite and mod_proxy improved a lot. In addition the project itself now has a new service for it’s users: via comments.apache.org you can send documentation comments to the project without the need to register for a bugzilla account and provide documentation patches. In addition there is now syntax highlighting in the documentation. One final hint: the project is very open and actively looking for new contributors - though they may be slow to respond on the user and dev list - they definitely are not unfriendly ;)

The first Apache Con day featured several presentations on NoSQL databases (track sponsored by Day software), a Hadoop track as well as presentations on Httpd and an Open source business track.

Since its inception Hadoop always was intended to be run in trusted environments firewalled from hostile users or even attackers. As such it never really supported any security features. This is about the change with the new Hadoop release including better Kerberos based security.

When creating files in Hadoop a long awaited feature was append support. Basically up to now writing to Hadoop was a one-of job: Open a file, write your data, close it and be done. Re-opening and appending data was not possible. This situation is especially bad for HBase as its design relies on being able to append data to an existing file. There have been efforts for adding append support to HDFS earlier as well as an integration of such patches by third party vendors. However only with a current Hadoop version Append is officially supported by HDFS.

A very interesting use case-wise of the Hadoop stack was presented by $name-here from $name. They are using a Hadoop cluster to provide a repository of code released under free software licenses. The business case is to enable corporations to check their source code against existing code and spot license infringements. This does not only include linking to free software under incompatible licenses but also developers copying pieces of free code, e.g. copying entire classes or functions into internal projects that originally were available only under copyleft licenses.

The speaker went into some detail explaining the initial problems they had run into: Obviously it's no good idea to mix and match Hadoop and HBase versions freely. Instead it is best practice to use only versions claimed to be compatible by the developers. Another common mistake is to leave parameters of both projects at their defaults. The default parameters are supposed to be fool-proof. However they are optimised to work well for Hadoop newbies who want to try out the system on a single node cluster and in a distributed setting obviously need more attention. Other anti-patterns include storing only tiny little files in the cluster thus quickly running out of memory on the namenode (that stores all file information including block mappings in main memory for faster access).

In the NoSQL track Jonathan Grey from Facebook gave a very interesting overview on the current state of HBase. Turns out that Facebook would announce only a few days after that their internal use of HBase for the newly launched feature of Facebook messaging.

HBase has adopted a release cycle including development/ production releases to get their systems into interested users' hands more quickly: Users willing to try out new experimental features can use the development releases of HBase. Those who are not should go for the stable releases.

After focussing on improving performance in the past months the project is currently focussing on stability: Data loss is to be avoided by all means. Zookeeper is to be integrated more tightly for storing configuration information thus enabling live reconfiguration (at least to some extend). In addition also HBase is targeting to integrate stored procedures like behaviour: As explained in Googles Percolator paper $LINK batch oriented processing get's you only so far. If data that gets added constantly it makes sense to give up on some of the throughput batch-based systems provide and instead optimise for shorter processing cycles by implementing event triggered processing.

On recommendation of one of neofonie's sys-ops I visited some of the httpd talks: First Rich Bowen gave an overview of unusual tasks one can solve with httpd. The list included things like automatically re-writing http response content to match your application. There is even a spell checker for request URLs: Given marketing has given your flyer to the press with a typo in the url, chances are that the spellchecking module can fix these automatically for each request: Common mistakes covered are switched letters, numbers replaced by letters etc. The performance cost has to be paid only in case no hit could be found – so instead of returning a 404 right away the server first tries to find the document by taking into account common mis-spellings.