Archive

Archive for August, 2012

FrOSCon 2012 - REST

August 29th, 2012 at 7:33pm

Together with Thilo I went to FrOSCon last weekend. Despite a few minor glitches and the “traditional” long BBQ line the conference was very well organised and again brought together a very diverse crowd of people including but not limited to Debian developers, OpenOffice people, FSFE representatives, KDE and Gnome developers, people with background in Lisp, Clojure, PHP, Java, C and HTML5.

The first talk we went to was given by JThijssen on REST in practice. After briefly introducing REST and going a bit into Myths and false believes about REST he explained how REST principles can be applied in your average software development project.

To set a common understanding of the topic he first introduced the four steps REST Maturity Model: Step zero means using plain old xml over http for rpc or SOAP. Nothing particularly fancy here - even to some extend breaking common standards related to http. Going one level up means modeling your entities as resources. Level two is as simple as using the http verbs for what they are intended - don’t delete anything on the other side just by using a GET request. Level three finally means using hypermedia controls, HATEOS and providing navigational means to decide on what to do next.

Myths and legends

Rest is always http - well, it is transport agnostic. However mostly it using http for transport.

Rest equals CRUD - though not designed for that it is often used for that task in practice.

Rest scales - as a protocol yes, however of course that does not mean that the backend you are talking to does. All Rest does for you is to give you a means to horizontally scale without having to worry too much about server state.

Common mistakes

Using Http verbs - if you’ve ever dealt with web crawling you probably know those stories of some server’s content being deleted just be crawling a public facing web site just because there was a “delete” button somewhere that would trigger a delete action through an innocent looking GET request. The lesson learnt of those: Use the verbs for what they are intended to be used. One commonly confused thing is the usage of PUT vs. POST. Common rule of thumb that also applies to the CouchDB REST API: Use PUT if you know what the resulting URL should be (e.g. when storing an entry to he database and you know the key that you want to use). Use POST if you do not care about which URL should result from the operation (e.g. if the database should automatically generate a unique key for you). Also make sure to use the error codes as intended - never return error code 2?? only to add an xml snippet to the payload that explains to the surprised user that an error occurred including an error code. If you really need an explanation of why this is considered bad practice if not plain evil, think about caching policies and related issues.

When dealing with resources a common mistake is to stuff as much information as possible into one single resource for one particular use case. This means transferring a lot of additional information that may not be needed for other use cases. A better approach could be to allow clients to request custom views and joins of the data instead of pre-generating them.

When it comes to logging in to your API - don’t design around HTTP - use it. Sure you can give a session id into a cookie to the user. However than you are left with the problem of handling client state on the server - which was supposed to be stateless so clients can talk to any server. You could store the logged in information in the client cookie - signing and encrypting that might even make it slightly less weird. However the cleaner approach would be to authenticate individual requests and avoid state altogether.

When it comes to URL design keep in mind to keep them in a format that is easy to handle for caches. An easy check would be to try and bookmark the page you are looking at. Also think about ways to increase the number of cache hits if results are even slightly expensive to generate. Think about an interface to retrieve the distance from Amsterdam to Brussels. The URL could be /distance/to/from - however given no major road issues the distance from Amsterdam to Brussels should be the same as from Brussels to Amsterdam. One easy way to deal with that would be to allow for both requests but to send a redirect to the first version in case a user requests the second. The semantics would be slightly different when asking for driving directions - there the returned answers would indeed differ.

The speaker also introduced a concept for handling asynchronous updates that I found interesting: When creating a resource hand out a 202 accepted response including a queue ticket that can be used to query for progress. For as long as the ticket is not yet being actively dealt with it may even contain cancellation methods. As soon as the resource is created requesting the ticket URL will return a redirect to the newly created resource.

The gist of the talk for me was to not break the Rest constraints unless you really have to - stay realistic and pragmatic about the whole topic. After all, most likely you are not going to build the next Twitter API ;)

Hacking ,

Video: Stefan Hübner on Cascalog

August 28th, 2012 at 8:49pm

Video: “Accessing Hadoop data with HCatalog and PostgreSQL”

August 20th, 2012 at 8:53pm

Open Source Meetup Berlin

August 19th, 2012 at 8:22pm

This evening the (to my knowledge first) Berlin Open Source Meetup took place at Prater (Bier-)garten in Berlin. There are lots of project specific meetings, a monthly Free Software meeting, quite some stuff on project management. However this was one of the rare occasions where you get Linux kernel hackers, Wikidata project members, Debian developers, security people, mobile developers as well as people writing on free software or making movies related to the topic around one table.

Despite the heat (over 30 degrees Celcius in Berlin today) over 30 people gathered for some food, cold beer, some drinks and lots of interesting discussions. Would be great to see another edition of this kind of event.

Event , ,

Spotted this morning…

August 16th, 2012 at 10:03pm

in front of my office:

Ever wondered how accurate navigable map data for your Garmin, your in-car navigation system (most likely), or maps.nokia.com are created? One piece of the puzzle is the car above collecting data for Navteq, a subsidary of Nokia.

Hacking , ,

Apache Hadoop Get Together Berlin - August 2012

August 15th, 2012 at 11:30pm

Despite beautiful summer weather roughly 50 people gathered at ImmobilienScout24 for the August 2012 edition of the Apache Hadoop Get Together (Thanks again for hosting the event and sponsoring drinks and pizza to ImmoScout as well as to David Obermann for organising the meetup.

Today there were three talks: In the first presentation Dragan Milosevic (also known from his talk at the Hadoop GetTogether and his presentation at Berlin Buzzwords) provided more insight as to how Zanox is managing their internal RPC protocols in particular when it comes to versioning and upgrading protocol versions. Though in principle very simple to do this sort of problem still is very common when starting to roll out distributed systems and scaling them over time. The concepts he described were not unlike what is available today in projects like Avro, Thrift or protocol buffers. However by the time they needed versioning support for their client server applications neither of these projects was a really good fit. This also highlights one important constraint: With communication being a very central component in distributed systems, changing libraries after an implementation went to production can be too painful to be followed through.

In the second presentation Stefanie Huber, Manuel Messner and Stephan Friese showed how Gameduell is using Hadoop to provide better data analytics for marketing, BI, developers, product managers et.al. Founded in 2003 they have a accumulated quite a bit of data consisting of micro transactions (related to payment operations), user activities, gaming results that need to be used for balancing games. Their team turned a hairy, complex system into a pretty clean, Hadoop based solution: By now all actions end up in a Hadoop cluster (with an option to subscribe to a feed for realtime events). Typically from there people would start analysis jobs either in plain map reduce or in pig and export the data to external databases for further analysis by BI people who preferred Hive as a query language as it is much closer to SQL than any of the alternatives. As of late they introduced HCatalog to support providing a common view on data for all three analysis options - in addition to allowing for a more abstract view of the data available that does not require knowing the exact filesystem structure to access the data.

After a short break in the last talk of the evening Stefan Hübner introduced Cascalog to the otherwise pretty Java-savvy crowd. Being based on Cascading Cascalog provides for a concise way of formulating queries to a Hadoop cluster (compared to plain map reduce). Also when contrasted with Pig or Hive what stands out is the option to easily and seemlessly integrate additional functions (both map- and reduce-side) into Cascalog scripts without switching languages or abstractions. Note: When testing Cascalog scripts, one project to look at is Midje.

Overall a really interesting evening with lots of new input, interesting discussions and new input. Always amazing to see what other big data applications people in Berlin are developing. It’s awesome to see so many development teams adopt seemingly new technologies (some even still in the Apache Incubator) for production systems. Looking forward to the next edition - as well as to the slides and videos of today’s edition.

Get Together , , , , , ,

Data Scientists - researchers’ persectives

August 3rd, 2012 at 8:35pm

“Data scientist” as a term has caught quite some attention as of late (together with all the big data, scalability and cloud hype). Instead of re-hashing arguments seen in other sources I thought it might make more sense to link to a few of the thought provoking posts I came across recently.

Science , ,

On Reading Code

August 2nd, 2012 at 3:14pm

“If you don’t have time to read, you don’t have the time or the tools to write.” –Stephen King

Quite a while ago GeeCon published the video taped talk of Kevlin Henney on “Cool Code”. This keynote is great to watch for everyone who loves to read code - not the one you encounter in real world enterprise systems - but the one that truely teaches you lessons:

GeeCON 2012: Kevlin Henney - Cool Code from GeeCON Conference on Vimeo.

Hacking , , ,

Apache Con returns to Europe

August 1st, 2012 at 8:41pm

In November Apache Con will come back to Europe. The event will take place in Sinsheim inviting foundation members, project committers, contributors and users to meet, discuss and have fun during the one week event.

Several meetups will be held the weekend before the main conference kicks off, watch out for announcements on your favourite project mailing list.

ApacheCon is still open for submissions until August 3rd - head over to the Call for submissions for more information. The conference is split into several tracks that are being handled individually: Apache Daily - Tools frameworks and components used on a daily basis, Apache Java Enterprise projects, Big Data, Camel in Action, Cloud, Linked Data, Lucene, Modular Java Applications, NoSQL Database, OFBiz (The Apache Enterprise Automation project), Open Office and finally Web Infrastructure (covering HTTPD, TomCat and Traffic Server, the heart of many Internet projects).

Make sure to mark the date in your calendar to meet with the people behind the ASF projects, learn more on how the foundation works and what makes Apache projects so particular compared to others. Join us for a week of fun and dense talks on all things Apache.

The Apache Feather logo is a trademark of The Apache Software Foundation.

Apache Con , , ,