Google App Engine

As you may have noticed last week, we announced the acquisition of a company called IMified and with that we brought in more opportunities and options for developers. Now you can create instant messaging “bots” (or “agents” or whatever you want to call them) that allow you to create applications that interact with users via IM. (More about what you can do with IMified in the acquisition announcement.)

Developer accounts are free over on www.imified.com. Please do sign up and check it out. We’d love to hear what you think of the service. (And you can safely assume that we’ll be evolving the service and adding more to it over the months ahead.)

One aspect of IMified that is a bit different from either Voxeo’s Evolution XML developer portal or our Tropo.com site is that with IMified you do need to host your application on a web server somewhere. With both Evolution and Tropo, you can host your application on your own server and simply point Evolution or Tropo to the URL of your app, but there is also the option with both services of hosting your apps directly within our hosted infrastructure. With IMified you do need to host your application somewhere and then provide the URL inside IMified when configuring your bot.

The beautiful thing about this, of course, is that it means you can write the bot application in whatever language you want on whatever operating system you want and using whatever tools you want. All your application has to do is use the IMified API to communicate with the IMified platform.

One place developers can host a web services app these days is certainly Google App Engine, and the IMified team just posted a knowledge base article showing an example bot in python hosted on Google App Engine. Courtesy of an IMified user named Barry, the source code is available in the IMified knowledge base. The IMified team notes that it’s also a good example of performing HTTP Authentication in python.

(And rather than include the source code here, I’ll just point you over to the IMified site.)

We’re excited about adding IMified to the Voxeo family of services and we’re definitely looking forward to seeing what people develop on the platform.

If you found this post interesting or helpful, please consider either subscribing via RSS, becoming a fan on Facebook or following us on Twitter.

Technorati Tags: im, imified, google, googleappengine, gae, python, voxeo

ShareThis

Since writing my set of function combinators for matching HTTP requests I’ve realised that there are better approaches.

While the combinators read nicely, once you start extracting attributes from the request things become messy, due to the hAnd combinator composing results as nested pairs.

Here’s an example:

skitchPutHandler =
    let
        storeImage p =
            let (p2, path) = p;
                (maybeContentType,imageData) = p2;
                ...
            in ...
    in
        matchMethod "PUT" `hThen` matchUrl "/skitch" `hThen`
            (getHeader "Content-Type") `hAnd` requireRawData `hAnd` requirePathInfo `hApply` storeImage;

Unpacking the values extracted from the request is cumbersome, and because many are Strings, error prone.

A better approach is to simply use the Maybe monad with the same extraction functions:

simpleSkitchPutHandler req =
    let
        maybeContentType = requireHeader "Content-Type" req;
        ...
    in
        matchMethod "PUT" req `anonymousBind` matchUrl "/skitch" req `anonymousBind`
            requireRawData req `bind` (\rawData ->
            requirePathInfo req `bind` (\path ->
            return $ storeImage path rawData (actualContentType path)));

Optional values are extracted in let expressions, and required values and tests are composed with the monadic bind operators in the main expression. You can make the function a bit more concise and ‘point free’ by writing a monad which composes the Maybe and Reader monads to supply the HTTP request parameter to functions implicitly:

readerMonadSkitchPutHandler =
    let
        ...
    in
        runHttpRequestM (
        method "PUT" `anonymousBind` matchUrlM "/skitch" `anonymousBind`
            rawData `bind` (\rawData ->
            pathInfo `bind` (\path ->
            (optional (header "Content-Type")) `bind` (\maybeContentType ->
            return $ storeImage path rawData (actualContentType path maybeContentType)))));

The function optional has the type HttpRequestM a -> HttpRequestM (Maybe a), that is, it allows a function to succeed by wrapping its result in a ‘Just’.

I’m not convinced that the improvement in readability gained by removing the request parameter is worth the increase in complexity.

So I’ll remove the combinators from my Http module and will rewrite Hangman using just the Maybe monad. I’ll keep the more complicated Monad on the back-burner in case I discover a compelling advantage.

I’d be interested to know what someone with a deeper understanding of functional programming would say.

A hobbyist's code of conduct to refactoring

The cardinal rule: Don't be a jerk

Corollary #1: If it ain't broke, don't fix it.

• The code has evolved historically towards the way it currently is, and it has never been too much of a problem.
  
• The project team has a different philosophy or architectural view towards development than you. Happens all the time, and that doesn't make it bad code. Feel free to ask the team why it is the way it is, but do not expect them to change it for you.
  
• You ran into a bug or an area with "TODOs". If it's an issue and the team agrees, feel free to improve it -- as long as you do it with the consent of the others.
  

Corollary #2: Honor the code's spirit and document the intentions of changes you make.

• changes that would make sense -- if the author had only documented why he or she is doing them
• too many concurrent modifications in one iteration
• combining a refactoring with writing new features
• incomplete or speculative refactorings (in other words, starting a change because "it might come in handy later" but never following through on that)
• major changes to the contract of a core class that require code changes throughout the project
• random changes just for the heck of it, or because one does not like the way a particular piece of code looks (see corollary #1). While it sometimes makes sense to clean up something small while you're in a particular class anyway, there is a fine line between cleanup and major modifications.

Corollary #3: Contributorship does not imply ownership

Corollary #4: Failure IS an option

The overall goal of my refactorings

• spawning threads
• network communication through anything but http
• storing data in files
• keeping data in a static field and expecting it to stick around (the next request might hit a completely different instance of my App Engine app)
• connecting via JDBC to a database
• longer living background processes
• anything that requires a "restart" of the system
• anything equivalent to a "global scan" (will the client ever need to get the list of all users?)
• anything with near-realtime requirements that relies on the accuracy of timestamps
• anything that uses native code or expects certain operating system commands to be available
• code that uses any class that may not be whitelisted
  

Today's refactoring in pictures

• a main class (JogreServer) creates a ServerSocket and listens in a loop to incoming connections
• for each incoming connection, it starts a new Thread (ServerConnectionThread) that then handles all the gaming logic.

"Every class should have a single responsibility: It should have a single purpose in the system, and there should be only one reason to change it."

• instead of managing sockets and threads, an AbstractConnectionThread is connected to a generic MessageBus object, which may choose any transport protocol (sockets, udp, http) it likes. The class no longer has any dependencies on sockets.
  
• AbstractConnectionThread has two remaining responsibilities: to manage the name of the user that the connection belongs to, and to provide a generalization for how to react to incoming messages from the MessageBus. The latter responsibility is represented by the MessageParser interface, which is what the MessageBus interfaces with.
• concrete subclasses like ServerConnectionThread still accept Sockets in their constructor (thus remaining compatible with the rest of the codebase), but they wrap the socket in a MessageBus before pushing it into the base constructor. I might choose to refactor that in a later stage, but if I do so, I can do it on an individual class basis, without breaking any of the other peer classes.
  

The refactoring in code

• AbstractConnectionThread (refactored)
• MessageBus (new)
• SocketBasedMessageBus (new)

• AdminClientConnectionThread
  
• AdminConnectionPanel
  
• AdminConnectionThread
• ClientConnectionThread
• GameConnectionPanel
• JogreLogger
  
• JogreServer
  
• ServerConnectionThread
  
• ServerGameController
  
• TestConnectionThread

• It should be as far away from my day job as it could possibly be, yet still involve Google App Engine.
• It should force me to look at the tools from a different angle and broaden my perspective.
• It should be about something I can "occasionally" mention in this blog and that people will hopefully enjoy reading.
• It should be something that I find fun to do.
  

Even if Google happens to stay out of the market, its products will help lower the entry-level for this industry even more. By using its products, a new breed of systems, based on Internet technologies and open standards will reach the market. Greater competition will improve the overall quality of the solutions offered, and the end user is going to benefit.

• A member of the team behind BuddyPoke shared some lessons learned behind his secret to scaling to millions of users. Unfortunately, I do not remember them all, but two of them were the use of sharding techniques for things like counters and to avoid using queries wherever possible. According to the presenter, BuddyPoke gets its data from the store almost exclusively through loading by primary key, and they avoid putting indexes on data wherever possible (<shameless-plug>if you'd like to use similar techniques in Java, check out my previous posts on a simple, key-based datastore api and efficient global counters. You can also download the sourcecode directly from this open source project.</shameless-plug>).
• Another developer is working on jiql, a JDBC driver for App Engine. In the long term, projects like this could be essential to ease of porting sql-based applications to App Engine, so I think it's worth checking out. I'd love to hear especially if anyone is using it to porting things like ActiveRecord in Rails or php-based stuff...
• There was a very nice demo on patching django-based applications to use the Jinja2 templating system, including a real life example on how it simplified life in a web application that the developer was building.

Version 1.2.1 of the App Engine SDK for Java is now available to download. This was the first update to our new Java language support and includes a number of bug fixes and feature additions including:

• Support for unindexed datastore properties of arbitrary types
• Embedded UTF-8 characters in JSPs now render correctly
• Increase in response size limit from 1MB to 10MB
• Support for Thread.setContextLoader()

There are a host of updates for the JDO/JPA layer as well. The project wiki has a full list of fixes and new features.

As always, we're very interested in your feedback with the SDK, especially during this early look period -- please feel free to share your experiences in the Java runtime discussion group.

Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.

Posted by Jason Cooper, App Engine Team

Introduction

My implementation of Hangman has only very simple data storage requirements, so my CAL module Datastore doesn’t cover the entire capabilities of Bigtable.

All the application needs is to be able to save a Game instance, put the key of that instance into a cookie in the user’s browser, and then retrieve and update that instance as the user makes each guess, or delete it if the user chooses to restart the game.

Important unexplored areas are:

• The creation of entities having another entity as a parent. This is important in Bigtable, because a transaction can only operate on entities in the same ‘entity group’ — that is, entities which share a common parent.

• The persistence of references to other entities, and special treatment of these references when performing operations on the entity containing the reference — that is, there is no support for modelling relationships between entities.

• No error handling support — any exception will result in a 500 response. In particular, ConcurrentModificationException should be treated differently, as this indicates that a transaction failed due to another transaction modifying the same records. A failure of this type should either be retried, or reported to the user in an application specific manner.

• The module is limited to storing types with a single constructor, although there is no difficulty in extending this to support multiple constructor ADTs.

It is implemented using the low level api, not JDO.

Datastore Module

The Datastore module provides a Monad instance for using the AppEngine data store, and a Storeable type class. The source is here.

The Storeable Type Class

Algebraic data types which need to be persisted must be instances of Storeable, which provides metadata needed to persist instances of the type. The metadata includes:

• The ‘kind’ of the record — analogous to the name of the table to store instances of the type in, not to kind in the type theory sense.

• The data store property names to use for each of the constructor arguments.

When using CAL in statically compiled mode the names of types and the names of constructor parameters are not easily available at runtime. The constructor arguments could simply be stored in properties named arg0...n, but this would make querying and debugging more difficult.

data public Store a = public Store kind :: String fields :: [String];

public class (Outputable a, Inputable a) => Storeable a where
    store :: a -> Store a;
    ;

Note that the type parameter a is not used in the declaration of the constructor arguments — it simply labels the store to ensure that the store used with, for example fromEntity below, is consistent with the type of Storeable we are expecting fromEntity to return. For this reason functions which construct a Store must be explicitly typed, like gameStore below, as otherwise no more specific type than Store a can be inferred.

So the instance for a game of hangman is:

data public Game = 
    private Game word :: String guesses :: String deriving Show, Inputable, Outputable;

gameStore :: Store Game;
public gameStore = Store "Org.Kablambda.AppEngine.Test.Game" ["word","guesses"];

instance Storeable Game where
    store = gameStoreA;
    ;

private gameStoreA a = gameStore;

The public gameStore function is used when we wish to query.

Converting between CAL values and AppEngine Entities

Two conversion functions are used, one in each direction. In both cases the bulk of the work is carried out in Java functions.

From a CAL value to an Entity

toEntity :: Storeable a => a -> Entity;
public toEntity !r =
    jObjectToEntity (store r).Store.kind (outputList $ (store r).Store.fields) (output r);

where:

foreign unsafe import jvm "static method org...jObjectToEntity" 
    public jObjectToEntity :: String -> JList -> JObject -> Entity;

The parameters passed to jObjectToEntity are the kind of entity to create, the list of names to use for the fields of the CAL value, and the CAL value, converted to a JObject by the output function. Storeable instances should all derive Outputable, so output r uses the default implementation of output to convert the algebraic data type instance to an instance of the CAL Java support class AlgebraicValue. The fields are transferred to the AppEngine entity like this:

Entity entity = new Entity(kind);
AlgebraicValue av = ...;
entity.setProperty(DC_NAME, av.getDataConstructorName());
entity.setProperty(DC_ORDINAL, av.getDataConstructorOrdinal());
for (String fieldName : fieldNames) {
    entity.setProperty(fieldName, av.getNthArgument(i++));
}

From an Entity to a CAL value

fromEntity :: Storeable a => Store a -> Entity -> a;
public fromEntity !s !e =
    input $ jEntityToJObject (outputList s.Store.fields) e;

where:

foreign unsafe import jvm "static method org...entityToJObject" 
   public jEntityToJObject :: JList -> Entity -> JObject;

jEntityToJObject reverses the process, creating an AlgebraicValue which is converted to a CAL value by the input function.

The Data Store Monad

Because data store operations operate via side effects we need to create a Monad instance to manage them. This is required or two reasons:

• We need to ensure that our data store operations happen in a definite sequence — our puts must happen before we commit the transaction, for example. Lazy evaluation won’t guarantee this without special attention.

• Some operation don’t produce results, or produce results which will be ignored, so lazy evaluation won’t cause these operations to happen at all.

We define the following functions for use with our Monad (the type of which is DSM):

put :: Storeable a => a -> DSM Key;

The put function creates a new record, returning the Key it was assigned by the data store.

update :: Storeable a => Entity -> a -> DSM Key;

The update function replaces a previously retrieved entity with a new value, returning the Key (which will always be the same as the old key). Note that the original Entity is used only for the value of its key — all the replacement fields come from the new Storeable value.

delete :: Entity -> DSM ();

The delete function deletes a previously retrieved entity.

find :: (Storeable a) => Store a -> Key -> DSM (Maybe (Entity,a));

The find function looks for a record using a Key and returns it or Nothing if it does not exist. The return value consists of a pair of values: the raw Entity and its value when converted to a Storeable instance. This allows update to be used later. An alternative design would have been to require Storeable instances to be able to store a key, but I think the current design better separates the concerns of data storage from the domain objects.

query :: (Storeable a, Outputable b) => Store a -> [(String,FilterOperator,b)] -> [(String,SortDirection)] -> DSM [(Entity,a)];

The query function returns a sorted list of (Entity,Storeable) pairs, based on a list of filter criteria applied to attributes of the records.

commit :: DSM ();

The runDSM function described below starts and commits the transaction within which the monad is being run, but if you need to operate on entities from more than one entity group you must commit the original transaction and start a new one using the commit function.

These functions are combined with the normal monad operators of bind and anonymousBind, and then the resulting function is ‘run’ with runDSM:

runDSM :: DSM a -> a;

Key Conversion

In the context of a web application we may wish to store keys in cookies or generated URLs. The following functions are provided for extracting Keys from Entity instances, and converting Keys to and from strings.

public keyToString :: Key -> String; 
public stringToKey :: String -> Key;
public getKey :: Entity -> Key;

Example

Suppose we want to retrieve a Game, add a letter to the set of guesses and store it again:

    updateGame :: Key -> Char -> Key;
    updateGame key letter =
        runDSM (find gameStore key `bind` (\p ->
            let (entity, game) = fromJust p;
            in update entity (addGuess letter game)
        ));

where addGuess returns a new game with the given letter added to its set of guesses. This function will terminate with an error if the Game is not found, that is, if find returns Nothing.

It has been an exciting month and a half for App Engine since the last community update. In addition to two SDK updates, Cron support, new Java runtime, etc., there have been a host of noteworthy community developments as well, only a few of which are mentioned below.

web2py Python framework now supports App Engine
We are always interested in frameworks that natively support App Engine, and the web2py framework recently came to our attention. Completely written in Python, web2py is described as a "free and open source full-stack enterprise framework for agile development of fast, secure and portable database-driven web-based applications." If your framework has native App Engine support let us know and we'll include you in a future blog post.

Alternative datastore backend for App Engine applications released
bdbdatastore is an alternative datastore backend for App Engine applications. Now at version 0.2, bdbdatastore has complete feature parity with the App Engine's production datastore, meaning you can run your App Engine application on your own hardware with bdbdatastore as the storage backend without having to change any of the code written previously to access App Engine's native datastore. Intended primarily for developers who want to host their own App Engine-based apps, bdbdatastore manages to be far more robust and scalable than the datastore included with the development server while being more manageable than larger backends like HBase and HyperTable.

App Engine supports White House town hall meeting
In late March, the White House hosted an online town hall meeting, soliciting questions from concerned citizens directly through its website. To manage the large stream of questions and votes, the White House used Google Moderator, which runs on App Engine. At its peak, the application received 700 hits per second, and across the 48-hour voting window, accepted over 104,000 questions and 3,600,000 votes. Despite this traffic, App Engine continued to scale and none of the other 50,000 hosted applications were impacted.

For more on this project, including a graph of the traffic and more details on how App Engine was able to cope with the load, see the Google Code blog.

App Engine "Sprint" held in Taipei
The Taipei App Engine Sprint was a coding event at Google Taipei office in mid-March. Seventeen Python developers split into five teams and worked for 10 hours straight to build a new application from scratch. in 10 hours. The winning application, Sight History, is a Picasa search-based application which displays a time distribution chart, slide show links, and a Google Map for photos queried from Picasa Web Albums. Only one of the four developers on this team was an active web developer, which illustrates how easy it can be to turn a good idea into a working web application with App Engine and other Google APIs.

Earth Hour 2009 site hosted on App Engine
Earth Hour, an initiative of WWF, encourages individuals, businesses, and governments to turn off their lights for one hour to show their support for action on climate change. The site aims to raise awareness of climate change and to show people and companies how easy it is to take action to avoid global warming. After launching this past December on App Engine, the site was able to scale seamlessly to handle over 450 hits per second during the Earth Hour event on March 28, making it one of the top App Engine applications served that day.

Posted by Jason Cooper, Google App Engine Team

• Rest in peace, Duke Nukem -- you have been one of my favorite college pastimes. As sad as it makes me, it shows however the importance of time to market: if your software project takes longer to develop than the atomic bomb, you might be in trouble. The Duke Nukem forever list has this and more examples of what has been accomplished in the same period of time.


• Here's a link to a recent talk called "what killed smalltalk" and the rant that made me aware of it ;-) Speaking of talks there is a thread on reddit where people recommend their favorite Google Tech Talks.


• If this has been way too serious for you, check out this Brief, Incomplete, and Mostly Wrong History of Programming Languages. My favorite fake history fact:

  Lambdas are relegated to relative obscurity until Java makes them popular by not having them.