Application Building For The Rest Of Us?

I meet lots of people at the University who want technology to help them with their work. With some people it's often a case of introducing them to something we already offer and/or guiding them through doing things a new way.

But often what people really want is bespoke custom software development. They don't want an off-the-shelf technology, because in many cases, it just isn't a good enough fit. They don't want a wiki or a blog but instead they might need a simple web app that does a simple thing.

The Problem With Custom Development

I'm a big fan of the idea of everyone being able to make their own software, not because I want everyone to become geeks, but instead I want making software to become easy enough for everyone to be able to do it.

One of the problems with developments is knowing where to start. At the moment the University offers a MySQL, CGI and PHP service which allows you to work within certain parameters. With these tools you could, in theory create quite a complex web application. Hooking your application into the University's authentication system for log ins might be problematic, other data sources such as a list of students studying English, more so.

The main problem with this approach is that it is so damn hard. You can't get away from the fact that writing web applications is difficult, it requires a knowledge of web servers, databases, programming languages, unix and HTML. And that is just to do something simple, before you introduce collaborative coding, caching, task logs, error reporting and the like.

The problem here, as I see it, is that whilst writing software, at first can seem simple enough, it's a trap. Before you know it, you're having to learn about very geeky things like data modelling or caching models just to do even simple things.

But there can many other problems with custom development from the University perspective...

• Keeping lots of small "solutions" working can be a nightmare as you upgrade part of the eco-system. It gets messy.
• Who has the responsibility to maintain each piece of code? It gets messy.
• Often, even well run projects can start simple and manageable and end in complexity and ultimate failure. It gets messy
• Everyone has different ways of working with different tools, and different needs for collaboration and backups and resilience. It gets really messy.

The answer to all this custom development messiness, might be to devolve development, or at least some of it, to people who want it to happen, you.


There are lots of good reasons for taking a devolved development development approach ...

• Messiness becomes localised. Rather than having to deal with the symptoms of messiness ( a locked down system, with a bottleneck in development etc ) you get to live with the mess you made. 
• You get to develop what you want, your way.

There are of course downsides to this approach ...

• The University could end with a devolved mess of processes and technology that NOBODY can understand. It could get messy.

So there is the problem. People often want really quick and simple custom development but the current approach to this is messy, and a solution to the messiness is a different kind of potential messiness. It's a messy never ending vortex that could do with tidying up a bit.

Is The Solution Google App Engine?

Google App Engine is an "in the cloud" web development and hosting system. It means that you can create a web application on your computer and then, when it's finished, run it and make it available to everyone at York.

It's not currently enabled for York at the moment as, because instead of paying a flat fee to have your application hosted, Google App Engine charges by bandwidth/ database usage/hits and so is currently disabled for york.ac.uk.

So why could Google App Engine (GAE) be useful at York? 

Well, in many ways GAE doesn't make the whole development process much easier. GAE isn't a point-and-click-to-create-an-application solution, you still need to know how to code ( in python or java) but it does make lots of the often forgotten parts of the development process simpler. 

For example,

• The database GAE uses is more like a spreadsheet than MySQL. This is so that if your database needs 20 million entries - it will still work. Google take care of scaling for you. And charge.
• If your web application gets very popular, with thousands of hits a second, Google will take of scaling for you. And charge.
• GAE can use simple permissions to make your app only available to York staff. Smaller applications are free.
• Lots of the complexities of creating a web application, like creating task queues, are already built in.

So, if you are already familiar with creating Django web application, then not much in GAE will be new.

But I Don't Want To Be A Programmer Tom!

You say that but I don't believe you. I take a wider view that everyone should be able to shape and control these computer things to do what we want.

It's really not your fault that currently "programming a computer" can be quite a difficult thing at times, which is why when I see anything that levels the playing field in technology terms I can get quite enthusiastic.

I get excited, not because it means there'll be a heap less mess in IT Services but because my experience shows me that if you give people good enough tools, they do amazing things with them.

What I tried to do

My aim was to quickly create a very simple aggregator in Google App Engine that would would pull together any (blog) feeds from  people at the University of York. There isn't (in my opinion) a good aggregator out there that doesn't require some sensitive hosting and effort, so I thought that the work involved might be minimal ( how hard could it be? )...

I thought it would mean that we ( meaning me, the Web Office and Anthony ) could have total control over the look and feel and maybe roll in a few "innovations" like tag clouds, maps perhaps, or search phrase based RSS feeds, so that "if anyone blogs about genomes at the university then add it to my news feed". That feature alone would be really useful in selling the usefulness of Google Reader.

What I actually did

I downloaded the SDK, read the documentation very carefully and set about getting myself a setup that I could envisage someone else ( as dim as I from a geeky perspective ) using.

Get The Right Editor

I quickly ditched using a text editor with the GoogleAppEngineLauncher.app because if you use the Eclipse based Aptana editor, not only do you get code completion but you also get step-thru debugging!

MacOS X users. When installing Aptana and creating an application this is currently (probably) the path to where the SDK is located. The tutorials I used were out of date...

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine

Found a "Bug" in Feedparser

This was almost a show stopper. There are lots of people complaining about it. The problem is that if you try to parse an RSS feed from Blogspot then Feedparser blows up. I discovered that, if you did this...

f = feedparser.parse(self.url)

...rather than this...

content = unicode(result.content, 'utf-8')

f = feedparser.parse(content)

... that it works fine. I only mention it because someone might find it useful. Being able to read blog feeds is kind of essential to an aggregator.

Got Into A Mess

So. This is the thing. 

If I am right, that Google AppEngine is a contender for people at the University to use it to develop small applications, then I need to be able to understand it to a level where I would be happy teaching it. Perhaps to people with little programming experience.

1. Google App Engine doesn't use a "normal" MySQL database, it uses something similar called the datastore. This is good and bad.

Different. Searching. Paging. Cost -> Caching. 

De-normalized the tag's count ( of entries using that tag ).

The End Result

The end result, which may or may not be running, is available here http://uoynews.appspot.com/ .

It's a simple aggregator that should update feeds, could be made to look very pretty, and could have some cute features ( like search query RSS feeds mentioned earlier ).

BUT, I have already had to think about costing and caching and database writes and have no idea how to implement text searching ( in a way that won't affect costing and caching and database writes ). I'd used up my quota for the day of database writes and only I was using the application!

In short, in trying to get to the simple stuff ( making a quick application ) I've had to both learn, and design around some quite hairy concepts. It's not quite the simple application making framework I'd hoped for. I think what I was looking for was Django-Lite™but 

It's knocked my belief that there could be an application creating tool for "the rest of us" because if anyone was going to create one, it'd be Google.

I feel a bit sad really. I really wanted App Engine to be a place where anyone could lash their ideas into something hosted online quickly, but the process felt all too familiar to someone that isn't that great at coding... It was bitty, messy and complex.

I've created the thing I wanted to create but I'm not proud of it. It's only 300 lines of python and a few template files but it already feels cumbersome and weighed down by the nitty gritty of geekery.