Genie in a Bottle

The usual suspects for open source database systems were easily identified in the past, but these days, things have changed. Instead of MySQL or PostgreSQL, people are talking about CouchDB, FluidDB, and MongoDB. These databases have a feature scope similar to classical SQL servers but offer far better performance and scalability. Although they can't compare with systems simple key/value store systems such as Memcached, this disadvantage is easily offset by the better feature scope in combination with what is still excellent performance. On top of this, the performance of these systems scales well horizontally. Instead of needing to buy expensive hardware, you can easily distribute a database over multiple systems without affecting functionality.

MongoDB is already on board with any major Linux distribution. In combination with Python and a web framework, you can thus quickly achieve impressive results. In this example, I will be using the Bottle micro web framework [1]. This very simple framework comprises a single file, which you can drop onto your own system using easy_install.

The Bottle framework is perfect for initial tests with MongoDB because there are no dependencies to resolve, except for the Python standard library, and you still get a fair amount of functionality for your money. It supports templates and comes with its own web server. The interface between the MongoDB service and the Bottle web framework in this example is PyMongo [2], which is also available out of the box with most distributions.

MongoDB splits a database into collections, which in turn store documents. The documents contain the database objects in JSON format. The good point is that the collections are not governed by a schema. Thus, you can simply transfer the desired data to the database without worrying about modifying the schema, which makes life much easier. The individual fields in a document can then be modified to suit your needs, and you can define indexes for frequently queried fields.

For an initial test, I will be using the Mongo shell to create a document (Listing 1).

01 # mongo
02 MongoDB shell version: 2.2.0
03 connecting to: test
04 > use football
05 switched to db football
06 > db.clubs.save({ Name:"FC Schalke 04", Colors:"Blue-white", Members:111000,   Address:[{Street:"Ernst-Kuzorra-Weg 1", Zip: 45891, City: "Gelsenkirchen"}] })
07 > db.clubs.save({ Name:"St. Pauli", Colors:"Brown-white", Members:15000,   Address:[{Street:"Heiligengeistfeld 1", Zip: 20359, City: "Hamburg"}] })
08 > db.clubs.save({ Name:"FC Nürnberg", Colors:"Red-White",   Address:[{Street:"Valznerweiherstrasse 200", Zip: 90480, City: "Nuremberg"}] })

You can see from this example that the records differ in terms of the members field: two records include this field, the third does not. This setup is not a problem because MongoDB does not have a fixed schema.

If you already have JSON-formatted documents, you can simply import them directly into the database:

# mongoimport -d football -c clubs --file /tmp/myclubs.json
connected to: 127.0.0.1
Mon Dec  3 08:43:26 imported 23 objects

It is also very easy to query an object via a specific field (Listing 2).

01 > db.clubs.find({Members: {"$gt": 100000}})
02 { "_id" : ObjectId("50bd2169a80e4531863ea2a8"),     "Name" : "FC Schalke 04",     "Colors" : "Blue-white",     "Members" : 111000,     "Address" : [  { "Street" : "Ernst-Kuzorra-Weg 1",    "Zip" : 45891, "City" : "Gelsenkirchen" } ] }

For a small web application, all you need now is a matching framework and a piece of software to connect the framework and MongoDB. I will be using the Bottle micro framework for the former, as already mentioned. To demonstrate the functionality, I will be accessing templates and views in the sample application, although this isn't strictly necessary. PyMongo provides the interface between the framework and the database and transfers the documents over the network in BSON format.

My application comprises a control file (Listing 3) and two view files defined as templates (Listings 4 and 5). Bottle expects the view files in the views folder below the actual application.

01 <!DOCTYPE html>
02 <html>
03
04 <head>
05 <title>My little Bottle example</title>
06 </head>
07
08 <body>
09 <p>Hi World..here is my little survey!</p>
10
11 <form action="/survey" method="POST">
12     What is your favorite club?<br>
13     <input type="text" name="club" size=40 value=""><br><br>
14     input type="submit" value="Submit"><br>
15 </form>
16
17
18 </body>
19 </html>

01 #!/usr/bin/python
02
03 import bottle
04 import pymongo
05
06 connection = pymongo.Connection('localhost', 27017)
07 db = connection.survey
08 clubs = db.clubs
09
10 @bottle.route('/')
11 def index():
12   return bottle.template('main')
13
14 @bottle.post('/survey')
15 def survey():
16   club = bottle.request.forms.get("club")
17   if (club == None or club == ""):
18     club="No club defined..."
19
20   clubs.save({"club":club})
21   results= clubs.find({"club":club})
22   results = results.count()
23
24
25   return bottle.template('result', {"club":club, "results":results})
26
27 bottle.run(host='localhost', port=8080, debug=1)

The call in line 6 of the survey.py file creates a connection object for communicating with the MongoDB service on the same system on which the web framework is running.

Lines 7-8 of Listing 3 then define a handler for the database and the desired collection. Lines 10 and 14 each define a route, that is, a URL with the matching code. The landing page (/) simply returns a template that shows the user a very simple survey page (main.tpl).

The Post instruction this results in is processed by a second URL (/survey). The results of the survey are stored in a document with a single field (club) in the database; the output the user sees simply confirms the team they selected as their favorite and shows how often this team was chosen by other users.

A variable (club) along with the results are passed in to the second template in line 25. The club variable assumes the value of the content from the input box in the first template, and contains the stated favorite team. results contains the results of a database search to find out how often this team is listed. Both variables are then used in the view template (Listing 5) in lines 9 and 10. In the browser, the application looks like Figures 1 and 2.

01 <!DOCTYPE html>
02 <html>
03
04 <head>
05 <title>Results</title>
06 </head>
07
08 <body>
09 <h1>Favorite club: {{club}}</h1><br>
10 Was mentioned {{results}} times thus far.
11 </body>
12
13 </html>

Figure 1: A simple input mask invites users to type their favorite team.

Figure 2: Users then see their choice on a new page, along with a statement on how often the team was chosen by other users.

This example shows how easy it is to develop a web application with a combination of Python, MongoDB, Bottle, and PyMongo. The example is deliberately simple, but I think it clearly illustrates how the individual components mesh. The example also assumes that MongoDB is only running on one server and that you are using Bottle's built-in web server. This will not be the case in a production environment, but the setup shown here is fine for development environments.

Since I wrote this article I have started running test applications as Platform-as-a-Service apps in OpenShift [3]. OpenShift now also has a cartridge that lets you install MongoDB as an add-on. If you are interested in exploring this topic more deeply, after this, admittedly, developer-focused introduction, you will find a treasure trove of useful documentation online [4], including a description of how to install MongoDB in OpenShift.