You're viewing blogs from Python only. RSS?

View all different categories

13th of December

Persistent caching with fire-and-forget updates

I just recently landed some patches on toocool that implements and interesting pattern that is seen more and more these days. I call it: Persistent caching with fire-and-forget updates

Basically, the implementation is this: You issue a request that requires information about a Twitter user: E.g. http://toocoolfor.me/following/chucknorris/vs/peterbe The app looks into its MongoDB for information about the tweeter and if it can't find this user it goes onto the Twitter REST API and looks it up and saves the result in MongoDB. The next time the same information is requested, and the data is available in the MongoDB it instead checks if the modify_date or more than an hour and if so, it sends a job to the message queue (Celery with Redis in my case) to perform an update on this tweeter.

You can basically see the code here but just to reiterate and abbreviate, it looks like this:

 tweeter = self.db.Tweeter.find_one({'username': username})
 if not tweeter:
    result = yield tornado.gen.Task(...)
    if result:
        tweeter = self.save_tweeter_user(result)
    else:
        # deal with the error!
 elif age(tweeter['modify_date']) > 3600:
    tasks.refresh_user_info.delay(username, ...)
 # render the template!

What the client gets, i.e. the user using the site, is it that apart from the very first time that URL is request is instant results but data is being maintained and refreshed.

This pattern works great for data that doesn't have to be up-to-date to the second but that still needs a way to cache invalidate and re-fetch. This works because my limit of 1 hour is quite arbitrary. An alternative implementation would be something like this:

 tweeter = self.db.Tweeter.find_one({'username': username})
 if not tweeter or (tweeter and age(tweeter) > 3600 * 24 * 7):
     # re-fetch from Twitter REST API
 elif age(tweeter) > 3600:
     # fire-and-forget update

That way you don't suffer from persistently cached data that is too old.

2nd of December

Python file with closing automatically

Perhaps someone who knows more about the internals of python and the recent changes in 2.6 and 2.7 can explain this question that came up today in a code review.

I suggest using with instead of try: ... finally: to close a file that was written to. Instead of this:

 dest = file('foo', 'w')
 try:
    dest.write('stuff')
 finally:
    dest.close()
 print open('foo').read()  # will print 'stuff'

We can use this:

 with file('foo', 'w') as dest: 
     dest.write('stuff')
 print open('foo').read()  # will print 'stuff'

Why does that work? I'm guessing it's because the file() instance object has a built in __exit__ method. Is that right?

That means I don't need to use contextlib.closing(thing) right?

For example, suppose you have this class:

 class Farm(object):
    def __enter__(self):
        print "Entering"
        return self
    def __exit__(self, err_type, err_val, err_tb):
        print "Exiting", err_type
        self.close()
    def close(self):
        print "Closing"

 with Farm() as farm:
    pass
 # this will print:
 #   Entering
 #   Exiting None
 #   Closing

Another way to achieve the same specific result would be to use the closing() decrorator:

 class Farm(object):
    def close(self):
        print "Closing"

 from contextlib import closing
 with closing(Farm()) as farm:
    pass
 # this will print:
 #   Closing

So the closing() decorator "steals" the __enter__ and __exit__. This last one can be handy if you do this:

 from contextlib import closing
 with closing(Farm()) as farm:
    raise ValueError

 # this will print
 #  Closing
 #  Traceback (most recent call last):
 #   File "dummy.py", line 16, in <module>
 #     raise ValueError
 #  ValueError

This is turning into my own loud thinking and I think I get it now. contextlib.closing() basically makes it possible to do what I did there with the __enter__ and __exit__ and it seems the file() built-in has a exit handler that takes care of the closing already so you don't have to do it with any extra decorators.

18th of November

Trivial but powerful tips for nosetests

I'm clearly still a nosetests beginner because it was only today that I figured out how to set certain plugins to always be on.

First of all you might like these plugins too: 

 $ pip install rudolf
 $ pip install disabledoc

Docs: rudolf and disabledoc

To get these gorgeous little tricks into every run of nosetests edit the file ~/.noserc and add the following: 

 [nosetests]
 with-disable-docstring=1
 with-color=1

That should make your life a little easier.

UPDATE:

I've since managed to shoot myself in both legs with messing around with nosetests plugins because I heavily rely on django-nose in Django. Long story short: be careful if you get strange import related errors!

8th of July

Slides about Kwissle from yesterdays London Python Dojo

Here are the slides from yesterday's London Python Dojo event.

I presented and demo'ed Kwissle to my fellow Python London friends and focused a lot on the technology but also tried to plug the game a bit.

Having seen that there's a lot of interest in "socket" related web applications about I thought this was a good chance to say that you don't need NodeJS and that tornadio is a great framework for that.

12th of June

Optimization story involving something silly I call "dict+"

Here's a little interesting story about using MongoKit to quickly draw items from a MongoDB

So I had a piece of code that was doing a big batch update. It was slow. It took about 0.5 seconds per user and I sometimes had a lot of users to run it for.

The code looked something like this:

  for play in db.PlayedQuestion.find({'user.$id': user._id}):
     if play.winner == user:
          bla()
     elif play.draw:
          ble()
     else:
          blu()

Because the model PlayedQuestion contains DBRefs MongoKit will automatically look them up for every iteration in the main loop. Individually very fast (thank you indexes) but because of the number of operations very slow in total. Here's how to make it much faster:

    for play in db.PlayedQuestion.collection.find({'user.$id': user._id}):

The problem with this is that you get dict instances for each which is more awkward to work with. I.e. instead of `play.winner` you have use `play['winner'].id`. Here's my solution that makes this a lot easier:

 class dict_plus(dict):

   def __init__(self, *args, **kwargs):
        if 'collection' in kwargs: # excess we don't need
            kwargs.pop('collection')
        dict.__init__(self, *args, **kwargs)
        self._wrap_internal_dicts()

    def _wrap_internal_dicts(self):
        for key, value in self.items():
            if isinstance(value, dict):
                self[key] = dict_plus(value)

    def __getattr__(self, key):
        return self[key]

   ...

  for play in db.PlayedQuestion.collection.find({'user.$id': user._id}):
     play = dict_plus(play)
     if play.winner.id == user._id:
          bla()
     elif play.draw:
          ble()
     else:
          blu()

Now, the whole thing takes 0.01 seconds instead of 0.5. 50 times faster!!

6th of April

TornadoGists.org - launched and ready!

Today Felinx Lee and I launched TornadoGists.org which is a site for discussing gists related to Tornado (python web framework open sourced by Facebook).

Everyone in the Tornado community seems to solve similar problems in different ways. Oftentimes, these solutions are just a couple of lines or so and not something you can really turn into a full package with setup.py and everything.

Sharing a snippet of code is a great way to a) help other people and b) to get feedback on your solutions.

The goal is to make it a very open and active project with lots of contributors. I'll be accepting and reviewing all forks but hopefully control will be opened up to all Tornado developers. Also, since the code is quite generic to any open source project Felinx and I might one day port this to rubygists.org or lispgists.org or something like that. After all, Github does all the heavy lifting and we just wrap it up nicely.

 

Older entries →