8th of October

New feature on Too Cool For Me: Everyone I follow

New feature on Too Cool For Me: Everyone I follow I've added a new feature to Too Cool For Me that lists all the users that you follow and splits them up into "Follows me" and "Too cool for me".

To try it you have to authenticate with Twitter (READ ONLY mode) then go to toocoolfor.me/everyone

This means you can use Too Cool For Me without having to use the Bookmarklet.

27th of September

Cateechee golf pictures

Cateechee golf pictures My friend Josh Ziff has uploaded pictures from the little golf tournament (we played scrambles) at Cateechee in Hartwell, Georgia a couple of weeks ago.

Check out the pictures here

25th of September

Too Cool For Me?

Too Cool For Me? Too Cool For Me? is a fun little side-project I've been working on. It's all about and only for Twitter. You login, then install a bookmarklet then when browsing twitter you can see who follows you and who is too cool for you.

For me it's a chance to try some new tech and at the same time scratch an itch I had. The results can be quite funny but also sad too when you realise that someone uncool isn't following you even though you follow him/her.

The code is open source and available on Github and at least it might help people see how to do a web app in Tornado using MongoDB and asynchronous requests to the Twitter API

23rd of September

Goodies from tornado-utils - part 3: send_mail

This is Part 3 in a series of blogs about various bits and pieces in the tornado-utils package. Part 1 is here and part 2 is here

send_mail

Code is here

First of all, I should say: I didn't write much of this code. It's copied from Django and modified to work in any Tornado app. It hinges on the same idea as Django that you have to specify what backend you want to use. A backend is an importable string pointing to a class that has the send_messages method.

To begin, here's a sample use case inside a request handler:

 class ContactUs(tornado.web.RequestHandler):

    def post(self):
        msg = self.get_argument('msg')
        # NB: you might want to set this once and for all in something 
        # like self.application.settings['email_backend']
        backend = 'tornado_utils.send_mail.backends.smtp.EmailBackend'
        send_email(backend,
                   "New contact form entry",
                   msg + '\n\n--\nFrom our contact form\n',
                   'noreply@example.com',
                   ['webmaster@example.com'],
                   )
        self.write("Thanks!")

The problem is that SMTP is slow. Even though, in human terms, it's fast, it's still too slow for a non-blocking server that Tornado is. Taking 1-2 seconds to send a message over SMTP means it's blocking every other request to Tornado for 1-2 seconds. The solution is instead save the message on disk in pickled form and use a cron job to pick up the messages and send them by SMTP instead, outside the Tornado process. First do this re-write:

    ... 
    def post(self):
        msg = self.get_argument('msg')
-       backend = 'tornado_utils.send_mail.backends.smtp.EmailBackend'
+       backend = 'tornado_utils.send_mail.backends.pickle.EmailBackend'
    ...

Now, write a cron job script that looks something like this:

 # send_pickled_messages.py
 DRY_RUN = False

 def main():
    from tornado_utils.send_mail import config
    filenames = glob(os.path.join(config.PICKLE_LOCATION, '*.pickle'))
    filenames.sort()
    if not filenames:
        return

    from tornado_utils.send_mail import backends
    import cPickle

    if DRY_RUN:
        EmailBackend = backends.console.EmailBackend
    else:
        EmailBackend = backends.smtp.EmailBackend
    max_count = 10
    filenames = filenames[:max_count]
    messages = [cPickle.load(open(x, 'rb')) for x in filenames]
    backend = EmailBackend()
    backend.send_messages(messages)
    if not DRY_RUN:
        for filename in filenames:
            os.remove(filename)

That code just above is butchered from a more comprehensive script I have but you get the idea. Writing to a pickle file is so fast it's in the lower milliseconds region. However, it depends on disk IO so if you need more speed, write a simple backend that writes instead of saving pickles on disk, make it write to a fast in-memory database like Redis or Memcache.

The code isn't new and it's been battle tested but it's only really been battle tested in the way that my apps use it. So you might stumble across bugs if you use it in a way I haven't tested. However, the code is Open Source and happily available for you to help out and improve.

21st of September

Goodies from tornado-utils - part 2: tornado_static

This is Part 2 in a series of blogs about various bits and pieces in the tornado-utils package. Part 1 is here

tornado_static

Code is here

This code takes care of two things: 1) optimizing your static resources and 2) bundling and serving them with unique names so you can cache aggressively.

The trick is to make your development environment such that there's no need to do anything when in "debug mode" but when in "production mode" it needs to be perfect. Which files (e.g. jquery.js or style.css) you use and bundle is up to you and it's something you control from the templates in your Tornado app. Not a config setting because, almost always, which resources (aka. assets) you need is known and relevant only to the templates where you're working.

Using UI modules in Tornado requires a bit of Tornado-fu but here's one example and here is another (untested) example:

 # app.py

 import tornado.web
 from tornado_utils.tornado_static import (
   StaticURL, Static, PlainStaticURL, PlainStatic) 

 class Application(tornado.web.Application):
    def __init__(self):
        ui_modules = {}
        if options.debug:
            ui_modules['Static'] = PlainStatic
            ui_modules['StaticURL'] = PlainStaticURL
        else:
            ui_modules['Static'] = Static
            ui_modules['StaticURL'] = StaticURL

        app_settings = dict(
            template_path="templates",
            static_path="static",
            ui_modules=ui_modules,
            debug=options.debug,
            UGLIFYJS_LOCATION='~/bin/uglifyjs',
            CLOSURE_LOCATION="static/compiler.jar",
            YUI_LOCATION="static/yuicompressor-2.4.2.jar",
            cdn_prefix="cdn.mycloud.com",
        )

        handlers = [
          (r"/", HomeHandler),
          (r"/entry/([0-9]+)", EntryHandler),
        ] 
        super(Application, self).__init__(handlers, **app_settings)

 def main(): # pragma: no cover
    tornado.options.parse_command_line()
    http_server = tornado.httpserver.HTTPServer(Application())
    http_server.listen(options.port)
    tornado.ioloop.IOLoop.instance().start()

 if __name__ == "__main__":
    main()

Note! If you're looking to optimize your static resources in a Tornado app you probably already have a "framework" for setting up UI modules into your app. The above code is just to wet your appetite and to show how easy it is to set up. The real magic starts to happen in the template code. Here's a sample implementation:

 <!doctype html>
 <html>
    <head>
        <title>{% block title %}{{ page_title }}{% end %}</title>
        <meta charset="utf-8">
        {% module Static("css/ext/jquery.gritter.css", "css/style.css") %}
    </head>
    <body>
       <header>...</header>
    {% block content %}
    {% end %}
    {% module Static("js/ext/head.load.min.js") %}
    <script>
    var POST_LOAD = '{% module StaticURL("js/dojo.js", "js/dojo.async.js") %}';
    </script>
    </body>
 </html>

What you get when run is a template that looks like this:

 <!doctype html>
 <html>
    <head>
        <title>My title</title>
        <meta charset="utf-8">
<link rel="stylesheet" type="text/css" href="//cdn.mycloud.com/combined/jquery.gritter.css.style.1313206609.css">
    </head>
    ...

(Note that this will create a whitespace optimized filed called "jquery.gritter.css.style.1313206609.css" in "/tmp/combined")

Have a play and see if it makes sense in your app. I do believe this can do with some Open Source love but so far it works great for me on Kwissle, DoneCal and TooCoolFor.Me

19th of September

Goodies from tornado-utils - part 1: TestClient

This is Part 1 in a series of blogs about various bits and pieces in the tornado-utils package.

tornado-utils is the result of me working on numerous web apps in Tornado that after too much copy-and-paste from project to project eventually became a repo standing on its own two legs.

TestClient

Code is here

This makes it possible to write integration tests that executes GET and POST requests on your app similarly to how Django does it. Example implementation:

 # tests/base.py

 from tornado_utils.http_test_client import TestClient, HTTPClientMixin
 from tornado.testing import LogTrapTestCase, AsyncHTTPTestCase

 class BaseAsyncTestCase(AsyncHTTPTestCase, LogTrapTestCase):
    pass 

 class BaseHTTPTestCase(BaseAsyncTestCase, HTTPClientMixin):

    def setUp(self):
        super(BaseHTTPTestCase, self).setUp()
        self.client = TestClient(self)

 # tests/test_handlers.py

 from .base import BaseHTTPTestCase

 class HandlersTestCase(BaseHTTPTestCase):

    def setUp(self):
        super(HandlersTestCase, self).setUp()
        self._create_user('peterbe', 'secret')  # use your imagination

    def test_homepage(self):
        response = self.client.get('/')
        self.assertEqual(response.code, 200)
        self.assertTrue('stranger' in response.body)

        data = {'username': 'peterbe', 'password': 'secret'}
        response = self.client.post('/login/', data)
        self.assertEqual(response.code, 302)

        response = self.client.get('/')
        self.assertEqual(response.code, 200)
        self.assertTrue('stranger' not in response.body)
        self.assertTrue('hi peterbe' in response.body)

You can see a sample implementation of this here

Note that this was one of the first pieces of test code I wrote in my very first Tornado app and it's not unlikely that some assumptions and approaches are naive or even wrong but what we have here works and it makes the test code very easy to read. All it basically does is wraps the http_client.fetch(...) call and also maintains a bucket of cookies

I hope it can be useful to someone new to writing tests in Tornado.

 

Older entries →
← Back