You're viewing blogs from Django only. RSS?

View all different categories

6th of December

Cryptic errors when using django-nose

After about 3 days of debugging using pdb, print and writing to a log file I've almost finally solve my bizarre errors I was getting when running a whole test suite. The error that it lead to was that Django refused to re-register models to the admin and the errors looked something like this: 

  ...
  File "/Users/peterbe/dev/MOZILLA/PTO/pto/urls.py", line 6, in <module>
    admin.autodiscover()
  File "/Users/peterbe/dev/MOZILLA/PTO/pto/vendor/src/django/django/contrib/admin/__init__.py", line 26, in autodiscover
    import_module('%s.admin' % app)
  File "/Users/peterbe/dev/MOZILLA/PTO/pto/vendor/src/django/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/Users/peterbe/dev/MOZILLA/PTO/pto/apps/users/admin.py", line 30, in <module>
    admin.site.register(UserProfile, UserProfileAdmin)
  File "/Users/peterbe/dev/MOZILLA/PTO/pto/vendor/src/django/django/contrib/admin/sites.py", line 85, in register
    raise AlreadyRegistered('The model %s is already registered' % model.__name__)
 AlreadyRegistered: The model UserProfile is already registered

Turns out to be independent of which Django project I ran and it was something no one else was able to reproduce on any machine with the exact same code.

After 2 days I found that there's a difference between a successful run and a failing run was how I specified (to nose) which module to load: 

 ./manage.py test users  # fails!
 ./manage.py test users.test  # works!

In both cases it finds the same tests. So it would either fail 10 times or work 10 times. Hmmm...

The bridging between nose and Django is done by awesome django-nose developed here at Mozilla by Django extraordinaire Jeff Balogh and it's a non-trivial piece of code as it depends on some really smart importing tricks and stuff which I haven't even begun to understand.

However, after so many trial and errors I finally discovered that the solution (for me) was to delete the ~/.noserc file. What's strange is that all it contained was: 

 [nosetests]
 with-doctest=1

I might never actually find out what went wrong. Ultimately I think a reason things went wrong was because it incorrectly populated sys.modules with excessive keys that would cause double imports of urls.py which in turn runs admin.autodiscover() but incorrectly does so twice.

Sorry for the rambling. And sorry for not actually finding the real bug. I did spent 2-3 days debugging this non-stop and hopefully some other poor frustrated person is going to see this and also look into the ~/.noserc for ways to fix it maybe.

1st of August

EmailInput HTML5 friendly for Django

Suppose you have a Django app with a login where people can only log in with their email address. Then use this widget on your login form:

 ## The input widget class
 class EmailInput(forms.widgets.Input):
    input_type = 'email'

    def render(self, name, value, attrs=None):
        if attrs is None:
            attrs = {}
        attrs.update(dict(autocorrect='off',
                          autocapitalize='off',
                          spellcheck='false'))
        return super(EmailInput, self).render(name, value, attrs=attrs)

 ## Example usage
 class AuthenticationForm(django.contrib.auth.forms.AuthenticationForm):
    """override the authentication form because we use the email address as the
    key to authentication."""
    # allows for using email to log in
    username = forms.CharField(label="Username", max_length=75,
                               widget=EmailInput())
    rememberme = forms.BooleanField(label="Remember me", required=False)

EmailInput HTML5 friendly for Django This input field does some cool stuff in the browser such as automatic validation in the browser as seen in this screenshot here.

More importantly it fixes a very annoying problem when surfing on a smartphone or a tablet like the iPad. As I'm about to type "someusername@mozilla.com" it first wants to start capitalized and which might fail the login. Also if the email address contains a word that it wants to correct like ("mozilla" -> "Mozilla") you have to click the little correct tooltip to tell the input is correct in verbatim.

Note to Djangonauts who want to use this and have a dual authentication backend that takes both usernames and email addresses, this form will make it impossible to log in as something called "admin" for example.

22nd of July

A taste of the Django on inside Mozilla, Sheriffs Duty

A taste of the Django on inside Mozilla, Sheriffs Duty One of the many great things about working for Mozilla is that everything we do is Open Source. Even our wiki is open (however we have an internal wiki for corporation boring stuff such as meeting rooms, HR etc.)

Last week I wrote an internal application for Mozilla's build engineers. Essentially it's a roster that lists one user per day and it's helped by being visualized as a calendar and as a vCal export. It's very unlikely that anybody outside Mozilla will find this particularly useful. But who knows, perhaps other companies have needs to take turns to sheriff build machines.

Anyway, the project was easy to write because we have something called Playdoh. It's a set of nifty and useful settings and a folder structure and it comes with a submodule called "playdoh-lib" which is stuffed with lots of useful packages that you'll most likely want to use. If you browse Playdoh on Github it might look like a lot of stuff but after a second look you'll see that there's actually almost no code. So don't you dare to play the "bloat card"! :)

What this app uses is TastyPie for the REST API which was awesome by the way.

For the authentication I used django-auth-ldap and some custom classes because at Mozilla we use email addresses instead of usernames.

To make the vCal export I use VObject which was easy to work with but has some usual syntax in places.

Jinja was used for the template rendering and it meant I had to do some tricks to use the django.contrib.auth.views.login view but with my templates. Might be worth looking into if people are interested.

The code has 98% test coverage but I had to upgrade to the latest nose to be able to run test coverage on app modules that have similar names to modules in the standard lib.

2nd of June

Test static resources in Django tests

At Mozilla we use jingo-minify to bundle static resources such as .js and .css files. It's not a perfect solution but it's got some great benefits. One of them is that you need to know exactly which static resources you need in a template and because things are bundled you don't need to care too much about what files it originally consisted of. For example "jquery-1.6.2.js" + "common.js" + "jquery.cookies.js" can become "bundles/core.js"

A drawback of this is if you forget to compress and prepare all assets (using the compress_assets management command in jingo-minify) is that you break your site with missing static resources. So how to test for this?


Read the whole text (367 more words)

22nd of February

Optimization of getting random rows out of a PostgreSQL in Django

There was a really interesting discussion on the django-users mailing list about how to best select random elements out of a SQL database the most efficient way. I knew using a regular RANDOM() in SQL can be very slow on big tables but I didn't know by how much. Had to run a quick test!

Cal Leeming discussed a snippet of his to do with pagination huge tables which uses the MAX(id) aggregate function.

So, I did a little experiment on a table with 84,000 rows in it. Realistic enough to matter even though it's less than millions. So, how long would it take to select 10 random items, 10 times? Benchmark code looks like this:

 TIMES = 10
 def using_normal_random(model):
    for i in range(TIMES):
        yield model.objects.all().order_by('?')[0].pk

 t0 = time()
 for i in range(TIMES):
    list(using_normal_random(SomeLargishModel))
 t1 = time()
 print t1-t0, "seconds"

Result:

 41.8955321312 seconds

Nasty!! Also running this you'll notice postgres spiking your CPU like crazy.

A much better approach is to use Python's random.randint(1, <max ID>). Looks like this:

  from django.db.models import Max
  from random import randint
  def using_max(model):
    max_ = model.objects.aggregate(Max('id'))['id__max']
    i = 0
    while i < TIMES:
        try:
            yield model.objects.get(pk=randint(1, max_)).pk
            i += 1
        except model.DoesNotExist:
            pass

 t0 = time()
 for i in range(TIMES):
    list(using_max(SomeLargishModel))
 t1 = time()
 print t1-t0, "seconds"

Result:

 0.63835811615 seconds

Much more pleasant!

UPDATE

Commentator, Ken Swift, asked what if your requirement is to select 100 random items instead of just 10. Won't those 101 database queries be more costly than just 1 query with a RANDOM(). Answer turns out to be no.

I changed the script to select 100 random items 1 time (instead of 10 items 10 times) and the times were the same:

 using_normal_random() took 41.4467599392 seconds
 using_max() took 0.6027739048 seconds

And what about 1000 items 1 time:

 using_normal_random() took 204.685141802 seconds
 using_max() took 2.49527382851 seconds

20th of February

Nice testimonial about django-static

My friend Chris is a Django newbie who has managed to build a whole e-shop site in Django. It will launch on a couple of days and when it launches I will blog about it here too. He sent me this today which gave me a smile:

"I spent today setting up django_static for the site, and optimising it for performance. If there's one thing I've learned from you, it's optimisation.

So, my homepage is now under 100KB (was 330KB), and it loads in @5-6 seconds from hard refresh (was 13-14 seconds at its worst). And I just got a 92 score on Yslow. I do believe I have the fastest tea website around now, and I still haven't installed caching.

Wicked huh?"

He's talking about using django-static. Then I get another email shortly after with this:

"correction - I get 97 on YSlow if I use a VPN.

I just found that the Great Firewall tags extra HTTP requests onto every request I make from my browser, pinging a server in Shanghai with a PHP script which probably checks the page for its content or if its on some kind of blocked list. Cheeky buggers!"

It's that interesting! (Note: Chris is based in China but hosts the test site in the UK)

 

Older entries →