You're viewing blogs from Linux only. RSS?

View all different categories

5th of April

Strange socket related error with supervisord

This took me a long time to figure out so I thought I'd share.

Basically, I'm a newbie supervisor administrator and I was setting up a new config and I kept getting these errors: 

 # supervisord -n
 2011-04-04 17:25:11,700 CRIT Set uid to user 1000
 2011-04-04 17:25:11,700 WARN Included extra file "/etc/supervisor/conf.d/gkc.conf" during parsing
 Error: Cannot open an HTTP server: socket.error reported errno.ENOENT (2)
 For help, use /usr/local/bin/supervisord -h

The reason was that in my config I had the line: 

 [unix_http_server]
 file=/var/lib/tornado/run/gkc.sock

but the directory /var/lib/tornado/run didn't exist. Creating that solved the problem.

Lesson learned from all this is that when specifying locations of .sock files always make sure the directories exist and that the current user can write to them.

25th of March

Bash tip of the day: ff

This is helping me sooo much that it would a crime not to share it. It's actually nothing fancy, just a very convenient thing that I've learned to get used to. ff is an executable script I use to find files in a git repository. Goes like this:

 $ ff list
 templates/operations/network-packing-list.html
 templates/sales/list_orders.html
 $ ff venue
 templates/venues/venues-by-special.html
 templates/venues/venues.html
 templatetags/venue_extras.py
 templatetags/venues_by_network_extras.py
 tests/test_venues.py

It makes it easy to super quickly search for added files without having to use the slow find command which would also otherwise find backup files and other junk that isn't checked in.

To install it, create a file called ~/bin/ff and make it executable:

 $ chmod +x ~/bin/ff

Then type this code in:

 #!/usr/bin/python
 import sys, os
 args = sys.argv[1:]
 i = False
 if '-i' in args:
    i = True
    args.remove('-i')
 pattern = args[-1]
 extra_args = ''
 if len(args) > 1:
    extra_args = ' '.join(args[:-1])
 param = i and "-i" or ""
 cmd = "git ls-files | grep %s %s '%s'" % (param, extra_args, pattern)
 os.system(cmd)

31st of December

ssl_session_cache in Nginx and the ab benchmark

A couple of days ago I wrote about how blazing fast the DoneCal API can be on HTTP (1,400 requests/second) and how much slower it becomes when doing the same benchmark over HTTPS. It was, as Chris Adams pointed out, possible to run ab with Keep-Alive on and after some reading up it's clear that it's a good idea to switch on shared ssl_session_cache so that Nginx's SSL TCP traffic can cache some handshakes.

With ssl_session_cache shared:SSL:10m : 

  Requests per second:    112.14 [#/sec] (mean)

Same cache size but with -k on the ab loadtest: 

 Requests per second:    906.44 [#/sec] (mean)

I'm fairly sure that most browsers with use Keep-Alive connections so I guess it's realistic to use -k when running ab but since this is a test of an API it's perhaps more likely than not that clients (i.e. computer programs) don't use it. To be honest I'm not really sure but it never the less feels right to be able to use ssl_session_cache to boost my benchmark by 40%.

It's also worth noticing that when doing a HTTP benchmark it's CPU bound on the Tornado (Python) processes (I use 4). But when doing HTTPS it's CPU bound on the Nginx itself (I use 1 worker process).

17th of May

Making output stay on stdout

This is fairly obvious stuff I guess but it has troubled me for a long time. Some programs on Linux don't spit out their results to stdout. Instead they start a little program similar to less. So what is a console nerd to do?

Pipe it cat! I don't know why I've never thought of this before: 

 $ psql -l | cat

11th of May

Upgrading to Ubuntu Lucid Lynx and downgrading to Python2.4 and Python2.5

So I upgraded to the latest Ubuntu Lucid Lynx 10.04 the other day and to my horror it removed Python 2.4 and Python 2.5. I rely more on those programs than I do on some silly Facebook connecting social widget crap. On my laptop I have lots of Zopes requiring Python 2.4 and I have about 10 active Django projects that rely on Python2.5. This fuckup by Ubuntu caused me to write this complaint.

So my estimeed colleague and Linux wiz Jan Kokoska helped me set things straight by showing me how to downgrade these packages to Karmic version and how to pin them in the apt preferences. First of all, make your /etc/apt/source.list look like this: 

 deb http://gb.archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ karmic-backports main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic-backports main restricted universe multiverse

 deb http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse
 deb-src http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

 deb http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse
 deb-src http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse

If you know what you're doing you might have other additional sources in there then keep those as is. Next thing to do is to update and upgrade: 

 # apt-get update
 # apt-get dist-upgrade

You should now see that it's intending to upgrade a bunch of juicy packages like python2.4-dev for example. To check that python2.4 is now getting in from Karmic run this: 

 $ apt-cache madison python2.4

Now for the trick that really makes the difference: 

 # apt-get install python2.4=2.4.6-1ubuntu3.2.9.10.1 python2.4-dbg=2.4.6-1ubuntu3.2.9.10.1 \
 python2.4-dev=2.4.6-1ubuntu3.2.9.10.1 python2.4-doc=2.4.6-1ubuntu3.2.9.10.1 \
 python2.4-minimal=2.4.6-1ubuntu3.2.9.10.1

The command is quite self-explanatory. You use the equal sign to basically say what version you want to install. If you now for example want to install something like python-profiler for your Python 2.4 since this isn't available as a PyPi package. First, find out what version you have to install: 

 $ apt-cache madison python-profiler | grep karmic

From that list you'll get a bunch of versions. Chose the one from karmic-updates or karmic-security. Then install it: 

 # apt-get install python-profiler=2.6.4-0ubuntu1

Now, to avoid this causing a conflict and thus be removed the next time you do an upgrade you need to pin it. Create a file called /etc/apt/preferences and put the following into it: 

 Package: python-profiler
 Pin: version 2.6.4-0ubuntu1
 Pin-Priority: 999

And that concludes it. A word of warning from Jan:

"he slight problem is that with this setup, suppose a big security flaw was found in python-imaging and got patched in karmic that is still supported... you wouldn't get the package update. That is because it's pinned and while asterisks can be used in the version number, we don't know in advance what the version will match and what the Lucid version that we don't want will match"

"so you basically lose security upgrades for affected packages"

"minor annoyance when you have one or two packages on a laptop, but a big deal if you have a dozen packages on 100 VMs on server"

Having written about this helps me remember it myself for the next time I need it. Also, hopefully it will help other people who get bitten by this. Hopefully this will shame the Canonical guys into action so that the next time they don't haste their deprecation process and actually think about who's using their products. I bet a majority of Ubuntu's users care more about programming or something like that than they do about the ability to buy music on Ubuntu One or whatever it's called.

8th of April

fcgi vs. gunicorn vs. uWSGI

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers: 

 fcgi (threaded)              640 r/s
 fcgi (prefork 4 processors)  240 r/s (*)
 gunicorn (2 workers)         1100 r/s
 gunicorn (5 workers)         1300 r/s
 gunicorn (10 workers)        1200 r/s (?!?)
 uwsgi (2 workers)            1800 r/s
 uwsgi (5 workers)            2100 r/s
 uwsgi (10 workers)           2300 r/s

 (* this made my computer exceptionally sluggish as CPU when through the roof)


Read the whole text (534 more words)

 

Older entries →