COPYFILE_DISABLE and python distutils in python 2.6
12 April 2014
My friend and colleague Jannis (aka jezdez) Leidel saved my bacon today where I had gotten completely stuck.
So, I have this python2.6 virtualenv and whenever I ran
python setup.py sdist upload it would upload a really nasty tarball to PyPI. What would happen is that when people do
pip install premailer it would file horribly and look something like this:
... IOError: [Errno 2] No such file or directory: '/path/to/virtual-env/build/premailer/setup.py'
What?!?! If you download the tarball and unpack it you'll see that there definitely is a
setup.py file in there.
Anyway. What happens, which I didn't realize was that within the
.tar.gz file there were these strange copies of files. For example for every
file.py there was a
Here's what the file looked like after a tarball had been created:
(premailer26)peterbe@mpb:~/dev/PYTHON/premailer (master)$ tar -zvtf dist/premailer-2.0.2.tar.gz -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 ./._premailer-2.0.2 drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/ -rw-r--r-- 0 peterbe staff 280 Mar 28 10:13 premailer-2.0.2/._LICENSE -rw-r--r-- 0 peterbe staff 1517 Mar 28 10:13 premailer-2.0.2/LICENSE -rw-r--r-- 0 peterbe staff 280 Apr 9 21:10 premailer-2.0.2/._MANIFEST.in -rw-r--r-- 0 peterbe staff 34 Apr 9 21:10 premailer-2.0.2/MANIFEST.in -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/._PKG-INFO -rw-r--r-- 0 peterbe staff 7226 Apr 11 15:51 premailer-2.0.2/PKG-INFO -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 premailer-2.0.2/._premailer drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/premailer/ -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 premailer-2.0.2/._premailer.egg-info drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/ -rw-r--r-- 0 peterbe staff 280 Mar 28 10:13 premailer-2.0.2/._README.md -rw-r--r-- 0 peterbe staff 5185 Mar 28 10:13 premailer-2.0.2/README.md -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/._setup.cfg -rw-r--r-- 0 peterbe staff 59 Apr 11 15:51 premailer-2.0.2/setup.cfg -rw-r--r-- 0 peterbe staff 280 Apr 9 21:09 premailer-2.0.2/._setup.py -rw-r--r-- 0 peterbe staff 2079 Apr 9 21:09 premailer-2.0.2/setup.py -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._dependency_links.txt -rw-r--r-- 0 peterbe staff 1 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/dependency_links.txt -rw-r--r-- 0 peterbe staff 280 Apr 9 21:04 premailer-2.0.2/premailer.egg-info/._not-zip-safe -rw-r--r-- 0 peterbe staff 1 Apr 9 21:04 premailer-2.0.2/premailer.egg-info/not-zip-safe -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._PKG-INFO -rw-r--r-- 0 peterbe staff 7226 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/PKG-INFO -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._requires.txt -rw-r--r-- 0 peterbe staff 23 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/requires.txt -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._SOURCES.txt -rw-r--r-- 0 peterbe staff 329 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/SOURCES.txt -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._top_level.txt -rw-r--r-- 0 peterbe staff 10 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/top_level.txt -rw-r--r-- 0 peterbe staff 280 Apr 9 21:21 premailer-2.0.2/premailer/.___init__.py -rw-r--r-- 0 peterbe staff 66 Apr 9 21:21 premailer-2.0.2/premailer/__init__.py -rw-r--r-- 0 peterbe staff 280 Apr 9 09:23 premailer-2.0.2/premailer/.___main__.py -rw-r--r-- 0 peterbe staff 3315 Apr 9 09:23 premailer-2.0.2/premailer/__main__.py -rw-r--r-- 0 peterbe staff 280 Apr 8 16:22 premailer-2.0.2/premailer/._premailer.py -rw-r--r-- 0 peterbe staff 15368 Apr 8 16:22 premailer-2.0.2/premailer/premailer.py -rw-r--r-- 0 peterbe staff 280 Apr 8 16:22 premailer-2.0.2/premailer/._test_premailer.py -rw-r--r-- 0 peterbe staff 37184 Apr 8 16:22 premailer-2.0.2/premailer/test_premailer.py
Strangly, this only happened in a Python 2.6 environment. The problem went away when I created a brand new Python 2.7 enviroment with the latest
So basically, the fault lies with OSX and a strange interaction between OSX and tar.
This superuser.com answer does a much better job explaining this "flaw".
So, the solution to the problem is to create the distribution like this instead:
$ COPYFILE_DISABLE=true python setup.py sdist
If you do that, you get a healthy lookin tarball that actually works to pip install. Thanks jezdez for pointing that out!
Buggy - A sexy Bugzilla offline webapp
13 March 2014
Buggy is a singe-page webapp that relies entirely on the Bugzilla Native REST API. And it works offline. Sort of. I say "sort of" because obviously without a network connection you're bound to have outdated information from the bugzilla database but at least you'll have what you had when you went offline.
When you post a comment from Buggy, the posted comment is added to an internal sync queue and if you're online it immediately processes that queue. There is, of course, always a risk that you might close a bug when you're in a tunnel or on a plane without WiFi and when you later get back online the sync fails because of some conflict.
The reason I built this was partly to scratch an itch I had ("What's the ideal way possible for me to use Bugzilla?") and also to experiment with some new techniques, namely AngularJS and localforage.
So, the way it works is:
You pick your favorite product and components.
All bugs under these products and components are downloaded and stored locally in your browser (thank you localforage).
When you click any bug it then proceeds to download its change history and its comments.
Periodically it checks each of your chosen product and components to see if new bugs or new comments have been added.
If you refresh your browser, all bugs are loaded from a local copy stored in your browser and in the background it downloads any new bugs or comments or changes.
If you enter your username and password, an auth token is stored in your browser and you can thus access secure bugs.
Pros and cons
The main advantage of Buggy compared to Bugzilla is that it's fast to navigate. You can instantly filter bugs by status(es), components and/or by searching in the bug summary.
The disadvantage of Buggy is that you can't see all fields, file new bugs or change all fields.
Everything is done using AngularJS. It's only my second AngularJS project but this is also part of why I built this. To learn AngularJS better.
I really wanted to use Grunt but it just didn't work for me. There are many positive things about Grunt such as the ease with which you can easily add plugins and I like how you just have one "standard" file that defines how a bunch of meta tasks should be done. However, I just couldn't get the concatenation and minification and stuff to work together. Individually each tool works fine, such as the grunt-contrib-uglify plugin but together none of them appeared to want to work. Perhaps I just required too much.
In the end I wrote a script in python that does exactly what I want for deployment. Its features are:
- Ability to fold all CSS minified into the HTML (since there's only one page, theres little reason to make the CSS external)
- A Git revision SHA into the HTML of the generated
- All files in
./client/static/copied intelligently into
- Images in CSS to be given hashes so they too can have far-future cache headers
So, the way I have it set up is that, on my server, I have a it run
python make.py and that generates a complete site in a
./dist/ directory. I then point Nginx to that directory and run it under
http://buggy-origin.peterbe.com. Then I set up a Amazon Cloudfront distribution to that domain and then lastly I set up a CNAME for
buggy.peterbe.com to point to the Cloudfront distribution.
One of the main features I want to add is the ability to add bugs that are outside your chosen products and components. It'll be a "fake" component called "Misc". This is for bugs outside the products and components you usually monitor and work in but perhaps bugs you've filed or been assigned to. Or just other bugs you're interested in in general.
Another major feature to work on is the ability to choose to see more fields and ability to edit these too. This will require some configuration on the individual users' behalf. For example, some people use the "Target Milestone" a lot. Some use the "Importance" a lot. So, some generic solution is needed to accomodate all these non-basic fields.
And last but not least, the Bugzilla team here at Mozilla is working on a very exciting project that allows you to register a certain list of bugs with a WebSocket and have it push to you as soon as these bugs change. That means that I won't have to periodically query bugzilla every 30 seconds if certain bugs have changed but instead get instant notifications when they do. That's going to be major! I confidently speculate that that will be implemented some time summer this year.
Give it a go. What are you waiting for? :) Go to http://buggy.peterbe.com/, pick your favorite products and components and try to use it for a week.