Grymt - because I didn't invent Grunt here
18 April 2014
grymt is a python tool that takes a directory full of .html, .css and .js and prepares the html for optimial production use.
For a teaser:
So why did I write my own tool and not use Grunt?!
Glad you asked! The reason is simple: I couldn't get Grunt to work.
Grunt is a framework. It's a place where you say which "recipes" to execute and how. It's effectively a common config framework. Like make.
However, I tried to set up a bunch of recipes in my Gruntfile.js and most of them worked well individually but it was a hellish nightmare to get it all to work together just the way I want it.
For example, the grunt-contrib-uglify is fine for doing the minification but it doesn't work with concatenation and it doesn't deal with taking one input file and outputting to a different file.
Basically, I spent two evenings getting things to work but I could never get exactly what I wanted. So I wrote my own and because I'm quite familiar with this kind of stuff, I did it in Python. Not because it's better than Node but just because I had it near by and was able to quicker build something.
So what sweet features do you get out of grymt?
You can easily make an output file have a hash in the filename. E.g.
vendor-64f7425.min.jsand thus the filename is always unique but doesn't change in between deployments unless you change the files.
It automatically notices which files already have been minified. E.g. no need to minify
somelib.min.jsbut do minify
You can put
$git_revisionanywhere in your HTML and this gets expanded automatically. For example, view the source of buggy.peterbe.com and look at the first 20 lines.
Images inside CSS get rewritten to have unique names (based on files' modified time) so they can be far-future cached aggresively too.
You never have to write down any lists of file names in soome Gruntfile.js equivalent file
$('<img>').attr('src', 'picture.jpg')for example.
The generated (aka. "dist" directory) contains everything you need. It does not refer back to the source directory in any way. This means you can set up your apache/nginx to point directly at the root of your "dist" directory.
So what's the catch?
It's not Grunt. It's not a framework. It does only what it does and if you want it to do more you have to work on grymt itself.
The files you want to analyze, process and output all have to be in a sub directory.
Look at how I've laid out the files here in this project for example. ALL files that you need is all in one sub-directory called
app. So, to run
grymtI simply run:
The HTML files you throw into it have to be plain HTML files. No templates for server-side code.
How do you use it?
pip install grymt
Then you need a directory it can process, e.g
./client/ (assumed to contain a .html file(s)).
For more options, check out
What's in the future of grymt?
If people like it and want to add features, I'm more than happy to accept pull requests. Some future potential feature work:
I haven't needed it immediately, yet, myself, but it would be nice to add things like coffeescript, less, sass etc into pre-processing hooks.
It would be easy to automatically generate and insert a reference to a appcache manifest. Since every file used and mentioned is noticed, we could very accurately generate an appcache file that is less prone to human error.
Spitting out some stats about number bytes saved and number of files reduced.
COPYFILE_DISABLE and python distutils in python 2.6
12 April 2014
My friend and colleague Jannis (aka jezdez) Leidel saved my bacon today where I had gotten completely stuck.
So, I have this python2.6 virtualenv and whenever I ran
python setup.py sdist upload it would upload a really nasty tarball to PyPI. What would happen is that when people do
pip install premailer it would file horribly and look something like this:
... IOError: [Errno 2] No such file or directory: '/path/to/virtual-env/build/premailer/setup.py'
What?!?! If you download the tarball and unpack it you'll see that there definitely is a
setup.py file in there.
Anyway. What happens, which I didn't realize was that within the
.tar.gz file there were these strange copies of files. For example for every
file.py there was a
Here's what the file looked like after a tarball had been created:
(premailer26)peterbe@mpb:~/dev/PYTHON/premailer (master)$ tar -zvtf dist/premailer-2.0.2.tar.gz -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 ./._premailer-2.0.2 drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/ -rw-r--r-- 0 peterbe staff 280 Mar 28 10:13 premailer-2.0.2/._LICENSE -rw-r--r-- 0 peterbe staff 1517 Mar 28 10:13 premailer-2.0.2/LICENSE -rw-r--r-- 0 peterbe staff 280 Apr 9 21:10 premailer-2.0.2/._MANIFEST.in -rw-r--r-- 0 peterbe staff 34 Apr 9 21:10 premailer-2.0.2/MANIFEST.in -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/._PKG-INFO -rw-r--r-- 0 peterbe staff 7226 Apr 11 15:51 premailer-2.0.2/PKG-INFO -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 premailer-2.0.2/._premailer drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/premailer/ -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 premailer-2.0.2/._premailer.egg-info drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/ -rw-r--r-- 0 peterbe staff 280 Mar 28 10:13 premailer-2.0.2/._README.md -rw-r--r-- 0 peterbe staff 5185 Mar 28 10:13 premailer-2.0.2/README.md -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/._setup.cfg -rw-r--r-- 0 peterbe staff 59 Apr 11 15:51 premailer-2.0.2/setup.cfg -rw-r--r-- 0 peterbe staff 280 Apr 9 21:09 premailer-2.0.2/._setup.py -rw-r--r-- 0 peterbe staff 2079 Apr 9 21:09 premailer-2.0.2/setup.py -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._dependency_links.txt -rw-r--r-- 0 peterbe staff 1 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/dependency_links.txt -rw-r--r-- 0 peterbe staff 280 Apr 9 21:04 premailer-2.0.2/premailer.egg-info/._not-zip-safe -rw-r--r-- 0 peterbe staff 1 Apr 9 21:04 premailer-2.0.2/premailer.egg-info/not-zip-safe -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._PKG-INFO -rw-r--r-- 0 peterbe staff 7226 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/PKG-INFO -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._requires.txt -rw-r--r-- 0 peterbe staff 23 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/requires.txt -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._SOURCES.txt -rw-r--r-- 0 peterbe staff 329 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/SOURCES.txt -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._top_level.txt -rw-r--r-- 0 peterbe staff 10 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/top_level.txt -rw-r--r-- 0 peterbe staff 280 Apr 9 21:21 premailer-2.0.2/premailer/.___init__.py -rw-r--r-- 0 peterbe staff 66 Apr 9 21:21 premailer-2.0.2/premailer/__init__.py -rw-r--r-- 0 peterbe staff 280 Apr 9 09:23 premailer-2.0.2/premailer/.___main__.py -rw-r--r-- 0 peterbe staff 3315 Apr 9 09:23 premailer-2.0.2/premailer/__main__.py -rw-r--r-- 0 peterbe staff 280 Apr 8 16:22 premailer-2.0.2/premailer/._premailer.py -rw-r--r-- 0 peterbe staff 15368 Apr 8 16:22 premailer-2.0.2/premailer/premailer.py -rw-r--r-- 0 peterbe staff 280 Apr 8 16:22 premailer-2.0.2/premailer/._test_premailer.py -rw-r--r-- 0 peterbe staff 37184 Apr 8 16:22 premailer-2.0.2/premailer/test_premailer.py
Strangly, this only happened in a Python 2.6 environment. The problem went away when I created a brand new Python 2.7 enviroment with the latest
So basically, the fault lies with OSX and a strange interaction between OSX and tar.
This superuser.com answer does a much better job explaining this "flaw".
So, the solution to the problem is to create the distribution like this instead:
$ COPYFILE_DISABLE=true python setup.py sdist
If you do that, you get a healthy lookin tarball that actually works to pip install. Thanks jezdez for pointing that out!
Buggy - A sexy Bugzilla offline webapp
13 March 2014
Buggy is a singe-page webapp that relies entirely on the Bugzilla Native REST API. And it works offline. Sort of. I say "sort of" because obviously without a network connection you're bound to have outdated information from the bugzilla database but at least you'll have what you had when you went offline.
When you post a comment from Buggy, the posted comment is added to an internal sync queue and if you're online it immediately processes that queue. There is, of course, always a risk that you might close a bug when you're in a tunnel or on a plane without WiFi and when you later get back online the sync fails because of some conflict.
The reason I built this was partly to scratch an itch I had ("What's the ideal way possible for me to use Bugzilla?") and also to experiment with some new techniques, namely AngularJS and localforage.
So, the way it works is:
You pick your favorite product and components.
All bugs under these products and components are downloaded and stored locally in your browser (thank you localforage).
When you click any bug it then proceeds to download its change history and its comments.
Periodically it checks each of your chosen product and components to see if new bugs or new comments have been added.
If you refresh your browser, all bugs are loaded from a local copy stored in your browser and in the background it downloads any new bugs or comments or changes.
If you enter your username and password, an auth token is stored in your browser and you can thus access secure bugs.
Pros and cons
The main advantage of Buggy compared to Bugzilla is that it's fast to navigate. You can instantly filter bugs by status(es), components and/or by searching in the bug summary.
The disadvantage of Buggy is that you can't see all fields, file new bugs or change all fields.
Everything is done using AngularJS. It's only my second AngularJS project but this is also part of why I built this. To learn AngularJS better.
I really wanted to use Grunt but it just didn't work for me. There are many positive things about Grunt such as the ease with which you can easily add plugins and I like how you just have one "standard" file that defines how a bunch of meta tasks should be done. However, I just couldn't get the concatenation and minification and stuff to work together. Individually each tool works fine, such as the grunt-contrib-uglify plugin but together none of them appeared to want to work. Perhaps I just required too much.
In the end I wrote a script in python that does exactly what I want for deployment. Its features are:
- Ability to fold all CSS minified into the HTML (since there's only one page, theres little reason to make the CSS external)
- A Git revision SHA into the HTML of the generated
- All files in
./client/static/copied intelligently into
- Images in CSS to be given hashes so they too can have far-future cache headers
So, the way I have it set up is that, on my server, I have a it run
python make.py and that generates a complete site in a
./dist/ directory. I then point Nginx to that directory and run it under
http://buggy-origin.peterbe.com. Then I set up a Amazon Cloudfront distribution to that domain and then lastly I set up a CNAME for
buggy.peterbe.com to point to the Cloudfront distribution.
One of the main features I want to add is the ability to add bugs that are outside your chosen products and components. It'll be a "fake" component called "Misc". This is for bugs outside the products and components you usually monitor and work in but perhaps bugs you've filed or been assigned to. Or just other bugs you're interested in in general.
Another major feature to work on is the ability to choose to see more fields and ability to edit these too. This will require some configuration on the individual users' behalf. For example, some people use the "Target Milestone" a lot. Some use the "Importance" a lot. So, some generic solution is needed to accomodate all these non-basic fields.
And last but not least, the Bugzilla team here at Mozilla is working on a very exciting project that allows you to register a certain list of bugs with a WebSocket and have it push to you as soon as these bugs change. That means that I won't have to periodically query bugzilla every 30 seconds if certain bugs have changed but instead get instant notifications when they do. That's going to be major! I confidently speculate that that will be implemented some time summer this year.
Give it a go. What are you waiting for? :) Go to http://buggy.peterbe.com/, pick your favorite products and components and try to use it for a week.
I do not deny it. I'm a YouTube fiend. I very rarely watch YouTube on my computer but a lot on my Apple TV and only tablet. It's
Here are some of my favorite YouTube channels that I subscribe to and encourage you to do the same if you aren't already and if there's something it appears you'll like too.
They started as clips that were around 1 minute but are now of variable length. I just adore Henry's voice and the topics he chooses. The animations are cute and even though seasoned with silly cat and dog references they really help to explain some of the most advanced subjects in physics.
Incidentally, this was the first channel I subscribed to once I figured that's the best way to get recurring content from channels I really liked.
This is a Brady Haran production that speaks directly to my mathematical aspirations. These aspirations aren't to solve any complex calculus problems but to keep that almost mystic infatuation alive I have with mathematics. There's something wonderfully down to earth and kind about the content which challenge you without patronizing you. By the way, my favorite interviewee, James Grime has his own channel now called singingbanana and also, by the way, and amazingly unattractive website.
Derek Muller is a brilliant video maker. Most of his videos are about science and it's mainly Derek holding his camera at arms length filming his pleasant face and talking about the perception or understanding of science. More so than the science itself. Actually some videos are not about how people (miss)understand science but speak directly to you and those are just brilliant. Usually sufficiently advanced to really get reallying thinking hard.
4. CGP Grey
The only, of my top favorite channels, that is not about natural science. These videos are on social science subjects you might never have thought to think about and not only that, but each and every one digs deep and misses very few facts. Similarly to SciShow, these videos require your full attention. Because what you learn from them is often so very valuable, I've revisited many videos. Some more than twice.
This is Henry Reich's (see above about MinutePhysics) second channel and the name of the channels fully describes what the videos are about. The animations are really magnificantly simple and rich at the same time. The subject matters in this videos are generally less advanced that those in MinutePhysics but often full of really interesting factoids to keep up your sleeve for dinner parties.
Hang Green is a gem! His geeky and passionate mannerisms is worth it just on its own. But you have to pay full attention because Hank speaks very fast. There is though an important undertone that isn't immediately obvious. There is this feeling of deeply researched facts. Even though you only understand a small part of it all (not to mention how little you remember!) it's inspiring that someone takes the time to do all the research.
A lot of subject matters are science oriented but more popular sciencey.
Another Brady Haran production, but this time more about physics and than Numberphile which is more about mathematics. Almost all videos are Brady interviewing doctors and professors in physics at the University of Nottingham. All very humble and approachable interviewees that, perhaps thanks to Brady's brilliant questions, the subjects are understandable but also very exciting because they're usually on matters that are very advanced and something more to look forward to than to enjoy in the moment.
This is a newcomer and I include it because they're of such high quality and adorable animations. To be honest I don't think I really understand what the various videos have in common. For example, one recent video is on quantum entanglement and another on the Dead Sea scrolls. Either way, every video is professional and highly enjoyable.
There are more channels I subscribe to and enjoy very much but the above list are my favorite ones. For example, I watch Jamie Oliver's Food Tube videos just as often but that's somehow more "obvious".
Actually I have many more channels on science and a bunch of computers and programming but I'm just simply not as passionate about them as I are with the channels mentioned above.
I really hope that by writing this it will inspire one or two fellow science nerdy readers to also discover some of the channels mentioned here.
Github Pull Request Triage tool
06 March 2014
Web development, AngularJS
Its goal is to try to get an overview of what needs to happen next to open pull requests. Or rather, what needs to happen next to get it closed. Or rather, who needs to act next to get it closed.
It's very common, at least in my team, that someone puts up a pull request, asks someone to review it and then walks away from it. She then doesn't notice that perhaps the integrated test runner fails on it and the reviewer is thinking to herself "I'll review the code once the tests don't fail" and all of a sudden the ball is not in anybody's court. Or someone makes a comment on a pull request that the author of the pull requests misses in her firehose of email notifictions. Now she doesn't know that the comment means that the ball is back in her court.
Ultimately, the responsibility lies with the author of the pull request to pester and nag till it gets landed or closed but oftentimes the ball is in someone elses court and hopefully this tool makes that clearer.
Here's an example instance: https://prs.paas.allizom.org/mozilla/socorro
Currently you can use prs.paas.allizom.org for any public Github repo but if too many projects eat up all the API rate limits we have I might need to narrow it down to use mozilla repos. Or, you can simply host your own. It's just a simple Flask server
About the technology
So it's a single page app that uses HTML5
pushState and an angular
$routeProvider to make different URLs.
The server simply acts as a proxy for making queries to
bugzilla.mozilla.org/rest and the reason for that is for caching.
Every API request you make through this proxy gets cached for 10 minutes. But here's the clever part. Every time it fetches actual remote data it stores it in two caches. One for 10 minutes and one for 24 hours. And when it stores it for 24 hours it also stores its last ETag so that I can make conditional requests. The advantage of that is you quickly know if the data hasn't changed and more importantly it doesn't count against you in the rate limiter.
What a book!
I defend that it took months to finish it with; it's not easy reading, I only read on the short train commute 3 days a week and I'm a really really slow reader.
Even though it was hard going at times I generally enjoyed every page. Some passages were like reading Latin but with English words. Some pages where thrilling and some pages where beautiful as poetry. Some short passages were so amazing that you have to stop and just take a quick smile-break.
Unlike many people I actually didn't know how the book plays out or how it ends and I'm NOT going to spoil that here, in case you too want to read it too, not knowing how it ends. The only thing I regret is reading the editors introduction which revealed something crucial to the plot line without any warning.
It was not until afterwards when I read about the book on Wikipedia (link contains spoilers) that I appreciated the many sub-plots and sub-contexts. For example, the many metaphysical and theological undertones. For one thing (this is NOT a spoiler), if you're going to read it pay extra attention to peoples' names.
One thing I can reveal is that the book is basically three books but you don't really notice that when it goes from one to the other. I can not imagine a modern day publisher allowing that to happen to a contemporary book with contemporary readers who have less attention span than a gold fish. However, I am glad I've read it because not only is it an entertaining book it's also a good exercise in modern life that not everything has to be so perfect and lean.
And a final tip to you who now feel inspired to read the book for the first time; it's an old-English book with lots of words you won't know and that's fine, but do take the time to look up some of the nautical words related to the ship because they re-appear again and again when you're reading action filled passages. Like bulwark, masthead and starboard.
What's the average number of domains a website depends on?
24 February 2014
For some time now, I've been running an experiment where I analyze how many different domains any website depends on. For example, you might have Google Analytics on your site (that's
www.google-analytics.com) and you might have a Facebook Like button (that's
s-static.ak.facebook.com) and you might serve your images from a CDN (that's
d1ac1bzf3lrf3c.cloudfront.net). That there is 3-4 distinct domains.
Independent of how many requests come from each domain, I wanted to measure how many distinct domains a website depends on so I wrote a script and started collecting random URLs across the web. Most of the time, to get a sample of different URLs I would take the RSS feed on Digg.com and the RSS feed on Hacker News on a periodic basis.
Take this page on The Toast for example, it depends on 143 different domains. Loading it causes your browser to make 391 requests, download 4.8Mb and takes 29 seconds (in total, not necessarily till you can start reading it). What were they thinking!?!
I think what this means is that website makers will probably continue to make websites like this. What we, as web software engineers, can not tell people it's a bad idea but instead to try to do something about it. It's quite far from my expertise but clearly if you want to make the Internet faster, DNS would be an area to focus on.
Test it out for yourself here: Number of Domains
For people familar with AngularJS, it's almost frighteningly easy to make a live-search on a repeating iterator.
Here's such an example: http://jsfiddle.net/r26xm/1/
Out of the box it just works. If nothing is typed into the search field it returns everything.
A big problem with this is that the pattern matching isn't very good. For example, if you search for
ter you get
More realistically you want it to only match with a leading word delimiter. In other words, if you type
ter you want it only to match
Teresa but not
Peter doesn't start with
So, to remedy that we construct a regular expression on the fly with a leading word delimiter. I.e.
Here's an example of that: http://jsfiddle.net/f4Zkm/2/
Now, there's a problem. For every item in the list the regular expression needs to be created and compiled which, when the list is very long, can become incredibly slow.
To remedy that we use
$scope.$watch to create a local regular expression which only happens once per update to
Here's an example of that: http://jsfiddle.net/f4Zkm/4/
That, I think, is a really good pattern. Unfortunately we've left the simplicity but we now have something snappier.
Unfortunately the example is a little bit contrived because the list of names it filters on is so small but the list could be huge. It could also be that we want to make a more advanced regular expression. For example, you might want to allow multiple words to match so as
ter ma should match
John Mayor and
Maria Connor. Then you could make a regular expression with something like
For seasoned Angularnauts this is trivial stuff but it really helped me make an app much faster and smoother. I hope it helps someones else doing something similar.
Because this bit me harder than I was ready for, I thought I'd make a note of it for the next victim.
In Python 2, suppose you have this:
Python 2.7.5 >>> items = [(1, 'A number'), ('a', 'A letter'), (2, 'Another number')]
Sorting them, without specifying how, will automatically notice that it contains tuples:
Python 2.7.5 >>> sorted(items) [(1, 'A number'), (2, 'Another number'), ('a', 'A letter')]
This doesn't work in Python 3 because comparing integers and strings is not allowed. E.g.:
Python 3.3.3 >>> 1 < '1' Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unorderable types: int() < str()
You have to convert them to stings first.
Python 3.3.3 >>> sorted(items, key=lambda x: str(x)) [(1, 'A number'), (2, 'Another number'), ('a', 'A letter')]
If you really need to sort by
1 < '1' this won't work. Then you need a more complex key function. E.g.:
Python 3.3.3 >>> def keyfunction(x): ... v = x ... if isinstance(v, int): v = '0%d' % v ... return v ... >>> sorted(items, key=keyfunction) [(1, 'A number'), (2, 'Another number'), ('1', 'Actually a string')]
That's really messy but the best I can come up with at past 4PM on Friday.
I have and have had many sites that I run. They're all some form of side-project.
What they almost all have in common is two things
- They have very little traffic (thus not particularly mission critical)
- I run everything on one server (no need for "spinning up" new VMs here and there)
Many many years ago, when current interns I work with were mere babies, I started a very simple "procedure".
On the server, in the user directory where the site is deployed, I write a script called something like
upgrade_myproject.shwhich is executable and does what the name of the script is: it upgrades the site.
In the server's
roothome directory I write a script called
restart_myproject.shwhich also does exactly what the name of the script is: it restarts the service.
On my laptop, in my
~/bindirectory I create a script called
UpgradeMyproject.sh(*) which runs
upgrade_myproject.shon the server and runs
restart_myproject.shalso on the server.
And here is, if I may say so, the cleverness of this; I use ssh to execute these scripts remotely by simply piping the commands to
ssh. For example:
#!/bin/bash echo "./upgrade_generousfriends.sh" | ssh -A email@example.com echo "./restart_generousfriends.sh" | ssh firstname.lastname@example.org
That's an example I use for Wish List Granted.
This means that all I need to do run a deployment is just type
UpgradeMyproject.sh[ENTER] and the simple little bash scripts takes care of everything else.
The reason I keep these on the server and not on my laptop is simply because that's where they naturally belong and if I'm ssh'ed in and mess around I don't have to exit out to re-run them.
Here's an example of the
upgrade_generousfriends.sh I use for Wish List Granted:
#!/bin/bash cd generousfriends source venv/bin/activate git pull origin master find . | grep '\.pyc$' | xargs rm -f pip install -r requirements/prod.txt ./manage.py syncdb --noinput ./manage.py migrate webapp.main ./manage.py collectstatic --noinput ./manage.py compress --force echo "Restart must be done by root"
I hope that, by blogging about this, that someone else sees that it doesn't really have to be that complicated. It's not rocket science and most complex tools are only really needed when you have a significant bigger scale in terms of people- and skill-complexity.
Keep it simple.
(*) The reason for the capitalization of my scripts is also an old habit. I use that habit to differentiate my scripts for stuff I install from any third parties.