You're viewing blogs from Zope only. RSS?

View all different categories

7th of March

To all Zope developers: Does this sound familiar?

I was reading this article about linkfluence moving from CouchDB to Riak

"Why we move away from CouchDB

We were already aware of Riak before we started using CouchDB, but we weren’t sure about trusting a new product at this point, so we decided, after some benchmark, to go for CouchDB.

After the first couple of months, it was obvious that this was a bad choice.

Our main problems with CouchDB is scalability, versionning and stability.

Once we store a document in CouchDB, we modify it at least twice after the original write. Each modification generates a new version of the document. This feature is nice for some use-cases, but we don’t need it, and there’s no way to disable it, so the size of our databases started to become really important. You’ll probably tell me “hey, you know you can compact your database ?”, and I’ll answer “sure”. The trouble is that we never managed to get it to compact an entire database without crashing (well, to be honest, with the last version of CouchDB we finally managed to compact one database).

The second issue is that one database == one file. When you have multiple small databases, this is fine. When you a have only a few databases, and some grow to more than 1TB, the problems keep growing too (and it’s a real pain to backup).

We also had a lot of random crashes with CouchDB, even if the last version was quite stable."

Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.

It's the exact same perspectives I've always had on ZODB:

1) It's really convenient and powerful

2) It being a single HUGE file makes it hard to scale

3) Versioning can be nifty but it's often not needed and causes headache with the packing

4) It works great but when it cracks it cracks hard and cryptically

28th of February

Massive improvement on sorting a fat list

IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.


Read the whole text (409 more words)

4th of June

Custom Fields in IssueTrackerProduct documentation written

Custom Fields in IssueTrackerProduct documentation written The Custom Fields feature started as a consultancy job in which we agreed the work can be open sourced as part of IssueTrackerProduct so I never got around to write an sensible high level documentation for it. Now I have! From the news piece about it:

"Custom Fields was a feature that was released almost a year ago but didn't have much documentation. Especially easy documentation that describes what it is and how it can be used. That has changed now.

In Custom Fields it is now described what they are and how they can become useful to you. It's such a powerful tool that very few "competing" issue/bug tracking systems can offer."

The written documentation is here: Custom Fields

Feedback appreciated.

15th of April

Lesson learned: Unicodifying request variables in Zope

This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

 UnicodeDecodeError: \
 'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

 >>> '\xc3' + 'string'
 '\xc3string'
 >>> u'\xc3' + u'string'
 u'\xc3string'

But you can't do this:

 >>> '\xc3' + u'string'
 Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: 
 ordinal not in range(128)

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

 /bla/bla?q=régime

And the template had this little innocent piece of code:

 <input tal:attributes="value request/q"/>

That's what f'ed everything up. So, I ended up having to add this:

 <input tal:attributes="value python:context.unicodify(request['q'])"/> 

With this little helper function in the base class:

 def unicodify(self, s):
    if isinstance(s, str):
        return unicode(s, 'utf8')
    return s

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

1st of October

When '_properties' gets stuck as a persistent attribute

Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:

 def manage_fixPropertiesProblem(self):
     """ fix so _properties becomes a class attribute instead """
     if '_properties' in self.__dict__.keys():
         del self._properties

     return "Awesome!"

30th of May

zope-memory-readings - Tracking Zope2's memory usage by URL

zope-memory-readings - Tracking Zope2's memory usage by URL I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.

I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?

Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.

Here is a sample report: 2008-05-30_16.47.32__3.8_minutes

 

Older entries →