I blog therefore I am (by Carlos de la Guardia)

17 Aug, 2008

After all these years, my name is finally on the list of presenters for the Plone Conference 2008. I've been meaning to go to a Plone (or Zope, or Python) Conference for the last few years, but the opportunity never materialized, so I forced myself to go by submitting a talk and hoping it was accepted.

It will be great to be there and finally meet a number of people that I have worked with over the years. I think one of the nice things of attending a Plone conference is that after you get back home the IRC chatrooms must seem more lifelike, as the mysterious IRC handles are associated with real faces and voices.

My talk will be about a project we have under way at the National Library of Congress of Chile, where we are using Plone as a CMS-only system and generating dynamic web sites from the published content. If you are interested in technlogies like WSGI, Repoze (who, bfg, etc.), ContentMirror, Deliverance or z3c.pt and want to see real world examples of their use, you might want to come and listen to my broken english for as long as you can take it.

Who knows? Hopefully someone other than me will get something out of this. For the time being, I'm just happy I'm finally going. See you there!

28 Jul, 2008

I've been very, very busy during the past few months, and my blog has suffered as a result. It's hard to come up with article ideas when you find yourself operating in deadline mode most of the time.

For the past few weeks, instead of sitting in front of a blank screen for a while when I get a few minutes, trying to think of something to write, I've been hammering away at some Python and Javascript code I wrote almost a year ago for an intranet project.

The result of those efforts is called collective.workflowed. It is a graphical workflow editor for Plone 3 and up, with full drag and drop support, property windows for actions and states, color highlighting of connected states and transitions and more.

The javascript code is based on the very powerful draw2d library, from the OpenJacob project. You should take a look at it if you ever want to create tools like this with Javascript.

When I was preparing the first release of this product, Martin Aspeli suggested that I take a look at collective.wtf for some easy to add functionality, and I'm glad he did. This great work by Martin has allowed me to include one-click XML and CSV workflow export facilities, plus a 'sanity check' option that spots problems with permission assignments on workflow states.

The funny thing is that all this work gives me a great reason to fire up my blog post editor and write about it, so I've come along full circle. If you work with Plone, I hope you find this useful. It really feels good to contribute to a project and community that have helped me make a living and take care of my family for the past couple of years.

7 Jun, 2008

Django has quickly become one of the leading web development platforms in the Python world. Still, having used Zope and Plone for years and accustomed to their power, I was unwilling to dedicate much time to it, other than reading a few tutorials and creating a simple application just to taste the waters.

This changed a bit when Google's App Engine came around, since its use of Django meant that I better learn a little more if I wanted to try it some time (I missed out on the first 10,000 invites).

That's why I eagerly accepted the offer from Packt Publishing to send me a review copy of their book Learning Website Development with Django, by Ayman Hourieh. A great opportunity to learn more about Django at the perfect time.

The book is aimed at web developers who are not looking to become experts in the framework, but just want to build a "2.0" web application quickly and painlessly. It is expected that they know some Python, but no great demands are placed on other previous knowledge.

Like most titles from Packt, the book wastes little time on introductions and quickly dives into the subject matter. The reader is put to work from the very beginning, and by page 20 Django is installed and the first project created and launched. I like this practical approach. All the code is available at the book's web site, so it's also easy to follow along.

The book is structured around the development of a social bookmarking application ("Social" and "2.0" seem to be mandatory buzzwords around Django, Rails and some other "new wave" frameworks). Without going into too much detail, each chapter introduces the reader to some new Django concepts needed to build specific features into the application. For example, there are chapters focusing on user management, searching, Ajax and administration interfaces.

By the end of the book, the reader has a working application for storing and sharing bookmarks, complete with tags, tag clouds, RSS, Ajax bells and whistles, voting, friend networks and an administrative interface. She also should end up with a good knowledge of what Django can do and how to use its features. A couple of final chapters on deployment and future directions complete the tour and give the reader some insight on what to do next with her newly gained Django proficiency.

Overall, I think the author explains the concepts well enough and achieves his objective of teaching beginners how to build an application with Django. However, if you have a good knowledge of web development and Python, you may find that you need a deeper source of knowledge about Django as a framework (which by the way is readily available at the free online Django book).

At just over 200 pages and at a price tag of US $40, I find the book somewhat expensive, but it covers the ground it promises, so if you just want to quickly get going with your "web 2.0" application, the book could be worth the price.

22 Apr, 2008

If you need some Plone or Zope 3 training and by any chance were considering Jazkarta's offerings next week (links below), I can get you a small discount and in the process help Alex Clark move plone.org to Plone 3.

I want to contribute to Alex's travel fund for the Paris sprint, but I have no money. However, Jazkarta will give me $50 for every person that signs up for the course using one of the links below. The person who subscribes to the course will also get a $50 discount.

I will donate any money I get from this to Alex Clark's fund. Here are the links:

Plone developer training (April 28-29)

Zope 3 developer training (April 30-May 1)

21 Apr, 2008

If you've been thinking about giving Grok a try for your cool new web application, but felt that the tutorial didn't cover enough ground to really get you started, take a look at the new primer on Grok by the tireless Martijn Faassen (note that this link points directly to a text version on the svn repository, web version should be coming soon).

In this document, aimed at developers, he explains in a very concise fashion all the parts that make up Grok and how they relate to each other. While doing this, he also explains the most important concepts of the Zope 3 Component Architecture (ZCA), which is something that the Zope 3 world wants the Python community to get their hands into (come on, give it a chance, it's only Python).

So now Grok can claim to make Zope 3 easy to use and the ZCA easier to understand, not bad for a band of cavemen.

26 Mar, 2008

Supervisor is a Unix process monitoring system that can automatically start and monitor any number of services for you and restart them whenever they stop unexpectedly. It is written in Python, and is very extensible. It includes a command line client program and also a web interface for easy remote process management.

Supervisor can be extended directly with Python or using any other language via XML-RPC. Its event notification protocol makes programming new event listeners a very simple task.

I used Supervisor recently to set up a Plone installation and I heartily recommend it to anyone who wants to monitor server processes like those used by a Plone site.

My site was using two Zope instances with ZEO, the Pound load balancer and the Varnish cache.
Supervisor allows me to handle all of these via its remote web interface, from where I can start, stop or even check the log files of any process. Supervisor lets you assign a priority to every program that it controls, so that it is possible to start or stop all services with one command in a specific order. Here is a screen capture of the web interface:

Supervisor can be installed with easy_install, which makes it very convenient to use with any Python installation. Integrating it into a Plone site, where zc.buildout is the favorite deployment mechanism, is also very easy. Here's what I added to my buildout.cfg file to use it:

parts = supervisor

[supervisor]
recipe = repoze.recipe.egg
eggs = supervisor

In addition to that, supervisor needs a configuration file. I added an etc directory to my buildout with the following supervisord.conf file in it:

[inet_http_server]
port=127.0.0.1:9999

username=admin
password=admin

[supervisord]
logfile=%(here)s/../var/log/supervisord.log
logfile_maxbytes=50MB
logfile_backups=10 
loglevel=info
pidfile=%(here)s/../var/supervisord.pid
nodaemon=false

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=http://127.0.0.1:9999

[program:zeo]
command = %(here)s/../parts/zeoserver/bin/runzeo
priority = 10

[program:zope_instance1]
command = %(here)s/../parts/instance1/bin/runzope
priority = 20
redirect_stderr = true

[program:zope_instance2]
command = %(here)s/../parts/instance2/bin/runzope
priority = 20
redirect_stderr = true

[program:pound]
command = bin/pidproxy %(here)s/../parts/pound/var/pound.pid %(here)s/../bin/pound
priority = 40
redirect_stderr = true

[program:varnish]
command = %(here)s/../bin/varnish
priority = 60
redirect_stderr = true

Note that supervisor starts its subprocesses via fork/exec and thus they must not dameonize. That's why on the example config Zope and ZEO are started using runzope and runzeo rather than zopectl and zeoctl.

For programs which launch another process separately and control that via PID files, like MySQL or Pound in the example above, the pidproxy script can take care of this. Just give it the launch script and the PID file locations in the config and everything will work OK.

Supervisor is a great solution for process monitoring and this post just scratches the surface of what is possible with it. I hope I can post later about using its extension mechanism. In the meantime, I recommend you try it out or at least read the manual and see what it can do.

21 Mar, 2008

Packt Publishing was generous enough to send me for review a copy of their book, Hacking Vim. I was really looking forward to this, but then some family matters prevented me from finishing the task within a reasonable amount of time. This is my late, late, late review.

Hacking Vim is a cookbook for becoming more productive using Vim and GVim. At 200 pages, it is not a large book, but it is packed with recipes and tips. The recipes range from simple stuff, like changing the color scheme, to advanced tasks, like creating and distributing Vim scripts. Some of the recipes apply only to GVim.

The book begins with an introduction, where the author quickly goes through the genealogy of Vim and explains how it differs from similar editors, like Elvis or Vi itself. This is short and to the point. It can easily be skipped if the reader wants to go straight to the recipes.

The recipes are organized into four categories, each devoted a separate chapter: personalization, navigation, production boosters and advanced formatting. Each chapter contains around ten to fifteen recipes. In addition, there are tips scattered all through the book. In general, I liked the format and the table of contents makes it easy to find a given recipe.

Personal favorites in the personalization chapter were spell checking, using abbreviations and modifying key bindings. In the navigation chapter I liked the explanations for search and markers. My favorite chapter was the one about production boosters, specially templates, tag lists, macros and opening files everywhere. On the formatting chapter I looked more into code formatting than text, since I use Vim mostly for development.

The final chapter is a tutorial for creating Vim scripts, covering everything from development to distribution. Instructions are included for creating the scripts using other programming languages, like Python and Ruby.

The appendices are more useful than in the average book. They include a survey of unusual Vim plugins, like games and mail clients. A sort of bonus recipe about storing the Vim configuration file online is also included.

Though I use Vim a lot, I'm not what you could call a power user. I don't think I'll ever write a Vim script, for example. Overall, however, I got some benefit from the book, even though serious practice is needed to really take advantage of some recipes.

That said, things like templates and tag lists can make the light Vim user much more productive without too much effort. Also, there are references to many Vim plugins that further simplify the work needed to get some stuff done. My Plone and Python development immediately became easier because of all this.

In conclusion, if you use Vim and are not an expert, I think this is a valuable book to own, though the US $40 price tag seems a little steep to me (it is also available as an e-book for half of that).

29 Feb, 2008

I recently gave a small introductory talk about Plone at the CONSOL (National Free Software Congress) here in México. The aim was to introduce new developers to Plone.

There were about a couple dozen students present, most of them quite familiar with Drupal or Joomla. They barely had heard about Plone or Python. No one had actually installed Plone at all.

The talk went reasonably well, but I was disappointed in that no one asked a question at the end. A couple of days later, however, one of the students contacted me by email to ask advice on how to install Plone.

He was specially interested in finding out how to upload what he did on his machine to a cheap hosting service shared by him and some friends. You see, with the poor economic situation for many students of state universities in México, the pooling of resources with other friends to have a web development testbed shows a lot of initiative. Even as I was writing to him that he just can't use Plone on that kind of hosting environment, I knew that would be the last of his experiences with Plone.

Of course, customers will always be able to use a real hosting solution for their projects, but new developers sometimes won't. In some countries, like México, this can have an impact in the adoption of Plone, since customers will be scared of going with it and then not finding any developers to maintain and extend their sites.

Now, I don't think that the solution is to try to make Plone run in cheap hosting environments. The effort is just not worth it. What could be done, a thing which would help Plone adoption at all levels, is to pursuit Martin Aspeli's vision of the Big Green Button that turns a Plone site into a static (or even potentially dynamic) version of itself that can be hosted with one of those $5 a month providers.

I have been looking at Entransit, from Enfold Systems, and I think it can do most of what Martin describes on his blog post. However, Entransit's out of the box experience is, according to Alan Runyan, very limited and just a small glimpse of what is possible with it.

Entransit also has little documentation and very few mentions on plone.org, but that could change fast, since it has been tagged as a possibility for delivering Plone content on the 4.0 release notes.

I am currently testing Entransit for a client's project. I'll let you know how it goes.

28 Feb, 2008

You often hear sports figures talking about giving a 110 per cent effort for some important game, but Plone walks the talk. Witness the following screen capture, which is proof that Plone coaxed at least one machine into giving this seemingly impossible effort.

Who says that Plone can't perform?

31 Jan, 2008

I have written a couple of tutorials for using Repoze under mod_wsgi:

• Install Plone 3 behind Apache and mod_wsgi using Repoze
• Installing and setting up Grok under mod_wsgi using Repoze

Give Repoze a try. WSGI will be an important part of the future of Zope and Repoze can get you there now.

28 Jan, 2008

Malthe Borch has a provocative take on what he calls a revolutionary approach to future Plone core development. Basically, create a parallel branch of Plone that uses the same recently componentized packages in the Plone core, but is otherwise free of any backwards compatibility requirements.

This idea is always attractive to developers. Why deal with all the cruft accumulated in years of evolution when you can start with an almost blank slate? A complete rewrite can lead to a cleaner, more modern implementation and in a shorter time too!

The problem is that, first, this is not entirely true. Second, you may lose many users while you do the grand rewrite and even after you are done (assuming some fairly flexible definition of done, because getting that rewrite to the dreamed-of perfect state envisioned at the beginning is really easier said than done).

We've had a very close complete rewrite experience in the Zope community in recent years. Zope 3 aimed to be the replacement for Zope 2, and some very good developers underwent the task of taking all the lessons learned from years of Zope development and distilling those into a new version of Zope that also took into account modern development paradigms.

The result? A new, great, powerful, successful web development library...which almost nobody knows or cares about outside the Zope community. Those who know that it exists are often very confused about its relationship to the old Zope ("please sir, could you tell me how to install Plone on my new Zope 3 instance?"). Heck, I have been following this story for years and still don't know whether to call Zope 3 a library or a framework.

Keeping the Zope name also backfired in unexpected ways, as the project inherited the perception of unwieldy and unpythonic (whatever that means) that it's namesake had acquired in the Python community by this time, thus limiting the number of new adopters.

I think the Zope 3 story shows us that it takes more than a name to maintain continuity between two otherwise disconnected pieces of software. You need an active and ongoing effort to make this connection apparent to outsiders or you will get a lot of confusion. Like Paul Everitt said in the comments to Malthe's post, until the Five project came along, there was no apparent bridge between the projects, thus the Zope 3 name turned out to be misleading.

The truth is, there are many uncertainties in a developers work to even think about working with a framework with an identity crisis. That's what you get with complete rewrites of frameworks with large user bases, because inevitably, a large portion of those users will stick with the old code. A good level of backward compatibility is necessary to maintain the user base. You just can't tell a user that all or most of his accumulated knowledge is no longer useful, even if it's for the better (in many cases, this means better for the developer, not the user).

To be succesful, a revolution needs to get the support of the masses. Thus, its ideals need at least to be based on the people's aspirations and needs. That's why I agree completely with Martin Aspeli's comment about the driving force needing to be at the users, integrators and businesses end. To me, an evolutionary path sprinkled with revolutionary ideas would be wiser than a complete break and rebuild strategy.

One last comment about starting from scratch: even a complete clean slate restart is no piece of cake to get right. The Django web application framework was started from the ground up by another group of talented Python developers and was in controlled use for some time in a production environment before its authors decided to open it up for the world. Not very long after going public, a drastic refactoring of some parts was deemed necessary (the so-called magic removal branch), to "remove warts that Django had accumulated over the years".

In Django's case, I think that refactoring was done early and worked fine, but the point is it was a fairly well controlled project and they still needed to remove warts. I doubt they could get away with something like that at this time. They have too broad an user base for that at this point. Books have been published, tutorials written, sites launched...there's no going back now.

I think Malthe would do a great good to the Plone community if he pursues his ideas and contrasts his approach with Plone frequently, so that some of his work can be integrated with the Plone core at some time. Vudo is a good name, I'd say. For better or worse, let Plone be Plone and continue its evolutionary path.

24 Jan, 2008

A recent post on Coderspiel mentions that Lucene does not use Lucene for indexing their own web site and goes on to criticize projects that don't use their own software on their sites. He asks:

"How is it that some fancy-pants framework is always the right tool for an abstract job and PHP is the right tool for a real job?"

He is mostly referring to Java projects under the Apache umbrella, but I think the question is interesting for the Python world too. Does a web application or project "lose points" if it doesn't use itself on its own site?

The Grok web framework has just launched a new web site, and there was some discussion on their mailing list about using Grok for this task, the dogfood approach, or "the right tool for the job". Ultimately, they settled on Plone, because they need CMS capabilities and Plone is, well, a CMS. Check out their site, I think it went well.

Speaking of Plone, they use Plone itself for their site too, though not the latest version. A prospective user can join the site easily and learn what Plone is all about while using it. In this case, the dogfood approach just works.

Zope, on the other hand, uses itself too (more precisely, a Frankenstein old Plone setup), but the results are far from appealing. For Zope, the dogfood approach seems counterproductive, though I would say that's not because of Zope itself.

Django's main site seems to use "the right tool for the job" and not itself for their main site, but and their showcase Django projects site both use Django. I should say that Django project related sites that I have seen are generally appealing regardless of the approach taken.

Pylons uses Confluence, a commercial wiki, and Turbogears, also sporting a nice looking site, uses moinmoin, at least for their documentation site (not sure what they use for the main site, but there's a chance it could be Turbogears itself).

Of course, how the site really looks is not dependent on the dogfood or right tool approaches, but the Zope case seems a reminder that if you are going to eat your own dogfood, you may want to spice it a little at least.

Overall, I cannot say I see a clear advantage for the dogfood approach. I think it has some value to show what you can do, but it all goes for nought if presented in the wrong way. Ultimately, the mission of a web site is to present the project, which can be accomplished fairly well without using it.

In the case of very specific technologies like Lucene, eating the dogfood may be more important, but as far as I can tell, it hasn't stopped or even slowed down adoption of the technology. Having access to a demo site can sometimes be useful for a developer, but easy to install packages and good documentation are really far more important.

Update: Simon Willison points out that Django's main site does indeed use Django. He and other commenters also rightly mention that many developer sites use a combination of tools, like issue trackers, wikis and svn frontends and it makes no sense to rewrite all of those just to eat your own dogfood. Reinout van Rees gently reminds me that I use the right tool approach for my own blog (I've had clients make fun of me because of this, actually).

22 Jan, 2008

The TIOBE programming community index has declared Python as the language of the year 2007, based on it's growth during the period as compared to other languages.

Without all the hype of last years wunderkind, Ruby, Python advanced from 8th to 6th place, overtaking Perl for the first time ever. Ruby lost one place after Delphi (!) climbed from 12th to 9th place and pushed last years fastest growing language into the 11th position, behind Javascript.

Python had a +2.04 percent growth for the year to get to 5.538 percent overall usage (as you can see, 'fastest growing' doesn't mean 'most used' by a long shot, so don't lose perspective here). Ruby's growth for the year? -0.17 percent. Its growth over the year of the big Ruby on Rails push? +2.15 percent. That's right, without all the noise, Python grew almost the same in 2007 as Ruby did in 2006.

There are several reason why, I think. First, keep in mind the TIOBE index ratings attempt to give an indication of programming language popularity and are based, according to the TIOBE site, on the world-wide availability of skilled engineers, courses and third party vendors. Search engines play a big part in the rating process, as can be seen in the definition at their site.

Now, what caused Python's popularity surge? I'd venture the following guesses:

• The popularity of strong agile web development frameworks, like Django, Pylons and Turbogears. Many a Python fan that had been lured away by RoR's seductive appeal came back to the language because of them.
• The growing use of Python as a system administrator's tool over Perl. Python, like Perl, is present in almost every system and has a nice standard library in place. Unlike Perl, Python is just easy.
• The constant if sometimes unaknowledged push of Zope and it's related technologies. Plone is one of the most heavily used Python based applications out there, and more than ten years later, Zope still continues to innovate and influence the Python development world.

Anyway, it's a great time to be a Python developer. No matter what part of the growth curve you and your favorite Python projects represent, please keep at it.

22 Jan, 2008

Doug Hellmann wonders what tools Python developers find more necessary to their work these days. In my case I would say vim, which got a lot of votes on Doug's comments page. Also, I would probably be out of business if Google didn't exist.

Of the tools mentioned on his site, I've used Wing IDE sometimes and I think it's cool, but somehow I always end up going back to vim. I might try out ipython one of these days, to find out the reason why it got a so many votes.

On the Zope/Plone front, I'd say zc.buildout is becoming one of the more important tools for development and deployment. I don't think I use any Zope or Plone specific tools besides that.

I am a Linux guy, but I guess we are becoming a minority on the Zope/Plone community with OSX growing in popularity (incidentally, call it envy, but I think the Mac experience has to be out of this world for Python developers to accept, and pay big bucks for, an OS with things like broken readline support for the Python interpreter). What tools do OSX developers find invaluable for Python work? Is there anything out there that makes Zope/Plone development easier on that OS?

21 Jan, 2008

I think Repoze is a potentially great thing for the Zope world, so I've been experimenting with getting different Repoze projects running under mod_wsgi and apache, which I think may become the standard deployment solution for production Python WSGI apps.

I recently wrote a step by step guide to setting up Zope and Plone in this manner. Now it's Grok's turn.

Grok is nearing a 1.0 release and is a web development framework which makes the following claims:

1. Easy to use.
2. For Python developers.
3. Powerful.
4. Based on Zope 3 technologies.

For those of you that will immediately jump and say that claim number four is incompatible with one and two above, please at least visit Grok's web site and give it a try. We are really talking about a Python web development framework here.

Getting Grok to run under mod_wsgi is very easy, but as of this writing you will need to check out repoze.grok from the subversion trunk, since it contains fixes to make the WSGI setup work using the included sample files instead of rolling your own.

Once you have the correct version, setting it up is very straightforward. Install mod_wsgi under Apache and make sure you have a Python 2.4 installation ready with setuptools already installed. Check my post on Plone behind mod_wsgi if you need more detailed instructions for this.

Next, install and run repoze.project:

$ easy_install-2.4 -f http://dist.repoze.org repoze.project

Repoze uses virtualenv to create sandboxes for the applications it runs, so that each one has an independent Python environment of its own and avoids egg conflicts with other applications. To create the Grok sandbox:

$ repozeproject repoze.grok --path=/home/cguardia/repoze/grok

Now all you need to do is copy the contents of the apache2.conf file from the etc directory in your sandbox into your Apache configuration (with the appropiate editsfor your site, of course) and rename the sample-users.zcml file in the same directory to users.zcml (or create you own) and Grok is ready to run.

Since this configuration is intended for production use, you will also need to setup a ZEO server and configure the zope process to connect to it. Just run 'bin/mkzeoinstance .' from your sandbox directory and add the required configuration to the zope.conf file. Here is an example:

<zodb_db main>
    cache-size 5000
    <zeoclient>
        server localhost:8100
        storage 1
        cache-size 20MB
        name zeostorage
        var $INSTANCE/var
    </zeoclient>
    mount-point /
</zodb_db>

That's all there is to it, you can now run Grok behind mod_wsgi by restarting the ZEO and Apache processes:

$ bin/zeoctl start

$ sudo apache2ctl start

You Grok application is ready to run under mod_wsgi, and perhaps be combined with other Python WSGI applications on the same site.