Dries Buytaert

State of Drupal presentation (August 2008)

Last week at DrupalCon Szeged I gave my traditional state of Drupal presentation. The video of the presentation is provided below, and you can download a copy of my slides (PDF, 11MB) as well.

The presentation discusses the results of the recent survey that I conducted; the survey ran for 30+ days and collected more than 1300 responses so it should provide a good idea of the community's current thinking. I'll provide more color and details about the survey results in a number of follow-up posts.

Drupal in the cloud

It is not always easy to scale Drupal -- not because Drupal sucks, but simply because scaling the LAMP stack (including Drupal) takes no small amount of skill. You need to buy the right hardware, install load balancers, setup MySQL servers in master-slave mode, setup static file servers, setup web servers, get PHP working with an opcode cacher, tie in a distributed memory object caching system like memcached, integrate with a content delivery network, watch security advisories for every component in your system and configure and tune the hell out of everything.

Either you can do all of the above yourself, or you outsource it to a company that knows how to do this for you. Both are non-trivial and I can count the number of truly qualified companies on one hand. Tag1 Consulting is one of the few Drupal companies that excel at this, in case you're wondering.

My experience is that MySQL takes the most skill and effort to scale. While proxy-based solutions like MySQL Proxy look promising, I don't see strong signals about it becoming fundamentally easier for mere mortals to scale MySQL.

It is not unlikely that in the future, scaling a Drupal site is done using a radically different model. Amazon EC2, Google App Engine and even Sun Caroline are examples of the hosting revolution that is ahead of us. What is interesting is how these systems already seem to evolve: Amazon EC2 allows you to launch any number of servers but you are pretty much on your own to take advantage of them. Like, you still have to pick the operating system, install and configure MySQL, Apache, PHP and Drupal. Not to mention the fact that you don't have access to a good persistent storage mechanism. No, Amazon S3 doesn't qualify, and yes, they are working to fix this by adding Elastic IP addresses and Availability Zones. Either way, Amazon doesn't make it easier to scale Drupal. Frankly, all it does is making capacity planning a bit easier ...

Then comes along Amazon SimpleDB, Google App Engine and Sun Caroline. Just like Amazon EC2/S3 they provide instant scalability, only they moved things up the stack a level. They provide a managed application environment on top of a managed hosting environment. Google App Engine provides APIs that allow you to do user management, e-mail communication, persistent storage, etc. You no longer have to worry about server management or all of the scale-out configuration. Sun Caroline seems to be positioned somewhere in the middle -- they provide APIs to provision lower level concepts such as processes, disk, network, etc.

Unfortunately for Drupal, Google App Engine is Python-only, but more importantly, a lot of the concepts and APIs don't map onto Drupal. Also, the more I dabble with tools like Hadoop (MapReduce) and CouchDB, the more excited I get, but the more it feels like everything that we do to scale the LAMP stack is suddenly wrong. I'm trying hard to think beyond the relational database model, but I can't figure out how to map Drupal onto this completely different paradigm.

So while the center of gravity may be shifting, I've decided to keep an eye on Amazon's EC2/S3 and Sun's Caroline as they are "relational database friendly". Tools like Elastra are showing a lot of promise. Elastra claims to be the world's first infinitely scalable solution for running standard relational databases in an on-demand computing cloud. If they deliver what they promise, we can instantly scale Drupal without having to embrace a different computing model and without having to do all of the heavy lifting. Specifically exciting is the fact that Elastra teamed up with EnterpriseDB to make their version of PostgreSQL virtually expand across multiple Amazon EC2 nodes. I've already reached out to Elastra, EnterpriseDB and Sun to keep tabs on what is happening.

Hopefully, companies like Elastra, EnterpriseDB, Amazon and Sun will move fast because I can't wait to see relational databases live in the cloud ...

Spam, OpenID and Mollom

There is an interesting discussion about spam and OpenID going on at Matt Mullenweg's blog. The discussion was triggered by the policy decision of social bookmarking site Magnolia to restrict signups to OpenID users. According to the site, 75% of new accounts were being created at Magnolia by spammers using automated tools (our friends the 'spambots'). They say that by restricting access to OpenID users, the rate of spam-account creation decreased. In the discussion, there is a lot of talk about whether OpenID should be used to fight spam, and whether it could be an effective spam-fighting tool in the long term.

Here are my thoughts. Spammers can create OpenIDs too, and a single sign-on system might be many a spammer's wet dream. It gives them easy access to millions of sites in one fell swoop.

Now, OpenID by itself can't prevent spam. All it does is provide a globally unique identifier for any given user on the planet. This is where a tool like Mollom comes in. At Mollom we're already maintaining an internal reputation for each OpenID account we encounter while assessing submitted content. Combine an identity system (OpenID) with a reputation system (Mollom) and it becomes a lot easier to separate spam users from non-spam users. Simon Willison said it best: "a trust system requires identity first". A globally unique identifier combined with reputation tools give us a powerful weapon to fight website spam. OpenID's attribute exchange might become Mollom's best friend ...

Similarly, Tim Berners-Lee is experimenting with combining FOAF ("friend of a friend") and OpenID to fight spam: you can only comment on Tim's blog if you are no more than a certain number of degrees of friendship away from him. Of course, it is a widely accepted theory that we are only six degrees away from everyone in the world so I do wonder how effective this would really be in the long run.

It is still early days in these debates and experiments, but for now, Mollom can already protect your login and submission forms with an image or audio CAPTCHA.

Either way, it is an interesting discussion that makes you wonder. Where will OpenID be in 3 years? Where do you think the website spam problem will be in 3 years? How will this affect online communities?

I have my own thoughts and predictions and it was one of the principal reasons for co-founding Mollom ...

State of Drupal presentation (March 2008)

Last week at DrupalCon Boston I gave my traditional state of Drupal presentation in front of 850 Drupalistas. The video of the presentation is provided below, and you can download a copy of my slides (PDF, 15 MB) as well. The video is available in alternative encoding formats from archive.org.

Topics I talked about: the Drupal 6 release, the state of our union, the need for a drupal.org redesign, the Drupal 7 killer release, the Drupal 7 development cycle, usability, test-driven development, the future of Drupal and the semantic web, etc. There is a lot of material in this presentation and during the course of the next few weeks, I plan to decompose this presentation in a number of extended blog posts. Stay tuned!

From infinite extensibility to infinite interoperability

Jeff Whatcott, Acquia's VP of marketing, wrote an interesting blog post about the term content management system, Microsoft SharePoint 2007, and Drupal's opportunity to be the poster child for the social software market space.

The funny thing is that I replied to Jeff's post back in 2006. In 2006, I agreed with Jeff -- and I still agree with Jeff today -- that (i) the term 'content management system' under sells what Drupal is capable of, (ii) that content management systems are consolidating to community and collaboration platforms, and (iii) that SharePoint is mind-boggling in more than one way.

From a content management system's point of view, we can summarize the current state of affairs as:

Web 1.0 = content management
Web 2.0 = Web 1.0 + user management + infinite extensibility

Jeff said it best when he wrote: the genius of what the [Drupal] community has done is to reduce all of the aspects of social software to their core DNA: content nodes and membership, and then build a platform that could be infinitely extended to allow the assembly of almost any styles of online social interaction.

But while Jeff rightfully sees a business opportunity for the Drupal community in the social publishing market, I tend to worry more about the fact that Drupal's key differentiator (i.e. bundling a wide variety of functionality into a single platform) becomes a commodity.

I want the Drupal community to stay ahead of the competition. I want to start implementing today what proprietary CMS vendors will implement in 2013. From a content management system's point of view, I believe, that means (and I really hate to use the term 'Web 3.0'):

Web 3.0 = Web 2.0 + infinite interoperability

which roughly translates to:

Web 3.0 = Web 2.0 + data portability + web service APIs

While the short-term business opportunity might be to go after the social publishing market, I strongly believe that the long-term business opportunity lies in the infinite interoperability and that spans well beyond the social software market.

Thanks to Open Source software and companies like Google, the cost of building Web 2.0 applications will approach zero. Contrary to what one might think, this actually creates a lot of business opportunities. Opportunities that are best monetized through web services. But for that to happen, ubiquitous and seamless interoperability is key.

State of Drupal presentation (September 2007)

Robert Garrigos from the Drupal Catalan user group just uploaded a video of the State of Drupal presentation that I gave at DrupalCon Barcelona 3 months ago. The presentation shares the results of a survey that I conducted in August; the survey ran for 30+ days, collected more than 1000 responses, and should do a pretty good job capturing the Drupal Zeitgeist.

Have a look at the video below (my presentation starts 12 minutes into the video) and/or download a copy of my slides (PDF, 14MB).

The future of snail mail

A video in Dutch about what we might tell our children about the world we lived in -- nostalgic memories of my childhood and why it is acceptable to close down more post offices. Via Bart De Waele.