Uriverse, a Drupal based website, was released in January 2010. Much of the data in Uriverse is based upon a data import from DBpedia, a semantic version of Wikipedia. Uriverse contains over 13M nodes and contains 90 languages covering around 3M primary subjects. This article is a case study of how the import was done and the challenges faced.

Drupal proves itself to be a flexible system which can handle large amounts of data so long as some bottlenecks are worked around and the hardware, particularly RAM, is sufficient to handle database indexes. The large data set was tedious to load but in the end the value added by Drupal made it worth it.

Motivation

DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web. DBpedia allows you to ask sophisticated queries against Wikipedia, and to link other data sets on the Web to Wikipedia data. We hope this will make it easier for the amazing amount of information in Wikipedia to be used in new and interesting ways, and that it might inspire new mechanisms for navigating, linking and improving the encyclopaedia itself.

Over the years I have had an interest in the semantic web and linked data in particular. The DBpedia project had always impressed me because it was an effort which allowed Wikipedia to be used as a hub of subjects for use in the semantic web. Wikipedia represents most of the commonly discussed subjects of our times, created and edited from the ground up by real people. It therefore forms a practical basis from which to build out relationships to other data sets. If you are looking for subjects to represent things you want to talk about the DBpedia is a good place to start. There is an increasing momentum around it as the linked data meme starts to spread.

The Linked Open Data Cloud

Not only has the DBpedia project formalized these subjects, it has extracted a large amount of information which was codified in the somewhat unstructured wikitext syntax used by Wikipedia. The knowledge within Wikipedia has been made explicit in a way that can be used (browsed, queried and linked) easily by other systems. The DBpedia data set therefore provides a convenient store for import into a CMS such as Drupal.

Choosing to import DBpedia into a content management system is not necessarily a natural thing to do. The RDF data model is very simple and flexible and allows for all kinds of data structures which may not fit well within a system such as Drupal. It may seem like I was attempting to get the worms back into the can after it had been opened. At times it did However, there was a good deal of regularity and structure within DBpedia. The provision of an ontology and a “strict” infobox mapping in version 3.4 made the importation process possible. Whilst not everything could be imported, most of DBpedia made it in. Concessions had to be made along the way, and the places where Drupal proved to be too inflexible are indicative of areas which could be improved in Drupal. More on that later.

I chose Drupal as the target platform because I had been impressed with the flexibility I had seen from the system. Unlike other popular blogging platforms/CMSs, Drupal has a decent way of specifying a schema through the Content Construction Kit (CCK). I believed that it was possible to mimic the basic subject-predicate-object structure of RDF. Drupal supports different content types with custom properties (strings, dates, integers, floats) and relationships between objects (node references, node referrer). It also has a relatively strong category system baked in which can be used for filtering. Drupal also offers a lot of other attractive features apart from the data modeling: users, permissions, themes, maps, ratings, timelines, data views, friendly URLs, SEO, etc. Utilizing these features was the carrot to get the data into the system.

The DBpedia data set

Version 3.4 of DBpedia is based upon a September 2009 dump from Wikipedia. Here’s a quick list of some of the data set’s statistics:

• 2.9 million things
• 282,000 persons
• 339,000 places
• 88,000 music albums
• 44,000 films
• 15,000 video games
• 119,000 organizations
• 130,000 species
• 4,400 diseases
• 91 different languages
• 807,000 links to images
• 3,840,000 links to external web pages
• 415,000 Wikipedia categories

The data set is provided in two main formats: N-Triples and CSV. The N-Triples format is suitable for loading into RDF stores with RDF software such as ARC. The CSV format was handy for quickly loading data into a relational database such as MySQL. In a data set as big as DBpedia it is essential that you do things the quickest way lest you will be spending weeks importing the data. Here are some very rough comparisons of import speed. NB: These are from memory and are very rough estimates.

Obviously if you are importing 100s of millions of triples these comparisons are important. (I can’t quite believe the last result but that is what I remember it to be.) Put your dumps and DB on different disks when doing imports! Mind you, adding the indexes after the data has been imported is still a very time consuming process requiring data to be copied to disk so the super fast results are a bit misleading.

Mapping to Drupal structures

The DBpedia data set can be divided into two parts: infobox and non-infobox. Infobox data includes all of the varied properties which are applied to each class as defined in the ontology. As such, it presents a substantial data modeling exercise to fit it into Drupal. Much of my time was spent analyzing the class hierarchy and properties to work out the sweet spot as to how content types and properties would be allocated. The non-infobox data is much more predictable and easier to import in standard ways.

Content Types, CCK and Importation Concessions

There were a number of areas where there was not a clear mapping between DBpedia and the features offered by CCK. The notes below refer to Drupal 6, and do not consider the Fields in Core initiative in Drupal 7.

Property Explosion

Drupal handles object properties through a system known as the Content Construction Kit (CCK). CCK offers a handy interface for defining a schema for various content types. Each object (node) is a instance of a single class (content type). Each node therefore has the properties of its content type.

Those of you familiar with the inner workings of CCK in Drupal 6 will understand the idiosyncrasies of the way CCK works behind the scenes. On the backend things work as expected up to a certain level and then they get a bit complicated. Properties for a content type are grouped together in a single database table, as you would expect. There are two exceptions to this rule. Firstly, if the property can have multiple values, it is stored in a separate table. This too is natural enough. Secondly, if two content types share a property then it is split out into its own table. This is a bit strange and can catch you unaware if you aren’t expecting it. However, it is sensible enough as it allows easy queries across content types on a single property.

Things become tricky when you have (i) lots of “multi” properties or (ii) lots of “shared” properties. Drupal needs to issue a new query on a different table to get the data back for that property. This is alright for most sites but has the potential to be a worry in the case of DBpedia where there are a massive amount of different relationships and in some cases relatively few instances of those relationships. ie. the data is not very dense. We are potentially talking about 100s of properties which would need to be retrieved.

Unfortunately, in these cases it make sense to pick only the properties where you get he most bang for your buck. Which “shared” properties are shared amongst the most types? Which “multi” properties have the most instances? Which “single” properties have the most density down the rows? Along the way we had to be pragmatic and pick and choose the properties we would support. At the end of the day this wasn’t a limitation of what CCK, rather a sensible decisions were made to stop the database exploding out into thousands of tables with very little data in them.

I don’t see an easy way to solve the table “explosion” problem save from moving to an RDF datastore such as that implemented by ARC. For data sets which have demands similar to DBpedia it makes sense to have something such as an RDF store in the backend. This conclusion is incredibly ironic given the lengths I have gone to to get the data into DBpedia. All was not lost however as the majority of data made it in and is usable within Drupal using standard techniques.

Interestingly this very issue seems to have plagued the backend design process for Drupal 7. According to the DADS Final Report (4th Feb 2009), “CCK 2 for Drupal will drop its variable schema and use the multi-value schema style for all fields.” Hmmm. I haven’t checked out Drupal 7 in this much detail but if this is true then the table explosion problem is going to be worse in Drupal 7. I’m not abreast of current developments here so I can’t comment further.

No sub classing properties

Drupal doesn’t allow for for sub classing content types. Each content type exists as its own base class. This means that we have to define base content types with all of the properties of the contained subclasses. The ontology in DBpedia can be up to four levels deep: Eurovsion Song Contest Entry is the deepest, preceded by Song, Musical Work, Work and finally Thing (a catch all at the root). This of course leads to base content types with many properties which will be null for the instance in question. The database tables would become very wide and have relatively low density of information.

The Taxonomy system does allow us to partially work around the sub classing problem. A class hierarchy maps nicely to a hierarchy of Terms in a Taxonomy. Further, multiple Terms can be applied to a Node making it possible to specify different classes for a node. It doesn’t cover the difficulty of property storage however.

No Multiple inheritance

When it comes to data modeling there are different ways to handle typing. The most simplistic and limited way is to allow instances to have a single class. This is the way Drupal currently works with content types and CCK. Each node belongs to a single content type and has the properties defined by CCK for that node. A more flexible way of modeling data allows for multiple inheritance where an instance can have more than one class.

Where an instance did straddle two base classes it was impossible to carry data for both types. This query shows all “people” who are “works” as well. I think these cases can be put down to DBpedia being a bit too promiscuous in the infoboxes it processes for each article. This isn’t a strong argument for multiple inheritance because the data is probably erroneous, however, it does demonstrate an area where modeling could be more flexible.

Compound Types

In some cases a compound type was required. For example, images data had three components: thumbnail link, depiction link and copyright info link. All three of these should have been considered one unit, however this is not possible through the standard CCK interface which handles atomistic primitives. Because these image properties applied to multiple content types, the end outcome was that they were represented by different database tables. It was very frustrating to know three queries were being issues when one (or none) would suffice. It is possible to define your own custom datatypes through a module but this is a fairly high barrier to jump.

Interfaces: A possible solution

Freebase is a collaboratively edited system of open data which is similar to DBpedia in many respects. The main difference is that Freebase allows users to create and edit content according to certain schemas. One of the very impressive aspects of the Freebase system is its ability to support multiple types, or co-types, for an object. From the Freebase data modeling guide we have:

A novel aspect of the Metaweb system is that instances may have multiple types. A single topic such as “Kevin Bacon” may have multiple types such as a Person, Film Actor, TV Actor and Musical Artist. and others. Since no single type could encapsulate such diversity, multiple types are required to hold all properties to fully describe Kevin Bacon and his life.

This approach has the advantage of grouping properties in a table allowing for fast retrieval and querying, as well as allowing for flexibility and sensible design. I believe that Drupal could benefit from the Freebase approach. The current content type + CCK model could remain in place AND be augmented by an interface system which allowed for grouping of properties for various types. To take the Kevin Bacon example, “Kevin Bacon” would be a Person content type with the base properties of Birthday and Deathday. “Kevin Bacon” would then have the FilmActor, TVActor and MusicalArtist interfaces which could be represented by separate tables on the backend. I believe that this offers good flexibility for those desiring a powerful system whilst maintaining simplicity for those who just need base types. It also solves a lot of the hand wringing which goes with the way some CCK tables are formed.

Import into Drupal

As a new comer to Drupal, by far the most disappointing aspect of the system was the lack of a clear and easy to understand API. I assumed that there would be a nice object abstraction which I could use to populate the system. I gradually came to understand that most development was done by examining the excretions of print_r() to determine what data was available at that particular moment. Where was the interface to content types, CCK and nodes? How could I create these programatically? There were times where I stopped and paused and considered a framework which was cleaner and more lightweight. The rich functionality was the thing that kept me though.

The large size of the data set pretty much dictated that it needed to be imported in the most efficient way possible. If I accepted a 1 second overhead for a save/update I would be waiting four months at least for the data to load. So, notwithstanding the state of the API in Drupal 6, a straight database import was the order of the day. After a bit of reverse engineering mucking around I had a few PHP/SQL scripts which could insert content pretty quickly, with insertion rates of around 1000 rows a second.

The import process followed these simplified steps for the various pieces of DBpedia.

• Import DBpedia data from CSV format into MySQL.
• Create a staging_node table with columns: title, language, teaser, content, url_alias, nid.
• DBpedia data populated into staging_node and cleaned.
• staging_node data copied into Drupal database.

The process was therefore semi automated with a number of scripts. It still took a few weeks to run through from start to finish. Importing 75M page links was the final time consuming process. I could only get throughput of about <200 rows a second as a url_alias to id lookup was required. This part of the process took around 7 days. Not something I want to repeat.

During the import I realized a few things about MySQL techniques which may come in handy for other people.

• Loading data into MySQL direct from file, with no index is very fast.
• It’s even faster if the source file is on a different disk to the DB.
• The overhead from very heavy/large select queries can be minimized by using mysql_unbuffered_query. The connection can be fragile though so be prepared with an internal counter so you know where the process got up to before it died. You can’t write to the table you are reading from and it is locked for other operations.
• Sometimes pure SQL is a good way to go: INSERT INTO SELECT
• Sometimes joining is no good and running a SELECT for each row is best (if caching and keys) are working.
• SHOW FULL PROCESSLIST is your friend.
• Sorting is your enemy.
• Dumping temp tables to disk must be avoided. Try tweaking your conf. You want to hear the cooling fans humming rather than the disk ticking away.

Drupal Features

Characteristics of the rich DBpedia data set provided a good foundation for using the following Drupal features:

• Image thumbnails pointing to Wikipedia display in the contents expanding to full depiction via thickbox.
• Geo cordinates for Nodes displayed in Google Map via Geo module.
• Geo Cordinates used with Flickr API to pull back Creative Commons images for places from Flickr.
• FiveStar ratings used on People, Places, Organizations and Music Genres.
• Views provided top rated lists of the most popular things.
• Simile Timeline widget used to display chronology of events.
• Solr used as search engine and filters on language and class.
• Solr recommends “More like this” based on Node content.
• Filtered Views provide lookups for titles, firstname, lastname and geo coordinates
• Various node properties allow for sorted list Views of richest people etc.

Performance Shortcomings

OK. So the data is in Drupal – were there any problems when it came to running the site? Yes, I ran into a few challenges along the way. Some were fixed, others worked around and others still remain a thorn in our side.

Database Indexes

The database is the most pressing concern when it comes to performance on a big site. If simple queries run slowly then the site will not function acceptably even for low traffic. The most important area is to ensure that indexes for key tables have been loaded into the key buffer. Lets look at a couple of simple selects with and without a primed key buffer.

If these indexes aren’t in RAM then the most basic of lookups will take a long time, ie. 0.07s. If you are on a page with a view with 50 nodes to look up then just getting the title and path out will take (0.07s * 50) 3.5 seconds. Note that this doesn’t include all the other processing Drupal must do. This in completely unacceptable and so it is mandatory to get these indexes into RAM. I recommend putting the following SQL into a file and running it every time MySQL is started up using the init-file variable in my.cnf.


USE drupal6;
LOAD INDEX INTO CACHE node;
LOAD INDEX INTO CACHE node_revisions;
LOAD INDEX INTO CACHE url_alias;
LOAD INDEX INTO CACHE term_data;
LOAD INDEX INTO CACHE term_node;


On massive sites you probably won’t be able to get all of the node indexes into the key buffer, even if you have been generous in its allocation (up to 50% of RAM). In this case I resorted to running a query which seems to get the node nids into the buffer whilst leaving out all the other indexes which aren’t used as much. It takes a while to run but does the trick.


select count(n.nid), count(n2.nid) from node n inner join node n2 on n.nid=n2.nid;


In the best case scenario we would all have RAM (money) to burn but unfortunately that’s generally not the case.

Core and Contributed modules

There are a few areas in Drupal which bogged down when handling huge data sets. As I developed the site I took note of problematic areas. Most of these areas are probably well known so we’ll just mention them briefly.

• In general a node load is very heavy. Lazy loading of CCK properties would make the system much faster if CCK didn’t have to be loaded. It would also mean the API could be used when speed is an issue. ie. when processing millions of nodes at once. During import and update, the solution is to work directly with the database. Just for a laugh I tried setting the status to 0 for a large node but the 404 still tried to load the whole node and then died.
• Editing a node with many multi properties it all but impossible. The edit page size is massive and RAM/CPU is hammered. Solution is not to edit! Real solution is to page multi properties in node edit.
• Viewing a page with many multi properties requires that the properties be paged. Looking up all those nodes gets slow quickly even with fast database. Solution is to use a module such as CCK Pager.
• Viewing list of content is not possible as SQL query relies on a join to user table. This query kills the database. Solution is to make a small hack to core to stop this join. If users were looked up with separate queries then this would be a better solution.
• Search is impossible from many angles. Queries for indexing are very slow. Database not designed to handle such large amounts of data. Solution is to let Solr handle search.
• Solr search design is generally good with data held in apachesolr_search_node. However, Solr indexing can be a drain if you exclude node types from search. The preparatory query to return the nodes will inner join to node and lead to a very slow query. It returns after a while (70s for me) so you can live with it. Definitely not something you want to be doing regularly on production server as CPU goes to 100%. Solution is (i) to replicate node type data in apachesolr_search_node or (ii) get the excluded nodes anyway and ignore them. First option is best.
• Taxonomy pages fail when there are many nodes in a category. SQL query is very slow. Solution is to let Solr show Taxonomy pages.
• Strangely, the Search module still was running even when Solr was handling search. Core had to be hacked to turn it off. There must be a better way but I couldn’t see it. Make sure the “search” tables aren’t populated when using Solr.
• Views queries can be slow with no extra indexes on content tables.
• Views queries use left joins which can be slow. Inner joins exclude a lot of rows you don’t need. Solution is to rewrite the Views SQL with a hook.
• Displaying nodes or teasers in Views can be very RAM intensive if the displayed nodes are very heavy with CCK fields. Solution is to use fields.
• The Voting API module stores data in an inefficient way leading to some hairy indexes on the table and some equally hairy queries in Views when trying to display ratings. Be careful.
• CCK node referrers datatype has a mandatory sort order specified. This kills query performance for large result sets.
• XML Sitemap has some queries which return massive data sets which need to be dumped to disk. I know this module is in the process of being reworked.

Solr: Some stats

Uriverse uses Solr as its search engine and it is a component which has performed remarkably well. The filtering capabilities of Solr are excellent and the speed at which results come back are very impressive, even for large corpuses. Since it runs as a service over HTTP it is possible to deploy it to a second server to reduce load on the web server/DB box.

It takes a while to index 10M articles (categories, redirects, disambiguations, photos and pages were excluded from the 13M) and it is a task not suited for the standard Drupal cron. A custom script was written to build the index so that it could hammer away almost constantly without upsetting the other tasks cron performs. Initially the script was written to be a long running process which would potentially run for months. However, a memory leak in Drupal meant that this was not possible. After 10 000 nodes RAM became to much for PHP and the script died. The solution was to limit the number of nodes processed each time. The script now processes around 8 000 nodes every 30 minutes. It therefore takes around a month to build the index.

On the web there are quite a few articles regarding memory usage with Solr. The JVM needs to be given enough room when started. These reports had me concerned because I am running 32 bit machines with only 3G at my disposal. Would an index of 10M articles run in a JVM limited to around 2G? What size would the index be on disk? These are the numbers for the 7883304 articles currently in the index:

Obviously these numbers are dependent on the average size of the title, body, fields and taxonomy. RAM is also affected by number of facets, sorting and server load. I therefore I have been very conservative in what I have indexed and have turned of sorting in the interface. It looks like the current RAM allocation will be sufficient.

Performance

Most Drupal performance best practices (opcode cache, aggregation, compression, Boost, Expires, database indexes, views and block caches) have been followed to get the most out of the site. A CDN has not been deployed because Uriverse doesn’t serve many images from its own server. The images that are served have an expires header and all other images come from Wikipedia and Flickr which probably have their own CDN solutions in place. Further optimizations would include going with MPM Worker + fcgid or Nginx. This will be required if traffic picks up and MaxClients is reached.

Two problematic areas remain. The first is the amount of RAM available for the database indexes. It would be nice to be able to increase that one day. Ordinary node page build times do come in at respectable times considering so this is not such a big issue. The second problematic area is some of the queries in Views. A bit more research is required here but it is likely that some Views will have to be dumped if they are hitting the disk with temp tables. Sometimes its easiest to forgo some functionality to maintain the health of the server.

Conclusion

All up the project has taken longer than expected – more than a few months. Most of the time was spent wrangling the data rather than fighting Drupal, although there were quite a few issues to work through, this being my first serious Drupal project. If I knew of the pain I would suffer from having to massage and prepare the data as well as the patience required to babysit the import process over days and weeks then I probably wouldn’t have commenced the project. That said, I am pleased with the outcome now that it is all in. I am able to leverage the power of many contributed modules to bring the data to life. There is a great sense of satisfaction seeing Solr return results from 10M articles in 90 languages, as well as the pretty theme, Google Maps, Thickbox, Similie timelines and Five Star ratings. I am humbled by the efforts of all Drupal contributors over the years. What I am left with now is a good platform which will form the basis of future data aggregation efforts.

Appendices

Class hierarchy

• Place
  • Populated Place
    • Country
    • Area
    • Municipality
    • City
    • Continent
  • Body Of Water
    • River
      • Canal
    • Lake
  • Historic Place
  • Mountain
  • Building
    • Airport
    • Station
    • Skyscraper
    • Bridge
    • Stadium
    • Shopping Mall
    • Lighthouse
    • Hospital
    • Launch Pad
    • Historic Building
  • Protected Area
  • Lunar Crater
  • World Heritage Site
  • Park
  • Island
  • Site Of Special Scientific Interest
  • Wine Region
  • Ski Area
  • Cave
• Infrastructure
  • Road
• Planet
• Person
  • Military Person
  • Athlete
    • Formula One Racer
    • IceHockey Player
    • Baseball Player
    • Wrestler
    • Basketball Player
    • Rugby Player
    • Cyclist
    • Figure Skater
    • Gaelic Games Player
    • Boxer’
    • Tennis Player
    • National Collegiate Athletic Association Athlete
    • Nascar Driver
    • Poker Player
    • Badminton Player
    • Cricketer
    • Football Player
    • Soccer Player
  • Office Holder
  • Scientist
  • College Coach
  • Monarch
  • Artist
    • Musical Artist
    • Actor
      • Adult Actor
      • Voice Actor
    • Writer
    • Comics Creator
    • Comedian
  • Playboy Playmate
  • Philosopher
  • Astronaut
  • Politician
    • Senator
    • President
    • Governor
    • Congressman
    • Prime Minister
    • Member Of Parliament
    • Chancellor
    • Mayor
  • Model
  • Journalist
  • Criminal
  • British Royalty
  • Architect
  • Fictional Character
    • Comics Character
  • Celebrity
  • Cleric
    • Saint
    • Christian Bishop
    • Cardinal
    • Pope
  • Judge
  • Football Manager
• Organisation
  • Band
  • Company
    • Record Label
    • Airline
  • Radio Station
  • Military Unit
  • SportsTeam
    • Soccer Club
    • Football Team
    • Hockey Team
  • Non-Profit Organisation
  • Legislature
  • Broadcast
  • Trade Union
  • Educational Institution
    • School
    • University
    • College
• Ethnic Group
• Work
  • Musical Work
    • Album
    • Single
    • Song
      • Eurovision Song Contest Entry
  • Film
  • Book
  • Video Game
  • Television Show
  • Television Episode
  • Software
  • Newspaper
  • Magazine
  • Musical
• Device
  • Automobile Engine
  • Weapon
• Mean Of Transportation
  • Ship
  • Aircraft
  • Automobile
  • Automobile Platform
  • Rocket
  • Space Shuttle
  • Spacecraft
  • SpaceStation
• AnatomicalStructure
  • Bone
  • Artery
  • Vein
  • Lymph
  • Nerve
  • Brain
  • Muscle
  • Embryology
• Olympic Result
• Event
  • Military Conflict
  • Music Festival
  • Convention
  • Year In Spaceflight
  • Space Mission
  • Film Festival
  • Sports Event
    • Grand Prix
    • Wrestling Event
    • Mixed Martial Arts Event
    • Race
    • Olympics
    • Womens Tennis Association Tournament
• Language
• Chemical Compound
• Species
  • Eukaryote
    • Plant
      • Green Alga
      • Moss
      • Club Moss
      • Fern
      • Cycad
      • Ginkgo
      • Conifer
      • Gnetophytes
      • Flowering Plant
        • Grape
    • Animal
      • Fish
      • Amphibian
      • Reptile
      • Bird
      • Mammal
      • Insect
      • Arachnid
      • Crustacean
      • Mollusca
    • Fungus
  • Archaea
  • Bacteria
• Protein
• Disease
• Drug
• Supreme Court Of The United States Case
• Website
• Music Genre
• Currency
• Beverage
• Award

Time to revisit the different types of Drupal sites to see where gains can be made. What type of site do you have? This quick reference recaps the previous articles and lists the areas where different types of Drupal sites can improve performance.

All Sites

• Get the best server for your budget and requirements.
• Enable CSS and JS optimization in Drupal
• Enable compression in Drupal
• Enable Drupal page cache and consider Boost
• Install APC if available
• Ensure no slow queries from rouge modules
• Tune MySQL for decent query cache and key buffer
• Optimize file size where possible

Server: Low resources

• Boost stops PHP load and Bootstrap
• Sensible module selection
• Avoid node load in views lists
• Smaller JVMs possibly if running Solr
• Nginx smaller than Apache
• mod_fcgid has smaller footprint over mod_php

Server: Farm

• Split off Solr
• Split off DB server, watch the latency
• With Cache Router select Memcache over APC for shared pools
• Master + slaves for DB
• Load balancing across web servers

Size: Many Nodes

• Buy more RAM for database indexes
• Index columns, especially for views
• Thoroughly check slow queries
• Warm up database
• Swap in Solr for search
• Solr to handle taxonomy pages

Activity: Many requests

• Boost or
• Pressflow and Varnish
• Nginx over Apache
• InnoDB on cache tables

Users: Mainly logged in

• View/Block caching
• CacheRouter (APC or Memcache)

Contention: Many Writes

• InnoDB
• Watchdog to file

Content: Heavy

• Optimized files
• Well positioned server
• CDN

Functionality: Rich

• Well behaved modules
• Not too many modules
• View/Block caching

Page browsing: Dispersed

• Boost over Varnish if RAM is tight

Audience: Dispersed

• CDN

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

The time for a page to render in a user’s browser is comprised of two factors. The first is the time it takes to build a page on the server. The second is the time it takes to send and render the page with all the contained components. This guide has mainly been concerned with the former – how to get the most from your server, however, it is estimated that 80% to 90% of page rendering time is taken up during the rendering phase.

It’s no good to serve a cached page in the blink of an eye if there are countless included files which need to be requested and many large images which need to be transported across the globe. Optimizing page rendering time can make a noticeable difference to the user and is the cream on the cake of a well optimized site. It is therefore important to consider and optimize this final leg of the journey.

Improving Drupal’s page loading performance

Wim Leers covers all the bases on how to improve loading performance.

High Performance Web Sites: Essential Knowledge for Front-End Engineers

Steve Souders, Chief Performance Yahoo! and author of YSlow extension, covers the Yahoo recommedations in this book.

High Even Faster Web Sites: Performance Best Practices for Web Developers

Another Steve Souders book covering Javascript (AJAX), Network (Image compression, chuncked encoding) and browser (CSS selectors, etc).

It is worthwhile reviewing Yahoo’s YSlow recommendations to see all of the optimizations which are possible. We cover selected areas where the default Drupal install can be improved upon.

Minimize HTTP Requests

Combined Files

The Out of The Box section covered the inbuilt CSS and JS aggregation and file compression. The use of “combined files” is a significant factor in Drupal’s relatively good score in the YSlow tests. Make sure you have this enabled.

All sites: Enable CSS and JS aggregation.

CSS Sprites

CSS Image Sprites are another method of cutting down the number of requests. This approach combines a number of smaller images into one large one which is then selectively displayed to the user through the use of background offset in CSS. It is a useful approach for thing such as small icons which can have a relatively large amount of HTTP overhead for each request. Something for the theme designers to consider.

Custom designs: Use CSS sprites if appropriate.

CSS Sprites: Image Slicing’s Kiss of Death

Overview of how CSS sprites work and how they can be used.

A lesson in the usefulness of CSS sprite generators

Covers commonly used spite generators.

Use a Content Delivery Network (CDN)

This is the number two recommended best practice.

A content delivery network (CDN) is a collection of web servers distributed across multiple locations to deliver content more efficiently to users. The server selected for delivering content to a specific user is typically based on a measure of network proximity. For example, the server with the fewest network hops or the server with the quickest response time is chosen.
http://developer.yahoo.com/performance/rules.html#cdn

Of all the CDN web services SimpleCDN seems to be getting positive press amongst Drupal folks as it is simple and cheap. It offers the “origin pull” Mirror Buckets service which will serve content from 3.9 cents to 1.9 cents per GB. At this price you will probably be saving money on your bandwidth costs as well as serving content faster.

The CDN integration module is the recommended module to use for integration with content delivery networks as it supports “origin pull” as well as push methods. It supports content delivery for a all CSS, JS, and image files (including ImageCache).

High traffic, geographically dispersed: use CDN

CDN integration module

Wim Leers’ fully featured module which integrates with a wide range of CDN servers.

SimpleCDN module

Simple CDN re-writes the URL of certain website elements (which can be extended using plugins) for use with a CDN Mirror service.

Drupal CDN integration: easier, more flexible and faster!

Slides covering advantages of CDNs and possible implementations.

mod_cdn

Apache2 module which shows some promise but not much info available for it with regards to Drupal.

Best Drupal CDN module?

Drupal Groups discussion.

On a related note many sites can benefit from judicial placement of the server if traffic tends to come from one place and no CDN is being used. Sites based out of the US may find the proximity of a site hosted in their area worth the extra cost of hosting.

Add Expires Headers

When a file is served by a web server an “Expires” header can be sent back to the client telling it that the content being sent will expire at a certain date in the future and that the content may be cached until that time. This speeds up page rendering because the client doesn’t have to send a GET request to see if the file has been modified.

By default the .htaccess file in the root of Drupal contains rules which sets a two week expiry for all files (CSS, JS, PNG, JPG, GIF) except for HTML which are considered to be dynamic and therefore not cachable.


# Requires mod_expires to be enabled.

# Enable expirations.
ExpiresActive On
# Cache all files for 2 weeks after access (A).
ExpiresDefault A1209600
# Do not cache dynamically generated pages.
ExpiresByType text/html A1



The Expires header will not be generated unless you have mod_expires enabled in Apache. To make sure it is enabled in Apache2 run the following as admin.


# a2enmod expires
# /etc/init.d/apache2 restart


Ensuring this is enabled will elevate your YSlow score by about 10 points or so.

All sites: Configue Apache correctly for fewer requests.

Gzip components

You can Gzip by enabling compression in the performance area of admin. Alternatively you could configure Apache to do it.

All Sites: Enable Gzip compression

Optimize Images

Binary files do not shrink significantly after Gzip compression. Gains can be made by ensuring that rich media such as images, audio and video are (i) targeted for the correct display resolution and (ii) have an appropriate amount of lossy compression applied. Since these files will generally only be downloaded once they do not benefit from caching in the client and so care must be taken to ensure that they are as small as reasonably possible.

All Sites: Compress binary files

Pngcrush

Pngcrush is an optimizer for PNG (Portable Network Graphics) files. It can be run from a commandline in an MSDOS window, or from a UNIX or LINUX commandline.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Benchmarking a system is a reliable way to compare one setup with another and is particularly helpful when comparing different server configurations. We cover a few simple ways to benchmark a Drupal website.

A performant system is not just one which is fast for a single request. You also need to consider how the system performs under stress (many requests) and how stable the system is (memory). Bechmarking with tools such as ab allows you to stress the server with many concurrent requests to replicate traffic when a site is being slashdotted. With a more customised setup they can also be used in more sophisticated ways to mimic traffic across a whole site.

Benchmarking and profiling Drupal

Documentation which covers tools of the trade including Apache Bench (ab) and SIEGE.

Apache Bench (ab)

ab is the most commonly used benchmarking tool in the community. It shows you have many requests per second your site is capable of serving. Concurrency can be set to 1 to get end to end speed results or increased to get a more realistic load for your site. Look to the “failed requests” and “request per second” results.

In order to test the speed of a single page, turn off page caching and run ab with concurrency of one to get a baseline.

ab -n 1000 -c 1 http://drupal6/node/1

To check scalability turn on the page cache and ramp up concurrent connections (10 to 50) to see how much the server can handle. You should also make sure keep alives are turned (-k) on as this leads to a more realistic result for a typical web browser. At higher concurrency levels making new connections can be a bottleneck. Also, set compression headers (-H) as most clients will support this feature.

ab -n 1000 -c 10 -k -H 'Accept-Encoding: gzip,deflate' http://drupal6/node/1

Drupal Performance Measurement & Benchmarking

Testing with ab and simple changes you can make within Drupal.

On Drupal Performance: Testing with Apache Benchmark

Covers server side tools and walks through ab options and use.

Benchmarking Authenticated Drupal Users with ApacheBench

Demonstrates how to pull out current session id and how to pass that to ab so that authenticated users can be tested.

Has anyone tried nginx caching with Drupal?

Illustrative discussion where different Drupal setups are benchmarked with ab.

JMeter

JMeter is a Java desktop app designed to test function and performance. It is the preferred testing tool of many administrators.

Drupal-Performance-Testing-Suite

Perl script which runs a jMeter test on Drupal and provides graphs.

Thoughts on benchmarking

Benchmarking is essential if you wish to have an objective comparison between different setups. However, it is not the final measurement with regards to performance. Remember that page rendering times are what are important for users and that too needs to be optimized. Also, benchmarks tend to be artificial in the sense that they often measure unrealistic situations. Will all of your requests be for one anonymous page only? Maybe in the Slashdot situation but there are other considerations obviously. Finally, it is easy to focus intently on the number, especially when it comes to caching scores, and forget that minor differences may not make so much of a difference to real life scenarios. Don’t forget the logged in user.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

There are a couple of projects which have made it easy to achieve performance gains by making slight amendments to core or packaging up the code in a helpful manner. Pressflow makes it possible to use a reverse proxy such as Varnish, amongst other things. Mercury packages Pressflow up as a Amazon EC2 image. The development of both these projects is the sign of a maturing ecosystem where serious deployments can easily be rolled out.

Pressflow

Pressflow is a distribution which attempts to bring many of the improvements discussed above (SQL improvements, Varnish) into a single package. Pressflow is a standard Drupal install which has had its core modified to fix bottlenecks and facilitate the use of advanced caching features. FourKitchens don’t regard Pressflow as a fork since many of the initiatives found in Pressflow are contributed back into the development of the head of Drupal.

So long as you haven’t hacked core yourself then using Pressflow is a simple matter of swapping out core drupal and replacing it with Pressflow.

In a nutshell Pressflow allows the following:

• Support for database replication
• Support for Squid and Varnish reverse proxy caching
• Optimization for MySQL
• Optimization for PHP 5

High Traffic, Varnish required: Easy setup.

Pressflow makes Drupal scale

Announcement covering the advantages and features of Pressflow.

Project Mercury

Project Mercury is an innovative project from Chapter Three which wraps up a tricked out PressFlow installation in to a preconfigured Amazon Machine Image (AMI) for use on Amazon EC2 instances.

The Pantheon Project Blazes Ahead

Hot of the press: Mercury will also be available for deployment on other servers, not just EC2. Further there will be a Mercury On Demand service at Rackspace.

The goal of this project is to make Drupal as fast as possible for as many people as possible. To that end, we are developing a pre-built Amazon Machine Image (AMI) which will allow anyone with an Amazon Web Services account to spin up an EC2 instance and see how all this works in real-time. The ultimate goal is a production-ready release that can be used for deploying real websites.

If you want to get started using the image all you need to do is signup to Amazon Web Services and then start up an instance of your choosing. You know are in control of a fully configured, scalable server. This sounds easy in practice, however, if you are considering going down this path there are a couple of considerations:

• Amazon is not the cheapest provider of bandwidth, RAM and storage. Other virtual servers have better deals. You are paying for the ability to spawn servers on the fly
• Persistent storage is an issue which needs to be overcome and managed if scaling out your web server.
• There a bit of a learning curve with some of the tricks of the trade when managing the servers.

Project Mercury and EC2 is a worthy combination if you really need the ability to serve massive amounts of traffic and also have the ability to temporarily scale out during peak times.

Project Mercury Benchmarks: 2000+ Requests Per Second!

Drupal is fast with APC + Page Cache. It is very fast with PressFlow and Varnish. NB. It would have been interesting to see how Boost went against Varnish for this test.

The configuration chosen by the project is interesting because it shows how other sites might go about setting up a scalable server. in brief the setup is as follows:

• Ubuntu 32 or 64 bit
• Pressflow
• APC for opcode cache
• CacheRouter and APC/Memcached for No SQL caches
• Varnish as reverse proxy

High Traffic, Varnish required, EC2 required: Easy setup considering.

Step-by-step: Setting up Varnish, Apache, APC and Solr Project Mercury Style

Step by step instructions for setting up Project Mercury on a Ubuntu server. Very helpful for admins wishing to install manually on their own server.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Several advanced options exist to take caching to the next level in Drupal. With advanced caching Drupal is able to scale to levels required by the most demanding of sites.

We have already discussed (i) page and block caching and (ii) other caches which come baked into the core of Drupal in the Out of the Box section. Whilst big gains can be made through enabling simple config options it is possible to make several more improvements to the way Drupal caches and serves data, making Drupal a system which can scale to serve 1000s of requests a second if the need arises.

Page Caching

The Bootstrap

Every time Drupal code needs to be run either through a web page or script, Drupal must undertake the boostrap process which has a certain amount of overhead. Generally this process cannot be avoided – you need to run code after all. However, there are a couple of cases where it is possible. Firstly, Normal Page caching avoids much of the bootstrap save for hook_boot() and hook_exit() hooks which are used by the statistics and throttle modules. Secondly, Aggressive Page Caching allows for all of the bootstrap to be avoided when serving content. Finally, contributed modules such as Boost and Varnish allow for the avoidance of PHP and bootstrap since they are operating at a stage before PHP is invoked. This is the main reason Boost and Varnish are so attractive as a page cache.

Boost

Boost is a page cache which works in a similar way to aggressive page caching in that it attempts to serve cached data without running the bootstrap. However, it is able to go one step further and avoid PHP being run as the redirect takes place in rewrite rules located in .htaccess. If the static file exists then it is served as the response. The end result is super fast response without running PHP or the bootstrap. This frees up the server to handle more requests from logged in users.

Boost is an attractive module because it is easy to install, has a lot of configuration options which give good control over cache building and invalidation. It is highly recommended.

One aspect of Boost which may be forgotten is that it isn’t an entirely a file based solution. If the operating system file cache is working well then there is a good chance that the “file” will come out of RAM rather than off disk. This makes Boost a competitive option when compared to other more complicated reverse proxy setups such as Squid or Varnish.

Because Boost is file based it is able to potentially cache a lot more data than a RAM based solution. If you have thousands or millions of pages to cache putting them all in RAM is probably not optimal. Best to put them on disk and save the RAM for DB indexes or possibly a CacheRouter cache.

All sites, especially big, infrequently changing, high traffic: Big gains. Easy setup.

Varnish

The Varnish HTTP Accelerator Integration module integrates Drupal with Varnish, a reverse proxy which sits in front of Apache, PHP and Drupal. Varnish stores cached content in RAM and avoids the overhead of Apache and the Drupal bootstrap. As such it offers very high performance for anonymous users on cached pages and is the preferred option for many sites where scaling is paramount.

Varnish requires either a patch to core to add HTTP headers, PressFlow or Drupal 7. Most people are therefore running Varnish in conjunction with PressFlow.

High Traffic: Big gains when performance critical.

Squid

Squid is a reverse proxy similar to Varnish and is in use on drupal.org. It doesn’t seem to be such a commonly deployed solution probably because PressFlow has been altered to work with Varnish which is higher performing apparently.

Varnish preferred over Squid.

Varnish vs Boost

A popular thread High Performance Drupal group is for the perfect recipe for page caching – whether to use Varnish or Boost. Varnish has the edge in speed over Apache+Boost as well as Nginx+Boost. Look at results published by brianmercer.

Varnish may have the edge in speed but it is more complex to install and requires a patched core. Some people may not want to run a non standard Drupal installation. Varnish also requires RAM to store the cached material – something which may be better spent elsewhere (database or max clients). Boost offers similar performance, is easy to install and has good control of cache invalidation and warmup. Boost is also able to serve files from RAM if the OS has cached them.

It is certainly up to you to decide which avenue to take. This guide is attracted to the relative simplicity of Drupal + Nginx + Boost over Pressflow + Varnish + Apache/Nginx. Nginx brings better performance to the web server as a whole, ie. pages not in the cache, and RAM savings if it is needed elsewhere. It must be stressed that each site has a different profile and the ultimate decision is up to you.

Has anyone tried nginx caching with Drupal?

Drupal Groups thread with lots of benchmarking. Final conclusion seems to be that Varnish vs Boost+Nginx is a pretty close thing.

Project Mercury: A pre-configured Drupal+Varnish EC2 AMI

Josh Konig of Chapter Three claims that Varnish is faster than Boost but gives no numbers.

Quick Drupal Cacherouter and Boost benchmarks

Dylan Tack likes Boost: “Response times are all close enough that it doesn’t really matter what caching backend you choose… The only factor that’s really relevant is how good your system’s cache expiration and regeneration logic is… it seems like Boost is the clear winner here as well.”

What’ recipe should i choose for best performance?

Discussion with participants split between Varnish and Boost depending on circumstances. Nginx+Boost seems pretty equal with Varnish.

Caching: Modules that make Drupal scale

Table of modules, performance gains and features.

Cache Router

Cache Router is a module which enables you to cache the Drupal cache tables (including views, blocks, menus, variables, and filters) in RAM. Drupal no longer has to hit the database to pull out this content – a big win for logged in users who might not be able to enjoy the advantages of a page cache. Cache Router therefore fills an important niche in your caching strategy.

Cache Router is able to do this via a number of backends including APC or Memcache. APC should be considered if you are running Drupal on a single node and only need the single local store. According to Josh Konig, APC “is less error-prone, more secure, and allegedly as fast (if not faster) than running memcached according to the folks at Facebook.” Memcache can be used to share cached data between or across multiple servers.

High Traffic, Logged in: Massive gains when page cache not hit.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Sometimes your installation will be slow for no apparent reason. Time to grab your toolkit and get started on finding and eliminating the problem. Here are some of the common problems faced by developers.

Database Queries

The database is generally the first place you look when trying to identify problems on a page. It is possible to identify problems through MySQl’s slow query log or through the query log of the Devel module.

High Performance MySQL: Optimization, Backups, Replication, and More

Must have book for anyone serious about getting the most from MySQL.

Views and indexes

The Views module makes building queries easy and sometimes a crucial part of the query will rely on a CCK column with no index. This tends to happen with sorting or filtering. Queries can run slow in these cases. The solution is to place an index on the column in question.

Large Sites: CCK need index

Views and Left Joins

The Views module will often design queries with LEFT JOIN rather than INNER JOIN, especially when joining from the node table to a content type CCK table. In many cases you might only want an inner join, especially when the node table is very big. In these cases it is possible to rewrite the query by hacking the query in hook_views_pre_execute.

Large Sites: Some View SQL inefficient

Ability to INNER JOIN to node for a specific field

Discussion of Views SQL for joining between node and content type tables.

Inefficient schema

The database design of some modules could possibly be improved. It is up to you the developer/administrator to ensure that you are happy with the internal design of contributed modules. If you find an inefficiency then submit an issue to the module, provide a patch or remove the module from your site.

Composite indexes

MySQL is limited to using one index per table when sorting and filtering. This can make it tricky when you wish to use multiple AND clauses or involve a sort and a filter at the same time. In these cases adding a composite index can get you out of trouble.

Problems in Core

In a number of cases there are problems in core of Drupal where queries are very slow on very big Drupal installations with millions of nodes. The author has experienced problems with:

• Inability to browse content in Admin section due to join from member to user table
• Inability to edit nodes with many CCK fields. Massive RAM usage when loading nodes.
• Taxonomy pages take a very long time to display
• Search system unable to index content

These problems can be avoided partly by swapping Solr in for search and having it override the taxonomy pages. The other problems you just have to live with

Drupal with millions of nodes

Some good research from Wesley Tanaka into problems with many nodes. Attitude from some here seems a little dismissive of the issues.

Slow Modules

Modules can also be badly written leading to poor performance in certain circumstances. Generally this happens when the module’s creator did not test the module against (i) large installations with many nodes (ii) complex installations with heavy nodes or detailed taxonomies. You generally will only run into these problems if your site is large. In some cases you can fix the module by creating more indexes in the DB. In others you just have to remove the module from your installation…. or submit a patch.

Some known offenders for large sites include:

• XML Sitemap
• Node access
• Taxonomy Browser
• Fivestar

How Drupal’s node_access table can negatively impact site performance

Scalability of the Taxonomy Browser module: Restricting number of terms

“Query from hell” with many joins leading to queries which never finish.

XML Sitemap 6.x-2.x: How Drupal modules can overload a site during cron, with solutions

XML Sitemap module needs to be configured correctly

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Bad performance can be down to decisions you have made during development because you haven’t been aware of natural limitations of the system or the way Drupal works. The more you poke about the more you will understand. Here’s a grab bag of things which may bite you.

Search

The search which comes inbuilt with Drupal has long been regraded as unsatisfactory for larger sites:

• failure to index large sites efficiently
• slow at returning results
• relatively limited feature set compared to dedicated search solutions

Part of the problem lies with the fact that standard relational databases such as MySQL and interpreted languages such as PHP are not well suited to handle the large indexes and filtering required for big datasets. Search can also be an intensive process if the corpus is large and the traffic high. Being able to move search off the main box will give more resources to Drupal to do other things. Both of the following solutions can help solve these issues.

Solr

Enter the Apache Lucene project and Apache Solr.

Welcome to Lucene!

Lucene “provides Java-based indexing and search technology, as well as spellchecking, hit highlighting and advanced analysis/tokenization capabilities.”

Welcome to Solr

“Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world’s largest internet sites”

The Apache Solr Search Integration module integrates Solr with Drupal. Installation requires a setup of a JVM and a container such as Tomcat or Jetty – something which is a little outside the usual LAMP configuration. Installation is not so difficult with some good documentation available. If this is beyond you then it may be worth checking out the Acquia Search hosted solution.

The beauty of Solr is that it is able to index massive datasets, millions if not 100s of millions of nodes, and return results surprisingly quickly. The faceted search (taxonomy, language, CCK, Author, Content Type) is its main selling point which will be sure to impress users. It really is a product which will take your site to the next level if search is important to you. More importantly Solr solves crippling scaling problems with core search as well as viewing taxonomy terms with many nodes.

Check out the following search for “Madonna” filtered by “Artistic Works” type and “English” language. Alternatively, here is Madonna filtered by “Person” type and “Italian” language. Italians do it better.

Solr is a natural fit for installation on another server because it runs as a web service over HTTP. This is a very easy way to take load of your main server, especially if search is a big part of your site. The disk requirements of the index and Java memory requirements for Solr can be significant on big sites so moving it off the main server may well be a necessity. My experience on Uriverse (a large amount of smallish nodes) would suggest that each node takes around 200 bytes of RAM in the JVM. Your requirements could vary dramatically from this but this may be helpful as a rough guide for those wanting to know about RAM consumption.

Large sites: Core search fails. Solr shines.

SolrPerformanceFactors

Solr documentation including optimization.

Scaling Lucene and Solr

Some very good notes on scaling Solr. File system cache is important as well as JVM and need 2+GB on top of JVM for big index.

how to reduce solr memory usage?

Smaller documents, less facets, no sorting.

How much disk space does optimize really take

Be careful of optimize – required disk space can double.

Google Custom Search

Another option for those users looking to switch away from the in-built search is Google Custom Search – a search service offered by Google where Google indexes your data and stores the results on their servers. You are able to search the data via a simple form on your site. Google Custom Search is free for individuals, who don’t mind ads with their results, with a business version starting at $100 pa. The Google Custom Search Module integrates it into Drupal by providing a block with the form.

Module bloat

Every module you add to a Drupal site leads to the consumption of more RAM reducing the number of simultaneous clients you can server. Modules also consume extra CPU. As a site designer you need to be aware that all modules added will have some cost on the performance of your site. Is the extra functionality worth the performance cost?

Server indigestion: The Drupal contributed modules “open buffet binge” syndrome

Minimal install of Drupal has Apache process around 17MB-33M. A bloated system has 93M bloating to 100MB after a few requests. This means fewer requests can be handled due to RAM consumption.

Feature rich sites: RAM wasted. Low max clients.

Node load

The loading of a node in Drupal can be a very quick/light or very slow/heavy process depending on the circumstances. It all comes down to (i) how much data is in the node and (ii) how fast that data comes back out of the database. If you have nodes with many CCK fields sitting in a database which hasn’t been able to cache all of the necessary indexes, a node load is something which you really want to avoid. Instead of lazily loading data when required a node load loads it all in at once! In extreme cases a node could take seconds to load.

This become a problem when you want to handle many nodes at once, such as when you want to display node teasers in a View for example. If you notice your page slowing down when displaying 50 teasers on a view then it is a good bet that the DB is getting hammered trying to load in all that data just to display the title, description and url_alias! In these cases skip the teaser and just do it with fields. You should notice a big improvement.

Similarly, performing massive (millions of nodes) imports for new/updated nodes can be prohibitive as well. In these rare cases you need to skip the API and execute SQL on the DB directly.

Many Nodes, Big Nodes: Speed and RAM suffers

CCK design

It’s important to know how to work within the confines of a system to get the most out of it so it is well worth investigating how data is stored in the backend for nodes and CCK fields. Smart, sensible design of CCK fields should ensure that database access is kept to a sensible level. Whilst this guide recommends designing a data model first and then worrying about data access times second, it is worth knowing the consequences of your decisions.

In a nutshell, CCK will create another table in the backend for multi fields and fields which are shared between content types. There’s no avoiding the first reason but in the case of shared fields, a separate DB query will need to be run for each shared property. This can be an issue on very large sites on nodes with a lot of properties when you want to keep DB queries to a minimum.

A couple of rules of thumb would be:

• Try to keep the properties of a content type encapsulated to that particular content type. eg. you might be tempted to share book.author and film.screenwriter in a single CCK field. Unless you are going to be doing queries across both then it makes sense to store the properties separately with their content types.
• If you are going to share properties between content types, then the more sharing that can be done the better. eg. a ‘geo-location’, ‘country’, ‘intended-audience’ are all possible candidates for stretching across several content types. This design is views friendly, aiding efficient queries across multiple content types.
• It is probably better to lean towards designing content types which use taxonomy types and have some null properties instead of creating too many ’sub-class’ content types to do the job. eg. lets say you have two possible types, ‘it-employee’ and ‘accounts-employee’, which share some properties (birthday, address) but not others (preferred-os). It probably would be better to add an employee-type taxonomy to do the sub-classing and have the unshared preferred-os property as optional. This ensures easy filtering using taxonomy and allows for fast employee retrieval from a single table.

Complex Data Model: Inefficient data

Module development

When designing your own modules you should have an eye to efficient design and use caching the Drupal way. Using the Drupal cache mechanism means that the custom caches are available to Cache Router to store in RAM (if appropriate).

It is also possible to use the “static” variable keyword to cache the variable for that particular script.

A beginner’s guide to caching data

Steps module developers can take to cache data the PHP/Drupal way.

Custom Modules: cache data the Drupal way

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Getting the most from your Drupal site means getting the most from your server – optimizing the various layers of the the LAMP stack. This includes the filesystem, database, web server, PHP, RAM and CPU. Tuning the LAMP stack is a major subject requiring a lot of study and practice to become proficient. It’s something you will probably never completely master Try Googling lamp performance tune for a few articles to whet your appetite. For now, we’ll cover a few of the major considerations for Drupal, although most of this advice would apply to any PHP web app running on Linux.

Server tuning considerations

Drupal documentation covering the basics.

Tuning LAMP systems, Part 1: Understanding the LAMP architecture

Intermediate article covering LAMP.

Tuning LAMP systems, Part 2: Optimizing Apache and PHP

Intermediate article covering Apache and PHP.

Tuning LAMP systems, Part 3: Tuning your MySQL server

Intermediate article covering MySQL.

Opcode cache

Opcode caches cache the compiled form of a PHP script in shared memory to avoid the overhead of parsing and compiling the code every time the script runs. This saves RAM and reduces script execution time.

Quite a bit of benchmarking has been done in the Drupal and PHP communities between APC, eAccelerator and XCache. eAccelerator may have the edge in raw performance, but it appears that APC is the preferred opcode cache in the Drupal community because it is well maintained and less buggy.

All sites: faster and less RAM. Moderate install.

Drupal web server configurations compared

APC gives 2x to 4x increase in throughput under load. PHP5 is around 10% slower.

PHP op-code caches / accelerators: Drupal large site case study

Op-code caches are a must for large sites serving many pages.

Benchmarking APC vs. eAccelerator using Drupal

eAccelerator is faster and smaller than APC. Both offer around 6x – 7x times speedup over PHP.

High PHP execution times for Drupal, and tuning APC for include_once() performance

Make sure apc.shm_size can fit the whole page else there will be no caching.

Database

There are a number of choices to be made when tuning your MySQL database server. The MySQLTuner script can be helpful for identifying outstanding issues you may be unaware of. It can be run on a functioning production server to see how your database is performing in the wild.

MySQLTuner

Perl script which is able to report on the operation of your MySQL installation and offer suggestions as to what can be fixed.

Tuning MySQL Performance with MySQLTuner

Helpful tutorial.

Storage Engine: InnoDB vs MyISAM

A default install of Drupal 6 installs the DB tables as MyISAM. This will change in Drupal 7 with the default set to InnoDB. A Drupal 6 installation may well have some InnoDB tables as modules may create new tables in the InnoBD engine. Your installation may therefore be a mix between the two engines.

In many places on the web you will read statements such as ‘All high performance Drupal sites run InnoDB”. This is not necessarily so as there are some cases where MyISAM may still be preferred although with recent changes to Drupal core the pendulum has swung to InnoDB as a sensible default.

A list of the main difference between the engines is as follows:/p>

• InnoDB is transactional (better integrity), MyISAM isn’t
• InnoDB more reliable (better recovery), MyISAM can be repaired
• InnoDB has row level locking (better concurrency), MyISAM locks tables
• InnoDB uses clustered indexes (faster access to data), MyISAM indexes just the keys
• InnoDB has a bigger memory footprint

In general, you would consider sticking with MyISAM if

• Memory footprint was an issue. If you have very big indexes which might only just fit into the key buffer then MyISAM could offer faster lookups.
• Most activity is read only.

InnoDB tables definitely should be used for all of the Drupal cache tables since this is where most contention is likely to occur.

Finally, it must be noted that Drupal was written based on the MyISAM engine and as such many queries were not optimized for InnoDB. The SELECT COUNT(*) is particularly slow in InnoDB because it must scan all rows to calculate the count. Many of these shortcomings have been removed in the PressFlow distribution and have since made their way back into core.

All sites: InnoDB for less contention on cache
Most sites: InnoDB for everything else
Big unchanging sites: MyISAM faster reads less RAM

MySQL Engines: MyISAM vs. InnoDB

InnoDB is a good fit for many cases and “in most cases, InnoDB is the correct choice for a Drupal site”. Very good comparison between the two engines.

MySQL InnoDB: performance gains as well as some pitfalls

InnoDB does row level locking but lookup is slower for some slow queries. NB. Pressflow distribution fixes some slow InnoDB queries.

InnoDB vs MyISAM vs Falcon benchmarks – part 1

Myth that MyISAM is faster than InnoDB in all cases.

Which Tables can be converted to InnoDB

High Performance discussion emphasizing that InnoDB should definitely be used for cache tables and complex joins in CCK if memory allows.

MySQL Configuration

There are a number of MySQL config variables which must be tweaked to suit your data. It is impossible to specify one set of options to suit all sites. A few rules of thumb are offered below.

Optimizing the mysqld variables

Clear article with some good rules of thumb for MySQL variables.

Key buffer

If you are running MyISAM tables then the key buffer is a very important variable to set. The key buffer stores table indexes in memory, allowing for fast lookups and joins. For large node, node_version and url_alias tables it is a must to have enough room to fit these tables into memory, otherwise your site will very slow on the most basic of operations: looking up nodes, titles and paths.

One rule of thumb is to set this buffer to somewhere between 25% and 50% of the memory on the server. To determine the best value up front sum the size of all the .MYI files.

MyISAM sites: most queries faster. Essential.

key_buffer_size

Documentation on the use of key_buffer_size.

Query cache

MySQL has a query cache which stores results up to a certain size in memory. The cache is very handy for quickly returning commonly accessed data when all other forms of caching (reverse proxies, page cache, Drupal caches) have not been invoked. Queries which may take sometime return almost instantly.

MySQL’s Query Cache

Covers config and operation of the query cache.

During the development and testing of a site the query cache can catch developers out since a query may appear to be performing quite well the second and subsequent times through. To really test a query you need to fire up mysql client (or phpmyadmin) and add the SQL_NO_CACHE option to the query to see the real time it takes. Don’t be fooled!

Query Cache SELECT Options

Documentation on the use of SQL_NO_CACHE.

The query cache is destroyed if any row in the table is changed and so it cannot be relied upon if tables are changing frequently. The cache shines when the are big tables which don’t change that often. Unless your site has such characteristics it is best to limit it so that it fits small unchanging tables and then some for the most popular queries. Examination of cache hit rates will show you if it needs to be extended or reduced.

query_cache_size

Documentation on the use of query_cache_size.

All sites: common queries faster

InnoDB Buffer Pool Size

If you are running InnoDB tables then it is essential to optimize the InnoDB Buffer Pool Size, increasing the memory to reduce query time. InnoDB is more memory intensive and so the pool will be larger than that used for MyISAM tables. MySQL documentation suggests that the size can be upped to 80% of physical memory. Anymore could lead to swap issues.

InnoDBsites: most queries faster. Essential.

innodb_buffer_pool_size

Documentation on the use of innodb_buffer_pool_size.

Other variables

Other variables worth tweaking include the following. See Optimizing the mysqld variables for more.

• table cache
• sort buffer
• read_rnd_buffer_size
• tmp_table_size

Database Warmup

A warm database will perform much better than a recently started one because its caches and buffers will be primed with keys and data. It therefore makes sense to warm up a DB every time the database is restarted. The best way to do this is to load in the indexes of commonly used tables. This guide recommends loading in node, node_revisions and url_alias. Taxonomy information could be good candidates as well.


USE drupal6;
LOAD INDEX INTO CACHE node;
LOAD INDEX INTO CACHE node_revisions;
LOAD INDEX INTO CACHE url_alias;
LOAD INDEX INTO CACHE term_data;
LOAD INDEX INTO CACHE term_node;


This SQL code can then be put in a script and run when MySQL restarts. It is possible to configure the init_file variable in my.cnf to tell mysql where to find the startup SQL.

init-file = /etc/mysql/init-file.sql

Many nodes: Most queries where index relied upon.

Index Preloading

How to use LOAD INDEX INTO CACHE t1.

init_file

How to use init_file variable to specify startup SQL.

Web server

Apache + MPM Prefork + mod_php is the default web server configuration in the LAMP stack. This combination does consume large amounts of RAM which can be a problem for handling many requests. It can also be quite heavy and slow for serving static content. Many administrators have looked to replace it with other combinations including multithreaded processes (MPM Worker) and external PHP (mod_fcgid) as well as swapping it out completely for another server such as Nginx. This guide has adopted the position that Apache problems can be ameliorated somewhat by removing unneeded modules, running fcgid to connect with PHP and using MPM Worker to enable multithreading per process. However, in some cases this won’t be enough and Nginx is a must.

Apache vs Nginx

Other Drupal users have replaced Apache with faster more lightweight (RAM and CPU) web servers such as Nginx and Lighttpd. Nginx is generally preferred over Lighttpd because of memory leaks in the latter. It is currently possible to run Nginx without losing any functionality in Drupal. Boost, a module based on .htaccess rules, now supports Nginx so it is feasible to run Nginx as the main web server. If you are constrained by CPU or have high loads then this certainly is an option worth considering.

Setting up Nginx is not trivial but it is reasonably straight forward if you are comfortable with compiling and patching. There are some good tutorials on the Web for user who want to do this.

Low resources, High Traffic, Many logged in: Possible to get more for less with Nginx.

Apache vs Nginx : Web Server Performance Deathmatch

“Nginx seems to compete pretty well with Apache and there doesn’t seem like there is a good reason not to use it especially in CPU usage constrained situations (ie. huge traffic, slow machines and etc).”

In reply to kbahey: apache vs nginx

Discussion and results over the pros and cons of Nginx vs various Apache setups.

How to get Drupal working with Nginx

Simple guide for installing and configuring Nginx on a server with only 256MB RAM. Uses FastCGI which may not be preferred method.

NGINX + PHP-FPM + APC = Awesome

“The following guide will walk you through setting up possibly the fastest way to serve PHP known to man…In this article, we’ll be installing nginx http server, PHP with the PHP-FPM patches, as well as APC.”

PHP-FPM – A simple and robust FastCGI Process Manager for PHP

Preferred way of connecting Nginx with PHP. Currently in PHP core for 5.3.2+ but not yet released. Requires patch to PHP 5.2.

Apache: Unneeded modules

It is possible to turn off unneeded modules in Apache to reduce memory footprint. This depends very much on your setup.

All sites: Good savings in RAM

What Apache2 modules can be disabled?

Lists of modules which should be enabled in Apache2.

Apache threading: MPM Worker (multi threaded) MPM Prefork

The use of MPM Worker allows for the handling of more requests due to multithreading in each process. It has a smaller memory footprint than Prefork and is faster. According to docs, Apache must be compiled with the --with-mpm argument in order to install Worker as “prefork” is the default on Unix systems.

RAM limited: Worker preferable to Prefork.

Compile-Time Configuration Issues

“Choosing an MPM” section covers differences between the two models.

Multi-Processing Modules (MPMs)

Apache documentation on installation.

Installing Apache2 and PHP5 using mod_fcgid

Hey, you don’t have to recompile Apache. Tutorial on how to install MPM Worker using apt-get with Apache2 on Ubuntu. Just the ticket.

mpm-worker versus mpm-prefork, and mod_php versus fastcgi

PreFork and FastCGI is still a win if you find that Worker is unstable due to long downloads as this person did.

Connectors: mod_php, FastCGI, mod_fcgid

The use of mod_php with Apache is the most common setup for calling PHP. mod_php works by embedding PHP into every Apache process. This has the disadvantage of a large memory footprint for each Apache process. FastCGI and mod_fcgid overcomes this problem and reduces resource utilization with no gains in performance.

kbahey lists the disadvantages of mod_php as follows:

• All PHP loaded into the process
• Heavy process even if flat file
• Many processes will hog RAM

Use mod_fcgid for lower memory and DB/Network connections

Drupal webserver configurations compared

The most common, Apache+mod_php is the slowest. Tests conducted with FastCGI which is faster. NB: FastCGI has subsequently suffered from stability issues.

Apache with fcgid: acceptable performance and better resource utilization

Informative article which comes out in favor of mod_fcgid over FastCGI and mod_php. This is the must read article if you wish to attempt fcgid.

Configure Apache for high performance on drupal 6

Some solid comments from kbahey from 2bits regarding stable setup: Apache, MPM Worker, fcgid, APC (code cache), memcache No SQL.

Apache MaxClients

The MaxClients parameter controls how many simultaneous clients Apache is able to serve. If it is set to high RAM will be chewed up and the Machine will go into swap. If it is set to low then your site will be unnecessarily limited by the number of clients it can serve. The setting of this value should be determined after consideration of (i) how much spare RAM is available on the server and (ii) how much RAM each Apache process consumes. Obviously you will want to maximize available RAM through frugal allocation of RAM to MySQL, JVM, etc and minimize the size of Apache process through techniques described above.

2bits provide the following formula:

MaxClients = (Total Memory - Operating System Memory - MySQL memory) / Size Per Apache process.

The only addition this guide would make is that it is important to leave some RAM free for the OS file buffer to allow efficient operation of the OS.

Tuning the Apache MaxClients parameter

How to set MaxClients param.

.htaccess

If you are running Apache then it is possible to either use .htaccess or the apache conf file to specify directives such as rewrite rules, etc. If you use .htaccess then Apache must look for .htaccess rules in the directory hierarchy for every request. This can take some time even if no rules are found. You may consider putting the rules in httpd.conf/apache2.conf if you are looking to eek out the most performance from your site.

.htaccess can slow down site if performance is crucial.

.htaccess vs httpd.conf

Evidence that .htaccess can slow a site by 6.6%.

RAM: A precious resource

Given the above, serious thought should be given to how the RAM on your box is to be divided up. In a nutshell we have the following apps contesting for their fair share:

• The JVM if you are running Solr
• MySQL query cache and key buffers
• Apache processes for client requests
• PHP if it runs outside Apache
• Memcached for holding Drupal caches
• The file system cache

Consider the following when deciding how to divide up your box:

• The JVM needs a certain amount or else Solr will crash.
• MySQL really should have indexes buffered for MyISAM and InnoDB. Use MySQLTuner. If they can’t fit then buy more RAM or (i) reduce max clients and (ii) forget about CacheRouter.
• Apache MaxClients should be set to consume available RAM.
• The file system cache needs to be big enough to allow smooth running of system.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.

Drupal has a number of features which allow for performance to be improved without the addition of external modules or for complicated configuration by administrators. A slow system can easily be turned into one which performs well under heavy loads.

Firstly, Drupal shortcuts the execution of unneeded code by using an internal caching system which stores results of expensive routines. Secondly, Drupal improves page rendering time by allowing for the aggregation static JS and CSS files. Thirdly, page output can be compressed saving download time. Fourthly, Drupal allows for manual configuration of the caching of Blocks and Pages which can improve performance significantly.

Drupal Core caching system

Drupal comes with a number of in-built caches which store the results of expensive calculations (strings) in the database so that they can be retrieved quickly later on. There are six caches enabled by default: cache, cache_block, cache_menu, cache_filter, cache_form and cache_page. Contributed modules are able to create their own caches for storing data which is handy for module designers. This default cache system provides improved performance across the whole app.

The caching system is pluggable and allows for custom storage engines to be substituted in for the default database implementation. Later in this guide you will see how the Cache Router module is able to swap in memory based storage in place of the database, making the retrieval of cached content that much faster.

All sites: Improved performance across the app. No config.

Aggregate and compress JS and CSS

Drupal’s modular system means that pages can have a large number of CSS and JS includes which results in a lot of client server communication – slowing the page draw time down. The problem can be alleviated by merging the files and then compressing them. This results in less includes and faster downloads. Up to 90% of download time can be attributed to downloading CSS, JS and images so it makes sense to aggregate and compress if possible.

During development it is advisable to keep this option turned off so that CSS and JS errors can be troubleshooted.

All users: Lower render times. One click config.

Minimize HTTP Requests

Yahoo place this at the top of their list for ways to reduce download time. 40%-6-% of users are first time users so client side caching is no help to them. Fewer HTTP requests are.

Page Cache for anonymous users

Pages for anonymous users can be cached, meaning that a full build of the page isn’t necessary for each new request which comes in, providing you with savings in CPU and DB load as well as giving the user much faster response times. This is a massive win for your website, especially if the majority of your page requests are from anonymous users. Basically it can help you survive a Slashdotting. Drupal offers “aggressive” and “normal” options – normal page caching is recommended for most websites. Other options for Page Caching are discussed below.

During development it is advisable to keep this option turned off so that any changes to logic or design can be troubleshooted.

Anonymous users: Big wins in speed and CPU. One click config.

Drupal vs Joomla: performance

An older article comparing Joomla and Drupal. Joomla faster on non-cached pages but caching makes Drupal win.

Block Cache

Enabling the block cache allows finer grained control over cached content. Caching blocks which don’t change frequently will enable speedups for logged in users who need a dynamic page built for them each request.

Logged in users: Moderate wins. Easy to implement.

Drupal guide to caching

Covers the various database tables which store data for Drupal’s caching system.

Page Compression

It is also possible to enable page compression for the pages sent. This will reduce the page size by 50% or more depending on the page. Users requesting big pages on slower connections will love this.

Slow connections: Massive win. Easy to implement.

How to make Drupal run 8.5x faster in 5 minutes…

Page cache provides a 3x speedup.

How I Survived a 2300% Traffic Increase With Drupal

Demonstrates that these out of the box techniques can be very effective even for sites on shared hosting.

This article forms part of a series on Drupal performance and scalability. The first article in the series is Squeezing the last drop from Drupal: Performance and Scalability.