Personal blog of Yzmir Ramirez

This week the Output Escaper component was heavily refactored and the Propel bundle was transformed into an independent bundle. In addition, the Symfony2 HTTP cache documentation was published.

Development mailing list

• Contexts for the Symfony DIC
• Remember Me Authentication
• Injecting variables into the layout
• making _format magic more useful
• [Doctrine2] issues with inheritance

symfony 1 development highlights

Changelog:

• r31247:
  [1.3, 1.4]
  fixed mbstring problem in sfFilesystem
• r31248:
  [1.3, 1.4]
  fixed sfI18nModuleExtract.class.php always assumes a file based message source

• r31250:
  [1.3, 1.4]
  added some missing information in the WDT

• r31253, r31254:
  [1.3, 1.4]
  fixed WDT injection

sfDoctrinePlugin:

• r31249:
  [1.3, 1.4]
  fixed memory leak in Swift_DoctrineSpool

Activity summary: 58 changesets, 13 bugs reported, 15 bugs fixed, 3 enhancements closed, 5 documentation defects reported, 1 documentation defect fixed, and 11 documentation edits

Symfony2 development highlights

Changelog:

• 6885f90:
  [HttpKernel\Security] fixed use statement and updated parameters constructor
• 3463f47:
  applied base64 encoding directly to the binary data instead of their hexadecimal representation
• 1d5ca49:
  [FrameworkBundle] refactored ide setting configuration
• e111652:
  [WebProfiler] fixed WDT display
• c065be8:
  [OutputEscaper] refactored the component
• e23c3cc:
  [OutputEscaper] made getEscaper*() methods more consistent with the way you can change the escaping strategy in __call()
• c448429:
  [HttpFoundation] fixed date format for HTTP headers (format must be RFC1123, not RFC2822)
• 7e6bdde:
  [TwigBundle] moved Form extension initialization as late as possible (it’s better for performance)
• 2b613f3:
  [FrameworkBundle] removed the need for decorating with SafeDecorator
• 3eee458:
  [OutputEscaper] replaced the JS escaper with the one from Twig
• 13f36b1:
  Removed logic that tried to avoid double-escaping
• 88d30f0, b9f33a6:
  removed Propel bundle (it has been moved as an independant bundle)

New developers for hire

• Wojtek Orzech (w.orzech@gmail.com): 26 years old PHP developer with almost 4 years of professional experience. Zend PHP5 certified engineer. I have been using Symfony for two years. I’m located in Cracow, Poland.

New plugins

• idlErrorManagement: set of functionnalities to manage errors on production systems: DB logging, emailing, user feedback, …
• sfCatalogoProdotti: product catalogue.
• cgHoptoadNotifier: submits bug reports to hoptoad and works with the api version 2.

Updated plugins

• sfDoctrineGuard:
  • updated french translations
  • fixed setupInheritance access
  • begin to merge sfDoctrineGuardExtraPlugin into sfDoctrineGuardPlugin
  • added more translations
  • use mailer compose function
  • added username or email validator to password forgot
  • removed flash translation in action you can do it in the template my mistake
  • moved mailer call into function
  • moved query into table class
  • fixed mail function
  • fixed password change function
  • add config param app_sf_guard_plugin_password_change_url and simplified code
• dcReloadedFormExtra:
  • added the change for credentials widget
  • added partial widget
  • updated README
  • added jquery ajax dependence
  • added the select jquery autocomplete widget
  • updated JavaScript
• sfTangoIcons:
  • fixed function name
  • updated documentation
  • added new helper functions
• ExtjsGenerator:
  • updated DateTimeField widget for edit panels to enable a time field
  • fixed a bug where the generator wasn’t passing the form and filter generators a dbMap
  • fixed a bug with mulitple panels batch_actions combo having the same id
  • fixed regression with foreign filter date widgets
  • added new Plain textfield widget and associated ux
  • switched formpanel to used edit configuration
  • renamed fieldset config from params_ to config_ for consistancy
• sfThrift:
  • updated directory structure
  • updated documentation
  • added Thrift protocol factory
• sfDoctrineActAsTaggable:
  • created a Taggable widget that extends the sfWidgetFormInput class

• sfPHPUnit2:

  • added task for compiling plugin core classes for external plugin usage
  • integrated lime lib to compiled file
• sfAssetsLibrary:
  • fixed pdf thumbnail generation
• sfGrid:
  • added support to custom columns (data is obtainded from invoking methods for the current object-row)

• sfAlyssaJqGrid:

  • added support for custom columns
• apostrophe:
  • you can now put :sf_culture in the a_page route if you wish
  • initial version of apostrophe site importer
  • changed image_tag() to url_for() because you can’t use image_tag() inside of an input element
  • aImporter now supports loading of richText and foreignHtml
  • Fixed a lot of search stuff, providing the underpinnings for several bug reports
  • put a mute on aZendSearch
  • there is now blog search in the blog sidebar, which is aware of both publication dates and the category restrictions of the blog engine page you are searching on
  • aMultipleSelect now has an “autocomplete” option
  • changed the cancel helper
  • menuToggle needs to accept functions for before and after open and closing
  • minifier now inserts a newline after each .js file minified
  • created a Taggable widget that extends the sfWidgetFormInput class
  • button slots accept an option for image
  • removed pages fixtures from the plugin
  • added a Focus parameter to the menuToggle so you can focus something after it opens
  • added a persistentLabel parameter to apostrophe.selfLabel so you can choose whether or not the label persists on focus
  • styled the New Post / New Event modal forms
  • refactored the enhanced media pager stuff out of indexSuccess
  • added blog slot stuff to a-area-slots.css
  • styled blog post pager to be visually consistent with media pager
  • fixed bug that was showing esc_raw in the search box
  • updated the aAdmin generator theme to more closely resemble the aBlogAdmin theme
• apostropheBlog:
  • the event form now has an ‘all day’ check box which toggles event time
  • the onChange autosave has been replaced by a setInterval autosave
  • save button has been added to the top of the form
  • the new post / event buttons now use the menuToggle like all of the other menus in apostrophe
  • on initial save, don’t override published_at if it has already been specified
  • don’t try to setTitle on the page object until it has been saved
  • removed stray references to the catTag area
  • added apostrophe-blog:generate-test-posts task, which creates 100 posts with random titles and text
  • implemented 20/50/100 pager

New symfony powered websites

• MyBestApulia: (English, Italian) Online hotel booking and tourist info in Apulia (or Puglia), Italy
• MyBestTrentino: (English, Italian) Online hotel booking and tourist info in Trentino-Alto Adige, Italy

They talked about us

• Doctrine behaviour LooseCoupling now collects garbage with a new deletion cascade
• Propel at Forum PHP Paris
• Diem is hiring!
• symfony.xrea.jp ????????
• A quick start guide to deploying symfony/doctrine applications to productive systems
• Utilisation de VHost pour l’accès au backend
• lyMediaManagerPlugin: FCKEditor and PDF thumbnails
• Lets talk about JSON in Symfony2
• One simple way how to use different symfony app.yml settings for development, preproduction and production servers
• How to customize the error pages in Symfony2
• Presentaciones Symfony en la PHP Barcelona 2010
• Video: IDE Autocompletion with Propel
• Symfony2 stuff
• Advanced symfony Techniques
• Unit testing symfony plugins with php unit
• Symfony MIME-type problem uploading image files
• Les tableaux associatif dans les fichiers de configuration .yml de Symfony
• Organizing larger symfony projects with plugins
• symfony insert raw sql using doctrine
• SYMFONY ?? ?Helpers?
• getInstance method for Doctrine tables
• symfony?????????????
• WordPress????????symfony???
• symfony: get installed plugin
• Bonnes pratiques Symfony: Refactorisons le code

If you’ve been around PHP for a while, you’ve probably heard about APC, the Alternative PHP Cache. Adding APC to an application usually results in improved application response times, reduced server load and happier users. This article will introduce you to APC and guide you through the process of integrating it with your PHP application.

Percona Community,

Percona Server version 5.1.51-rel11.5 is now available for download.

The main purpose of this release is to update the current Percona stable release to the latest version of MySQL 5.1.

Functionality Added or Changed

•  Percona Server 5.1.51-rel11.5 is now based on MySQL 5.1.51.
•  New Features Added: None
•  Other Changes: None

Bugs Fixed

•  Bug #661354 – Fixed a problem compiling query_cache_with comments for 5.1.51-rel11.5. (Oleg Tsarev)
•  Bug #661844 – Fixed a problem with server variables failing test for 5.1.51-rel11.5. (Oleg Tsarev)

The Release Notes for this and previous releases can be found in our Wiki. 

The binary packages are available on our website.
The latest source code for Percona Server, including the development branch, can be found on LaunchPAD.

Please report any bugs found at Bugs in Percona Server.
For general questions, use our Pecona-discussions group, and for development questions our Percona-dev group.
For support, commercial, and sponsorship inquiries, contact Percona.

Entry posted by Fred Linhoss |
No comment

Add to: | | | |

Joey Rivera posted a tutorial on his blog recently discussing how to authenticate with the Twitter API using Zend_Oauth. Click on in for all the details, facts and probably an opinion or two.

This is the third in a series on what’s seriously limiting MySQL in core use cases (links: part 1, 2, 3). This post is about the way MySQL handles connections, allocating one thread per connection to the server.

MySQL is a single process with multiple threads. Not all databases are architected this way; some have multiple processes that communicate through shared memory or other means. It’s cheap to create a connection to MySQL, because it just requires creating a thread (or taking one from a cache). This is generally so fast that there isn’t really the need for connection pools as there is with other databases, at least not in the same way. Windows in particular has had excellent threading support practically forever; Linux has very good threading now, but that wasn’t always the case.

However, many development environments and programming languages really want a connection pool. They’re just built that way (I’m looking at you, Java). And many others use persistent connections by default, so that a connection isn’t really closed when it’s closed; it’s kind of like a connection pool, except that the connection is persisted from request to request within the same process, rather than being shared with whichever request needs a connection.

Connection pools and persistent connections combined with a large number of application servers can lead to a situation where the database server has a very large number of connections open to it, most of which are doing nothing. It’s not uncommon for me to see a server with 1000 to 5000 connections open, and maybe one to three are actually running queries on average. These connections originate from dozens to hundreds of application server instances. When you have a heavily sharded or otherwise horizontally scaled application, it’s not only easy to get into this pickle, it’s really hard or impossible to avoid it.

And with 5000 connections open, you get 5000 threads in the server. That increases the overhead from thread scheduling, and potentially memory usage as well. I feel like I’m forgetting some reasons that this matters — please fill in whatever’s missing in the comments.

There can be more than one solution to this problem, but the one that’s actually partially implemented is a pool of threads, which was originally coded for MySQL 6.0, but is available now in MariaDB.

Unfortunately it isn’t a full solution, because it can cause undesirable lock-out or waiting, and the specific implementation has a scalability bottleneck on multicore servers. Mark Callaghan has done much more investigation of the pool of threads than I have. There are more details in this blog post by Mark, and two followup blog posts from Tim Cook (1, 2).

Thanks for the great comments on the last post. Some of them were good guesses. Remember that the context for this series isn’t micro-limitations or edge-case badness (even if they are serious in some cases), but rather a focus on shortcomings in the main use cases for the server. There are a lot of things MySQL doesn’t do well, but it doesn’t matter that much, because that’s not what it’s designed for. Wrong tool, wrong use, NotABug. I’m thinking of the lack of sort-merge joins or intra-query parallelism, for example. It would be lovely to have those things, if you’re running a data warehouse on MySQL, and in some cases for other uses too (note that most databases that do have these query plans usually try to use nested-loop joins whenever possible, because of things like the lower startup cost for the query). But MySQL isn’t a data warehouse DBMS first and foremost. It’s a general-purpose OLTP database server that runs well on affordable hardware and is great for Web usage. It’s so good, in fact, that it can be used for tons of other things such as… data warehousing. But it isn’t a Netezza or Paraccel, and if it were, it wouldn’t be a great OLTP web database too.

MySQL replication is one of the core, fundamental features — and it’s single-threaded and relies on the binary log, which are two major limitations. And it has subqueries, which are a core, fundamental part of SQL — but it’s bad at certain kinds of them. That’s why I listed those as major limitations. And because MySQL is a multi-threaded database for Web usage that tends to be used in sharded environments with tons of application servers, which creates a situation with many thousands of connections to the database, and because it doesn’t handle that very well, I list its one-thread-per-connection design as a serious limitation.

Entry posted by Baron Schwartz |
13 comments

Add to: | | | |

My regular readers may have already noticed I don’t even dare writing code without TDD/BDD-ing it from left to right. In my Zend Framework-based project, we currently have 600+ PHPUnit test cases, most of them for models and controllers, but quite some also for view helpers, bootstrap and even some views.

The whole suite is taking around 90 seconds to run on my machine against a MySQL database (modelled with Zend_Db* in an active record/table data gateway manner).

In this post I’ll be showing how I managed to decrease the time from ~90 seconds to ~30 seconds using a PHPUnit listener and database transactions.

NOTE: This is a copy of the original article on using PHPUnit listeners to wrap each test case in a database transaction available under this link .

A couple of weeks ago I blogged about Sharing an auto_increment value across multiple MySQL tables. In the comments, a few people wrote in to suggest alternative ways of implementing this.  I just got around to benchmarking those alternatives today across two large EC2 machines:

(Measured in transactions/second – higher is better)

What is the conclusion?  With the exception of my original option2, they actually all perform fairly similar.  The Flickr and Option1 tests perform marginally better.  Test “arjen2″ is option2, but with a MyISAM table — it suffers a little because EC2 can be a little high for latency, and there’s one additional round trip.  Test arjen2005 is not too dissimilar from the Flickr solution, but uses a MySQL stored function.

Full Disclosure.

Entry posted by Morgan Tocker |
7 comments

Add to: | | | |

This is the third in a series on what’s seriously limiting MySQL in certain circumstances (links: part 1, 2). This post is about subqueries, which in some cases execute outside-in instead of inside-out as users expect.

It’s easy to pick on subqueries in MySQL, so I’ll try to be gentle. The following query will surprise users unpleasantly:

select * from a where a.id in (select id from b);

Users expect the inner query to execute first, then the results to be substituted into the IN() list. But what happens instead is usually a full scan or index scan of table a, followed by N queries to table b. This is because MySQL rewrites the query to make the inner query dependent on the outer query, which could be an optimization in some cases, but de-optimizes the query in many other cases. NOT IN(SELECT …) queries execute badly, too. (Note: putting a literal list of items in the IN() clause performs fine. It’s only when there is a SELECT inside it that it works poorly.)

The fix for this has been in progress for a few years, and Sergey Petrunia committed working code to the stalled 6.0 release. But it’s not quite clear whether that code was a complete solution. It has not been in any GA or RC release, so it hasn’t been used widely.

To be fair, many other database servers also have poor subquery performance, or have had it in the past and have fixed it. And many MySQL users have learned to simply write JOINs instead, so it isn’t that much of a limitation. But it would be a big improvement if it were fixed.

See if you can guess what limitation number 4 will be!

Entry posted by Baron Schwartz |
17 comments

Add to: | | | |

The parameter sort_buffer_size is one the MySQL parameters that is far from obvious to adjust. It is a per session buffer that is allocated every time it is needed. The problem with the sort buffer comes from the way Linux allocates memory. Monty Taylor (here) have described the underlying issue in detail, but basically above 256kB the behavior changes and becomes slower. After reading a post from Ronald Bradford (here), I decide to verify and benchmark performance while varying the size of the sort_buffer. It is my understanding that the sort_buffer is used when no index are available to help the sorting so I created a MyISAM table with one char column without an index:

and I inserted 100k rows with this simple script:

I know, I could have used the uuid() function of MySQL. For the benchmark, I used an old PII 350 MHz computer, I think for such CPU bound benchmarks, an old computer is better, if there are small differences, they’ll be easier to observe. I varied the sort_buffer_size by steps of 32 KB and I recorded the time required to perform 12 queries like ‘select * from sorttest order by data limit 78000,1′ with, of course, the query cache disabled. I also verified that during the whole process, the computer never swapped and I pre-warmed the file cache before the benchmark by doing “alter table sorttest engine=myisam;”. The script used for the benchmark is the following:

which in addition to output the total time, output the number of Sort_merge_passes, which will be useful to interpret the results. The figure below shows a graphical representation of the results.

The first we can notice by looking at the graph is that the expected correspondence between the time for the queries and the number of sort merge passes. For the small values of the sort buffer size, below 440KB, there are many sort merge passes and the time for the queries hover around 18s. Above 440K, as the sort merge passes drops to 1, there is a large drop of the time for the queries below 14s. Then, as the sort buffer size is further risen, the performance gain is negative up to the point, around 6.4MB where no more sort merge pass are required and then, the time for the queries loses all dependency over the sort buffer size. I am still trying to figure out why the number of sort merge passes felt to zero at 6.4MB since the total size of the table is less than 3MB. That seems to be a pretty high overhead per row (~37 bytes) but if the structure has a few pointers, that can add up to such amount of bytes pretty quickly.

The important point here, at least for the Linux, glibc and MySQL versions I used and for the test I did, there doesn’t seem to be an observable negative impact of the glibc memory allocation threshold at 256KB. I’ll try to find ways to repeat this little experiment with the other per session buffers just to confirm the findings.

OS: Ubuntu 10.04 LTS, Linux test1 2.6.32-21-generic-pae #32-Ubuntu SMP Fri Apr 16 09:39:35 UTC 2010 i686 GNU/Linux
MySQL: 5.1.41-3ubuntu12.6-log

P.S.: Gnumerics for graphs is so much better than OpenOffice Calc.

Entry posted by Yves Trudeau |
7 comments

Add to: | | | |

The Symfony Day conference is an annual event held in Cologne, Germany and hosted by Interlutions. It was held on October 8th, 2010 and it is the second year the conference was put together and it was a major success to say the least! I was honored to give a talk about the latest Doctrine project, the MongoDB Object Document Mapper. Several other great talks were given by some of the top Symfony community members, including a talk by our fearless leader, Fabien, on The State of Symfony2. You can view the event on joind.in if you attended. Below you will find all of the talks given and the presentations for each of them:

• Doctrine MongoDB Object Document Mapper by Jonathan H. Wage
• PHP, symfony and software lifecycles by Pierre Joye
• Unit testing symfony plugins with PHPUnit by Christian Schaefer
• Advanced symfony Techniques by Kris Wallsmith
• Framework or CMS by Gaylord Aulke
• The State of Symfony2 by Fabien Potencier

Stefan Koopmanschap also gave a workshop on getting to know Symfony and it was well received by its attendees. We will also have training workshops at the upcoming Symfony Live 2011 conferences so if you would like to attend one be sure to check it out now!