Personal blog of Yzmir Ramirez

Just few short hours remain for Early Bird Discount for Percona Live, New York City. This is your last chance to grab the ticket at almost 50% discount, for $129 instead of $259. We have a Great Session Lineup and outstanding networking opportunities !

We just pushed to sysbench support for workload against multiple tables ( traditionally it used only single table).

It is available from launchpad source tree lp:sysbench .

This is set of LUA scripts for sysbench 0.5 ( it supports scripting), and it works following way:

- you should use --test=tests/db/oltp.lua to run OLTP test
i.e. prepare

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=25 prepare

run:

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=25 --num-threads=5 run

oltp.lua should understand most options that available for regular sysbench –test=oltp

there are couple other scripts, like
oltp_simple
select
insert
delete
update_index
update_non_index

to support different OLTP modes.

Most interesting: all scripts support --oltp-tables-count=N ( default 1), in this case N sbtest tables will be used.

Additional parameter --oltp-secondary is taken from Mark Callaghan’s sysbench tree , in this case
table sbtest will be created with KEY xid (ID) , instead of PRIMARY KEY (ID)

Now, as we have N tables, we may want to prepare them in parallel, to save some time for preparation
I implemented that in script parallel_prepare.lua

so instead of

./sysbench --test=tests/db/oltp.lua --oltp-tables-count=25 prepare

you may use

./sysbench --test=tests/db/parallel_prepare.lua --oltp-tables-count=64 --num-threads=8 run

( please note that oltp-tables-count should be multiplier of num-threads)

A month ago, I
announced
that the final version of Symfony2 was delayed. At that time, we were eagerly
waiting for the merge of the new form framework. And I can tell you that it
was worth the wait. Bernhard has done a fantastic job and I’m confident that
you will love the new version.

All features of Symfony 2.0 are now available. That means that we can finally
enter the long-awaited beta/RC release cycle that will eventually lead to the
release of the shiny 2.0 version.

Beta means that all features are not stable yet. Beta also means that we will
break backward compatibility only for good reasons and that all the breaks
will be explained and documented in the UPDATE file to ease the migration path
of existing applications.

Download Symfony 2.0 Standard Edition beta1,
read the quick
tour, or dive into the book.

Oh, and don’t forget to give us feedback.
Tell us about the remaining
pain points, create tickets for
bugs, contribute
patches, …

Like with the preview releases, we will release often, probably once a week.
And depending on the feedback we will have, I expect the beta phase to last a
month or so. The first release candidate will be released when all blocking
bugs will be closed and when the documentation will be complete.

New Features

Variable Changes

Bug Fixes

1. Bug #757749 – Using ALTER TABLE to convert an InnoDB table to a MEMORY table could fail due to a bug in innodb_expand_fast_index_creation.patch. (Alexey Kopytov)
2. Bug #764395 – InnoDB crashed with an assertion failure when receiving a signal on pwrite(). The problem was that InnoDB I/O code was not fully conforming to the standard on POSIX systems. Calls to fsync(), pread(), and pwrite() can be interrupted by a signal. In such cases, InnoDB would crash with an assertion failure, rather than just restarting the interrupted call. (Alexey Kopytov)
3. Bug #766236 – A crash was occurring in some cases when innodb_lazy_drop_table was enabled with very large buffer pools. (Yasufumi Kinoshita)
4. Bug #733317 – SYS_STATS internal table of XtraDB has been expanded for supporting ‘innodb_stats_method’ from InnoDB-plugin. (Yasufumi Kinoshita)

Please report any bugs found at Bugs in Percona Server.

For more information, please see the following links:

• Downloads: Binary distributions, Percona Software Repositories and Experimental build for MacOS
• Installation instructions
• Consulting, support, training, and custom development
• Release notes
• Documentation
• Developer discussion group

Good news for all Percona Software users out there. There are more ways how you can get access to Percona Software. If you’re using Scalr (See also Scalr Project on Google Code) you might be interested in Percona Server roles support Scalr team has announced recently. If you’re Mac user and looking for complete development environment for PHP and MySQL you can check out MNPP which also includes lightweight Ngnix web server instead of Apache.

Now about any Open Source projects including or based on Percona Server or Xtrabackup ? Let us know !

With the release of the first beta approaching fast, our main focus has
switched from adding new features to polishing existing ones.

Symfony2 has many great features. But sometimes, configuration feels a bit too
complex, or the error messages are not friendly enough. We care a lot about
these problems as these issues can lead to a lot of frustration.

Recently, Ryan and I have spent our time tweaking error messages, simplifying
the code and the configuration, adding more documentation, and making things
more consistent throughout the framework. The goal is to ease the learning
curve and make things that people will need on a day to day basis simpler.

Let’s see some examples of what we’ve done so far.

Better Twig Error Messages

Twig is a great time saver thanks to its concise syntax and its great
automatic output escaping feature. Each time I need to edit an “old” template
written in PHP, I’m really happy that this is just about legacy code (and
apparently I’m not the only
one)

One problem you might have encounter though is when something goes wrong
during template compilation or at runtime. Sometimes, the exception message
does not mention the original template name, nor the line where the problem
happened; and of course, the PHP stack trace only has information about the
cached PHP file. That makes debugging harder than it has to be. This has been
“fixed” recently. Now, Twig tries very
hard
to always tell you where the error occurred in the original template file.

And on a totally unrelated note, Twig now also enjoys native support in
PHPStorm.

Better Configuration Error Messages

What I’ve described above also apply to configuration errors. Here, we are
talking about configuration files (in XML or YAML) instead of Twig templates.
For instance, if you use a non-defined parameter in a service definition, you
will have the following error
message:

The service "%s" has a dependency on a non-existent parameter "foo".

And here is the old one:

You have requested a non-existent parameter "foo".

The new message instantly tells you what you need to look for.

Simpler Doctrine Configuration

Most projects will ever only use one database. But the Doctrine configuration
syntax is “optimized” for when you need to be able to manage several database
connections. This makes the configuration verbose, non-trivial to understand,
and hard to teach:

doctrine:
    dbal:
        connections:
            default:
                dbname:   dbname
                user:     root
                password: ~

As of today, if you only need one database connection, you can use this
simpler syntax:

doctrine:
    dbal:
        dbname:   sfweb
        user:     root
        password: ~

Configuring the ORM is also a pain point as every time you add a third-party
bundle, you must remember to add it to the Doctrine mapping configuration. If
you forget, you will have an error that won’t really help you understand what
you need to do to fix it.

If you only use one entity manager (which is again the most common use case),
you can now just tell Doctrine to automatically load all bundle’s mapping
configuration:

orm:
    auto_mapping: true

For the record, here is the “old” minimal ORM configuration:

    orm:
        entity_managers:
            default:
                mappings:
                    BlogBundle: ~

The good news is that you can still fallback to the old syntax for the rare
occasions where you need more than one database connection or more than one
entity manager.

This post is really just a sample of what we are doing on a day to day basis.
The good news is that *you** can help us make the framework better. Whenever
you hit a wall because an error message is not clear, or because a syntax
looks awkward, please tell us about it; write an email to the Symfony
developer mailing-list and we
will try to work together to make things better and easier. Expect more
similar changes in the coming weeks.

Now that you’re done paying the Tax Man (for the North Americans reading this blog post), it’s time to pay it forward by helping an open source project close some issues! For those who haven’t put the reoccurring event in their calendar, this announcement is for you: Zend Framework Monthly Bug-hunt is here again! Tomorrow , Friday and Saturday of April (the 28th , 29th and 30th 2011 ), we’ll be hosting our monthly bug hunt. For those of you unfamiliar with the event, each month, we organize the community to help reduce the number of open issues reported against the framework.

There are many ways to slice and aggregate metrics of activity on a system such as MySQL. In the best case, we want to know everything about the system’s activity: we want to know how many things happened, how big they were, and how long they took. We want to know precisely when they happened. We want to know resource consumption, instantaneous status snapshots, and wait analysis and graphs. These things can be expensive to measure, and even more importantly, they can require a lot of work to analyze. And sometimes the analysis is only valid when very stringent conditions are satisfied. What metrics can we gather that provide the most value for the effort?

There’s a relationship known as Little’s Law (sometimes erroneously referred to as the fundamentals of queueing theory), that lays out some simple properties between various metrics. One of the properties of Little’s Law is that it holds regardless of inter-arrival-time distributions and other complex things. And you can build upon it to create surprisingly powerful models of the system.

Let’s begin with a query. It arrives, executes for a bit, and then completes.

Then some time elapses, and another query arrives, executes, and completes.

To add the minimal amount of extra complexity, let’s say that another query arrived while query 2 was executing, and completed after query 2 completed. Now we have the following picture, fully annotated. The arrivals and completions are denoted by “a” and “c”.

If we start at the left end of the Time arrow, and call it the beginning of the observation period, and observe the system until the right end of the arrow, we have our observation interval. If we travel from left to right along the arrow, we can count the “residence time” as the time during which at least one query is running. This is the period from 1a to 1c, plus the period from 2a to 3c. I am in the habit of calling this the “busy time” for historical reasons, too.

It’s easy to compute this by just creating a counter, a running total duration, and a last-event timestamp. Every time there is an arrival or completion, look at the counter of currently executing queries. If it is greater than 0, increment the duration by the time elapsed since the last arrival or completion. Then increment the counter of currently executing queries if the current event is an arrival, and decrement it if it’s a completion. Finally, set the last-event timestamp to the current time.

Now, if we add another duration variable, we can get what I call the “weighted time.” (I have not heard a better phrase for this.) This is similar to the residence time, but instead of just adding the time elapsed since the last event, we multiply the time elapsed by the number of currently executing queries and add it. This is effectively the “area under the curve” — the blue-colored regions in the picture above. (Each query has a height of 1.)

Finally, we can simply count the number of arrivals or completions. It doesn’t matter which; Little’s Law is based on a long-term average, where the two are equal.

All together, we have the following four fundamental metrics:

1. The observation interval.
2. The number of queries in the interval.
3. The total time during which queries resided in the system — the residence time.
4. The total execution time of all queries — the “weighted time.”

We can maintain these variables in the system, and sample them at will by simply looking at the clock and writing down the three variables at the start and end of whatever observation period we choose.

What is this good for? We can use these together with Little’s Law to derive these additional four long-term average metrics:

1. Throughput: divide the number of queries in the observation interval by the length of the interval.
2. Execution time: divide the weighted time by the number of queries in the interval.
3. Concurrency: divide the weighted time by the length of the interval.
4. Utilization: divide the residence time by the length of the interval.

This is extremely powerful. These are the key metrics we need to model scalability with the Universal Scalability Law, perform queueing analysis, and do capacity planning. And we obtained all of this without needing to analyze anything complicated at all. This is just addition, subtraction, multiplication, and division of some constantly increasing counters and durations!

There’s more. We derived our four long-term average metrics from the four fundamental metrics, and we derived those in turn from the arrivals and completions. In a future article, I’ll show you what else we can learn by observing arrivals and completions. And I’ll present on this at Percona Live, where I’ll show you how to tie it all together quickly to answer real-world questions about practically any system, whether it’s instrumented with the variables I mentioned or not, with shockingly little effort.

PS: I’ve edited to reflect Dr. Gunther’s comments, which I solicited. Also, readers should know that these are the four metrics that I think of as fundamental — I’m not referring to any established standard. PPS: updated to clarify “counter” versus “duration” as per Mr. Wilson’s comments.

The spilled beer had barely had time to dry on the floor at The PHP Community Conference last week before Brian Moon had already cranked out a blog post on everybody’s favorite whipping boy, PHP Frameworks.

I have just finalized the tour schedule for my “SQL Performance Basics Workshop for Developers“.
The most important tips and tricks every developer must know—in just 3 hours. It takes place in your office because every developer—the entire team—should take part.

The tour covers Austria, Germany and Switzerland. Just let me know if there isn’t a scheduled date for your area. I’m sure we can arrange something. Small teams can also take the virtual workshop.

Hope to see you soon.

Workshop Details