Jan
29
2015
--

GHOST vulnerability (CVE-2015-0235) Percona response

Cloud security company Qualys announced Tuesday the issues prevalent in glibc since version 2.2 introduced in 2000-11-10 (the complete Qualys announcement may be viewed rel="nofollow" href="http://www.openwall.com/lists/oss-security/2015/01/27/9" rel="nofollow">here). The vulnerability, CVE-2015-0235, has been dubbed “GHOST.”

As the announcement from Qualys indicates, it is believed that MySQL and by extension Percona Server style="text-decoration: underline;">are not affected by this issue.

Percona is in the process of conducting our own review into the issue related to the Percona Server source code – more information will be released as soon as it is available.

In the interim the current advisory is to update your glibc packages for your distributions if they are in fact vulnerable. The C code from the Qualys announcement may aid in your diagnostics, section 4 of rel="nofollow" href="https://www.qualys.com/research/security-advisories/GHOST-CVE-2015-0235.txt" rel="nofollow">this document or via rel="nofollow" href="https://gist.github.com/Oneiroi/8622ff9ff26854ccd4ed#file-qualys-cve-2015-0235-c" rel="nofollow">this gist. I also wrote a very quick python script to help identify processes which may be running libc that you can access rel="nofollow" href="https://gist.github.com/Oneiroi/8622ff9ff26854ccd4ed#file-find-libc-in-memory-of-processes-run-as-root-py" rel="nofollow">here.

Compiling the above and executing it will yield an output indicating if your glibc version is believed to be vulnerable or not vulnerable.

Distribution Resource Resource Links

    1. RedHat BZ: rel="nofollow" href="https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2015-0235" rel="nofollow">https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2015-0235
    2. RedHat EL5 Errata: rel="nofollow" href="https://rhn.redhat.com/errata/RHSA-2015-0090.html" rel="nofollow">https://rhn.redhat.com/errata/RHSA-2015-0090.html
    3. RedHat EL6 / 7 Errata: rel="nofollow" href="https://rhn.redhat.com/errata/RHSA-2015-0092.html" rel="nofollow">https://rhn.redhat.com/errata/RHSA-2015-0092.html
    4. Ubuntu USN: rel="nofollow" href="http://www.ubuntu.com/usn/usn-2485-1/" rel="nofollow">http://www.ubuntu.com/usn/usn-2485-1/ (affects 10.04 12.04)
    5. Debian security tracker: rel="nofollow" href="https://security-tracker.debian.org/tracker/CVE-2015-0235" rel="nofollow">https://security-tracker.debian.org/tracker/CVE-2015-0235

Distributions which use musl-libc ( rel="nofollow" href="http://www.musl-libc.org/" rel="nofollow">http://www.musl-libc.org/) are not affected by this issue.

Acknowledgements

rel="nofollow" href="https://www.qualys.com/" rel="nofollow">Qualys

Robert Barabas – Percona /> Raghavendra Prabhu – Percona /> Laurynas Biveinis – Percona

The post rel="nofollow" href="http://www.percona.com/blog/2015/01/29/ghost-vulnerability-cve-2015-0235-percona-response/">GHOST vulnerability (CVE-2015-0235) Percona response appeared first on rel="nofollow" href="http://www.percona.com/blog">MySQL Performance Blog.

Jan
29
2015
--

Huddle Co-Founder Alastair Mitchell Steps Down As CEO, Morten Brogger Steps In

cloud On the heels of a $51 million round of funding, cloud collaboration company Huddle today announced a changing of the guard to lead the company to its next stage. Co-founder Alastair Mitchell is stepping down from his role as CEO and assuming a new position as president and CMO. Taking his place is Morten Brogger, a veteran exec whose past roles included CEO of Mach, which he helped sell… Read More

Jan
29
2015
--

Multi-threaded replication with MySQL 5.6: Use GTIDs!

MySQL 5.6 allows you to execute replicated events in parallel as long as data is split across several databases. This feature is named “Multi-Threaded Slave” (MTS) and it is easy to enable by setting slave_parallel_workers to a > 1 value. However if you decide to use MTS without GTIDs, you may run into annoying issues. Let’s look at two of them.

Skipping replication errors

When replication stops with an error, a frequent approach is to “ignore now and fix later.” This means you will run SET GLOBAL sql_slave_skip_counter=1 to be able to restart replication as quickly as possible and later use pt-table-checksum/pt-table-sync to resync data on the slave.

Then the day when I hit:

mysql> show slave status;
[...]
Last_SQL_Error: Worker 0 failed executing transaction '' at master log mysql-bin.000017, end_log_pos 1216451; Error 'Duplicate entry '1001' for key 'PRIMARY'' on query. Default database: 'db1'. Query: 'INSERT INTO sbtest1 (id, k, c, pad) VALUES (0, 5320, '49123511666-22272014664-85739796464-62261637750-57593947547-00947134109-73607171516-11063345053-55659776318-82888369235', '11400300639-05875856680-20973514928-29618434959-69429576205')'
Exec_Master_Log_Pos: 1005432

I tried to use the trick:

mysql> set global sql_slave_skip_counter=1;
mysql> start slave;

But:

mysql> show slave status;
[...]
Last_SQL_Error: Worker 0 failed executing transaction '' at master log mysql-bin.000017, end_log_pos 1216451; Error 'Duplicate entry '1001' for key 'PRIMARY'' on query. Default database: 'db1'. Query: 'INSERT INTO sbtest1 (id, k, c, pad) VALUES (0, 5320, '49123511666-22272014664-85739796464-62261637750-57593947547-00947134109-73607171516-11063345053-55659776318-82888369235', '11400300639-05875856680-20973514928-29618434959-69429576205')'
Exec_Master_Log_Pos: 1005882

Note that the position reported with Exec_Master_Log_Pos has moved forward, but I still have my duplicate key error. What’s wrong?

The issue is that the positions reported by SHOW SLAVE STATUS are misleading when using MTS. Quoting the rel="nofollow" href="http://dev.mysql.com/doc/refman/5.6/en/show-slave-status.html" rel="nofollow">documentation about Exec_Master_Log_Pos:

When using a multi-threaded slave (by setting slave_parallel_workers to a nonzero value in MySQL 5.6.3 and later), the value in this column actually represents a “low-water” mark, before which no uncommitted transactions remain. Because the current implementation allows execution of transactions on different databases in a different order on the slave than on the master, this is not necessarily the position of the most recently executed transaction.

So the solution to my problem is first to make sure that there is no execution gap, and only then to skip the offending event. There is a specific statement for the first part:

mysql> start slave until sql_after_mts_gaps;

And now I can finally skip the error and restart replication:

mysql> set global sql_slave_skip_counter=1;
mysql> start slave;
mysql> show slave statusG
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

The last thing to do is of course to resync the slave.

Backups

If you cannot trust the output of SHOW SLAVE STATUS to get the current binlog position, it means that taking a backup from a slave with parallel replication is tricky.

For instance, if you run mysqldump --dump-slave=2 to get the binlog position of the master, mysqldump will first run STOP SLAVE and then SHOW SLAVE STATUS. Is stopping the slave sufficient to avoid execution gaps? Actually, rel="nofollow" href="http://bugs.mysql.com/bug.php?id=74528" rel="nofollow">no.

The only option then seems to be: run STOP SLAVE followed by START SLAVE UNTIL SQL_AFTER_MTS_GAPS, followed by mysqldump while replication is stopped. Not very handy!

GTIDs to the rescue!

The solution for both issues is to use GTIDs.

They help when you want to skip an event because when using GTIDs, you must explicitly specify the transaction you will be skipping. It doesn’t matter whether there are execution holes.

They also help for backups because mysqldump takes the position from gtid_executed which is updated at each transaction commit (XtraBackup does that too).

Conclusion

If your application uses several databases and if you’re fighting with replication lag, MTS can be a great feature for you. But although GTIDs are not technically necessary, you’ll be exposed to tricky situations if you don’t use them.

Is everything rosy when using both GTIDs and MTS? Not exactly… But that will be the topic for a separate post!

By the way, if you are in the Brussels area this weekend, come see me and other great speakers at the rel="nofollow" href="https://fosdem.org/2015/schedule/track/mysql_and_friends/" rel="nofollow">MySQL and friends devroom at rel="nofollow" href="https://fosdem.org/2015/" rel="nofollow">FOSDEM!

The post rel="nofollow" href="http://www.percona.com/blog/2015/01/29/multi-threaded-replication-with-mysql-5-6-use-gtids/">Multi-threaded replication with MySQL 5.6: Use GTIDs! appeared first on rel="nofollow" href="http://www.percona.com/blog">MySQL Performance Blog.

Jan
28
2015
--

Investors Throw Datadog A $31M Bone

Dog holding two one hundred dollar bills in its mouth like a bone. Datadog, a cloud service that helps customers monitor infrastructure and software, whether all in the cloud or a hybrid on-premises-cloud environment, announced $31M in Series C funding today. The round was led by Index Ventures with help from RTP Ventures, Openview Partners and what they referred to as “other equity holders.” Index helped fund the Series B round last February,… Read More

Jan
28
2015
--

Amazon WorkMail Takes On Microsoft And Google For Enterprise Email And Calendars

aws_logo Amazon has a new product called WorkMail debuting today (via Forbes), which is an email and calendaring service that aims to provide those tools for corporate customers. The tech is based on Amazon Web Services, and aims to best the reigning champs (which include Microsoft, and to a lesser extent, Google) in terms of ease of use and security. Read More

Jan
27
2015
--

Percona University: Back to school Feb. 12 in Raleigh, N.C.

id="attachment_12663" style="width: 310px" class="wp-caption alignright"> href="http://www.percona.com/blog/wp-content/uploads/2013/01/PMySQLU.jpg"> class="size-medium wp-image-12663" src="http://www.percona.com/blog/wp-content/uploads/2013/01/PMySQLU-300x224.jpg" alt="Percona CEO Peter Zaitsev leads a track at the inaugural Percona University event in Raleigh, N.C. on Jan. 29, 2013." width="300" height="224" /> class="wp-caption-text">Percona CEO Peter Zaitsev leads a track at the inaugural Percona University event in Raleigh, N.C. on Jan. 29, 2013.

About two years ago we held our first-ever href="http://www.percona.com/news-and-events/percona-mysql-university/raleigh-nc-campus-2013">Percona University event in Raleigh, N.C. It was a great success with high attendance and very positive feedback which led us to organize a number of similar educational events in different locations around the world.

And next month we’ll be back where it all started. On February 12, Percona University comes to Raleigh – and this time the full-day educational event will be much more cool. What have we changed? Take a look at href="http://www.percona.com/news-and-events/percona-university/raleigh-united-states-2015" >the agenda.

First - this is no longer just a MySQL-focused event. While 10 years ago MySQL was the default, dominating choice for modern companies looking to store and process data effectively – this is no longer the case. And as such the event’s theme is “Smart Data.” In addition to MySQL, Percona and MariaDB technologies (which you would expect to be covered), we have talks about Hadoop, MongoDB, Cassandra, Redis, Kafka, SQLLite.

However the “core” data-store technologies is not the only thing successful data architects should know – one should also be well-versed in the modern approaches to the infrastructure and general data management. This is why we also have talks about Ansible and OpenStack, DBaaS and PaaS as well as a number of more talks about big-picture topics around architecture and technology management.

Second – this is our first multi-track Percona University event – we had so many great speakers interested in speaking that we could not fit them all into one track, so we have two tracks now with 25 sessions which makes that quite an educational experience!

Third – while we’re committed to having those events be very affordable, we decided to charge $10 per attendee. The reason for this is to encourage people to register who actually plan on attending – when hosting free events we found out that way too many registered and never showed up, which was causing the venues to rapidly fill past capacity and forcing us to turn away those who could actually be there. It was also causing us to order more food than needed, causing waste. We trust $10 will not prevent you from attending, but if it does cause hardship, just drop me a note and I’ll give you a free pass.

A few other things you need to know:

This is very much a technically focused event. I have encouraged all speakers to make it about technology rather than sales pitches or marketing presentations.

This is low-key educational event. Do not expect it to be very fancy. If you’re looking for the great conference experience consider attending the href="http://www.percona.com/live/mysql-conference-2015/home">Percona Live MySQL Conference and Expo this April.

Although it’s a full-day event, you can come for just part of the day. We recognize many of you will not be able to take a full day from work and may be able to attend only in the morning or the afternoon. This is totally fine. The morning registration hours is when most people will register, however, there will be someone on the desk to get you your pass throughout the day.

Thinking of Attending? Take a look at the day’s href="http://www.percona.com/news-and-events/percona-university/raleigh-united-states-2015" >sessions and then href="http://www.percona.com/news-and-events/percona-university/raleigh-united-states-2015" >register as space is limited. The event will be held at North Carolina State University’s McKimmon Conference & Training Center. I hope to see you there!

The post rel="nofollow" href="http://www.percona.com/blog/2015/01/27/percona-university-back-to-school-feb-12-in-raleigh-n-c/">Percona University: Back to school Feb. 12 in Raleigh, N.C. appeared first on rel="nofollow" href="http://www.percona.com/blog">MySQL Performance Blog.

Jan
26
2015
--

Here’s What You Need To Know From Microsoft’s $26B Quarter

microsoft-data Microsoft reported its fiscal second-quarter financial performance today, laid out its forecasts for its future performance and took questions from a number of analyst. It was a flurry of data, so let’s take a moment and dig through the big points that matter. Read More

Jan
26
2015
--

Sources Say IBM Planning On Laying Off 12,000 Over Next Year

IBM Logo Rumors have been swirling — and IBM’s stock has been rising — on reports that the company is planning a huge round of layoffs of up to 26% of its workforce, or 118,000 people. TechCrunch has been digging and has found out from two separate sources that there will in fact be job cuts, but not anywhere close to this magnitude. Multiple sources tell TechCrunch the 118,000… Read More

Jan
26
2015
--

Percona Toolkit 2.2.13 is now available

href="http://www.percona.com/blog/wp-content/uploads/2013/03/Percona_ToolkitLogoVert_RGB.png"> class="alignright wp-image-13951" style="margin: 7px;" src="http://www.percona.com/blog/wp-content/uploads/2013/03/Percona_ToolkitLogoVert_RGB-300x249.png" alt="Percona Toolkit" width="180" height="149" />Percona is pleased to announce the availability of  href="http://www.percona.com/software/percona-toolkit">Percona Toolkit 2.2.13.  Released January 26, 2015. Percona Toolkit is a collection of advanced command-line tools to perform a variety of MySQL server and system tasks that are too difficult or complex for DBAs to perform manually. Percona Toolkit, like all Percona software, is free and open source.

This release is the current GA (Generally Available) stable release in the 2.2 series. It includes multiple bug fixes for pt-table-checksum with better support for Percona XtraDB Cluster, various other fixes, as well as continued preparation for MySQL 5.7 compatibility. Full details are below. Downloads are available  href="http://www.percona.com/downloads/percona-toolkit/2.2.13/">here and from the  href="http://www.percona.com/doc/percona-server/5.5/installation.html">Percona Software Repositories.

New Features:

  • href="http://www.percona.com/doc/percona-toolkit/2.2/pt-kill.html">pt-kill now supports new --query-id option. This option can be used to print a query fingerprint hash after killing a query to enable the cross-referencing with the href="http://www.percona.com/doc/percona-toolkit/2.2/pt-query-digest.html">pt-query-digest output. This option can be used along with --print option as well.

Bugs Fixed:

  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1408375" rel="nofollow">1408375: Percona Toolkit was vulnerable to MITM attack which could allow exfiltration of MySQL configuration information via --version-check option. This vulnerability was logged as rel="nofollow" href="http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2015-1027" rel="nofollow">CVE 2015-1027
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1019479" rel="nofollow">1019479: href="http://www.percona.com/doc/percona-toolkit/2.2/pt-table-checksum.html">pt-table-checksum now works with ONLY_FULL_GROUP_BY SQL mode.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1394934" rel="nofollow">1394934: running pt-table-checksum in debug mode would cause an error.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1396868" rel="nofollow">1396868: regression introduced in Percona Toolkit 2.2.12 caused href="http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html">pt-online-schema-change not to honor --ask-pass option.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1399789" rel="nofollow">1399789: pt-table-checksum would fail to find Percona XtraDB Cluster nodes when variable href="http://www.percona.com/doc/percona-xtradb-cluster/5.6/wsrep-system-index.html#wsrep_node_incoming_address">wsrep_node_incoming_address was set to AUTO.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1321297" rel="nofollow">1321297: pt-table-checksum was reporting differences on timestamp columns with replication from 5.5 to 5.6 server version, although the data was identical.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1388870" rel="nofollow">1388870: pt-table-checksum was showing differences if the master and slave were in different time zone.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1402668" rel="nofollow">1402668: pt-mysql-summary would exit if Percona XtraDB Cluster was in Donor/Desynced state.
  • Fixed bug rel="nofollow" href="https://bugs.launchpad.net/bugs/1266869" rel="nofollow">1266869: href="http://www.percona.com/doc/percona-toolkit/2.2/pt-stalk.html">pt-stalk would fail to start if $HOME environment variable was not set.

Details of the release can be found in the  href="http://www.percona.com/doc/percona-toolkit/2.2/release_notes.html#v2-2-13-released-2015-01-26">release notes and the  rel="nofollow" href="https://launchpad.net/percona-toolkit/+milestone/2.2.13" rel="nofollow">2.2.13 milestone at Launchpad. Bugs can be reported on the Percona Toolkit rel="nofollow" href="https://bugs.launchpad.net/percona-toolkit/+filebug" rel="nofollow">launchpad bug tracker.

The post rel="nofollow" href="http://www.percona.com/blog/2015/01/26/percona-toolkit-2-2-13-now-available/">Percona Toolkit 2.2.13 is now available appeared first on rel="nofollow" href="http://www.percona.com/blog">MySQL Performance Blog.

Jan
26
2015
--

MySQL benchmarks on eXFlash DIMMs

In this blog post, we will discuss MySQL performance on eXFlash DIMMs. href="http://www.percona.com/blog/2014/08/12/benchmarking-exflash-with-sysbench-fileio/">Earlier we measured the IO performance of these storage devices with sysbench fileio.

Environment

The benchmarking environment was the same as the one we did sysbench fileio in.

CPU: 2x Intel Xeon E5-2690 (hyper threading enabled) /> FusionIO driver version: 3.2.6 build 1212 /> Operating system: CentOS 6.5 /> Kernel version: 2.6.32-431.el6.x86_64

In this case, we used a separate machine for testing which had a 10G ethernet connection to this server. This server executed sysbench. The client was not the bottleneck in this case. The environment is described in greater detail at the end of the blog post.

Sysbench OLTP write workload

href="http://www.percona.com/blog/wp-content/uploads/2015/01/exflash_sysbench_oltp_tp_partial.png"> class="alignnone size-full wp-image-27989" src="http://www.percona.com/blog/wp-content/uploads/2015/01/exflash_sysbench_oltp_tp_partial.png" alt="exflash_sysbench_oltp_tp_partial" width="780" height="780" />

The graph shows throughput for sysbench OLTP, we will examine properties only for the dark areas of this graph: which is the read/write case for high concurrency.

Each table in the following sections has the following columns /> class="table-responsive">
style="width:100%; " class="easy-table easy-table-cuscosky " > column explanation storage The device that was used for the measurement. threads The number of sysbench client threads were used in the benchmark. ro_rw Read-only or read-write. In the whitepaper you can find detailed information about read-only data as well. sd The standard deviation of the metric in question. mean The mean of the metric in question. 95thpct The 95th percentile of the metric in question (the maximum without the highest 5 percent of the samples). max The maximum of the metric in question.

Sysbench OLTP throughput

class="table-responsive">
style="width:100%; " class="easy-table easy-table-cuscosky " > storage threads ro_rw sd mean 95thpct max eXFlash DIMM_4 128 rw 714.09605 5996.5105 7172.0725 7674.87 eXFlash DIMM_4 256 rw 470.95410 6162.4271 6673.0205 7467.99 eXFlash DIMM_8 128 rw 195.57857 7140.5038 7493.4780 7723.13 eXFlash DIMM_8 256 rw 173.51373 6498.1460 6736.1710 7490.95 fio 128 rw 588.14282 1855.4304 2280.2780 7179.95 fio 256 rw 599.88510 2187.5271 2584.1995 7467.13

Going from 4 to 8 eXFlash DIMMs will mostly mean more consistent throughput. The mean throughput is significantly higher in case of 8 DIMMs used, but the 95th percentile and the maximum values are not much different (the difference in standard deviation also shows this). The reason they are not much different is that these benchmark are CPU bound (check CPU idle time table later in this post or the graphs in the whitepaper). The PCI-E flash drive on the other hand can do less than half of the throughput of the eXFlash DIMMs (the most relevant is comparing the 95th percentile value).

Sysbench OLTP response time

class="table-responsive">
style="width:100%; " class="easy-table easy-table-cuscosky " > storage threads ro_rw sd mean 95thpct max eXFlash DIMM_4 128 rw 4.4187784 37.931489 44.2600 64.54 eXFlash DIMM_4 256 rw 9.6642741 90.789317 109.0450 176.45 eXFlash DIMM_8 128 rw 2.1004085 28.796017 32.1600 67.10 eXFlash DIMM_8 256 rw 5.5932572 94.060628 101.6300 121.92 fio 128 rw 51.2343587 138.052150 203.1160 766.11 fio 256 rw 72.9901355 304.851844 392.7660 862.00

The 95th percentile response time for the eXFlash DIMM’s case are less than 1/4 compared to the PCI-E flash device.

CPU idle percentage

class="table-responsive">
style="width:100%; " class="easy-table easy-table-cuscosky " > storage threads ro_rw sd mean 95thpct max eXFlash DIMM_4 128 rw 1.62846674 3.3683857 6.2600 22.18 eXFlash DIMM_4 256 rw 1.06980095 2.2930634 3.9170 26.37 eXFlash DIMM_8 128 rw 0.42987637 0.8553543 1.2900 15.28 eXFlash DIMM_8 256 rw 1.32328435 4.4861795 6.7100 9.40 fio 128 rw 4.21156996 26.1278994 31.5020 55.49 fio 256 rw 5.49489852 19.3123639 27.6715 47.34

The percentage of CPU being idle shows that the performance bottleneck in this benchmark was the CPU in case of eXFlash DIMMs (both with 4 and 8 DIMMs, this is why we didn’t see a substantial throughput difference between the 4 and the 8 DIMM setup). However, for the PCI-E flash, the storage device itself was the bottleneck.

If you are interested in more details, href="http://form.percona.com/Diablo-Benchmarking.html" rel="nofollow">download the free white paper which contains the full analysis of sysbench OLTP and linkbench benchmarks.

The post rel="nofollow" href="http://www.percona.com/blog/2015/01/26/mysql-benchmarks-exflash-dimms/">MySQL benchmarks on eXFlash DIMMs appeared first on rel="nofollow" href="http://www.percona.com/blog">MySQL Performance Blog.

Written by in: benchmarks,eXFlash DIMMs,MySQL,MySQL benchmarks,Peter Boros,Primary,sysbench,Zend Developer |