Dec
15
2014
--

MySQL Tutorials: A time to learn at Percona Live 2015

The many hours of intensive tutorials, led by some of the top minds in MySQL, have always been a major draw each year to the Percona Live MySQL Conference and Expo. And 2015’s event will be no exception.

Percona Live 2015 runs April 13-16 in Santa Clara, Calif. and the first day is dedicated to the classroom – so bring your laptops for the combined 45 hours of learning. MySQL tutorials are included with the full-conference pass but a “tutorial-only pass” is also available. Super-saver registration discounts have been extended until Dec. 19. Here’s a look at this year’s tutorials lineup. (There will be a couple more announced in January).


And that’s just on Monday! There will be much more over the four days of the Percona Live MySQL Conference and Expo 2015. I posted a sneak peek of the full Percona Live (initial) roster a couple weeks ago. And remember, super-saver registration discounts have been extended until Dec. 19 so register now – and don’t forgot your laptop (and power cord)!

The post MySQL Tutorials: A time to learn at Percona Live 2015 appeared first on MySQL Performance Blog.

Dec
10
2014
--

Recover MySQL root password without restarting MySQL (no downtime!)

Disclaimer: Do this at your own risk! It doesn’t apply if you’re using Pluggable authentication and certainly won’t be usable if/when MySQL system tables are stored on InnoDB

Recover your root password with care!

Recover your root password with care!

What is the situation?

The situation is the classic “need to recover MySQL root password” but you cannot restart MySQL (because it is the master production server, or any other reason), which makes the –skip-grant-tables solution as a no-no possibility.

 What can I do?

There is a workaround, which is the following:

  •  Launch another instance of mysqld, a small one (without innodb).
  •  Copy your user.[frm|MYD|MYI] files from the original datadir to the datadir of the new instance.
  • Modify them and then copy them back to the original location.

That simple? No, but close. Here is the step by step:

Step by step recovery

  1. Create a new datadir and run mysql_install_db for the new datadir. This one will be removed at the end. Don’t forget to change ownership to mysql user and group:
    [root@machina dbdata]# mkdir datadir
    [root@machina dbdata]# chown -R mysql:mysql datadir/
    [root@machina dbdata]# mysql_install_db --datadir=/dbdata/datadir/ --user=mysql
    Installing MySQL system tables...OK
    Filling help tables...OK
  2. Launch the new instance. Be careful with the datadir path, the socket file and the port number. Also, disable InnoDB, you won’t need it, just add –skip-innodb AND –default-storage-engine=myisam:
    [root@machina datadir]# /usr/sbin/mysqld --basedir=/usr --datadir=/dbdata/datadir --plugin-dir=/usr/lib/mysql/plugin --skip-innodb --default-storage-engine=myisam --socket=/var/run/mysqld/mysql2.sock --port=3307 --user=mysql --log-error=/dblogs/log/error2.log --pid-file=/dbdata/data/mysql.pid &
  3. Copy the user.* files from the original mysql instance (the ones that you need to modify) to the new instance’s datadir and login to this instance of mysql:
    [root@machina ~]# cp /dbdata/data/mysql/user.* /dbdata/datadir/mysql/cp: overwrite `/dbdata/datadir/mysql/user.frm'? y
    cp: overwrite `/dbdata/datadir/mysql/user.MYD'? y
    cp: overwrite `/dbdata/datadir/mysql/user.MYI'? y
    [root@machina datadir]# mysql --socket=/var/run/mysqld/mysql2.sock -p
    Enter password:
    Welcome to the MySQL monitor.  Commands end with ; or g.
  4. Execute a “flush tables” command, so the user table will be “reopened” and you can see the data and verify:
    mysql2> flush tables;
    mysql2> select user, host, password from user where user like 'root';
    +------+--------------------------------------+------------------------------------------+
    | user | host                                 | password                                 |
    +------+--------------------------------------+------------------------------------------+
    | root | localhost                            | 696D727429CC43695423FA5F2F0155D92A0AAC08 |
    | root | 127.0.0.1                            | 696D727429CC43695423FA5F2F0155D92A0AAC08 |
    | root | %                                    | 696D727429CC43695423FA5F2F0155D92A0AAC08 |
    +------+--------------------------------------+------------------------------------------+
    3 rows in set (0.00 sec)
  5. Now, update the password field with the desired value:
    mysql2> update mysql.user set password='*696D727429CC43695423FA5F2F0155D92A0AAC08' where user like 'root';
    Query OK, 3 rows affected (0.00 sec)
    Rows matched: 3  Changed: 3  Warnings: 0
  6. Verify again:
    mysql2> select user, host, password from user where user like 'root';
    +------+--------------------------------------+-------------------------------------------+
    | user | host                                 | password                                  |
    +------+--------------------------------------+-------------------------------------------+
    | root | localhost                            | *696D727429CC43695423FA5F2F0155D92A0AAC08 |
    | root | 127.0.0.1                            | *696D727429CC43695423FA5F2F0155D92A0AAC08 |
    | root | %                                    | *696D727429CC43695423FA5F2F0155D92A0AAC08 |
    +------+--------------------------------------+-------------------------------------------+
    3 rows in set (0.00 sec)
  7. Flush privileges and verify that the new password is correct, by logging in again:
    mysql2> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
  8. Now that we have made the changes, we can move back the user.* files to the original location, being extremely careful with owner and privileges:
    [root@machina ~]# cd /dbdata/datadir/mysql/
    [root@machina mysql]# cp user.* /dbdata/data/mysql/; chown mysql:mysql /dbdata/data/mysql/user.*; chmod 660 /dbdata/data/mysql/user.*
    cp: overwrite `/dbdata/data/mysql/user.frm'? y
    cp: overwrite `/dbdata/data/mysql/user.MYD'? y
    cp: overwrite `/dbdata/data/mysql/user.MYI'? y
  9. At this moment, you can shutdown the new mysql instance since is no longer needed. Be very very careful so you don’t end up shutting down your original mysqld!:
    [root@machina datadir]# mysqladmin --socket=/var/run/mysqld/mysql2.sock -p shutdown
    Enter password:
    141120 06:59:14 mysqld_safe mysqld from pid file /dbdata/data/mysql.pid ended
  10. Now, the last step is to execute a “FLUSH PRIVILEGES” in the original mysqld. Since we cannot yet access it, we need to send a SIGHUP signal to mysqld. MySQL responds to this signal by reloading the grant tables and flushing tables, logs, the thread cache, and the host cache, so choose wisely the moment of the day when you want to send the SIGHUP since the performance might be degraded (look at “flush tables” ).The way to send SIGHUP is to execute “kill” command with the -1 flag:
    [root@machina datadir]# kill -1 $(/sbin/pidof mysqld)
  11. Finally, login into MySQL as root!:
    [root@machina datadir]# mysql -p
    Enter password:
    Welcome to the MySQL monitor.  Commands end with ; or g.
    Your MySQL connection id is 101208
    mysql1> select user, host, password from mysql.user where user like 'root';
    +------+--------------------------------------+-------------------------------------------+
    | user | host                                 | password                                  |
    +------+--------------------------------------+-------------------------------------------+
    | root | localhost                            | *696D727429CC43695423FA5F2F0155D92A0AAC08 |
    | root | 127.0.0.1                            | *696D727429CC43695423FA5F2F0155D92A0AAC08 |
    | root | %                                    | *696D727429CC43695423FA5F2F0155D92A0AAC08 |
    +------+--------------------------------------+-------------------------------------------+
    3 rows in set (0.00 sec)


    You can see your schemas? of course you can! your databases are okay!

    mysql1> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | mysql              |
    | percona            |
    | testing            |
    +--------------------+
    4 rows in set (0.03 sec)

We’ve successfully recovered the MySQL root password without the need to restart MySQL and thus avoid downtime.

I hope you never face this situation, but in case you do, there’s a workaround to recover your access! Is there another way to perform this?

Share it with the world!

The post Recover MySQL root password without restarting MySQL (no downtime!) appeared first on MySQL Performance Blog.

Nov
12
2014
--

Log rotate and the (deleted) MySQL log file mystery

Did your logging stop working after you set up logrotate? Then this post might be for you.

Archive

Archive your log files!

Some time ago, Peter Boros wrote about Rotating MySQL Slow Logs safely, explaining the steps of a “best practice” log rotate/archive. This post will add more info about the topic.

When running logrotate for MySQL (after proper setting the /etc/logrotate.d/mysql conf file) from anacron, there’s a situation that you might potentially face if the user and password used to execute the “flush logs” command is stored in, for example, /root/.my.cnf file.

The situation:

You might find out that you have a new MySQL log file ready to receive data, but nothing is being written to it.

Why did this happen?

The logrotate script is executed, but the postrotate fails to successfully flush logs. If this happened to you, you might think, “I’ve lost my slow log file!” The good news: You didn’t lose it. What just happened is that your MySQL log file is no longer visible from the filesystem perspective, but the file still exists and is still receiving data.

So where is it? How can I find it again?

Through the file descriptor. If your mysqld still running, you can find your log under /proc/[pid of mysqld process]/fd path:

[root@hostname]# cd /proc/$(/sbin/pidof mysqld)/fd
[root@hostname fd]# ls -lh | grep deleted
lrwx------ 1 root root 64 Oct 21 11:39 131 -> /tmp/MLQKbznR (deleted)
lrwx------ 1 root root 64 Oct 21 11:39 26 -> /tmp/ib95UPJ8 (deleted)
lrwx------ 1 root root 64 Oct 21 11:39 5 -> /tmp/ib9nYywT (deleted)
lrwx------ 1 root root 64 Oct 21 11:39 501 -> /var/log/mysql/log-slow-queries.log.1 (deleted)

And how big is it? lsof can give us the answer with the file descriptor number, which for this example is 501:

[root@hostname fd]# /usr/sbin/lsof -p $(/sbin/pidof mysqld) -ad 501
COMMAND  PID  USER   FD   TYPE DEVICE  SIZE/OFF     NODE NAME
mysqld  2813 mysql  501u   REG  253,0 976746174 70516762 /var/log/mysql/log-slow-queries.log.1 (deleted)

The output of lsof tell us that this file size is 976746174 bytes, which is 931MB aprox.

Can I recover the file contents?

Yes, you can. You just need to use the “cat” command and knowing the File Descriptor number. In this case, is 501:

cat /proc/$(/sbin/pidof mysqld)/fd/501 > /path/to/new/logfile.log

Remember that once you execute a success “flush logs” commands on the MySQL client, the old contents will disappear, so do this prior any further log rotation.

How did this happen?

Let’s examine the logrotate script:

/var/log/mysql/log-slow-queries.log {       
   create 600 mysql mysql       
   daily       
   rotate 3       
   missingok       
   compress       
   sharedscripts   
   postrotate       
      if test -x /usr/bin/mysqladmin &&  
           /usr/bin/mysqladmin ping &>/dev/null       
      then
           /usr/bin/mysql -e 'select @@global.long_query_time into @lqt_save; set global long_query_time=2000; set global slow_query_log = 0; select sleep(2); FLUSH LOGS; select sleep(2); set global long_query_time=@lqt_save; set global slow_query_log = 1;' > /var/log/mysqladmin.flush-logs 2>&1
        fi   
   endscript
}

Everything seems okay, except for one thing: When executing from cron, the HOME term environment variable will be blank. Meaning: /usr/bin/mysql won’t be able to find the file with the access credentials (user and password) and thus cannot execute the “flush logs” command.

What is the solution?

Add the HOME variable to the postscript line: env HOME=/root/

env HOME=/root/ /usr/bin/mysql -e 'select @@global.long_query_time into @lqt_save; set global long_query_time=2000; set global slow_query_log = 0; select sleep(2); FLUSH LOGS; select sleep(2); set global long_query_time=@lqt_save; set global slow_query_log = 1;'  > /var/log/mysqladmin.flush-logs 2>&1

Can I get an alert if this happens to me?

Yes! With the Percona Nagios Plugin pmp-check-mysql-deleted-files. The Percona Nagios Plugin, which like all Percona software is free, looks at the files that the mysqld process has open and warns if any of them are deleted that shouldn’t be. For example: a slow MySQL log file that has being deleted by a poorly written logrotate script. (Download the Percona Nagios Plugin here)

In conclusion: Don’t fall into a situation where you suddenly realize, to your horror, that you’ve lost your slow MySQL log file. And if you do, relax: Recover your MySQL log file contents and add the proper monitoring alert. Problem solved!

The post Log rotate and the (deleted) MySQL log file mystery appeared first on MySQL Performance Blog.

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com