Oct
21
2016
--

Percona Responds to East Coast DDoS Attack

ddos

ddosAs noted in several media outlets, many web sites have been affected by a DDoS attack on Dyn today. Since Percona uses Dyn for its DNS server, we are experiencing issues as well.  

The attack has impacted the percona.com web site availability and performance, including all related services such as our forums, blogs and downloads.

Our first response was to wait it out, and trust the Dyn team to deal with the attack — they have to handle issues like this all the time, and are generally pretty good at resolving these issues quickly. This was not the case today.

As such, to restore service, we have added another DNS provider (DNS Made Easy). This has restored connectivity for the majority of users, and the situation should continue to improve as the changed list of DNS servers propagates (check current status).

Our customer support site, Zendesk, has also been impacted by the DDoS attack. We are using a similar strategy to remedy our support site. You can see the current status for Zendesk’s domain resolution here.

For additional information about this incident from Dyn check out official Dyn incident status page

If you’re a Percona customer and have trouble accessing our Customer Support portal, do not hesitate to call or Skype us instead.

Thank you for your patience. We will provide updates as the situation develops.

Aug
23
2016
--

How to stop offending queries with ProxySQL

halt_man

ProxySQLThis blog discusses how to find and address badly written queries using ProxySQL.

All of us are very good in writing good queries. We know this to always be true! ?

But sometimes a bad query escapes our control and hits our database. There is the new guy, the probie, who just joined the company and is writing all his code using SELECT * instead of WHERE. We’ve told him “STOP” millions of times, but he refuses to listen. Or a there is a new code injection, and it will take developers some time to fix and isolate the part of the code that is sending killing queries to our database.

The above are true stories; things that happen every day in at least few environments.

Isolating the bad query isn’t the main problem: that is something that we can do very fast. The issue is identifying the code that is generating the query, and disabling that code without killing the whole application.

That part can take days.

ProxySQL allows us to act fast and stop any offending query in seconds. I will show you how.

Let us say our offending query does this:

SELECT * from history;

Where history is a table of two Tb partitioned by year in our DWH.

That query will definitely create some issue on the database. It’s easy to identify this query as badly designed.

Unfortunately, it was inserted in the ETL process that uses a multi-thread approach and auto-recovery. Now when you kill it, the process restarts it. After, it takes developers some time to stop that code. In the meantime, your reporting system serving your company in real-time is so slooow (or down).

With ProxySQL, you can stop that query in one second:

INSERT INTO mysql_query_rules (rule_id, active, match_pattern, error_msg, apply) VALUES (89,1,'^SELECT \* from history$','Query not allowed',1);
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

Done, your database never receives that query again! Now the application gets a message saying that the query is not allowed.

And look, it’s possible to do things even better:

INSERT INTO mysql_query_rules (rule_id, active, match_digest, flagOUT, apply) VALUES (89,1,'^SELECT \* FROM history', 100, 0);
INSERT INTO mysql_query_rules (rule_id, active, flagIN, match_digest, destination_hostgroup, apply) VALUES (1001,1, 100, 'WHERE', 502, 1);
INSERT INTO mysql_query_rules (rule_id, active, flagIN, error_msg, apply) VALUES (1002,1, 100, 'Query not allowed', 1);
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

In this case, ProxySQL checks for any query having SELECT * FROM history. If the query has a WHERE clause, then it redirects it to the server for execution. If the query does not have a WHERE it stops the query and sends an error message to the application.

Conclusion

This is a very basic example of offending query. But I think it makes clear how ProxySQL helps any DBA in stopping them quickly in the case of an emergency.
This gives the DBAs and the developers time to coordinate a better plan of action to permanently fix the issue.

References

https://github.com/sysown/proxysql
http://www.proxysql.com/2015/09/proxysql-tutorial-setup-in-mysql.html
https://github.com/sysown/proxysql/blob/v1.2.2/doc/configuration_howto.md
https://github.com/sysown/proxysql/blob/v1.2.2/INSTALL.md

Powered by WordPress | Theme: Aeros 2.0 by TheBuckmaker.com