Personal blog of Yzmir Ramirez

 We’ve seen leaks of a new Office for Mac 2015 suite circulate, but the new version of Outlook for Mac is already here, at least for Office 365 subscribers. The overhauled email client includes a new look and feel, as well as search that extends to archived mail (whether stored online or on corporate servers), better all around performance, push email support and more. Microsoft also… Read More

 Until today, cloud infrastructure company Joyent had raised a total of $120 million in a number of funding rounds, with its 2012 $85 million Series D round being the latest and largest. There aren’t all that many companies that go past the Series D stage, but cloud platform provider Joyent today announced that it has raised an additional $15 million from existing investors Intel… Read More

If you’re following this blog, it’s quite likely you’re already aware of the Percona Live London 2014 conference coming up in just a few days. Just in case, though (you know, if you’re still looking for an excuse to sign up), I wanted to put a spotlight on the tutorial to be delivered by my esteemed colleagues Frédéric Descamps (@lefred) and Kenny Gryp (@gryp), and myself.

The past two years at Percona we’ve been spending a substantial amount of time working with customers taking their first steps into creating Highly Available MySQL environments built on Galera. Percona XtraDB Cluster allows you to get it up and running very fast, but as any weathered “HA” DBA will tell you, building the cluster is only the beginning. (Percona XtraDB Cluster is an open source (free) high-availability and high-scalability solution for MySQL clustering.)

Any cluster technology is likely to introduce a great amount of complexity to your environment, and in our tutorial we want to show you not only how to get started, but also how to avoid many of the operational pitfalls we’ve encountered. Our tutorial, Percona XtraDB Cluster in a nutshell, will be taking place on Monday 3 November and is a full-day (6 hours) session, with an intense hands-on approach.

We’ll be covering a great deal of practical topics, such as:

• Things to keep in mind when migrating an existing environment over to PXC
• How to manage and maintain the cluster, keeping it in good shape
• Load balancing requests across the cluster
• Considerations for deploying PXC in the cloud

Planning on attending? Be sure to come prepared! Given the hands-on approach of the tutorial, make sure you bring your laptop with enough disk space (~20GB) and processing power to run at least 4 small VirtualBox VM’s.

We look forward to seeing you there!

The post Get a handle on your HA at Percona Live London 2014 appeared first on MySQL Performance Blog.

 Andy Rubin, best known for his work on Android at Google, is leaving the company. According to The Wall Street Journal, Rubin will build an incubator for what it describes as companies working with “technology-hardware” products.
Google confirmed the departure to us with a prepared statement from Google CEO Larry Page: “I want to wish Andy all the best with what’s next. Read More

 The promise of the cloud, whether private or public, has always been the potential to bill users or lines of business for specific usage, but it’s proven difficult to do in practice. Today, Cloudyn introduced a new tool called Cost Allocation 360° that attempts bring some clarity to cloud billing, whether it’s in the data center or the public cloud. Cloudyn CEO Sharon… Read More

 Nokia made its name in mobile phones, breaking new ground and holding the title of world’s biggest handset maker for a number of years before the onslaught of competition from OEMs like Samsung and Apple reversed its fortunes. Now that Nokia has sold that business on to Microsoft, the pared-down, enterprise-heavy remaining Nokia is looking for the next big thing in communications.… Read More

With 1.28 billion active users, Facebook MySQL database engineers are active and extremely valuable contributors to the global MySQL community. So naturally they are also active participants of Percona Live MySQL conferences! And next week’s Percona Live London 2014 (Nov. 3-4) is no exception. (Register now and use the promotional code “Facebook” to save £30!)

I spoke with Facebook database engineers Yoshinori “Yoshi” Matsunobu and Shlomo Priymak about their upcoming sessions along with what’s new at Facebook since our last conversation back in April.

---

Tom: Yoshi, last year Facebook deployed MySQL 5.6 on all production environments – what have you and your team learned since doing that? And do you have a few best practices you could share? I realize you’ll be going into detail during your session in London (MySQL 5.6 and WebScaleSQL at Facebook), but maybe a few words on a couple of the bigger ones?

Yoshi: MySQL 5.6 has excellent replication enhancements to use in large-scale deployments. For example, crash safe slave makes it possible to recover without rebuilding a slave instance on server crash. This can greatly minimize slave downtime, especially if your database size is large. There are many other new features such as GTID, multi-threaded slave, streaming mysqlbinlog and we actively use them in production.

For InnoDB, Online DDL is a good example to ease operations. Many MySQL users are doing schema changes by switching masters. This can minimize downtime but requires operational efforts. Online DDL made things much easier.

—

Tom: Facebook is an active and extremely valuable part of the overall MySQL community and ecosystem – what are some of the key features and improvements you’ve contributed in the past year since moving to MySQL 5.6?

Yoshi: For InnoDB, I think online defragmentation and faster full table scan are the most valuable contributions from Facebook in 5.6. I have received very positive feedback about faster InnoDB full table scan (Logical ReadAhead). My colleague Rongrong will speak about something interesting regarding online defragmentation at Percona Live London. For Replication, we have done many optimizations to make GTID and MTS work without pain. Semi-Synchronous mysqlbinlog and backported Loss-Less semisync from MySQL 5.7 are very useful when you use Semi-Synchronous replication.

—

Tom: Shlomo, your sesson, “MySQL Automation at Facebook Scale,” will be of great interest to DBAs at large and growing organizations considering that Facebook has one of the world’s largest MySQL database clusters. What are the two or three most significant things that you’ve learned as a database engineer operating a cluster of this size? And has anything surprised you along the way (so far)?

Shlomo: This is a great question! We like to speak of “10x” at Facebook when thinking of scaling. For example, what would you do differently if the number of servers you had was 10x more than what it is? This type of mental exercise is surprisingly useful when working with systems at scale. If you, or any of the readers, try to extrapolate this about systems you manage, there will be things you’ll be imagining about how a system like this would be – and you won’t be too far from our reality in many aspects.

You’d imagine that we automate much of the single units of work, like master/slave failover, upgrades and schema changes. You’d suspect we have automated fault detection, self managing systems, good alarming and self remediation. You’d presume that if you’re used to running a command on 100 machines, you’ll now be running it on 1000. At least that’s what I thought to myself, so these are not the things that surprised me. There are a few fundamental shifts in one’s thinking when you get to these sizes, which I didn’t foresee.

The first one is that there is absolutely no such thing as “one-off.” If there is a server somewhere that hits a problem every three years, and you have 1000 servers, this will be happening daily! Take it to 10,000 servers, and you can see absolutely nothing is a “one-off”. We can’t write things off as “worst case, I’ll get an SMS.” Whatever it is, we have to chase it down and fix it. Not just that – to deploy a fix at scale can require writing fairly large amounts of code, a fix that could be deployed manually by a DBA in smaller environments.

The second one is adapting to constraints which are very pragmatic and tangible. If you’re on AWS, you’re pretty much isolated from things like worrying where your servers are physically located, when they go over their lifetime, and if the firmware on the switch in the rack needs to be upgraded. If you’re a small shop and have a few racks up in a co-lo, hardware maintenance is just not as frequent, but it becomes more painful as you grow.

At Facebook, we run our own datacenters! We need to work around interesting challenges, such as running datacenters that have highly variable compositions of server hardware. Since we have so many servers, something is always going on. Racks of servers need to be moved. Whole clusters need to be rebuilt or refreshed, to be made better, faster, stronger.  New datacenters are constructed, others decommissioned.

Tom: And this is where automation comes into the picture, right?

Shlomo: We have had to build a lot of automation to make these operations seamless, and we work closely with the Site Ops teams on the ground to coordinate these logistically complicated processes.

Another thing my team does in this space is planning capacity and hardware purchases. Since we build our own servers, the turnaround time between ordering and getting machines is quite long, so proper planning is paramount. Buy too much, and you’ve wasted millions of dollars. Buy too few servers, and there won’t be space for user growth and upcoming projects. The sheer scale makes these decisions more complicated and involved.

These things have actually made my job much more interesting, and I think I’d find it hard to adjust to a smaller environment.

—

Tom: Last April Facebook announced a move to the newly created WebScaleSQL. Yoshi, do you have an update on where WebScaleSQL is today? And I know it’s early, but has there been any impact on Facebook yet?

Yoshi: WebscaleSQL is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale. Collaboration is nothing new to the MySQL community. The intent is to make this collaboration more efficient.

We are based on the latest upstream (currently MySQL-5.6.21), and added many features. We added patches to improve InnoDB performance around compression LRU flushing, locking, NUMA Support, and doublewrite. We statically link Semi-Sync based on lessons learned at Facebook environments (plugin-lock caused hot mutex contentions). We have many upcoming features such as async clients.

We will continue to track the upstream branch that is the latest, production-ready release (currently MySQL 5.6). We are continuing to push the generally useful changes we have from all of the participants.  If you think you have something to contribute, get in touch!

—

Tom: I remember being surprised earlier this year when you told me there was usually just one MySQL Operations team member on call at any given time thanks to “robots.” How many robots did your team build and what do they do? Oh, and should rank-and-file DBAs around the world be worried about losing their day jobs to these robots?

Shlomo: Instead of becoming obsolete as some fear, our team is shifting its focus from smaller to larger problems, as we rise higher in the levels of abstraction. Our team has progressed with the requirements of the role. From being a team of DBAs that automate some of their work, we have become more like Production Engineers. We design, write and maintain MySQL/Facebook-specific automation that does our work for us.

While we build these software “robots” to do our work, we also have to maintain them. The job of the oncall is to fix these robots when they malfunction, and that can sometimes be difficult due to the size of our codebase.

In regards to employment concerns, I’d say our work has become more interesting, and the amount has increased. It definitely did not decrease, so if Facebook is indicative of other companies, jobs are not at risk just yet. Speaking of jobs – if what we’re doing sounds interesting, we’re hiring!

Oh, and as for details about these “robots” – that’s the topic of my talk next week in London. Come and hear me speak if you want to know more!

—

Tom: Yoshi, you also will host a session titled “Fast Master Failover without Data Loss.” I don’t want to give too much away, but how did you get failover to work at scale – across vast datacenters?

Yoshi: Master failure is a norm at Facebook, because of the large amount of servers. Without automation, it is not realistic for a limited number of people to manage. We have a very interesting infrastructure to automate failure handling at Facebook scale. To automate stuff, reliability is important. Unreliable automation makes engineers spend lots of time fixing things manually, and that increases downtime. It is also important to define what to automate and what we shouldn’t automate. Define failure scenarios and write good test cases and continuously integrate. There are multiple failure scenarios like the ones below and you’ll hear about each in detail at my session:

– mysqld crash
– mysqld stalls
– kernel panic and reboot
– error spikes caused by H/W failure
– error spikes caused by bad application logic
– rack switch down
– multiple rack switches down
– datacenter down

—

Tom: What other sessions, keynotes or events are you looking forward to at Percona Live London 2014? And are you guys planning on attending the MySQL Community Dinner?

Yoshi: “MySQL 5.7: Performance and Scalability Benchmark” (led by Oracle MySQL performance architect Dimitri Kravtchuk). And yes, we’re looking forward to meeting with people at MySQL Community Dinner!

—

Tom:  Thanks again Yoshi and Shlomo for taking the time to speak with me and I look forward to seeing you both in London next week!

And readers, I invite you to register now for Percona Live London using the promotional code “Facebook” to save £30. I also hope to see you at the MySQL Community Dinner next Monday (Nov. 3). Space is limited so be sure to reserve your spot now and join us aboard our private double-decker bus to the restaurant.

I’d also like to thank the Percona Live London 2014 Conference Committee for putting together a terrific event this year! The conference committee includes:

• Dailymotion’s Cédric Peintre, conference chairman
• Percona’s David Busby
• MariaDB’s Colin Charles
• ebay Classifieds Group’s Luis Motta Campos
• Booking.com’s Nicolai Plum
• Oracle’s Morgan Tocker
• Spil Games’ Art van Scheppingen

The post Facebook MySQL database engineers ready for Percona Live London 2014 appeared first on MySQL Performance Blog.

 Microsoft took an axe to another slice of its employees today, cutting around 3,000 jobs according to Mary Jo Foley. TechCrunch heard through a separate source that the layoffs were imminent, but was not able to confirm their size. The company released a short statement indicating that it has nearly wrapped “all [of] the 18,000 reductions announced [it] in July.” Today’s… Read More

Gwen Gardner is author of the popular young adult Indigo Eady Paranormal “Cozy” Mystery series that began with Givin’ Up the Ghost. With Halloween approaching, I wanted to mention her latest ghostly book: Secret Keepers.

 

About Secret Keepers:

 In Second Death, Indigo and her team of sleuths escape the clutches of underground evil.

In Ghostly Gallows, Indigo and the gang managed to dodge the falling ruins and free those spirits trapped by the evil spell that bound them to Gallowsgate Prison.

Now in Secret Keepers, even though Indigo speaks to ghosties, there is so much about the afterlife she doesn’t understand. Such as…how can spirits be sick? They’re dead, for goodness sake! But once again Indigo is forced to go where she made it a point to never go: the hospital. There, the mystery of the sick and missing spirits explodes into a frenzied battle for souls where dawn may bring life or final death.

Indigo Eady:

Much to her chagrin, Indigo Eady is a celebrity among spirits. She’s the girl who can see and speak with them. She has a proven track record in helping spirits find their murderers, so it’s no surprise when she’s drafted onto the Missing Paranormal Committee. Who better to help them than the experienced teenage ghost whisperer- investigator and her friends?

---

Gwen explains her Excerpt

In honor of Halloween, I’m giving you a short excerpt from a chapter called Asylum. It’s a haunted hospital after all, and a lot of the book ending takes place there with the final showdown against the demons and dark spirits.

This scene is where Indigo, Badger and Simon are approaching the hospital for the first time. Councilman Diggles has become ill enough to be admitted (yes, he’s a ghost, and yes, apparently ghosts can get sick!), so they are going to brief him on the updates of their investigation into the missing paranormals.

Simon, as usual, is worried about the crazy…

 

We stared up at the gray and dilapidated building. It was almost like the village had shrunken in on itself, with all the abandoned buildings that sprang up around the edges of Sabrina Shores. Or maybe I had just never noticed them before. Stucco and plaster peeled and fell off in chunks. Four stories high, what was still standing, and every window broken, the jagged edges like monsters ready to take a bite.

I shivered.

Why did I always seem to be hanging out in broken down buildings lately? Look at the bright side, Eady. It’s better than being underground.

We approached the door. Oval shaped and shored up with metal strips riveted to the heavy wood, probably oak. The strips were once painted black according to the specks remaining. Now it was rusted and the door splintered.

Next to the door a sign hung sideways by one nail, squeaking slightly in the breeze. Simon lifted the loose side and read. “Belgrave Asylum.” He turned to me. “Oh great. The crazy potential just kicked up a notch. As if we didn’t have enough already, now we’re expected to enter the insane asylum?”

“Short answer?” I asked. “Yes.” I pushed on the door and it creaked open relatively easy. The boys and I stood just inside, our shadows standing before us while we waited for our eyes to adjust to the dark.

---

GWEN GARDNER

Gwen Gardner is a native Californian living in sunny San Diego, where her love of reading and writing led to a BA in English literature. Life is now complete with her husband, two dogs and a daily call from her daughter.

Since ghosts feature prominently in her young adult Indigo Eady Paranormal “Cozy” Mystery series, she has a secret desire to meet one face to face – but will run screaming for the hills if she ever does. Gwen adores travel and experiencing the cultures and foods of different countries. She is always up for an adventure and anything involving chocolate – not necessarily in that order.

Connect with Gwen here: Blog / Twitter / Facebook / Pinterest / Amazon / B&N

 Secret Keepers available here: Amazon / B&N / Google Play / iTunes

 Twitter and IBM announced a significant partnership today that will involve Twitter sharing its data with IBM for integration into IBM’s enterprise solutions, including the Watson cloud platform. The deal means IBM will gain access to the Twitter “firehose,” allowing businesses to incorporate insights gained from the social network into their decision-making… Read More