What a n00b! (mysql)

MySQL Secure Installation Using Ansible

Wyatt — Sat, 23 Jan 2016 03:56:28 GMT

Recently I was setting up MySQL using Ansible, and wanted to ensure the mysql_secure_installation script or an equivalent was run to get rid of the default users and db. Turned out that writing a task in Ansible wasn't all that bad using the check_implicit_admin feature of the MySQL plugins, it could even be idempotent.

- name: delete anonymous MySQL server user for {{ ansible_nodename }} mysql_user: login_user=root login_password='{{ mysql_root }}' check_implicit_admin=yes user="" host={{ item }} state="absent" with_items: - "" - "{{ ansible_nodename }}" - localhost - name: Change root user password on first run mysql_user: login_user=root login_password="{{ mysql_root }}" check_implicit_admin=yes name=root password={{ mysql_root }} priv=*.*:ALL,GRANT host={{ item }} with_items: - "{{ ansible_nodename }}" - 127.0.0.1 - ::1 - localhost - name: remove the MySQL test database action: mysql_db login_user=root login_password="{{ mysql_root }}" db=test state=absent

MySQL Memory Usage Growth After Lots of Table Creates/Drops

Wyatt — Tue, 18 Sep 2012 02:23:22 GMT

I should clarify somewhere here that "Lots" in the title means something on the order of half a million table creations and drops per week. This is pretty unique to our standard usage of MySQL, and I would assert unique for most.

We've been having troubles for a while now with a cluster of MySQL nodes which consume a never-ending amount of memory over time until it eventually runs out, forcing us to restart MySQL if we catch it in time, or reboot the box if we don't. After doing tons of searches (and looking through countless bug reports), it seemed like it could have been a few memory leaks, fixed in late 5.1 or early 5.5 land. As this cluster was slightly older and not updated regularly, we pushed things over to new nodes and up to a modern version of 5.5. However, this didn't seem to solve the problem. Every week or so, we'd fail over to a secondary node and restart things. The biggest hint towards it being caused by all the creates is that the "leak" flowed through replication and the secondary nodes would run out within several hours of the primary.

As it turns out, the answer was quite simple. According to the Percona MySQL Performance Blog, the MySQL dictionary cache memory is allocated with each opened table, and never given back - without any limit. This isn't so bad I suppose, except if you're opening something like half a million tables per week and you just so happen to have a machine with a finite amount of memory. Then you might have a bit of trouble.

The fix is bad news if you're a MySQL community-build user. There's no way to limit it without switching to Percona Server (which I highly recommend over the stock MySQL builds, anyway). If you have installed Percona Server in favor of MySQL you can set innodb_dict_size_limit to limit the growth. This is '0' by default which mimics current stock MySQL. Note that this is a soft limit, so if you still have that many tables open you'll just need moar memories.

I should also note that while this setting says it's dynamic, it's a danger zone. I set this to a not-terribly-aggressive-yet-still-lower-than-current-dictionary-size value on the non-active node and everything came to a screeching halt with this in the error log:

--Thread 1264216384 has waited at dict0dict.c line 744 for 688.00 seconds the semaphore:

Enabling InnoDB Compression on MySQL 5.5

Wyatt — Fri, 31 Aug 2012 02:04:15 GMT

I got a quick little lesson this morning in a few MySQL internals while trying to enable InnoDB compression. On the surface, it seems pretty easy. The MySQL documentation just says you can do:

ALTER TABLE table ROW_FORMAT=COMPRESSED;

However, for me this did nothing. The size of the table didn't change at all. After a bit of digging, I figured out that you have to set the default file format for InnoDB to Barracuda which has compression enabled. This is disabled by default, according to the documentation:

Although Oracle recommends using the Barracuda format for new tables where practical, in MySQL 5.5 the default file format is still Antelope, for maximum compatibility with replication configurations containing different MySQL releases.

Pretty sneaky, but easy enough to fixup:

set global variable innodb_file_format = Barracuda;

add it to your my.cnf (/etc/my.cnf on most distros):

... innodb_file_format = Barracuda ...

After that, the ALTER TABLE converted the table and cut the size on disk in half. Very nice.

Since this table is rarely read, and even more rarely written to, I didn't end up doing much by way of performance testing or tuning. However, I should also note blog entry I found that seems to indicate that performance degrades as the size of the buffer pool increases when compression is enabled. The workaround is to use multiple buffer pools. Fortunately, we're already did this as a workaround for issues with slow DROP TABLE performance on older versions of 5.5 when innodb_file_per_table is enabled.

Percona Live MySQL Conference 2012 - day 1

Wyatt — Thu, 12 Apr 2012 05:14:15 GMT

This week I am fortunate enough to be sent to the Percona Live Conference by my company. I've been sitting here tonight, pondering and recalling sessions for the day; trying to think of all those things that I learned I'm was doing wrong or better tools I could be using. And, since I don't blog enough lately, here we go!

Nothing earth-shattering so far, but some takeaways from sessions I hit up today:

Building a Multi-Master, Mult-Region Database Infrastructure in Amazon EC2

When I first realized this was going to be just a demo of some commercial software (Tungsten Enterprise), not a "lessons-learned" type of session I was pretty disappointed (I didn't read the program carefully enough!). As the speaker got deeper into the talk, though, I became more and more impressed with the technology. Continuent has built some interesting technology, letting you not only failover (and failback!) individual nodes easily, but also entire clusters. They also have some pretty hot backup/restore/replication functionality as well.

One to Many: The Story of Sharding at Box

Definitely the most engaging session of the day for me, personally. The session was a pretty open story about Box's migration from a single database architecture into a sharded architecture (which seems pretty new for them). There were quite a few gotcha's along the way, especially on the application side. One of the more relevant pieces for me was in their clever use of the Tungsten Replicator to move databases between clusters in a fairly ad-hoc fashion.

The 5 Minute DBA: MySQL DBA 101 for Non-DBA's

While I wouldn't exactly call myself a "Non-DBA".. I know enough to be break things worse than they already are, and I am certainly not a full-time DBA. This was a great session a great overview of the basics (needed occasionally), and a solid overview of some of the tools available in the Percona Toolkit that I know I should be using but currently am not. For shame.

Backing up Facebook

I have to be brutally honest and say attending this session was easily the biggest letdown of the day. I was hoping for more from Facebook on this one. We spent about 25 minutes listening to them talk about how they use mysqldump every day on every database server.. in a non-consistency-ensuring way. And when a pointed question was asked about what would happen when they would need to recover a host server to a point in time? The question was basically brushed off saying they'd use some other mechanism to restore (presumably Xtrabackup or something similar).. I thought that's what we were here for? Whatever. I did have some interesting hallway conversation with a couple of fine folks about how perhaps there's some interestingness happening on the database that they're not wanting to talk about that solves this problem for them. That very well could be (and probably is) the right answer, but then why are they at a conference giving a session on doing backups? Ugh. Again.. whatever.

Scaling MySQL Databases for the Web

This was another title that decieved me ('fool me twice', or something like that..). It was less about scaling MySQL and more about a tool developed in-house at YouTube that currently front-ends all MySQL traffic there, doing some intelligent caching and optimization.

One more thing

I usually don't enjoy or look forward to walking around in the exhibit halls at conferences like this, but I did talk with one very interesting vendor that I hadn't heard of before. NuoDB seems to be working on a pretty interesting product that's just about ready to release. I don't really have the time to dive into details about how it supposedly works, but they seem to understand and handle the scaling-back part of elasticity better than most other database vendors I've talked with. They also have a replication strategy that's similar to the way bittorrent works which is a pretty awesome idea. There seems to be nothing open (as in source) about the product, except they want others to write various language bindings for them, so we'll see how well this thing takes off. Definitely an interesting product to watch, though.