While there is no magic bullet for MySQL 5.7 database tuning, there are a few areas that can be focused on upfront that can dramatically improve the performance of your MySQL installation. While much information has been published on this topic over the years, I wanted to break down some of the more critical settings that anyone can implement – with no guesswork required.

Depending on the version of MySQL you are running, some of the default values used in this post may differ from your install, but the premise is still largely the same. We will focus on MySQL 5.7 for the purposes of this article.

Initial MySQL performance tuning can be broken down to the following categories:

• Tuning for your hardware
• Tuning for best performance/best practices
• Tuning for your workload

MySQL 5.7 Database Tuning

Tuning For Your Hardware

Depending on the hardware you have installed MySQL on, some variables need to be set based on the machine (or VM) specifications. The following variables are largely dependent on your hardware:

innodb_buffer_pool_size

• Generally, set to 50% – 70% of your total RAM as a starting point.
• It does not need to be set any larger than the total database size.

innodb_log_file_size

• This is generally set between 128M – 2G.
• It does not need to be any larger than the buffer pool.

innodb_flush_log_at_trx_commit

• Setting to “1” (default in 5.7) gives the most durability.
• Setting to “0” or “2” will give more performance, but less durability.

innodb_flush_method

• Setting this to O_DIRECT will avoid the performance penalty from double buffering.

Tuning For Best Performance / Best Practices

innodb_file_per_table

• Setting this to “ON” will generate an independent InnoDB table space for every table in the database.

innodb_stats_on_metadata

• Setting this to “OFF” avoids unnecessary updating of InnoDB statistics and can greatly improve read speeds.

innodb_buffer_pool_instances

• A best practice is to set this to “8” unless the buffer pool size is < 1G, in which case set to “1”.

query_cache_type & query_cache_size

• Setting both of these to “0” will entirely disable the query cache.

Tuning For Your Workload

To tune further, more information will be required. The best way to gather this information is to install a MySQL monitoring/graphing tool like Percona Monitoring and Management (PMM). Once you have a tool installed, we can dive into the individual metrics and start customizing based on the data.

I would recommend starting with one of the most impactful variables – the innodb_buffer_pool_size.  Compare the RAM and number of free pages on your instance to the total buffer pool size. Based on these metrics, you can determine if you need to increase or decrease your overall buffer pool size setting.

Next, take a look at your metrics for the InnoDB Log File usage. The rule of thumb is that your log files should hold approximately one hour of data. If you see that the data written to the log files hourly exceeds the total size of the log files, you would want to increase the innodb_log_file_size variable and restart MySQL. You could also verify with “SHOW ENGINE INNODB STATUS;” via the MySQL CLI to assist in calculating a good InnoDB log file size.

In Summary

While this is by no means a comprehensive article on MySQL 5.7 database tuning, the suggestions above should clear some of the low hanging fruit and get your system closer to an ideal setup. As with all database tuning, your process should be an ongoing one based on current information.

• Examine the settings proposed above, and implement if they make sense for your environment/workload.
• Install a good monitoring tool to give insight into the database (Percona Monitoring and Management is our suggestion).
• Stay current on your monitoring graphs to determine other areas where you may need to tune.

Recently I was looking for some simple way to analyze and create graphs from data stored in MySQL tables. I know Grafana can also plot graphs from MySQL tables, but somehow it felt like overkill and you are not using the traditional SQL syntax there. So I was searching for another solution, and it actually took me a while to find it. I might have used the wrong keywords when searching, or the SEO is not the best for this site/project, but in the end, I found Metabase.

In this blog post, I would like to highlight this open source project because I found it very simple and it does the job perfectly for what I wanted it for.

What Is Metabase?

I would describe Metabase as a tool that can provide you a graphical interface to create business intelligence and analytics graphs in minutes. It does not require any previous SQL knowledge, and it has a nice interface where you can simply make analytics with just a few clicks.

Is It Only for MySQL?

It supports a lot of backend databases:

• MySQL
• Postgres
• Druid
• SQL Server
• Redshift
• MongoDB
• Google BigQuery
• SQLite
• H2
• Oracle
• Vertica
• Presto
• Snowflake
• SparkSQL

Installation

It is very simple; you can just call the JAR file or run a docker container and it is done. You can run on AWS Elastic Beanstalk, or on Heroku, or even on Kubernetes.

After installation, you only have to add your datasource and you are good to go.

Creating a Graph

Using a test IMDB database, we simply graph how many movies were created per year. It takes only a few seconds and a few clicks. Of course, we can write a query by ourselves as well, and it is able to plot the result without any problems.

It Has a Lot of Features

Metabase has a lot of features and settings, but here I will just mention a few:

• Work with large databases and configure when and how they should scan the data.
• It can handle multiple users.
• It can handle permissions.
• It works with Slack and you can create questions/graphs from Slack.
• You can create dashboards where you can add your graphs.
• You can share dashboards and graphs.
• Activity log
• Alerts

Conclusion

If you are looking for a very simple data visualization or business analytics tool, I would definitely recommend taking a look at Metabase. Please let us know what you think if you are already using it!

As discussed in part one, this series is meant to go over some of the biggest differences you will likely encounter when moving from on-premise to the cloud.  In this portion, I’ll look at the architectural and operational differences that are commonly seen.

Operational and Architectural Differences

One of the most common issues I see revolves around clients attempting to achieve a 1:1 mapping of their current environment in AWS.  While a carbon copy would seem like the easiest approach, depending on the current on-prem solution, it can actually be much more challenging.  Rather than heavily modifying an existing infrastructure (think scripts, automation, etc.), it is often much easier (and cleaner) to leverage the tools already in AWS.

High Availability

As mentioned earlier, a common approach to achieve HA is to leverage floating (or virtual) IPs and/or some form of a load balancer.  With the differences in networking, using an IP-based solution is generally very challenging. This complicates tools such as Orchestrator or keepalived that are common in on-premise deployments.

A more “AWS friendly” approach for HA would be to leverage an internal network Elastic Load Balancer (ELB) in conjunction with a layer 7 tool such as ProxySQL.  As an ELB is already a highly available component within AWS, you can create a target group of software load balancers that can intelligently route requests to the appropriate database server.  This gives your application a highly available endpoint with multiple layers of redundancy behind it and follows the AWS best practices. Here is a quick example of how this paradigm shift may look:

Backups/Disaster Recovery

Looking at an on-premise deployment, backups and DR generally follow this standard pattern:

• Backups written either locally (and moved) or directly to some network-attached storage
• DR achieved by replicating to another physical data center

In terms of backups, there are several different approaches that can be taken.  This includes both the process and storage options available. Some options include:

• Writing to an Elastic Block Storage (EBS) volume and detaching it or taking a snapshot
• Writing to an Elastic File Store (EFS) mount point attached to the instance (NFS equivalent)
• Writing to an EBS volume and moving to S3 (flexibility in storage tier, retention, etc)

How you manage backups is highly dependent on existing processes.  In the case where you are actively restoring backups for other environments, a portable EBS volume or EFS mount may be a good option.  If you hope to never use your backups, then S3 (with an automatic rotation to Glacier) may be a more cost-effective alternative.

In AWS, DR is generally similar in that there is replication to another location.  However, the big difference here is that you don’t need to provision another data center, as this is managed through regions. Simply spin up an instance in a different region, start replication, and you have a national or global DR strategy in place.

In what has been a common theme, the alternatives you have allow you to tailor your processes to existing AWS components are vast.  Choosing a native AWS component or best practice will generally be a much better (and often cost-effective) option than trying to mirror an on-premise deployment.

Automation/Orchestration

This is a segment that is often fairly different and requires some significant changes.  While AWS offers tools like OpsWorks for Chef Automate, sometimes that may not be the best fit.  Much of the automation will require the use of the AWS CLI or API. Launching instances, registering them with ELBs (Elastic Load Balancer), creating and attaching EBS volumes: all of these actions can be done programmatically.  However, if you are already dealing with a large on-premise deployment, much of this will need to be re-written for AWS.

There are some tools such as Auto Scaling Groups that can help when combined with AMI images of your servers.  For intermediate layers (ProxySQL, for example), you can ensure two or more instances (for HA) using simple Auto Scaling Groups.  The actual database layer is generally more complicated, but certainly not impossible.

Summary

In conclusion, it is best to say that architecture in AWS will probably look somewhat different than an on-premise deployment.  While this could spell additional work up front, it is best to embrace the differences and design your system for the cloud. Even though it is sometimes possible to directly mimic an existing infrastructure, you will likely end up with a fragile deployment that gives more headaches in the long run.  Different doesn’t mean “worse”. Rather, it just means that you have a clean slate and the opportunity to leverage all the benefits of the cloud!

If you or your team is preparing for such a move, don’t hesitate to reach out to Percona and let us help guide you from start to finish!

In this blog, we will test a new security feature added in ProxySQL 2.0.9. Since a time ago, we have had the ability to block queries using mysql_query_rules table matching a group of queries using reg exp like a blacklist. Check out a previous blog for how to config “ProxySQL Firewalling” using the mysql_query_rules table.

You can improve a whitelist using the mysql_query_rules table, but it is difficult if you have hundreds of queries.

ProxySQL 2.0.9 introduces two new tables for the firewall whitelist algorithm:

Admin> SELECT name AS tables FROM main.sqlite_master WHERE type='table' AND name IN ('mysql_firewall_whitelist_rules','mysql_firewall_whitelist_users') ORDER BY name;
+--------------------------------+
| tables                         |
+--------------------------------+
| mysql_firewall_whitelist_rules |
| mysql_firewall_whitelist_users |
+--------------------------------+
2 rows in set (0.00 sec)

How Does It Work?

If you have the param mysql-query_digests enabled, you have all your queries logged in the stats.stats_mysql_query_digest table. The idea is to collect as much traffic as possible to identify normal traffic.

Example

Let’s gets started, so I’ll show you how to configure it using a little example to simulate your traffic:

set mysql-query_digests = 1;
set admin-stats_mysql_query_digest_to_disk = 1;

LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

Now we will prepare some databases and tables, and we are going to insert some data to log that traffic into the stats.stats_mysql_query_digest table. Remember, we need to connect through ProxySQL.

mysql -u user1 -p123456 -h 127.0.0.1 -P 6033

create database db_firewall_test1;
create database db_firewall_test2;

use db_firewall_test1
create table table1 (id int auto_increment, col1 varchar(100), col2 varchar(100), primary key(id));
insert into table1 values (1, 'aa', 'bb');
insert into table1 values (2, 'cc', 'dd');
select * from table1;

use db_firewall_test2
create table table2 (id int auto_increment, col1 varchar(100), col2 varchar(100), primary key(id));
insert into table2 values (1, 'ee', 'ff');
insert into table2 values (2, 'gg', 'hh');
select * from table2;

After that, I recommend checking if those queries are in the stats_history.history_mysql_query_digest table and you should see something like this for the MySQL user “user1”:

admin ((none))>select distinct(digest), username, digest_text from stats_history.history_mysql_query_digest WHERE username = 'user1' ORDER BY dump_time;
+--------------------+----------+------------------------------------------------------------------------------------------------+
| digest             | username | digest_text                                                                                    |
+--------------------+----------+------------------------------------------------------------------------------------------------+
| 0x02033E45904D3DF0 | user1    | show databases                                                                                 |
| 0x0C8B490E5847BD8A | user1    | create database db_firewall_test2                                                              |
| 0xEC489734142A4C69 | user1    | create database db_firewall_test1                                                              |
| 0xBA3A41A6FF5DFE4E | user1    | create table table1 (id int auto_increment, col1 varchar(?), col2 varchar(?), primary key(id)) |
| 0x99531AEFF718C501 | user1    | show tables                                                                                    |
| 0x236BBB7F360888D6 | user1    | insert into table1 values (?, ?, ?)                                                            |
| 0xA6D96F2525BD0179 | user1    | select * from table1                                                                           |
| 0x620B328FE9D6D71A | user1    | SELECT DATABASE()                                                                              |
| 0x907932FE528CCC74 | user1    | select * from table2                                                                           |
| 0xA61313D2974A4080 | user1    | insert into table2 values (?, ?, ?)                                                            |
| 0xA74FAF80A4A93A29 | user1    | create table table2 (id int auto_increment, col1 varchar(?), col2 varchar(?), primary key(id)) |
| 0x92A70B0CCDCE0ACB | user1    | SELECT * FROM table2 WHERE ?=?                                                                 |
| 0x0D15E3987E9E24B1 | user1    | select col1 from table2                                                                        |
| 0xFF15DBC009550EA3 | user1    | show dataases                                                                                  |
| 0xD82EC15EB253E71B | user1    | SELECT * FROM table1 WHERE ?=?                                                                 |
| 0x226CD90D52A2BA0B | user1    | select @@version_comment limit ?                                                               |
| 0x9DFADD97FB0AF9C7 | user1    | show create table table1                                                                       |
| 0xFDA30626AB3B972E | user1    | select col2, col1 from table1                                                                  |
+--------------------+----------+------------------------------------------------------------------------------------------------+

Pay attention to the above list, to the column “digest“, as we will need it to create the whitelist.

After the collection, we can now continue and configure the firewall tables. ProxySQL introduces two new tables for firewall whitelist.

admin ((none))>SELECT name AS tables FROM main.sqlite_master WHERE type='table' AND name IN ('mysql_firewall_whitelist_rules','mysql_firewall_whitelist_users') ORDER BY name;
+--------------------------------+
| tables |
+--------------------------------+
| mysql_firewall_whitelist_rules |
| mysql_firewall_whitelist_users |
+--------------------------------+
2 rows in set (0.00 sec)

The first one “mysql_firewall_whitelist_rules” identifies a specific user to apply the firewall whitelist algorithm and determines the default action for each user. There are three modes for each user (from ProxySQL documentation):

OFF : allow any query
DETECTING : allow any query, but queries not explicitly enabled in table mysql_firewall_whitelist_rules generate an error entry in the error log
PROTECTING : allows only queries explicitly enabled in mysql_firewall_whitelist_rules , and block any other query

We will enable the firewall on ProxySQL:

SET mysql-firewall_whitelist_enabled = 1;
SET mysql-firewall_whitelist_errormsg = 'ProxySQL Firewall blocked this query';

LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;

Note: Take care – if you enabled the firewall, before configuring the above tables, that you will block all users and queries. I’m doing this only for test purposes.

We can continue and enable the MySQL user “user1” to accept all traffic.

INSERT INTO mysql_firewall_whitelist_users (active, username, client_address, mode, comment) VALUES (1, 'user1', '', 'OFF', '');

LOAD MYSQL FIREWALL TO RUNTIME;
SAVE MYSQL FIREWALL TO DISK;

Now this “user1” can run any query, existing or not, in the table mysql_firewall_whitelist_rules table.

We will update the rule to mode = DETECTING, and in this case, “user1” can run any query, but if the query doesn’t exist in the table mysql_firewall_whitelist_rules, it will generate an error entry in the proxysql.log file.

UPDATE mysql_firewall_whitelist_users SET mode='DETECTING' WHERE username = 'user1';

LOAD MYSQL FIREWALL TO RUNTIME;
SAVE MYSQL FIREWALL TO DISK;

And we can run any query, for example:

mysql -u user1 -p123456 -h 127.0.0.1 -P 6033

mysql> use db_firewall_test1
Database changed
mysql> select * from table1;
+----+------+------+
| id | col1 | col2 |
+----+------+------+
| 1 | aa | bb |
| 2 | cc | dd |
+----+------+------+
2 rows in set (0.00 sec)
mysql> select col2, col1 from table1;
+------+------+
| col2 | col1 |
+------+------+
| bb   | aa   |
| dd   | cc   |
+------+------+
2 rows in set (0.01 sec)

This is the warning we can see in the proxysql.log file. We cannot see the query statement but we can see the digest:

2020-03-28 21:26:01 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall detected unknown query with digest 0x620B328FE9D6D71A from user user1@127.0.0.1
2020-03-28 21:26:10 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall detected unknown query with digest 0xA6D96F2525BD0179 from user user1@127.0.0.1
2020-03-28 21:27:32 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall detected unknown query with digest 0xFDA30626AB3B972E from user user1@127.0.0.1

Anyway, “user1” can run any query and new queries will be logged into the stats_history.history_mysql_query_digest table.

Finally, the last option is mode = PROTECTING, and it will check if the query (in this case, it will check digest) we are running exists in mysql_firewall_whitelist_rules, and we are going to update the mode.

UPDATE mysql_firewall_whitelist_users SET mode='PROTECTING' WHERE username = 'user1';

LOAD MYSQL FIREWALL TO RUNTIME;
SAVE MYSQL FIREWALL TO DISK;

And now run any query that exists or not into stats_history.history_mysql_query_digest and check the output. Let’s take a look:

mysql -u user1 -p123456 -h 127.0.0.1 -P 6033

mysql> use db_firewall_test1;
Database changed
mysql> select * from table1;
ERROR 1148 (42000): ProxySQL Firewall blocked this query
mysql> show tables;
ERROR 1148 (42000): ProxySQL Firewall blocked this query

Perfect! As we can see, those queries are blocked and showing the message “ProxySQL Firewall blocked this query“.

Also, there are warnings in the proxysql.log file. The three first warnings are from this statement “use db_firewall_test1”, as the “use” statement will not be logged into the stats_history.history_mysql_query_digest table.

2020-03-28 21:37:30 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x620B328FE9D6D71A from user user1@127.0.0.1
2020-03-28 21:37:30 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x02033E45904D3DF0 from user user1@127.0.0.1
2020-03-28 21:37:30 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x99531AEFF718C501 from user user1@127.0.0.1
2020-03-28 21:37:44 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0xA6D96F2525BD0179 from user user1@127.0.0.1
2020-03-28 21:37:52 Query_Processor.cpp:1905:process_mysql_query(): [WARNING] Firewall blocked query with digest 0x99531AEFF718C501 from user user1@127.0.0.1

Now, we can choose which queries from stats_history.history_mysql_query_digest we want to permit to run from the user “user1”.

The next step is to copy one or all the digests and insert them into mysql_firewall_whitelist_rules table. For this test, I’ll just copy/insert only three digests for test purposes.

From stats_history.history_mysql_query_digest:

| 0x0D15E3987E9E24B1 | user1    | select col1 from table2                                                                        |
| 0x907932FE528CCC74 | user1    | select * from table2                                                                           |
| 0xA6D96F2525BD0179 | user1    | select * from table1                                                                           |

Insert into mysql_firewall_whitelist_rules:

INSERT INTO mysql_firewall_whitelist_rules VALUES (1, 'user1', '', 'db_firewall_test1', 0, '0x0D15E3987E9E24B1', '');
INSERT INTO mysql_firewall_whitelist_rules VALUES (1, 'user1', '', 'db_firewall_test1', 0, '0x907932FE528CCC74', '');
INSERT INTO mysql_firewall_whitelist_rules VALUES (1, 'user1', '', 'db_firewall_test1', 0, '0xA6D96F2525BD0179', '');

LOAD MYSQL FIREWALL TO RUNTIME;
SAVE MYSQL FIREWALL TO DISK;

And test again…

mysql> use db_firewall_test1
Database changed
mysql> select * from table1;
+----+------+------+
| id | col1 | col2 |
+----+------+------+
| 1 | aa | bb |
| 2 | cc | dd |
+----+------+------+
2 rows in set (0.00 sec)

mysql> use db_firewall_test2
Database changed
mysql> select * from table2;
+----+------+------+
| id | col1 | col2 |
+----+------+------+
| 1 | ee | ff |
| 2 | gg | hh |
+----+------+------+
2 rows in set (0.00 sec)

mysql> select col1 from table2;
+------+
| col1 |
+------+
| ee |
| gg |
+------+
2 rows in set (0.00 sec)

mysql> select id from table2;
ERROR 1148 (42000): ProxySQL Firewall blocked this query

It works! My last query was to check if it’s working, because that query was not added in the mysql_firewall_whitelist_rules table.

If you want to add all your users and queries in the whitelist, I recommend the following queries to collect all your traffic for a long time:

INSERT INTO mysql_firewall_whitelist_users (active, username, client_address, mode, comment) SELECT DISTINCT 1, username, '', 'DETECTING', '' FROM mysql_users;

INSERT INTO mysql_firewall_whitelist_rules
(active, username, client_address, schemaname, flagIN, digest, comment)
SELECT DISTINCT 1, username, client_address, schemaname, 0, digest, ''
FROM stats_history.history_mysql_query_digest;

LOAD MYSQL FIREWALL TO RUNTIME;
SAVE MYSQL FIREWALL TO DISK;

Conclusion

This is a solid approach to build a good whitelist, but remember, before/after any deploy from your application with new queries keep in mind you need to add those new queries into the mysql_firewall_whitelist_rules table.

I think it’s good to add permission to one user to run only one or more queries.

Last but not least, take care that the next two queries are the same for you but have a different digest. Check the following example:

mysql> select * from table2; --# 0x907932FE528CCC74
+----+------+------+
| id | col1 | col2 |
+----+------+------+
| 1 | ee | ff |
| 2 | gg | hh |
+----+------+------+
2 rows in set (0.00 sec)

mysql> select * from db_firewall_test2.table2; --# 0x2EDC8B093C50BE48
ERROR 1148 (42000): ProxySQL Firewall blocked this query

I hope this is helpful to you. Thanks so much.

Interested in learning more?

Be sure to get in touch with Percona’s Training Department to schedule a hands-on tutorial session for all things ProxySQL. Our instructors will guide you and your team through the install and setup processes, learn how to manage read-write splitting, chain rules together, implement query firewalling, query caching, high-availability and much more.

As a Technical Account Manager, or TAM for short, I am fortunate to work with some large clients.  Most all of them are in some state of increasing their usage of containerization.  There are a number of reasons for this, but let’s explore a few of the biggest reasons.

Why Containers?

Almost everyone has deployed Virtual Machines (VMs) for years.  Certainly, the benefits of virtualization are obvious, but what about containers?  I still hear the phrase “Databases do not belong in containers!” quite often, but is it valid?  That is hard to say and likely falls back to the quintessential statement of “It depends…”  Containers, as you likely know, is a more lightweight implementation of virtualization than VMs.

Containers provide multi-tenancy application options on physical hosts and are especially useful for implementing immutable applications that can be deployed and destroyed at will.  They can be run in isolation of other services with packaged libraries and dependencies, all in one container.  In a nutshell, containers virtualize the host operating system whereas traditional VMs virtualize the hardware.

Another huge benefit of containerization is in automation.  Containers make it easy to deploy containerized databases that are pre-configured and ready for use.  Then, when the packaged container is no longer needed, it can be easily destroyed.  No mess, no fuss!

Need to work with lower environments but want to push changes up through to production?  No problem.  Containers allow you to automate deployment in a consistent manner and ensure there are no differences between environments.  Add to this the fact that you can incorporate a CI/CD (Continuous Integration/Continuous Delivery) pipeline into the process and you have a home run!

Using Containers with Cloud Providers

Compatibility with multiple cloud providers is another bonus.  When enterprise leadership wants to migrate from one cloud provider to another, or even wants to move to or from on-premises cloud hosting, it is hard to beat containers.  Containers are portable and can be moved as needed.  This avoids vendor lock-in.

Scalability is also a huge reason I hear from clients.  The ability to scale up or scale down is critical, especially given the rapidly evolving situation the world is currently facing and its ever-changing demands on information technology.  Containers, paired with automation, make it easy to deploy additional servers as demand increases and then scale back down once it subsides.

Cost is another major factor.  Since containers can share infrastructure, there are often substantial cost savings to be had with them versus VMs.

As to why many companies are not all-in yet with containers, there are few reasons that I have heard.  One thing mentioned often is performance, especially in the networking stack.  This is something that is improving with time but can be a potential bottleneck in heavy applications.  Depending upon which components of your application are deployed within the container, and whether the container has to do a lot of communication outside itself, this may or may not be an issue.  Again, the “It depends…” mantra applies.

Security, due to a lack of isolation, is another concern as multiple containers most likely are deployed on a particular host.  If the host machine is breached, the containers are in jeopardy of being breached as well.

Moreover, since containers on a physical host must share the same Operating System, there is a lack of flexibility in choosing a different OS.  For those administering the OS, this could well be considered a perk since there will be fewer installations of the OS to manage.

Performance tuning is another concern I often hear, but that is the same with VMs.  The problem of multi-tenancy on a physical host is a challenge to performance tuning for either approach of virtualization.  Unless you have the luxury of physical machines, you will face the dilemma either way.

Probably one of the biggest hurdles for containerization is resistance to progress.  Many are simply resistant to change.  It is new and therefore scares a lot of people.

With all of the above said, I lean towards the conservative end of the spectrum.  I think containers are fantastic for deployment to Dev, QA, Staging, and other non-production environments.  Although I am not at all opposed to deployment in Production (and work with large scale enterprises already doing so), I do think that caution is still warranted and as always, you should thoroughly test all deployments before they go into production with workloads simulating expected production levels.

Still not convinced databases should be in containers?  I doubt you will have the luxury to avoid them much longer.  As the technology advances, container usage is on the rise and nothing I have seen suggests otherwise!

We recently released an experimental version of percona-dbaas command-line tool with the goal of significantly simplifying the deployment of database instances in Kubernetes.

We have Kubernetes Operators for Percona XtraDB Cluster and Percona Server for MongoDB, which provide great flexibility, but they also come with the complexity of managing configuration and deployment files, so we wanted to make it even simpler to deploy database instances. We found it hard enough to get a MySQL instance up and running in Kubernetes, and then a whole different process to get MongoDB up, and we thought it should be a unified set of commands. This is how the percona-dbaas CLI tool was born. You can get more information on installation of the percona-dbaas-cli tool and in this blog post, I wanted to review the basic functionality it provides.

1. Kubernetes Setup

First of all, the tool assumes you have your Kubernetes cluster installed and running. For example, I will use Google Kubernetes Engine. For the local system, you will need to have Google Cloud SDK installed, and for the Ubuntu-based system (which I used), the instructions are here.

After the initial setup, let’s make sure we have access to the cluster from kubectl and have enough nodes running:

kubectl get nodes
NAME                                       STATUS   ROLES    AGE    VERSION
gke-cluster-1-default-pool-43590ffc-4nbn   Ready    <none>   158m   v1.14.10-gke.24
gke-cluster-1-default-pool-43590ffc-9hd4   Ready    <none>   158m   v1.14.10-gke.24
gke-cluster-1-default-pool-43590ffc-n0tv   Ready    <none>   158m   v1.14.10-gke.24
gke-cluster-1-default-pool-43590ffc-qqdh   Ready    <none>   158m   v1.14.10-gke.24

And the extra permission grant step is needed for our Operators:

kubectl create clusterrolebinding cluster-admin-binding-$USER --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)

2. Basic Operations

As we have the Kubernetes cluster running, now there are basic operations to create database instances:

> percona-dbaas mysql create-db db-cluster1 
Starting.....................................[done]
Database started successfully, connection details are below:
Provider:          k8s
Engine:            pxc
Resource Name:     db-cluster1
Resource Endpoint: db-cluster1-proxysql.default.pxc.svc.local
Port:              3306
User:              root
Pass:              S4QN2Szz0f6oMML7
Status:            ready

To access database please run the following commands:
kubectl port-forward svc/db-cluster1-proxysql 3306:3306 &
mysql -h 127.0.0.1 -P 3306 -uroot -pS4QN2Szz0f6oMML7

The cluster was installed, and we can even access it with the mysql command line via kubectl port-forward.

Let’s see what was really deployed:

> kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
db-cluster1-proxysql-0                             3/3     Running   0          18m
db-cluster1-pxc-0                                  1/1     Running   0          18m
db-cluster1-pxc-1                                  1/1     Running   0          17m
db-cluster1-pxc-2                                  1/1     Running   0          16m
percona-xtradb-cluster-operator-7766bb49dc-x4zg7   1/1     Running   0          18m

So we can see we deployed three nodes of Percona XtraDB Cluster with ProxySQL running in front. And now if we access it from mysql command line:

> kubectl port-forward svc/db-cluster1-proxysql 3306:3306 &
> mysql -h 127.0.0.1 -P 3306 -uroot -pS4QN2Szz0f6oMML7
mysql: [Warning] Using a password on the command line interface can be insecure.
Handling connection for 3306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 282
Server version: 5.7.28 (ProxySQL)

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

We see that we actually connected to the ProxySQL instances.

To destroy the cluster:

> percona-dbaas mysql delete-db db-cluster1 
ARE YOU SURE YOU WANT TO DELETE THE DATABASE 'db-cluster1'? Yes/No
ALL YOUR DATA WILL BE LOST. USE '--preserve-data' FLAG TO SAVE IT.
Yes
Deleting...............[done]

It’s a very similar step to create a MongoDB instance from the command line tool:

> percona-dbaas mongodb create-db db-cluster1 
Starting........................[done]
Database started successfully, connection details are below:
Provider:          k8s
Engine:            psmdb
Resource Name:     db-cluster1
Resource Endpoint: db-cluster1-rs0.default.psmdb.svc.local
Port:              27017
User:              clusterAdmin
Pass:              t3gzVf3KPt1Q5a27
Status:            ready

To access database please run the following commands:
kubectl port-forward svc/db-cluster1-rs0 27017:27017 &
mongo mongodb://clusterAdmin:t3gzVf3KPt1Q5a27@localhost:27017/admin?ssl=false

3. Extra Configuration

It is possible to customize the deployment with additional options.

Case 1: Deploy without ProxySQL:

> percona-dbaas mysql create-db db-cluster1  --options="proxysql.enabled=false"
Starting....................................[done]
Database started successfully, connection details are below:
Provider:          k8s
Engine:            pxc
Resource Name:     db-cluster1
Resource Endpoint: db-cluster1-pxc.default.pxc.svc.local
Port:              3306
User:              root
Pass:              XZLwgMxiaq3uHeqSouk
Status:            ready

To access database please run the following commands:
kubectl port-forward pod/db-cluster1-pxc-0 3306:3306 &
mysql -h 127.0.0.1 -P 3306 -uroot -pXZLwgMxiaq3uHeqSouk

You actually will have three pods running:

db-cluster1-pxc-0                                  1/1 Running 0 18m

db-cluster1-pxc-1                                  1/1 Running 0 17m

db-cluster1-pxc-2                                  1/1 Running 0 16m

You can choose which one you want to connect to. And if we connect to the instance:

> kubectl port-forward pod/db-cluster1-pxc-0 3306:3306 &
> mysql -h 127.0.0.1 -P 3306 -uroot -pXZLwgMxiaq3uHeqSouk
mysql: [Warning] Using a password on the command line interface can be insecure.
Handling connection for 3306
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.7.28-31-57 Percona XtraDB Cluster (GPL), Release rel31, Revision ef2fa88, WSREP version 31.41, wsrep_31.41

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>  show global status like 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+

We connected directly to a Percona XtraDB Cluster node, and there are three nodes are running.

Case 2: Create only a single node cluster without ProxySQL:

> percona-dbaas mysql create-db db-cluster1  --options="proxysql.enabled=false,allowUnsafeConfigurations=true,pxc.size=1"
Starting...........[done]
Database started successfully, connection details are below:
Provider:          k8s
Engine:            pxc
Resource Name:     db-cluster1
Resource Endpoint: db-cluster1-pxc.default.pxc.svc.local
Port:              3306
User:              root
Pass:              rBenwO6cL1aHZtaRK5
Status:            ready

To access database please run the following commands:
kubectl port-forward pod/db-cluster1-pxc-0 3306:3306 &
mysql -h 127.0.0.1 -P 3306 -uroot -prBenwO6cL1aHZtaRK5

To check that we have only one node running:

> kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
db-cluster1-pxc-0                                  1/1     Running   0          80s
percona-xtradb-cluster-operator-57cdc94964-scvf5   1/1     Running   0          5m6s

Case 3: Create a cluster which is accessible outside Kubernetes.

This particular case will work only with Google Kubernetes Engine and other providers that can assign a public IP to a LoadBalancer.

So you can run:

> percona-dbaas mysql create-db db-cluster1  --options="proxysql.serviceType=LoadBalancer"
Starting......................................[done]
Database started successfully, connection details are below:
Provider:          k8s
Engine:            pxc
Resource Name:     db-cluster1
Resource Endpoint: db-cluster1-proxysql.default.pxc.svc.local
Port:              3306
User:              root
Pass:              KtwBG3qzvVR9onQe6
Status:            ready

To access database please run the following commands:
kubectl port-forward svc/db-cluster1-proxysql 3306:3306 &
mysql -h 127.0.0.1 -P 3306 -uroot -pKtwBG3qzvVR9onQe6

Now there should be a public IP assigned to the cluster, and to find it, we run:

> kubectl get services
NAME                           TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)             AGE
db-cluster1-proxysql           LoadBalancer   10.8.3.197   34.71.177.219   3306:30887/TCP      8m3s

So now we can connect to the cluster from anywhere:

mysql -h 34.71.177.219 -P 3306 -uroot -pKtwBG3qzvVR9onQe6

Just make sure you create users with secure passwords!

4. Customize my.cnf

This is not really a documented feature yet (so consider this is a hack), but sometimes we want to make some adjustments to my.cnf, for example, to increase buffer pool size if we know that nodes have enough memory to accommodate a bigger buffer pool.

For this we create our or my.cnf:

[mysqld]
innodb_buffer_pool_size=2G

Deploy it to a configmap:

kubectl create configmap mycluster1-pxc --from-file=my.cnf

And create the cluster after that:

percona-dbaas mysql create-db mycluster1

After we connect to to the cluster:

mysql> show global variables like 'innodb_buffer_pool_size%';
+-------------------------------------+----------------+
| Variable_name                       | Value          |
+-------------------------------------+----------------+
| innodb_buffer_pool_size             | 2147483648     |
+-------------------------------------+----------------+

We have nodes with 2GB buffer pool size.

In the command above, “mycluster1” specifies the configuration that will be applied to “mycluster1” cluster and “pxc” specifies you’re supplying MySQL/PXC configuration through my.cnf.

Conclusion

Our percona-dbaas tool is the next step in simplifying the deployment for Percona XtraDB Cluster and Percona Server for MongoDB. It is still in an experimental state, and we are asking for your feedback to see what we can improve before we get to the official release.

This is a three-part blog series. Be on the lookout for parts two and three in the coming days.

There was no reason to plan for it, but the load on your system increased 100%, 300%, 500%, and your MySQL database has to support it. This is a reality many online systems have to deal with these days. This series focuses on dealing with the unexpected high traffic event as it is happening.

There are also a lot of things you can do proactively, which we covered in “Prepare Your Databases for High Traffic on Black Friday.”

First, let’s see what impact a traffic spike can cause on the database — what problems will your application engineering team likely see?

• Poor query response time
• Elevated error rate (connecting to the database and executing queries)
• The database is down (unavailable)
• Incorrect (outdated) data due to replication lag or batch jobs unable to complete

The immediate goal of handling the traffic spike is to eliminate these issues as quickly as possible, which for most teams means focusing on “low hanging fruit” — solutions that can be deployed within hours, or days, and which do not require a massive application or infrastructure changes.

The good news is that for the majority of applications, you can gain several times database capacity by taking some simple actions:

1. Scale your Cloud Instance Size 

Complexity: Low
Potential Impact: High

If you’re running in the cloud (or some virtualized environment), going to a larger instance size (also known as “tuning by credit card”) is often the easiest thing you can do. It is one of the most expensive things you can do, but it is a short term action you can take while you proceed to implement other performance optimization actions.

Note: Databases tend NOT to scale linearly so do not fall into a false sense of security — if your cloud vendor has 10x larger instances available, do not expect it will handle 10x traffic. It may be a lot less depending on workload.

2. Deploy More MySQL Slaves/Replicas

Complexity: Medium
Potential Impact: High

If your workload is read-intensive, deploying more replicas can be a great way to improve performance. Do not know what kind of workload you have? Reviewing “Is it Read Intensive or Write Intensive Workload” should help you figure that out.

Deploying replicas is not enough though; you need to make sure your application is capable of routing traffic to them. Some applications may find it easy to enable this functionality on the application level. For others, deploying ProxySQL and using its read-write splitting functionality may be a better choice.

In many cases, you can even move complete applications to use replicas: reporting applications or applications utilizing MySQL Full-Text Search are often great candidates.

Keep in mind, MySQL replication is asynchronous which means replicas will have data propagated with delay (sometimes significant) therefore, only route queries to replicas which can tolerate not up to date data and make sure to monitor for replication lag and health.

3. Deploy ProxySQL for Connection Management and Caching

Complexity: Medium
Potential Impact: High

ProxySQL is a great tool to help manage MySQL traffic, especially during traffic spikes. One way ProxySQL can help is through connection pooling so the application does not run out of connections and does not overload MySQL by having too many concurrent connections.

If you’re running Percona Server for MySQL or MariaDB, you can also enable ThreadPool which can allow a MySQL instance to handle 100k+ connections directly.

Another feature of ProxySQL which can be even more helpful during traffic spikes is ProxySQL Query Cache, which allows you to cache results of the queries for a period of time.

As you find queries that do not absolutely need to serve completely up to date results, route them to MySQL replicas you can cache the same queries for added benefit.

4. Disable Heavy Load Applications Features 

Complexity: Medium
Potential Impact: Medium 

Management and development teams will often hate such ideas, but it is a great tool to have. Not all application features provide the same value or are used with the same frequency, yet it is advanced, rarely applied features which often can be the most costly, as not a lot of time has been spent optimizing them. Disabling them, at least temporarily, while you go through traffic spikes or finding a time to optimize them, is often a great thing to do.

It does not have to be user-facing features — think if there are any internal-facing reports which you can live without?

5. Check for Resource Bottlenecks 

Complexity: Low
Potential Impact: High 

Hardware-level databases are likely to be bottlenecked by one (or more) primary resources — CPU, memory, disk, or network. If you’re running Percona Monitoring and Management (PMM), you can see these in the Node Summary Section in your MySQL Instance Summary Dashboard.

If a particular resource is saturated, you can typically get better performance by increasing that resource, though focusing on reducing that resource usage is another thing to consider. For example, a high CPU usage problem is often better solved by optimizing your queries rather than by getting a faster CPU.

6. Get More Cores or Faster Cores

Complexity: Low
Potential Impact: Medium

An important thing to know about MySQL is, it can only use a single CPU core to do most of the work of running a single query, which means getting more CPU cores often does not make your slow queries or batch jobs running many queries sequentially run faster. If that is your problem, you need to focus on getting faster CPU cores or you may need to focus on getting more cores.

But how do you know what kind of workload you’re running?

Take a look at your CPU Usage, CPU Saturation, and Max Core Usage in the Node Summary in Percona Monitoring and Management (or comparable graph in your favorite monitoring system).

If CPU Usage is high (excluding IOwait), and if Normalized CPU Load is 2 or more, your system would benefit from having more CPU cores available to run a workload.

If, however, Max CPU Core Utilization is pegged closer to 100% and your CPU usage is not high, you should focus on faster cores.

For example, if you’re running on AWS, Cloud C5 instance type offers higher CPU performance compared to the General Purpose M5 Instance type.

Another thing to watch out for when it comes to CPU, especially in cloud and virtualized environments, is “CPU Stealing” — it may leave your MySQL instance with a lot fewer CPU resources than CPU frequency and core number can indicate. Read “Choose Your EC2 Instance Type Wisely on AWS” for more detail.

This is a three-part blog series. Parts two and three, with numbers 7-18,  coming soon.

Join us on Wednesday, April 15th, 2020, at 1:00 pm EDT for an hour-long roundtable with Peter Zaitsev, CEO and Matt Yonkovit, CXO, of Percona.

• Is your business experiencing a slowdown? Matt will share five easy ways your company can cut costs quickly, without reducing staff.
• Or, perhaps your company is actually seeing an unexpected increase in traffic? Learn how to prepare for or survive a surge in your traffic.
• Peter will also share tips for how to remove bottlenecks caused by high traffic events in your MySQL database.
• Finally, with the sudden shift to remote working, Peter and Matt will share their personal experience being part of and leading a remote company, with Percona being almost entirely remote for the past 13 years.

Don’t miss out! Register for the Percona Roundtable Now!

Register Now

This is a three-part blog series that focuses on dealing with an unexpected high traffic event as it is happening. Part one of this series talked about the first six things you can do, and in part two, we will be discussing numbers seven through twelve.

7. Get More Memory

Complexity: Low
Potential Impact: High

If your data does not fit into memory well, your MySQL performance is likely to be severely limited. If your data already fits in well, adding even more memory will not provide any performance improvements.

Even when you’re running on very fast storage, such as Intel Optane or directly Attached NVMe Storage, accessing data in memory is still more than an order of magnitude faster.

How do you know if you have enough memory? Look at Memory Utilization and I/O Activity.

I/O Activity is actually the first item I would look at. If, as in this case, you have no read IO to speak about, all your data is in the cache — either MySQL’s data caches or Operating System file cache. Write activity, however, will not be eliminated completely even if all data fits in the cache as database modifications need to be recorded on the disk.

Typically you will not look to eliminate Read IO Completely — it will just require too much memory in most cases, and not needed. However, you want to make sure Read IO does not impact your performance substantially. You can do this by ensuring your disk load is manageable or, if you have Percona Monitoring and Management (PMM) installed, you can check how much disk reads impact your specific queries performance in Query Analytics.

Note: While you will get some value from simply adding more memory because it will be used by Operating System as a cache, to get most of the newly available memory, you should configure MySQL to use it. Innodb_buffer_pool_size is the most important variable to consider. 80% of memory is often used as a rule of thumb but there is more to that. 

One thing you should be mindful of, as you configure MySQL to take advantage of all your memory, is to ensure you do not overcommit memory and MySQL does not run out of virtual memory (as it may crash or be killed by Out of Memory (OOM) Killer).

You also want to make sure there is no significant swapping activity (1MB/sec or more) but some swap space use is OK. Check out “In defense of swap: common misconceptions” for more details.

8. Move To Faster Storage

Complexity: Medium
Potential Impact: High

When your data size is small, fitting it into memory is the best way to scale reads. If your database is large, it can become impractical and a faster disk may be a better choice. Additionally, you get Writes which need to be handled even if you have a lot of memory. This old but still valid article goes into detail on this topic.

With CPUs, you need to know whenever you need more or faster cores, the situation with storage is even more complicated. You need to understand the difference between throughput (IOPS) and latency (check out this fantastic article on the topic) as well as the difference between read and write performance.

One way to look at IO performance is by looking at the Number of IOPS storage being served or the Bandwidth of IO activity.

It is helpful if you know what the limits of your storage are and if you’re close to or running into them. You may not know the exact performance storage can provide. In this case, it is helpful to take a look at Disk IO Load which roughly shows you how many IO operations are in flight at the time.

If you see this number in tens or hundreds, chances are your disk is overloaded. The problem with storage, unlike CPU, is that we have no way to know what is the “natural level of concurrency,” when requests can proceed in parallel, or when queueing needs to happen.

Take a look at request latency for reads and writes and see if they are any different from the time before traffic spike. Also, read and write latencies can be independently impacted and should be looked at separately.

How much can a faster disk impact your queries’ performance? From the reads standpoint, you can check PMM Query Analytics as I explained in the 7. Get More Memory section, but for writes, it is more complicated.

Write to InnoDB Redo Log, or more specifically, persisting it on disk through fsync() is a very common bottleneck. You will see if this happens in your system by looking at the number of pending fsyncs (MySQL Innodb Details dashboard, Innodb Disk IO Section).

If it is close to 1 all the time, you likely have a bottleneck with disk flushing. To improve the situation you will need storage with better write (fsync()) latency. You can adjust your MySQL configuration to reduce durability guarantee or adjust your workload to group queries in smaller numbers of transactions.

What faster storage options are available? Intel Optane SSD or NVMe storage tends to offer the best performance and fastest and most predictable latency. However, if you use those solutions, especially in the cloud, make sure you use some form of replication for data redundancy.

If you need to use network storage, look for throughput optimized options such as AWS EBS io1 volume type. Traditional “general purpose” gp2 volumes can be a lot more cost-effective, but they have a lower peak performance.

9. Check Your Network

Complexity: Low
Potential Impact: High

When checking if a network is a bottleneck in your traffic spike event, you need to be looking at Bandwidth, Latency, and Errors.

Networks tend to be more complicated than other resources because all these have to be measured for different clients separately. For example, clients running on “localhost” tend not to have a problem, however, clients running in other parts of the world communicating with your database will have issues.

Network bandwidth, at least when it comes to the local node, is rarely the problem.

Rarely, applications retrieve large result sets and saturate the network. Network backups and other large data transfers can saturate the network causing the slowness of other user transactions.

The latency between your client and database server can be roughly measured by the “ping” or “mtr” tool. If you have a 10Gb network you may expect 0.2ms in the same data center. It is generally slightly higher in the cloud providers within the same availability zone. Different high availability zones come with higher latency and latency between distant regions can be 100ms and can have significantly higher variance than the local network.

In this case, we see the path between client and server passes through only one router (and perhaps a few switches) with an average latency of 1.5ms and no packet lost.

You should keep your application server and your database as close as possible — in the same availability zone if possible, but surely in the same region for latency-sensitive applications.

When it comes to errors, TCP retransmission is your worst enemy as it can add very significant latency.

If you’re seeing increased retransmission rates during your traffic spike event, chances are there are problems on the network level which need to be addressed.

10. Locate and Optimize Queries Which Cause the Load

Complexity: Medium
Potential Impact: High

Locating and optimizing bad queries is one of the highest value activities you can do as it provides long-term benefits. Unlike beefing up your hardware, it does not require additional investment (other than time).

If you’re running Percona Monitoring and Management you should take a look at the Query Analytics tool, which by default sorts queries by the load they generate.

Examining and optimizing queries in this order is a fantastic way to make your system run faster. In some cases, like commit query, you can’t really optimize the query itself, but you can speed it up through hardware changes or MySQL configuration.

Check out query execution details:

And Explain Plan to see if and how this query can be optimized:

MySQL Query Optimization is too complex a topic to cover in one blog post. I would consider learning to read EXPLAIN and attending a Webinar.

11. Add Missing Indexes

Complexity: Low
Potential Impact: High

Full query optimization may require changes to how a query is written, which requires development and testing time that may be hard to obtain. This is why, as the first pass, you may want to focus on adding missing indexes only. This does not require application changes and is reasonably safe (with rare exceptions), and should not change the results of the query.

Check out this webinar for some additional details.

12. Drop Unneeded Indexes

Complexity: Medium
Potential Impact: Medium 

Over time, it is common for database schema to accumulate indexes that are duplicate, redundant or unused. Some are added by mistake or misunderstanding, others were valuable in the past but aren’t anymore as the application changed.

You can read more about redundant and duplicate indexes in this blog post. The pt-duplicate-key-checker from Percona Toolkit is also a great tool to find them.

An unused index is a bit more complicated and risky — just because there was no query which needed this index in the last week, does not mean there is no monthly or quarterly report which needs one.

The blog post, Basic Housekeeping for MySQL Indexes, provides a recipe on how to find such indexes. If you’re running MySQL 8, you may consider making such an index invisible for a while before dropping it.

This is a three-part blog series. Part one can be found here, and part three will be published soon.

Percona Server for MySQL 8.0.19 includes the experimental simple LDAP authentication plugin, which allows for direct authentication using an LDAP server. Similarly to the data masking plugin added in 8.0.18, this plugin was also added as a community contribution from Francisco Miguel Biete.

Authenticating with LDAP

In earlier versions, Percona Server for MySQL supported the built-in authentication mechanism of MySQL and also included a port of the PAM plugin from MySQL enterprise.

As LDAP servers can be configured as backends for PAM, this already provided a way to authenticate SQL users using LDAP, by adding PAM as an intermediate layer — requiring setting up both the auth_pam plugin and configuring PAM to use LDAP for authenticating mysqld users.

With this plugin this process becomes simple, as LDAP authentication can be enabled by 3 simple steps:

1. Install the LDAP authentication plugin
2. Configure the LDAP server, as described in the plugin’s documentation
3. CREATE a user using the LDAP authentication, or ALTER existing users

A simple example looks like the following:

INSTALL PLUGIN authentication_ldap_simple 
 SONAME 'authentication_ldap_simple.so';
SET GLOBAL authentication_ldap_simple_server_host = 'hostname';
SET GLOBAL authentication_ldap_simple_bind_base_dn
  = 'ou=people,dc=example,com=com';
CREATE USER ldapuser IDENTIFIED WITH ldap_simple 
  BY 'cn=ldapuser,ou=people,dc=example,com=com';

Authentication Performance

By having three possible different authentication methods in the server, one might wonder if there is a performance benefit in using one over another. For example, since using LDAP is possible with two different plugins, which one is better?

To analyze this, we performed simple measurements. The measurement used a C++ client program created for this purpose, which connected to the server, executed a SELECT on an empty table, then disconnected — doing this repeatedly over and over, in multiple threads and processes.

We included both multiple processes and multiple thread test cases to verify that possible locks in the MySQL client library do not affect the test result: with this, we were able to verify that both multi-threaded and multi-process clients result in the same throughput.

The above graph shows the results of our measurements. It displays the execution time compared to the single process single-threaded execution of the default authentication test case, using a logarithmic scale.

We had to use a logarithmic scale because we noticed performance issues with the PAM plugin and also with all tested methods under high contention.

The first issue shows that the PAM plugin has a slow throughput compared to anything else: in the single-threaded test, it performed 15 times slower than the default plugin, and with 10 threads it was 57 times slower. We omitted PAM results with larger concurrency numbers as this is already above any other result.

This clearly shows the advantage of the new LDAP authentication plugin compared to the earlier PAM plugin.

For the other methods, the execution times only increase by 15-30% up to 20 threads or processes which results in a rapidly increasing throughput level. At its peak, the default plugin was able to handle around 100 000 connections per second, while the LDAP plugin measured at 60 000.

Above 20 concurrent tests, we can see a rapid increase of the execution time reaching more than 20 times the baseline at 100 threads, which results in a decreasing overall throughput.

This is also where we can see the advantage of thread pooling: while in the low concurrency levels (below 50) it performed worse than the default threading implementation, with 100 concurrent threads it performed more than 25% better: while the default threading implementation was only able to handle 23 000 login requests per second there, with thread pooling the server was able to perform more than 30 000 requests per second.

A Note on TCP Limitations

As shown, the tests initiated several hundred thousand connections within seconds. The LDAP authentication plugin creates new TCP connections for each authentication attempt, which can quickly result in the exhaustion of the entire port range. To use the LDAP authentication in a scenario like this, the net.ipv4.tcp_tw_reuse sysctl setting has to be 1.

Conclusion

Overall we can see that the LDAP authentication performs similarly to the default authentication method, and is viable for use in heavily loaded environments.

A four-part blog series by Matt Yonkovit, Percona Chief Experience Officer.

Baseline Data and the Size of the Market

Over the last few years, we have seen some drastic changes in both the database and the open source industry. We have seen the proliferation of open source databases, we have seen the impact of the cloud, and we have seen massive growth in adoption and usage as companies turn away from proprietary software in an effort to find something better.

So where do we stand now and where are we headed? What are the trends? Which databases are growing and which are not?

Before we start, I wanted to comment on Percona’s position in this debate.

Percona offers software, support, and services for open source databases. We have our own offerings for MySQL, MariaDB, PostgreSQL, and MongoDB. While our own software is important to us, we have spent the last 13 years building our reputation on ensuring that businesses can successfully use open source software, even if it’s not ours. Ultimately, our goal is to ensure users and customers choose the best open source database software to meet their needs and goals. Sometimes that’s our software, sometimes it’s not.

We are proud of our reputation and our willingness to help find the most scalable, performant, and efficient solutions for our customers.

The Database Market as a Whole

Gartner published data last year which pegged worldwide database management system revenue at around $46 billion, with about 18% overall year-on-year growth. Cloud systems accounted for a huge 68% of that growth. Our own open source survey backed-up this growth in cloud adoption.

However, many of the analysts and press I talk with disagree on how fast this segment is growing. I believe this partly comes from a disconnect in how analysts define a “cloud database.”

Some people use “cloud database” to refer to any database running on a cloud service, some only refer to those running in a cloud providers’ database as a service (DBaaS). Either way, this growth is a game-changing trend.

Despite this, not everything will move exclusively to the cloud in the next five, or even ten, years. Even the most aggressive cloud models show hold-outs.

In fact, if you model industry growth, even if 20% of future databases are on-premises versus 80% in the cloud in five years (which is an aggressive figure in my view), that still leaves a $25 billion database industry either on-premises or running ‘on the edge’. This is a huge amount.

Hybrid and Multi-Cloud Environments

An interesting development is that we are not seeing companies moving exclusively towards one provider. Instead, many are ending up in hybrid environments and multi-clouds. Our own survey data showed that 61% of enterprises already have hybrid environments.

Many companies are deploying a business strategy that calls for portability, ensuring they can run in different environments. In fact, analysts and researchers have started to view this run anywhere/run everywhere requirement as a must for future application design.

Open Source Database Market Growth

When we look specifically at the open source database space it is harder to find published research on the subject of growth (other than “a lot!”) Gartner published some insight on the market as a whole, but it did not talk specifically about size and growth.

Many open source database companies are not public and cloud providers, and companies such as Oracle, do not break out their specific revenue in relation to open source databases.

From talking with different companies, cloud providers, analysts, etc, and looking at available market data, the best estimate I can give is that this segment is in the range of $7-8 billion per year. This includes spending on open source software databases from cloud providers.

Open source growth rates also appear to be much higher than the industry average, seemingly around 45-50% overall. Some vendors are seeing even faster growth. MongoDB reported that in FQ3 their Atlas database-as-a-service accounted for 40% of their total revenue (up from 21% in the previous year). Atlas revenue in the latest quarter grew by more than 185%.

The database space is a big market and getting bigger. Cloud and open source databases are growing even faster than the industry average.

My next blog in this series on The State of the Open Source Database Industry in 2020 is available next week and will discuss migrating from proprietary databases to open source.

This is a three-part blog series that focuses on dealing with an unexpected high traffic event as it is happening. Part one can be found here, and part two can be found here.  

13. Configure MySQL Server Properly

Complexity:Medium
Potential Impact: High 

A poorly configured MySQL Server can cause severe issues, especially under high load during a traffic spike, yet getting the basics right is not that hard. While MySQL Server has more than 400 variables you can tune, you rarely need to change more than 10-20 of them to get 95% of the possible performance for your workload.

This blog post covers the most important basics.

14. Purge Unneeded Data

Complexity: Medium
Potential Impact: Medium

Everything being equal, the more data you have the slower the database will operate. Getting rid of (or archiving) data you do not need to have available online is a great way to improve performance.

In many cases, you find applications keep various logs in the database going back many years, where they have little use beyond a few weeks or few months.

Pt-archiver from Percona Toolkit is a great tool to archive old data.

Note: Although after the purge is completed you will have leaner, faster, database archiving, the process itself takes additional resources and is not something you should do while your database is already overloaded.

15. Complete Database Maintenance

Complexity: Medium
Potential Impact: Medium

When things were quiet, you can get by without maintaining your database. However, as such, database statistics may be outdated and your tables may be fragmented and otherwise not in the most optimal state.

Run OPTIMIZE TABLE on your tables to re-build them to become more efficient and to update stats.

To run OPTIMIZE for all tables you can run mysqlcheck –optimize -A.

Keep in mind, optimizing is likely to impact your system even more than purging old data so you may not want to do it during high load. A good approach may be to remove your replicas (slaves) from serving the traffic and running the process on them one by one when promoting one of those slaves to a master.

16. Check Your Background Jobs

Complexity: Medium
Potential Impact: Medium

Background jobs such as Backup, Maintenance, Report Generation, and Large Data Loads are often not very well optimized — they can be run in a slower period where the MySQL Server can handle the additional load. During traffic spikes, they may be causing database overload and downtime.

Another problem with background jobs running during traffic spike events is overlapping or snowballing. If your background job normally runs 15 minutes and you scheduled two of them at 2 AM and 3 AM, only one of them usually runs at a time. However, due to the additional load, it now may take two hours to run, and you could have several background jobs running at the same time, causing additional load and possible data corruption.

Check your background jobs and ask the following questions:

• Do I need this background job or can it be postponed?
• Can this job be run on a replica? Running different jobs on different replicas can be a great solution!
• Have you scheduled your batch jobs to make sure they are not overlapping?
• Is it possible to optimize a background job? Optimize the queries it uses or if you’re doing a backup with mysqldump you should use Percona Xtrabackup instead, which is a lot more efficient.
• Can you limit the resources this job uses? For example, limiting the concurrency (number of parallel connections) a batch job uses. Or, if you’re running Percona Xtrabackup and it impacts your server performance you can Throttle Backups.
  

17. Check for Data Hotspots

Complexity: High 
Potential Impact: High

Some applications scale very well with hardware scaling, others not so much. The difference typically is whenever applications rely on “hotspots” — data that needs to be updated so frequently it becomes a bottleneck. For example, if you create a single counter in the database where every single transaction will need to update it, it would not scale well.

There are many different types of hotspots, and some of them are hard to find and diagnose. The most common one is similar to described above and shows itself as high row-level lock waits (and high deadlock rate).

Within Percona Monitoring and Management you can check the MySQL Innodb Details dashboard to see what portion of time overall is spent waiting on row-level locks:

Or see the rollback rates:

Note that different applications could have different normal values for these if you have seen them outside of the norm during your high traffic event.

You can also examine what specific queries are having long row-level lock waits:

Reducing hotspots may be as easy as adding a better index, or it could be more difficult, requiring application re-engineering. Regardless, I included it here because if you have designed an application with very bad data hotspots, easier optimization techniques mentioned may not work for you.

18. Configure Your Application Server Correctly

Complexity: Medium
Potential Impact: Medium

While configuring MySQL Server it is extremely important to use the proper settings on your application server-side. You want to make sure you’re using persistent connections and not re-connecting for every tiny transaction, especially if you’re using TLS/SSL for database connections. If you’re using a connection pool, make sure it is configured correctly, especially if you’re not using ProxySQL or Threadpool. Specific performance optimization recommendations will vary depending on the programming language, ORM framework, or connection pool used — go Google for them!

Summary

This is quite a list of recommendations, and indeed there is a lot of things you can do during traffic spike events to bring things under control. The good news is you will not need to follow every one of these suggestions to get performance gains and ultimately delight your customers with fantastic application performance (or at least make your development team happy when the database is not the problem). Look at these recommendations as a menu — see what is easiest for you to apply in your environment and what is likely to provide the biggest gain, and use them to guide your actions!

This is a three-part blog series. Part one can be found here, and part two can be found here.  

Last month, our new Director of Product wrote a blog about all the cool things she learned about how we track Percona Monitoring and Management (PMM) adoption and even went into what conclusions we can draw from that.  As you can imagine, I was excited she geeked out on what little data we gather just like I do; you can make much better decisions even with just a little information!  So why are we talking about it again? Well, we’ve enhanced the telemetry endpoint to enable us to deliver even more value in the future and wanted to make sure we do so as transparently as possible.

So What Exactly Are We Doing?

We’ve stood up a new telemetry micro-service (I’ll have to get one of our architects to blog about that, but guess what we’ll be using to monitor and alert with!) that will let us eventually retire the legacy endpoint after we run them in parallel for a bit. This one will be FAR more flexible than the current one and will allow us to maintain separation of purely anonymous data and that of our subscribers (which isn’t actually built yet but baby steps).

Why Are We Doing It?

Well, to deliver better service, of course.  The basic anonymous data helps us determine where we need to focus our development dollars and understand both the makeup and uptake of the various versions of PMM.  This helps determine if we need to release patches to legacy versions that may not technically be supported all the way to how the frequency of release impacts adoption. We also are looking to deliver MORE services and many of them will require we know a little more about the user.  That subscriber service I mentioned earlier is just that. When it’s done you’ll be able to register for a free Percona account which will allow us to gather slightly more detailed information (don’t freak out, I have a whole section on transparency coming up) and turn that into more actionable information for you (did you know that MySQL 8.0.12 is vulnerable to 105 CVE’s most with a CVSS of 4.0 or greater…how many servers do you have that are impacted by that?)

Ummm…Can You Talk About That Transparency Part?

As far as the currently collected anonymous data we gather only the following: PMM Version, Installation Method (Docker*, AMI, OVF), and Uptime.  We do not capture anything that would make you identifiable (we get Country Code from the submitting IP address before it’s discarded) however we do create an “instance ID” which is just a random string we generate using UUID v4 to decide if we’re hearing from a new instance or an existing one for figuring out instance upgrades and the like.  We’ve also put a landing page up at the new endpoint check.percona.com to clearly identify what this service is, what it’s collecting, and how you can turn it off. Finally, we’re delaying the first reporting of a new instance by 24 hours to allow sufficient time to disable the service for those that do not wish to share any information. The documentation has also been drastically improved: we had a link to “more information” that contained no relevant information (fixed!) and are highlighting how to disable telemetry more prominently in our online docs.  And of course, we’re also an open source company, you can look at the logic for what we’re gathering and how by looking here and here.

So what’s next?

That’s the exciting part, our goal with Percona Monitoring and Management is to be able to drive you to take action!  I mentioned letting you know about CVEs but that’s the tip of the iceberg. Percona has always made its money through selling our database expertise via services, support and consulting while giving away our software.  Registered users and current customers will now be able to get some of that expertise delivered and applied directly to their environments. Our roadmap is overflowing with ideas to help you not only maintain awareness of what’s going on in your database infrastructure but to be more proactive about spotting problems before they become outages!

While I hope you are equally as excited about what we’re working on, but if you just don’t feel comfortable sharing any information, we wouldn’t be true to our word if we didn’t include instructions here on how to opt-out: You can easily switch this off via the on/off switch for Call home (<=PMM 2.3) or Telemetry (>=PMM 2.4) on the PMM Settings panel off the main dashboard.

For those of us who exist almost entirely on the command line, you can also disable telemetry on container initialization by adding ‘-e DISABLE_TELEMETRY=1’ to your ‘docker run’ command.

If you have thoughts on additional insights you’d like to get from Percona or on our approach, please comment below.

*Prior to PMM 2.4 we detected the OS of the PMM server and made the decision on our server if it was Docker vs AMI vs OVF now we only capture Docker, AMI or OVF.

This article describes how to install ProxySQL from the Percona Software Repositories.

Percona Software Repositories contain a ProxySQL version that has passed rigorous Percona tests and is known to be compatible with other Percona Software. We recommend using those builds whenever you’re running ProxySQL with Percona Server for MySQL, Percona XtraDB Cluster, MySQL, or MariaDB.

To install the Percona Repository for DEB-based distributions (Debian and Ubuntu) follow these instructions, and for RPM-based distributions (RedHat Enterprise Linux and Derivatives) follow these instructions.

At the time of this writing, ProxySQL 2 is the latest major release of ProxySQL and it is recommended for new installations. You can also install older ProxySQL version 1 by using “proxysql” instead of “proxysql2” as the package name.

To Install ProxySQL on DEB-based distributions, run:

sudo apt install proxysql2

To Install ProxySQL on RPM-based distributions, run:

sudo yum install proxysql2

After installation, ProxySQL will not be started automatically.  To start it, use:

sudo service proxysql start

Warning – Never run ProxySQL in productions with default credentials. You can change those credentials by editing the /etc/proxysql.cnf configuration file or (assuming you have the mysql client installed) by running:

mysql -uadmin -padmin -h127.0.0.1 -P6032 --prompt='\u (\d)>' main

admin (main)>UPDATE global_variables SET variable_value='admin:NEWPASSWORD' WHERE variable_name='admin-admin_credentials';
admin (main)>LOAD ADMIN VARIABLES TO RUNTIME;
admin (main)>SAVE ADMIN VARIABLES TO DISK;

This will set credentials for the ProxySQL Administrative user to user “admin” with the password “NEWPASSWORD”.

Bonus Tip:  If you do not want to remember this long command to access your ProxySQL admin interface, consider saving it to your .bashrc:

echo "alias psqladmin=\"mysql -uadmin -pNEWPASSWORD -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin> '\"" >>$HOME/.bashrc

You can then proceed to configure ProxySQL according to your use-case.   

Further Reading:

• ProxySQL Query Cache – What it is and How it Works 
• Using ProxySQL to improve MySQL Scalability 
• Multiplexing in PostgreSQL 
• How to use pt-heartbeat for replication lag monitoring with ProxySQL
• How to use ProxySQL 2 on Percona XtraDB Cluster for Failover

As I interface with large clients as a Technical Account Manager (TAM) at Percona, one thing I have noticed is the evolution of the DBA role.  With the rise of the cloud, DBAs seem to be spending less time doing traditional DBA roles of performance tuning, configuring backups, and so on.  Instead, DBAs are moving more from System DBA roles to more application-centric (App DBA) duties.  The cloud has largely taken over the System DBA functions and automated many of them.

So What Is the Next Evolution of the DBA Role?

How long before App DBA functions, such as query tuning, become a thing of the past?  Already, work is progressing in the MySQL arena for developing the ability for code to recommend (and perhaps even automatically implement) indexes to optimize query traffic and as well as find redundant indexes.  Auto-tuning of configuration settings is also becoming “a thing” for MySQL, and why not?  For years, MySQL monitors have suggested changing certain values by monitoring system metrics and applying formulae.  Why not allow monitors to make the changes as needed as well, especially with the increasing role of Artificial Intelligence?  Where will this evolve?  The sky is the limit.

One thing I am seeing with my clients is the increasing demand for DBAs to foresee issues and resolve them proactively.  Technology in this arena is evolving, and monitoring tools are becoming more predictive in their approach.  In the past, monitoring has largely been reactive and is even still mostly the case today.

Want to Be a Real Asset and Be Prepared for the Future?

The DBA needs to be able to see trends across applications, especially when working with different teams who may be facing the same challenges.  Often, teams are distributed and the DBA may be one of the few roles that can span across teams and bring institutional knowledge across the various teams.

As the role evolves, and databases become even more fault-tolerant, highly available, self-healing, and so on, the DBA must evolve to remain highly relevant.  When the DBA is highly-engaged from the initial development of the application, they can advise from the onset of schema design, query writing, indexing, and so on.  Unfortunately, many organizations still bring the DBA Teams in after the application goes to production and expects them to solve problems after it is too late to make app changes.  But, what else can DBAs do?

I believe DBAs need to be able to see the bigger picture of the database and be able to work more as architects.  If this is done, DBAs will be able to predict bottlenecks in the database layer and drive solutions to resolve them – before the code is even written.  As it stands today, few DBAs have the luxury of looking into the future and are instead consumed with only the immediate present; but, I am not sure this will always be the case.  After all, who knows the database layer better than the database experts?  Why, then, do more organizations allow developers to make all the decisions on database deployment when that is not their primary forte?

Again, I think the evolution of the DBA role will facilitate a shift in focus as time is freed up from mundane tasks that become more highly automated.  Are you able to drive this kind of change in your organization?  Are you looking this far forward and seeing how your role will evolve?  I think you should.  As more DBA responsibilities become consumed by the cloud, we need to take on different ones.  I don’t foresee the DBA role going away, but we need to always be ready to adapt as technology advances.  If we don’t, we may be left behind.  The need for a database expert will likely be there for the foreseeable future, but it may not always look the same; at least that is what I am seeing in the market.

What do you think? Let us know in the comments!

In this blog, I want to evaluate Group Replication Scaling capabilities in cases when we increase the number of nodes and increase user connections.

For testing, I will deploy multi-node bare metal servers, where each node and client are dedicated to an individual server and connected between themselves by a 10Gb network.

Also, I will use 3-nodes and 5-nodes Group Replication setup.

Hardware specifications:

System | Supermicro; SYS-F619P2-RTN; v0123456789 (Other)
Service Tag | S292592X0110239C
   Platform | Linux
    Release | Ubuntu 18.04.4 LTS (bionic)
     Kernel | 5.3.0-42-generic
Architecture | CPU = 64-bit, OS = 64-bit
  Threading | NPTL 2.27
    SELinux | No SELinux detected
Virtualized | No virtualization detected
# Processor ##################################################
 Processors | physical = 2, cores = 40, virtual = 80, hyperthreading = yes
     Models | 80xIntel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
     Caches | 80x28160 KB
# Memory #####################################################
      Total | 187.6G

For the benchmark, I use sysbench-tpcc 1000W prepared database as:

./tpcc.lua --mysql-host=172.16.0.11 --mysql-user=sbtest --mysql-password=sbtest --mysql-db=sbtest --time=300 --threads=64 --report-interval=1 --tables=10 --scale=100 --db-driver=mysql --use_fk=0 --force_pk=1 --trx_level=RC prepare

The configs, scripts, and raw results are available on our GitHub.

The workload is “in-memory,” that is, data (about 100GB) should fit into innodb_buffer_pool (also 100GB).

For the MySQL version, I use MySQL 8.0.19.

Results

Let’s review the results I’ve got. First, let’s take a look at how performance changes when we increase user threads from 1 to 256 for 3 nodes.

Interesting to see how the results become unstable when we increase the number of threads. To view it in more detail, let’s draw the chart with the individual scales for each set of threads:

As we can see, there are a lot of variations for threads starting with 64. Let’s check 64 and 128 threads with a 1-sec resolution.

It looks like there are cyclical processes going on, with periodic drops to 0. It seems like it is related to this bug.

3 nodes vs. 5 nodes

Now let’s check the performance under 5 nodes (comparing to 3 nodes)

There does not seem to be a huge difference; only when there are stable results with 8-16 threads, we can see a decline for 5 nodes. For threads 64 to 256, when the variance is prevailing, it is hard to notice the difference.

Conclusions

From my findings, it seems that Group Replication handles extra nodes quite well in this workload, but the multiple threads are problematic.

I am open to suggestions on how the performance of multiple threads can be improved.

March is not only the month when a pandemic closed all borders and people had to stay home on the quarantine; it’s also the month when daylight saving time change happens. For some regions, this is not only a time change but also a switch to a different timezone. For example, New York uses EST during winter and EDT during summer. If you use the system timezone and do not restart the MySQL server or a PXC node after the switch, you may notice that the change was not implemented.

You may end up with a situation where some of your cluster nodes still use the time zone before the switch (e.g. EST) and others use the timezone after the change (e.g. EDT).

$ date
Sun Mar  8 03:03:28 EDT 2020

$ ./bin/mysql  -h127.0.0.1 -P3373 -uroot test
...
EDT node> show variables like '%zone%';
+------------------+--------+
| Variable_name    | Value  |
+------------------+--------+
| system_time_zone | EDT    |
| time_zone        | SYSTEM |
+------------------+--------+
2 rows in set (0.00 sec)

$ ./bin/mysql  -h127.0.0.1 -P3372 -uroot test
...
EST node> show variables like '%zone%';
+------------------+--------+
| Variable_name    | Value  |
+------------------+--------+
| system_time_zone | EST    |
| time_zone        | SYSTEM |
+------------------+--------+
2 rows in set (0.00 sec)

Should you worry about it?

Nope!

MySQL initiates the

system_time_zone

To demonstrate this, let’s look at a simple table holding timestamp values:

CREATE TABLE `t1` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB

First, let’s check existing data; it is the same no matter which value of the

system_time_zone

EDT node> select * from t1;
+----+---------------------+
| id | ts                  |
+----+---------------------+
|  2 | 2020-03-08 01:03:53 |
|  4 | 2020-03-08 01:03:54 |
|  6 | 2020-03-08 01:03:55 |
+----+---------------------+
3 rows in set (0.00 sec)

EST node> select * from t1;
+----+---------------------+
| id | ts                  |
+----+---------------------+
|  2 | 2020-03-08 01:03:53 |
|  4 | 2020-03-08 01:03:54 |
|  6 | 2020-03-08 01:03:55 |
+----+---------------------+
3 rows in set (0.00 sec)

If we add a new row, the valid timestamp will be inserted on both nodes:

EST node> insert into t1 values();
Query OK, 1 row affected (0.01 sec)

EST node> select * from t1;
+----+---------------------+
| id | ts                  |
+----+---------------------+
|  2 | 2020-03-08 01:03:53 |
|  4 | 2020-03-08 01:03:54 |
|  6 | 2020-03-08 01:03:55 |
|  8 | 2020-03-08 03:02:22 |
+----+---------------------+
4 rows in set (0.00 sec)

EDT node> select * from t1;
+----+---------------------+
| id | ts                  |
+----+---------------------+
|  2 | 2020-03-08 01:03:53 |
|  4 | 2020-03-08 01:03:54 |
|  6 | 2020-03-08 01:03:55 |
|  8 | 2020-03-08 03:02:22 |
+----+---------------------+
4 rows in set (0.00 sec)

As you can see, the row, inserted on the node, started before the timezone change, has the same value on both nodes.

The same happens if we insert a new row on the node, started after the time change:

EDT node> insert into t1 values();
Query OK, 1 row affected (0.01 sec)

EDT node> select * from t1;
+----+---------------------+
| id | ts                  |
+----+---------------------+
|  2 | 2020-03-08 01:03:53 |
|  4 | 2020-03-08 01:03:54 |
|  6 | 2020-03-08 01:03:55 |
|  8 | 2020-03-08 03:02:22 |
|  9 | 2020-03-08 03:02:32 |
+----+---------------------+
5 rows in set (0.00 sec)

EST node> select * from t1;
+----+---------------------+
| id | ts                  |
+----+---------------------+
|  2 | 2020-03-08 01:03:53 |
|  4 | 2020-03-08 01:03:54 |
|  6 | 2020-03-08 01:03:55 |
|  8 | 2020-03-08 03:02:22 |
|  9 | 2020-03-08 03:02:32 |
+----+---------------------+
5 rows in set (0.00 sec)

Conclusion

If you use

time_zone=SYSTEM

system_time_zone

In this blog, I want to evaluate Group Replication Scaling capabilities to handle several writers, that is, when the read-write connection is established to multiple nodes, and in this case, two nodes. This setup is identical to my previous post, Evaluating Group Replication Scaling Capabilities in MySQL.

For this test, I deploy multi-node bare metal servers, where each node and client are dedicated to an individual server and connected between themselves by a 10Gb network.

I use the 3-nodes Group Replication setup.

Hardware specifications:

System | Supermicro; SYS-F619P2-RTN; v0123456789 (Other)
Service Tag | S292592X0110239C
   Platform | Linux
    Release | Ubuntu 18.04.4 LTS (bionic)
     Kernel | 5.3.0-42-generic
Architecture | CPU = 64-bit, OS = 64-bit
  Threading | NPTL 2.27
    SELinux | No SELinux detected
Virtualized | No virtualization detected
# Processor ##################################################
 Processors | physical = 2, cores = 40, virtual = 80, hyperthreading = yes
     Models | 80xIntel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
     Caches | 80x28160 KB
# Memory #####################################################
      Total | 187.6G

For the benchmark, I use sysbench-tpcc 1000W prepared database as:

./tpcc.lua --mysql-host=172.16.0.11 --mysql-user=sbtest --mysql-password=sbtest --mysql-db=sbtest --time=300 --threads=64 --report-interval=1 --tables=10 --scale=100 --db-driver=mysql --use_fk=0 --force_pk=1 --trx_level=RC prepare

The configs, scripts, and raw results are available on our GitHub.

The workload is “in-memory,” that is, data (about 100GB) should fit into innodb_buffer_pool (also 100GB).

For the MySQL version, I use MySQL 8.0.19.

I use the following command line:

./tpcc.lua --mysql-host=172.16.0.12,172.16.0.13 --mysql-user=sbtest --mysql-password=sbtest --mysql-db=sbtest --time=$time --threads=$i --report-interval=1 --tables=10 --scale=100 --trx_level=
RR  --db-driver=mysql --report_csv=yes --mysql-ignore-errors=3100,3101,1213,1180 run

This establishes an active connection to TWO separate nodes in Group Replication.

To make sure that transactions do not work with stale data, I use

group_replication_consistency=’BEFORE’

Results

Let’s review the results I’ve got. First, let’s take a look at how performance changes when we increase user threads from 1 to 256 for 3 nodes.

Interesting to see how the throughput becomes unstable when we increase the number of threads. To see it in more detail, let’s draw the chart with the individual scales for each set of threads:

As we can see, there are a lot of variations for threads starting with 2. Let’s check the 8 and 64 threads with 1-sec resolution.

There are multiple periods when throughput is 0.

These are the results with 1-sec interval:

393
610.99
603.01
3140.97
4822.97
4865.93
2454.05
1038
939.99
1340.02
1549.01
1561
626
0
0
66
0
0
63
0
0
69
0
0
268
369
367
331
385
356
258
0
0

There is basically an 11-second long stall when the cluster could not handle transactions.

Conclusion

Unless I missed some tuning parameters which would improve the performance, it is hard to recommend a multi-writer setup based on Group Replication. Though Multi-primary mode is disabled by default in MySQL Group Replication, you should not try to enable this.

We are excited to announce Percona Live ONLINE, a stand-alone virtual event we’re bringing to the community. It’s totally free of charge, and available for everyone to attend, no matter where you are in the world!

Percona Live ONLINE is not intended to replace our in-person Percona Live conferences in Europe and North America. We’re still on track to bring you Percona Live Europe in Amsterdam, October 19-21, 2020.

However, given the current circumstances, it seemed like a good time to introduce an online version of Percona Live to a wider audience. We’ll be featuring topics and speakers showcasing the latest trends in open source database technology.

The conference will be a true follow-the-sun event running for a full 24 hours kicking off at 10:00 am Eastern time on May 19, 2020, with some exciting keynote speakers. This allows users in EVERY timezone to be able to participate — like a hackathon, but for database content! Look for the full agenda to be released next week.

How Are We Selecting the Speakers?

Normally we’d run a call for papers but as we’d already gone through a selection process for our Austin event, we’re pleased that many of the speakers we have reached out to have said that they would be willing to support this great new online initiative.

With just a single track planned for Percona Live ONLINE, we’re currently working on creating a balanced agenda that will embrace MySQL, MongoDB, MariaDB, PostgreSQL, Kubernetes, cloud… and more!

There will be a limited number of speaking spaces so if you don’t receive an invite the first round, we will look at a second event and hope to get those speakers on the agenda then!

Will We Be Featuring Sponsors for Percona Live ONLINE?

The companies that sponsor the in-person Percona Live events enable us to deliver conferences of high standards in desirable locations. Without them, we simply could not present conferences to the standards our community has come to expect. In return, sponsors get an opportunity to engage with attendees and showcase their products and services. They know that they’ll get a great return on their investment!

Percona Live ONLINE is a new concept and heavily focused on the community aspect. Although we are confident that it will be a huge success, without known outcomes, we have decided not to solicit sponsorship for this event. We will revisit this for future events if we think that we can offer benefits to sponsors and participants alike.

How Can I Access the Conference?

We encourage you to pre-register to ensure you receive the latest information on the agenda, as well as streaming links for the conference. It is totally free to register and participate! Pre-register today!

Been There, Got the T-Shirt?

Many of those who have attended Percona Live conferences collect our unique event T-shirts.

Well, fear not! We will have a limited-edition t-shirt this year as well, but you’ll have to earn it! Make sure to follow the chat during the conference live streams to find out how to get one.

We’ll also be offering some cool prizes during the event. Stay tuned for more news!

Where Can I Interact With Other Online Attendees During the Conference?

We will have dedicated chat rooms and a live chat feed throughout the conference. Please pre-register so that we can send you details on how to access these.

Will I Be Able to Access Recordings If I Miss Some Sessions or Fall Asleep During the Event?!

Recordings of sessions will be made available shortly after the conference ends so you’ll be able to access them later. But honestly… we don’t think anyone is going to be sleeping!

We look forward to seeing you all virtually on May 19! Use #PerconaLive to stay up to date with the latest news and happenings!

Pre-Register Today!

The backup and restore of a complete database is an extensive exercise. But what if you need to restore just one table which has been mistakenly modified by an incorrect query? Help is at hand with Percona XtraBackup.

For our purpose, we will take a test database having tables created by the sysbench tool. The 8.0 versions of Percona XtraBackup and Percona Server for MySQL have been used in this test.

Restore Single Table

Here we will take the backup of the sbtest2 table and restore it. The initial checksum of the table is given below:

8.0.19>CHECKSUM TABLE sbtest2;
+--------------+-----------+
| Table        | Checksum |
+--------------+-----------+
| test.sbtest2 | 905286813 |
+--------------+-----------+
1 row in set (0.01 sec)

Take a backup of a single InnoDB table using the option: --tables

./xtrabackup --user=backup --password='Bspass!4%' --backup --tables=sbtest2 --target-dir=$HOME/dbbackup_PS8_table -S $HOME/PS130320_8_0_19_10_debug/socket.sock --datadir=$HOME/PS130320_8_0_19_10_debug/data

XtraBackup copies the table file sbtest2.ibd in the backup directory (dbbackup_PS8_table/test) along with other files required to prepare the backup.

You can also give patterns in the --tables option and XtraBackup will take backups of all tables matching the pattern. If there are many tables to be backed up, then these can be specified as a list in a text file with the option --tables-file. There is also an option to exclude tables using the --tables-exclude option.

Now prepare the backup with the extra --export option. This is a special option that will prepare the table configuration.

./xtrabackup --prepare --export --target-dir=$HOME/dbbackup_PS8_table

After preparation, the files sbtest2.ibd and sbtest.cfg are available in the backup directory. To restore this table, we have to first remove the existing tablespace from the database.

8.0.19>ALTER TABLE sbtest2 DISCARD TABLESPACE;
Query OK, 0 rows affected (0.20 sec)

Now copy the table files from the backup directory (dbbackup_PS8_table/test/sbtest2.*) to Percona Server data directory (PS130320_8_0_19_10_debug/data/test). 

Note: Before copying the files, disable selinux. After the files are copied, change the ownership of the copied files to the mysql user, if the backup user is different.

Finally, import the tablespace.

8.0.19>ALTER TABLE sbtest2 IMPORT TABLESPACE;
Query OK, 0 rows affected (1.12 sec)

The checksum of the table after the restore is:

8.0.19>CHECKSUM TABLE sbtest2;
+--------------+-----------+
| Table        | Checksum |
+--------------+-----------+
| test.sbtest2 | 905286813 |
+--------------+-----------+
1 row in set (0.02 sec)

The table is restored successfully.

Another method is to take the backup of the whole database and use it to restore one or more tables. Here, backup is simply done using the--backup option.

./xtrabackup --user=backup --password='Bspass!4%' --backup --target-dir=$HOME/dbbackup_PS8 -S $HOME/PS130320_8_0_19_10_debug/socket.sock --datadir=$HOME/PS130320_8_0_19_10_debug/data

Prepare the backup using--export option.

./xtrabackup --prepare --export --target_dir=$HOME/dbbackup_PS8

Next, discard the table’s tablespace, copy the table files from backup directory to the Percona Server data directory, and import the tablespace.

For MyISAM tables, the backup and prepare process is the same as above, and the only difference is that the table needs to be dropped and then restored using the IMPORT TABLE statement.

Restore Entire Schema/Database

We can take the backup of a database schema and restore it using the same process as described above.

Take a backup of the database using the --databases option.

./xtrabackup --user=backup --password='Bspass!4%' --backup --databases=test --target-dir=$HOME/dbbackup_PS8_db -S $HOME/PS130320_8_0_19_10_debug/socket.sock --datadir=$HOME/PS130320_8_0_19_10_debug/data

For more than one database, specify the databases as a list, such as --databases="db1 db2 db3". The databases can also be specified in a text file and used with the option --databases-file. To exclude a database from the backup, use the option --databases-exclude.

Prepare the backup using --export option.

./xtrabackup --prepare --export --target-dir=$HOME/dbbackup_PS8_db

Now remove the tablespace of all InnoDB tables in the database using ALTER TABLE <table name> DISCARD TABLESPACE.

Copy all table files from the backup dir (dbbackup_PS8_db/test/*) to the mysql data dir(PS130320_8_0_19_10_debug/data/test).

Note: Before copying the files, disable selinux. After the files are copied change the ownership of the copied files to mysql user if the backup user is different.

Finally, restore the tables using ALTER TABLE <table name> IMPORT TABLESPACE;.

This will restore the tables to the time of the backup. For a point in time recovery, binlogs can be further applied to the database, though care should be taken to apply only those transactions which affect the tables being restored.

The advantage of using this method is that the database server need not be stopped. A slight disadvantage is that each table needs to be restored individually, though it can be overcome with the help of a script.

Conclusion

It is easy to backup and restore a table or a database using Percona XtraBackup, by using just a few commands.