While we can act on the advice of the WHO and our governments, there’s no doubt that COVID-19 leaves many of us feeling pretty helpless. And while it might seem a small personal contribution, as a community we can lend our computing power to Folding@home, an open source project.

Percona is happy to promote this initiative and add in our guidance on how you can easily contribute to this research initiative.

What is Folding@Home? What Are They Trying to Do?

Folding@home (F@h) is simulating the dynamics of COVID-19 proteins to hunt for new therapeutic opportunities. To do this, they need a lot of CPU power. They have created an internet-based public volunteer computing project, similar in concept to SETI@Home. All outcomes of F@h are open source. If you want to learn more, take a look:

Coronavirus – What we’re doing and how you can help in simple terms

GitHub: FoldingAtHome

How Can You Help?

You can help by sharing part of your CPU for performing computing of their Working Units (WU).

Your computer periodically requests protein data to be analyzed, and the servers respond by assigning a Work Unit to the FAH client and securely deliver it via the internet to your computer. When the client completes its assignment, it returns its results to the Stanford servers and more work will be assigned. Your computer may complete a Work Unit in a few hours, or a few days, depending on a number of factors.

How Do You Do That?

Below, we have detailed how we set it up at Percona. You are very welcome to join the Percona group at F@h, team number 241445. Or, you can set up a new team.

Linux:

We prepared a Docker container for our users. This isolates the F@h client from the rest of the computer, so it can’t access any data outside the container (make sure that you are running the latest version of Docker).

Windows and Mac Users:

You should also use Docker if you are running on a corporate-owned computer. It’s fine, however, to install the client on any personal computer.

Corporate Computer Users:

Note: Check with your security team to confirm you are allowed to run this on your corporate computer. DO NOT RUN THIS WITHOUT THEIR EXPLICIT PERMISSION.

This example uses the Percona Team ID, 241445. Download an F@h client wrapped in a Docker container, then spawn an instance by executing:

docker run -p 7396:7396 covperc/fah:7.5.1 --gpu=false --smp=true --user=your_name --passkey=your_password --team=241445

The user and passkey are needed only to identify you, and your account is created after fetching the first sample. If you prefer, you can omit those and be an anonymous contributor:

docker run -p 7396:7396 covperc/fah:7.5.1 --gpu=false --smp=true --team=241445

It will also enable a web panel on 7396/tcp port.

Personal Computer Users:

Windows:

Windows Installer

Windows Instructions

Mac:

Mac Installer

Mac Instructions

Both Windows and Mac:

Make sure that you have selected ‘Any diseases’ which means COVID-19 (as it has the highest priority).

How Do I Know I’m Working on a COVID-19 Related Project?

If you are running one of these projects, then you are helping to fight COVID-19 :

• CPU: 14328 – 14329 – 14530 – 14531
• GPU: 11741 to 11764

Getting a Lot of Errors?

A couple of examples:

13:15:35:WARNING:WU00:FS00:Failed to get assignment from '18.218.241.186:80': No WUs available for this configuration
13:15:35:ERROR:WU00:FS00:Exception: Could not get an assignment

That’s fine! It just means that currently there are no new samples for you. Just keep the process running in the background. It will eventually fetch a new WU and the process will restart.

This is an unprecedented event, and we hope it will soon get better, but in the meantime, thanks for all you do to help — even if it seems small, it all makes a difference!

Recently, a client of ours, Duolingo, was using Aurora was reaching the max connection limit of 16,000 (that is Aurora’s hard limit for all instance classes). In this case, Percona recommended implementing ProxySQL to manage connections (now their max connections only peak to 6000!).

Duolingo’s main application is run in AWS ECS (Elastic Container Service) so they decided to attach ProxySQL as a sidecar container to the main application container.

What is a Sidecar?

The sidecar design concept consists of a main application container plus a second container running a service to support, enhance, or extend the main container. The second container is “attached” to the main container, much like a sidecar attaches to a motorcycle.

The most common example you’ll likely find is a web server plus a log forwarding service (pictured above). The web server is the main container, the log forwarder is the sidecar, and in this case they actually share data (the volume is mounted to both containers). The web server writes the data, while the log forwarder just reads and sends it along (perhaps to some aggregation service like Splunk).

Sidecars don’t have to just be logs. It could be for monitoring or even a MySQL proxy…

Launch A ProxySQL Sidecar

Let’s set up a WordPress container (our main container) and a ProxySQL 2 sidecar with an Aurora cluster backend.

Prerequisites

• Docker and Docker Compose installed.
• An Aurora cluster (I used 2.07.0) with at least one writer and one reader instance.
• A ProxySQL “monitor” user and “wordpress” user created on Aurora.

mysql> CREATE USER 'monitor'@'%' IDENTIFIED BY 'MonPass123^';
mysql> GRANT REPLICATION CLIENT ON *.* TO 'monitor'@'%';

mysql> CREATE USER 'wordpress'@'%' IDENTIFIED BY 'WPPass123^';
mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE ON wordpress.* TO 'wordpress'@'%';

• A “wordpress” database created on Aurora.

mysql> CREATE DATABASE wordpress;

Configuration

1. Create a project directory and change into it.

$ mkdir wordpress ; cd $_

2. Create a Dockerfile to build our sidecar.

• This will install ProxySQL2 from ProxySQL repositories and copy our (soon to be created) ProxySQL config file from our host into the image.
• Note: ProxySQL v1.4.x cannot detect Aurora failovers so v2.0.x is required.

$ vim Dockerfile

FROM debian:buster-slim

RUN apt-get update ; apt-get -y install lsb-release gnupg2 curl

RUN curl -L 'https://repo.proxysql.com/ProxySQL/repo_pub_key' | apt-key add - && echo deb https://repo.proxysql.com/ProxySQL/proxysql-2.0.x/$(lsb_release -sc)/ ./ | tee /etc/apt/sources.list.d/proxysql.list

RUN apt-get update ; apt-get -y install proxysql

COPY proxysql.cnf /etc/proxysql.cnf

3. Create our ProxySQL configuration file.

• When adding your nodes to the mysql_servers, you must use the instance endpoint and not the Aurora reader/writer VIPs.
• The reason is after a failover ProxySQL will move the individual instances around within the hostgroups to direct read/write traffic. It determines an instance’s role itself based on innodb_read_only and does not need to wait for Aurora to update the VIPs.
• This is advantageous. Our client reported failovers taking as little as 15 seconds verse 30 – 60 seconds for Aurora’s DNS failovers.

$ vim proxysql.cnf

datadir="/var/lib/proxysql"
errorlog="/var/lib/proxysql/proxysql.log"

admin_variables={
admin_credentials="admin:admin"
mysql_ifaces="0.0.0.0:6032"
}

mysql_variables={
monitor_username="monitor"
monitor_password="MonPass123^"
monitor_read_only_timeout=60000
monitor_connect_timeout=60000
monitor_ping_timeout=60000
mysql_ping_timeout_server=500
}

mysql_servers=({
hostname="my-aurora-instance-endpoint-1.zzzzg1gfxyo9.us-west-2.rds.amazonaws.com"
port=3306
hostgroup=1
},{
hostname="my-aurora-instance-endpoint-2.zzzzg1gfxyo9.us-west-2.rds.amazonaws.com"
port=3306
hostgroup=2
})

mysql_users=({
username="wordpress"
password="WPPass123^"
default_hostgroup=1
})

mysql_replication_hostgroups=({
writer_hostgroup=1
reader_hostgroup=2
check_type="innodb_read_only"
comment="Aurora Wordpress"
})

4. Create our Docker Compose file to bring it all together.

• This calls the “Dockerfile” to build our ProxySQL image, download the latest WordPress image, and start the containers.
• WordPress depends_on ProxySQL. This instructs Docker Compose to start ProxySQL first. If WordPress launched first, it could not connect to Aurora.
• We are using Docker’s internal naming to define the database endpoint. It’s simply the ProxySQL container name – WORDPRESS_DB_HOST: proxysql:6033
• Take a look at the volumes mounted to each container. It is not required in this case, but I mounted the ProxySQL volume to the WordPress container as well.

$ vim docker-compose.yml

version: '3'

services:
   wordpress:
     image: wordpress:latest
     container_name: wordpress
     depends_on:
       - proxysql
     ports:
       - "8080:80"
     volumes:
       - wordpress:/var/www/html
       - proxysql:/var/lib/proxysql
     environment:
       WORDPRESS_DB_HOST: proxysql:6033
       WORDPRESS_DB_USER: wordpress
       WORDPRESS_DB_PASSWORD: WPPass123^
       WORDPRESS_DB_NAME: wordpress
     restart: always

   proxysql:
     build: .
     container_name: proxysql
     volumes:
       - proxysql:/var/lib/proxysql
     restart: always
     command: ["proxysql", "-f", "-D", "/var/lib/proxysql", "-c", "/etc/proxysql.cnf"]

volumes:
    wordpress:
    proxysql:

5. Now we’re ready to bring up the containers. This will take several minutes as Docker needs to download and build our images.

$ docker-compose up -d
...
Creating network "wordpress_default" with the default driver
Creating proxysql ... done
Creating wordpress ... done

Review

Once finished, check that the containers are up.

$ docker ps -a
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                  NAMES
79babc5c1c0e        wordpress:latest     "docker-entrypoint.s…"   23 seconds ago      Up 21 seconds       0.0.0.0:8080->80/tcp   wordpress
d8ea54cc02e5        wordpress_proxysql   "proxysql -f -D /var…"   24 seconds ago      Up 22 seconds                              proxysql

Then visit your WordPress site! http://127.0.0.1:8080/

Summary

The sidecar design pattern is an easy solution to SPOF (single point of failure) because each application container gets its very own ProxySQL. For applications that are already containerized this is a simple and effective approach. Using Kubernetes? You can launch a sidecar container into the same pod as your application container.

Lastly, there is an official ProxySQL Docker image on Docker Hub you can pull instead of building your own!

Duolingo is the most popular language-learning platform worldwide, with over 300 million total learners. Their mission is to make education free and accessible to everyone. They offer 94 total courses across 38 distinct languages, from the world’s top most-spoken languages to endangered languages like Hawaiian, Navajo, and Scottish Gaelic. Duolingo is available on iOS, Android and web at www.duolingo.com.

Percona Monitoring and Management 2.4.0 (PMM), released several days ago, added support for custom extensions of the Prometheus configuration file. Before explaining what that is and how to use it, let me tell you a bit of history.

PMM 1.4.0, released 2.5 years ago, added support for hooking external Prometheus exporters into PMM’s Prometheus configuration file. That was quite tricky to implement. Back then, information about monitoring services was stored in Consul, and the Prometheus configuration file prometheus.yml contained consul_sd_config section for using it. We could not use the same mechanism for storing external exporters configuration as it wasn’t flexible enough – for example, there was no place to save custom scrape intervals or HTTP basic authentication credentials. At the same time, we could not move information about monitoring services out of Consul because pmm-admin relied on it being there. Then some customers depended on manual prometheus.yml edits that should be preserved across PMM Server updates, including comments and their placements, a feature that wasn’t supported by our YAML parser library. In the end, we decided to add and remove scrape_configs sections for external exporters via a simple regexp-based text parser and templater. As you can imagine, patching a YAML configuration file with regular expressions is not the most stable approach; in fact, we broke it once just by replacing tabs with spaces. So, while designing PMM 2.0, we decided that the Prometheus configuration will be fully managed by PMM Server’s component called pmm-managed; its database (called Inventory database) is the single source of truth, and manual edits are not supported to avoid the abovementioned problems.

Prometheus Configuration Extensibility

Of course, we understood that Prometheus configuration extensibility is a very desirable feature for many of our users, and we are happy to announce the first iteration of it. While generating prometheus.yml file, pmm-managed now checks if there is a file with a fixed path /srv/prometheus/prometheus.base.yml. If it can be read and parsed, most information from it is preserved and used in the generated configuration: global section is preserved if present, scrape_configs and rule_files arrays are merged together, and alerting, remote_write, and remote_read settings are preserved as-is. For example, it allows one to use long-term storage for Prometheus data like VictoriaMetrics by adding the following to the base file:

---
remote_write:
  - url: http://1.2.3.4:8428/api/v1/write

scrape_configs:
  - job_name: victoria-metrics
    honor_timestamps: true
    scrape_interval: 5s
    scrape_timeout: 4s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets:
          - "1.2.3.4:8428"
        labels:
          instance: pmm-server

Here we add VictoriaMetrics storage running on 1.2.3.4:8428 via remote_write. We also add monitoring for it. Then we should tell pmm-managed to generate a new combined file. The simplest way to do that is to use our change settings API with an empty body:

curl -X POST https://username:password@pmm-server/v1/Settings/Change

Normally, that API is used to update PMM Server configuration values like data retention interval, but it also causes pmm-managed to regenerate Prometheus configuration file, and reload it if needed. After running this command, you will notice a change in /etc/prometheus.yml.

Now you may wonder why I said that “most” configuration from the base file is preserved. While implementing this feature, we used Prometheus source code for configuration parser. Unfortunately, it is deeply tied to their service discovery mechanisms that include Kubernetes, and the Kubernetes dependencies list is bigger than the rest of PMM. So we decided to leave some things out of this release. Specifically, all service discovery mechanisms except static_configs are not supported. In the next release, we will use our own code for configuration parser. It is a separate Go library that can be useful for the community.

Oh, and what about re-introducing proper external exporters that can be added with pmm-admin? They are coming; I can’t comment on when exactly they will be shipped, but they are high in our priority list. prometheus.base.yml is only the first step.

Last week I wrote about How IT Can Scale and Manage Costs in an Economic Downturn. In that blog, I talked about ways to cut costs and prepare your systems for what could be next. With any massive world-changing event, there are going to be companies that have to scale down and weather the proverbial storm and try to wait out the current situation. Industries like airlines, travel, brick and mortar retail and entertainment (i.e. movie theaters, theme parks) are greatly impacted as business dwindles, but it is not the end for these industries — just a bump in the road.

This week I want to talk about the companies that are actually seeing an overwhelming increase in systems usage and traffic. During a crisis like the one we are experiencing, the shift in how people interact and behave has accelerated the usage of certain services and introduced people into new ways of working, living, and interacting. In many ways, it’s forcing the modernization of many jobs and people.

For example, my wife is a middle school science teacher. Over the years she has gone into school teaching six classes a day, going through each class one by one and reteaching the next class the same concepts the previous class heard. Sure, over the years new technologies were introduced to help with grading, teaching, etc. but largely, the foundations of standing in front of a class and teaching in-person have remained. Because of COVID-19, the school was forced to convert to all online and digital (at least for a portion of the year). This caused my wife and her fellow teachers to adapt and accelerate their usage and knowledge of technology. It also opened up a brand new set of problems, like how do you do science experiments online?

Some Industries are Experiencing Massive Growth

A shift is happening across the globe in every industry but some industries are seeing massive growth and an increase in usage, such as:

• Online chat, video, and collaboration tools: Think about how many people are now using Zoom, Bluejeans, Slack, MS Team, Mattermost, etc., to communicate. Video and chat technologies are becoming a critical component to support the world.
• Grocery and pharmacies: Are there busier or more vital physical spaces in the world right now?
• Online retailers and shipping companies: As the shift in shopping is forced online, these are taking off even more than ever.
• E-learning and online education: As more and more colleges and K-12 schools are forced to send kids home, e-learning has taken off.
• Streaming services: What would we all do without access to Netflix, Hulu, Disney, or Amazon Video?
• Gaming companies: We need to pass those weekends somehow, without leaving the house.
• Medical manufacturers, telemedicine, and pharmaceutical companies: Without a doubt, there are now critical needs here.

As these industries see exponential growth, we are starting to see new problems and challenges popping up that need to be solved. For example:

• Streaming services have had to slow down their services to prevent overloading the internet.
• Xbox live had multiple outages recently, impacting service.
• Epic Game’s Fortnite had outages as well.
• Robinhood had outages during a market crash.
• Microsoft teams had outages, affecting users in Europe.
• IBM Cloud had a partial mystery outage.
• Even the video app Zoom has had outages!

These problems are not new — last year, two out of five people experienced slowdowns that impacted production while one out of five reported unplanned outages — but the massive shift in routines and workloads is making these issues more frequent and with more impact than they were in the past.

With the Shifting Workload, Is Your Company Prepared for a Surge?

News is evolving and changing daily, affecting people’s attention and priorities. Masses of people are being forced to shift their routines in a matter of minutes or hours. That means many websites, e-tailers, applications, or services can see their traffic and applications go from 0 to 100 in milliseconds. As we have seen, many of these companies’ infrastructure will not able to handle the surge. It is often hard to quantify the impact of an outage, but in today’s world, a slowdown or outage may have life-altering impacts beyond what it was just three months ago.

Some important things to consider:

• Realize a slowdown feels like an outage to many people. Over the years, people have done a variety of studies on this, showing people now have less than the attention of a goldfish or that you will lose 50% of your traffic if your app is unresponsive for more then a second or two.
• Depending on what your application is, some people may consider your service vital. An outage can have cascading impacts on your business and reputation.
• Workloads are changing so rapidly you may not be prepared for the shift. Slack, for instance, grew 40% since the start of February and Microsoft teams more than doubled their active users since last year! Most companies don’t plan for doubling or tripling traffic in days or even in a few months. They may have dreamt of it, but in reality, a reasonable progression would have been more plausible — not today!

What can you do now?

If you are already seeing growth and opportunities in the current market, it’s important to ensure your systems are set up to be elastic with the ability to grow and shrink as the demand occurs. This is harder (but not impossible) if you have not set things up ahead of time. From the database perspective, ask yourself these questions:

• • Can you add or remove slaves or nodes from your cluster easily?
  • Are you spread across availability zones or providers to prevent one area or provider from causing you an outage?
  • Do you have tools set up to quickly find and fix issues as they occur?
  • Have you tested your systems under load?
  • Do you have a plan in place to test the performance of the new code before it makes it into production?
  • Do you have regular ongoing performance and scalability audits on your systems?
  • How often are you reviewing your workload for changes in the patterns?
  • When was the last time you tested your failover?
  • What happens if your hosting provider has an outage?
  • What happens if the database traffic doubles or triples overnight?

Hopefully, you have prepared as best you can, but even if you haven’t, it’s never too late to start testing, tuning, and optimizing your systems. There are plenty of great tools and companies that can help accelerate these efforts, including Percona. Our teams are here for you, and we’ll continue to keep you updated and informed. Contact us if you’d like to talk more about ways we can help scale and manage your databases in this new tech world.

How Has COVID-19 Impacted Your Company?

We want to know how your company has been impacted by COVID-19.

Over the past ten years, those in charge of running enterprise databases saw their focus shift from simply ensuring access and availability to overseeing architecture, design, and scalability.

Site reliability engineers (SREs) and/or DBAs suddenly found themselves responsible for not only ensuring the company’s primary revenue engine stayed up but also for ensuring it could scale effectively as the business grew.

This was a huge shift for traditional enterprise DBAs, who used to just be responsible for keeping the database up, keeping it backed up, and reacting to issues as they arose.

Today, enterprises look to capitalize on new revenue opportunities afforded by the shift towards cloud-based technologies. Specific database environment challenges and requirements arise as a result of this shift.

In this new world, application outages lead directly to lost revenue.

Even worse, outages can lead to customer loss, giving your competitors the chance to benefit from your misfortune. To keep your revenue flowing, every part of your company’s critical infrastructure needs to be planned out, redundancy should be built-in, and a future-proof architecture should be created, which can flex to scale when needed.

The more issues you can plan for before launch, the less chance of a catastrophic outage later on.

The Need for Database Expertise

When a company moves to the cloud, the cloud provider often takes care of much of the operational automation and mundane day-to-day tasks (for example, using database as a service (DBaaS) options such as Amazon RDS, Microsoft Azure, or Google Cloud).

However, contrary to popular belief, this service does not eliminate the need for dedicated database expertise. In fact, this skill set becomes even more important as someone needs to design and tune the database to support the application. They also need to understand how to build the modular pieces available in the cloud into a cohesive, scalable unit that meets the needs of the application and the company.

Having efficient database expertise in place has a high impact on the potential performance of your database and, as a result, clear ROI implications.

The Changing Role of the DBA

As the image above indicates, the role of the DBA has evolved. At Percona, we have been monitoring this shift in focus. Over 50% of our customer support tickets are now related to application design issues, query performance, or database infrastructure design.

However, when you consider the maturity of database products and the technological advances that impact databases, this makes sense.

More stable versions of open source databases such as MySQL, PostgreSQL, and MongoDB, coupled with advances in homegrown automation and cloud-based infrastructure, reduce the frequency of bugs that lead to crashes. Instead, outages and database issues are increasingly a result of unwise design decisions, bad code, or unforeseen edge cases. DBAs must adapt to keep up with technological advances and to have a proactive impact on the performance of their databases.

Utilizing Specialist Database Support

At Percona, we recognize and embrace the changing requirements of modern database deployments. We have been providing database expertise on SaaS and the cloud since inception. We also recognize that the needs of our clients who choose to run on a DBaaS platform are different to those managing their own full-stack database deployments.

To help our clients make the most of their DBaaS platforms, we created Percona DBaaS Support to ensure your business is getting the most out of your DBaaS service and cloud migration.

We have helped thousands of customers successfully deploy open source databases on-premises and in the cloud, achieving enhanced performance, cost savings, and improved ROI. That same deep database expertise is available when you deploy MySQL, MariaDB, MongoDB, or PostgreSQL on a managed DBaaS platform. We are an AWS Advanced Partner, and have partnerships with Google Cloud and Microsoft Azure.

You can still rely on your cloud provider for operational break/fix support, but you can choose to augment it with Percona’s proven world-class database expertise. Percona focuses specifically on database design, development, and tuning. These are areas typically not addressed by cloud providers.

Additionally, we also offer Percona Managed Database Services. This is a flexible, managed database service delivering exceptional enterprise-grade expertise across a variety of high-performance enterprise database environments.

You can use Percona’s expert DBAs to keep your cloud-based database running at peak performance, allowing your technical team to focus on core business activities. Percona Managed Database Services means you can be confident that your database performance is being proactively monitored around the clock. This helps reduce critical incidents and ensures your database is meeting business goals.

Finally, we are all currently encountering a challenging global situation. Many companies are starting to look at ways of reducing costs, trimming budgets, and minimizing the impact on their business. Our recent blog, How IT Can Scale and Manage Costs in an Economic Downturn, gives companies guidance on steps they might want to consider to mitigate the impact on their business.

Please contact us for details on how we can help you optimize your cloud database environments, ensuring they run as efficiently as possible to support the applications that drive your business success.

We recently helped migrate a production PostgreSQL database running on a given DBaaS platform… to another DBaaS platform. When it comes to vendor “lock-in”, some providers are easier (and even friendlier) to deal with than others, but it is never a straightforward process. While in a traditional environment we would usually approach this problem by creating a replica of the server and switching over database requests at the right moment, the situation complicates when it is not possible to create a direct replication flow outside the provider’s platform.

In this post, we’ll share the approach we used for this particular case. It is not intended to provide a step-by-step, one-size-fits-all solution, but we hope it might serve you as “inspiration” when dealing with a similar situation. You may find some parts of our approach more useful than others; consider the ones that suit your unique needs and be creative. As always, regardless of the plan you design, make sure you test all involved steps in a testing/staging environment first as in practice we often step into yet another stone we didn’t know was laying in our path.

Intermediating DBaaS Instances

When we cannot have direct replication between DBaaS instances, we can still attempt to “chain it” somehow through an intermediate player (read: server):

source DBaaS → Intermediate server → target DBaaS

In this particular case, the source DBaaS was producing daily base backups and streaming WAL segments to externally accessible storage, an S3 bucket, for archiving purposes. We took advantage of this to create an intermediate PostgreSQL server from the latest base backup available and configured it to consume the WAL segments available on this storage. Be mindful of where to install this intermediate server: you may want to install it either closer to the source DBaaS or yet closer to the target DBaaS instead. When making your choice, consider network bandwidth as well as the availability of common tools by the infrastructure provider.

However, this intermediate server was not an exact PostgreSQL replica: even though it had to operate with hot_standby enabled it would only be as up-to-date as the latest WAL file it had available to consume. And since it had to operate as a standby server we couldn’t have the target DBaaS replicate directly from it using logical replication; we would have to wait and only set it as a replica once the intermediate PostgreSQL server operating in standby mode had consumed all WAL files and was promoted as primary. To minimize application downtime, the trick here was to perform the migration in two steps: 

1. Bring the intermediate server as in sync with the source DBaaS as possible, stop all connections to the source DBaaS, allow the intermediate server to consume the latest WAL segments generated, promote it as primary, and finally direct database requests to it.
2. Configure the target DBaaS as a replica of the intermediate-now-turned-primary server using logical replication, wait until this new replica has caught up, promote the new replica as primary and failover production traffic to it.

We discuss each of these steps in more detail in the next sections.

Creating the Intermediate PostgreSQL Server

We started the process by installing a Percona Distribution for PostgreSQL instance in a separate server that could access both the S3 bucket and the target DBaaS instance. For this, we configured the server with the Percona repository and followed the installation instructions. We made sure to use the same major PostgreSQL release running on the source DBaaS instance and to have PostgreSQL configured accordingly, with custom operational settings (such as work_mem) replicated from the source DBaaS instance’s configurations while adjusting others (such as shared_buffers) to our intermediate server specs. Even though this server would be theoretically holding production traffic for only a brief amount of time you should make it at least as capable as the current production instance. Three other settings you should make sure to adjust correctly are:

• hot_standby = ‘on’
• wal_level = ‘logical’
• listen_address to either 0.0.0.0 or the IP address corresponding to the network interface that can reach the target DBaaS instance.

You should also make sure that any related firewalls allow communication between the parts, including pg_hba.conf files; the intermediate server must be able to access the target DBaaS instance and vice-versa.

Populating the Intermediate Server

We took advantage of having daily binary backups available in the S3 bucket and restored the latest one in the intermediate instance. For this, we used wal-e (though the more modern wal-g should work as well). Their Github page contains simple installation instructions using pip. We ended up using snap instead, which had the binary file installed under /snap/bin/wal-e. 

In order to safely store the S3 credentials, we put them under /etc/wal-e.d/env and used envdir (from the daemontools package) to pass them along to wal-e. To validate that we were able to access the storage we tried listing the available backups with the following command:

$ envdir /etc/wal-e.d/env /snap/bin/wal backup-list

We then proceeded with the first step on populating the intermediate server, which was to restore the latest available backup. Make sure PostgreSQL is stopped, make a copy of the configuration files (if necessary) and clean up the data directory (in this example, /var/lib/postgresql/11/main) before you proceed with the fetching of the backup:

$ envdir /etc/wal-e.d/env /snap/bin/wal-e backup-fetch --blind-restore /var/lib/postgresql/11/main/ LATEST

If the backup includes configuration files inside the data directory, you may need to adjust those. Pay special attention to the recovery.conf file: it should indicate the server will run in standby mode and how it can access WAL files to catch up with the primary, which in our case are also available in the S3 bucket. Here’s how our recovery.conf file looked like:

standby_mode = on
restore_command = 'envdir /etc/wal-e.d/env /snap/bin/wal-e wal-fetch %f %p'

With that file in place, you can now start PostgreSQL on the intermediate server again: considering all things go well, it should run and process all WAL files available until it has consumed them all, and then stop again. Check the PostgreSQL error log / syslog to certify that was the case.

Switching Over Traffic to the Intermediate Server

When the archive_timeout variable is set to a value greater than zero in the source DBaaS server it may trigger the creation of a new WAL file every archive_timeout seconds when the server is under low (write) activity and not naturally generating WAL segments during this period. This is a good hint to understand the rate at which WAL segments are produced and can thus be consumed by a standby server such as our intermediate PostgreSQL instance. When planning a maintenance window to switch over the traffic to the intermediate server, try to have it as in sync with the primary as possible, which you can accomplish by keeping the recovery.conf file in place and starting PostgreSQL again.

Once the time to switchover arrives, terminate all existing connections to the primary, make sure no new ones are created and observe the server activity for a few seconds. When in doubt, you may create some artificial load to force to produce new WAL segment(s), such as with:

create table xlog_switch as select '0123456789ABCDE' as col1 from generate_series(1,1000000);
drop table xlog_switch;

Start PostgreSQL one more time in the intermediate server with the recovery.conf file still in place to consume the latest WAL segments. When it stops, delete or rename the recovery.conf file and start it one last time. Make sure everything looks alright then promote is as primary:

$ pg_ctl -D /var/lib/postgresql/11/main promote

You can validate the promotion worked by checking the server is no longer in recovery mode:

select pg_is_in_recovery();

You can now redirect your database requests to the intermediate server.

Setting up the new primary as a publisher 

Logical replication in PostgreSQL uses a publish and subscribe model so you must configure a publisher covering all the tables in the database(s) you want to have replicated in the target DBaaS. This process may be as simple as connecting to the target database (percona in our example here) and creating a publisher (which we will call percpub):

\c percona
CREATE PUBLICATION percpub FOR ALL TABLES;

An important point regarding logical replication performance: there must be a way to uniquely identify each tuple to which changes are being applied, ideally through a Primary Key or Unique Key (NOT NULL). When a target table lack either of these, REPLICA IDENTITY will have to be set to FULL and the entire tuple composition will be used to identify it, which makes the comparison process more expensive, incurring higher overhead.

Configuring the Target DBaaS as a Replica

First, make sure that this instance is also configured with wal_level = ‘logical’ or you won’t see any replication taking place, which is not as evident a requirement as one would expect. The changing of this setting is accomplished in different ways according to the provider involved.

Besides tuning other PostgreSQL settings of the target DBaaS accordingly, create the target database and a user that can access it. Once that is done, you’ll be able to build the schema and schema objects of the target database by taking a pg_dump with the -s option (“dump only the schema, no data“) from the intermediate server and restoring it on the target DBaaS instance. This can be accomplished with a single command that encapsulates both actions, which should be run from the intermediate server:

$ pg_dump -h localhost -U postgres -d percona -p 5432 -Fc -s |pg_restore -d percona -h your.target.dbaas.com -p 5432 -U postgres

Make sure to adjust the values for database, hostname, and user both in the command above and in the one that follows below.

Once the schema is completed you can create the subscription:

CREATE SUBSCRIPTION percsub CONNECTION 'host=intermediate.server.com dbname=percona user=postgres password=secret port=5432' PUBLICATION percpub;

This should do the trick; the target DBaaS instance will start initial data copy and then sync all changes from the intermediate server. You may follow its advancements by querying:

select * from pg_stat_replication;

A couple of important considerations:

1. Beware that it may be difficult to create a logical replication slot on a busy, high-transaction server. You may want to do it immediately after promoting the intermediate server as master and before the application starts connecting to it. This cannot be done earlier because logical decoding is currently not possible on a standby server.
2. Logical replication does not replicate DDLs; make sure you freeze any changes in the database schema until you’ve completed the migration.

Final Switchover

Once the target DBaaS has caught up with the primary you can plan a final switchover by promoting this replica as a master and redirecting production traffic to it.

And that’s it: we were able to complete the migration following the above steps. We hope you will find them useful as well; you may let us know in the comments section below.

Last week I did a webinar on MySQL Troubleshooting and Performance Optimization with Percona Monitoring And Management v2 (PMM2).

There was a tremendous amount of interest and many more questions than I could answer, so I’m answering them in this blog post instead.

Q: What are the red and white dots on the last column in PMM Query Analytics?

This is a graphical visualization of the query response time variance; we see minimum query response time and maximum query response time as the ends of the orange line. The white line shows the average response time and the red dot estimates 99 percent response time. This feature allows us to visually see the variance of a given query execution; if the orange bar is short all query executions take a similar amount of time, and if it is long, as in this example, the variance is large. You can see exact numbers with mouseover (in this case, taking anywhere from less than 0.1ms to 55sec to execute – this is quite a range!).

Q: Where can I find information about the integration with ClickHouse?

Stay tuned. I’m planning to write a couple of blog posts about it.  For now, I think this webinar will help you to get started

Q: Can you send the queries you are using?

I’m planning to write a separate blog post about using Clickhouse directly with Percona Monitoring and Management and I will share queries in it.

Q:  Using PMM can we determine the optimal  Redo logfile size for performance?

Percona Monitoring and Management does not give advice on what optimal Redo Log Size is, but the “Innodb Details” dashboard is there to help you. In the very top the dashboard you will see this image:

If this number does not exceed 50-60 percent over the last 24 hours, you would not likely benefit from the larger redo log file. If you’re looking for more detailed information, check out “Innodb Checkpointing and Flushing” section in the same dashboard and you will see this graph:

If the orange line is pushing close to the red line, the Redo log file is too small for your workload to be run efficiently. I would strive for not exceeding 50 percent of your log file space use as a good Redo log size, but also keep in mind a larger Redo log file means longer crash recovery time.

Q: Can these views be set to a very recent time frame, like the most recent 5 minutes, or a recent hour?  And, how quickly are the newest data/measures refreshed?

If you look at the query information it is captured in one-minute intervals (buckets), and with additional propagation delay, I would expect up to two minutes lag. Operational metrics are often available with just a few seconds of delay. Here is a view showing Query Data for the last five minutes:

Q: Hi, is (there) a way to check the blocked queries and blocking queries during a window whether the read/writes went into lock state?

You can see how many queries are spent in the locked state but you do not know which query or queries have been causing that wait.

Showing which queries caused the row-level lock wait is not easy as often it may not be one query which is “guilty” but rather, many queries interacting. It may not be a “query” at all, but a transaction having locked rows waiting on the application side. This is an important problem and we will look into solving it in the future.

Q: Can you summarize what Redis/Memcache would look like (if at all) in PMM2?

Percona Monitoring and Management v2 only supports monitoring of MySQL (and MariaDB), PostgreSQL, and MongoDB at this point. Redis and Memcached monitoring are not currently supported. The plan is to enable adding support for their monitoring in PMM2 through the support of External Exporters as was supported in PMM v1. This feature is already in development. And in the more distant future, through some sort of extension/plugin interface allowing to implement query information capture for technologies not supported by PMM out of the box.

Q: How much load does PMM have on DB?

Q: Is this tool to be used on production servers? Or on staging/debug server?

Q: Any recommendation on not making it kill the instance or add too much load on the instance we are monitoring?

These similar questions I thought are best answered together. The design goal for PMM is to have no more than 1 percent overhead in normal circumstances, so you can safely run it in production.

Having said that, it is best practice to run Percona Monitoring and Management in production as well as in your development and staging QA environments.   If the development team can find bad queries, bad schema, or other mistakes before pushing them to production, it is the best outcome.

You never catch all bad queries though, and even if you do there are cases when “good queries” behave badly due to some problems with statistics or other operational issues. To catch these and resolve problems quickly you need to run PMM in production.

In some exceptional cases, PMM may generate more than the target 1 percent of the load. In this case, you can configure resolution, as lower resolution means lower load.

You also disable Table Statistics if you have a large number of tables. Run pmm-admin  add mysql –help for details. In general, “Do No Harm” is the primary objective for Percona Monitoring and Management, so if you’re observing PMM negatively impacting your database performance in any way, please let us know by filing a bug.

Q: Is it able to detect/recommend config changes to better tweak the parameters based on the server resources, ex.: cache, innodb_buffer_pool_*,?

Percona Monitoring and Management does not give you direct advice right now, but this is in the plans. All server variables (like innodb_buffer_pool) are regularly collected so you can often see if the value changes in various dashboards.

One of the “advanced” things you can do is to compare this value among different servers in the Advanced Data Exploration Dashboard:

In this case, I see three servers having a buffer pool size of 256MB and one only 32MB, for some reason. I can also compare the instances to see if there are any differences between them:

Q: How could one correlate running queries with deadlocks and IO operations? It looks like the stats can’t be grouped using custom rules.

This question is not completely clear to me, so I’ll just show what PMM2 can do. If you’re looking for top queries based on the lock waits or disk IO, you can do that:

Changing the Lead column to “InnoDB IO Read Wait” will show the queries which are the most commonly blocked on the disk IO.  You can also sort by amount for the read IO query is doing. We can’t attribute write IO directly to queries so it is not available.

Q: Is there a way to monitor what’s currently being executed in MySQL DB?  Also, SQLs that are executed in a particular time frame (like History Data)?

Right now, there is no feature to see currently running queries (like SHOW PROCESSLIST) but you can view the history of the queries for any period of time. Just use the calendar in the top right corner:

Q: Where do we can find “Transaction History Length” details?

Look into the MySQL Innodb Details -> Innodb Undo Space and Purging section:

The highlight – maximum Transaction History Length is also available right at the top of that dashboard.

Q:  Is there documentation to upgrade from PMM 1.0 to 2.0?

There is no support for an in-place upgrade from PMM 1.0 to PMM 2.0. Instead, what you usually would do is set up PMM2 agent to run concurrently with PMM1 until the time you have enough data in PMM2 to switch over.

I understand it is not ideal but significant data structure changes in PMM2 made conversion rather impractical.

Q: Can we monitor MongoDB without having root access on the host to install client?

Yes.  You do not have to run the “agent” at the same host as the database, you can set the agent to monitor remote instance, but in this case, you will not have access to the Operating System level metrics, but only those which the database server provides. (This last limitation does not apply to Amazon RDS and Aurora monitoring due to CloudWatch integration).

Q: Is there still SSL restriction to monitor Postgres RDS in AWS?

No, this restriction has been lifted in PMM2.  Note, Amazon RDS for PostgreSQL does not yet support integration with CloudWatch metrics.  This feature will be added in the future PMM2 release.

Q: Is there anything that PMM can do while VividCortex can not?

This is a hard question to answer as both PMM and VividCortex are constantly adding new features. The most important difference is that you roll your own version of Percona Monitoring and Management, which means it can be used in secure environments where sending even sanitized queries into the cloud is not acceptable.  This also gives you much more cost controls.

Because PMM is open source and built on Grafana and Prometheus, there are many more extension possibilities for advanced users.

In-depth query analyses is another feature that we’re particularly proud of in PMM, for example, you can zoom in on queries which are run by a particular user from a particular hostname at a particular schema with PMM:

Q: Is it advisable to install PMM in the same server as the database and is it heavy?

In a typical production environment, it is not recommended, but in a development environment, it is quite a common setup. To monitor a single database server, PMM would need very little CPU (less than 5 percent of a single CPU core) and 1GB of RAM will be enough.

Q: It has so many graphs so which one is most helpful and how often should we be viewing PMM from a daily point of view?

Yes. The number of graphs in PMM can be intimidating; we built Percona Monitoring and Management in part for the Percona team to be able to resolve the most complicated problems, so it goes into a lot of detail.

The most important dashboards would be the Query Analytics and also dashboards available from “Home Dashboard”:

You can click on the CPU usage, for example, to see the system overview or database connections or QPS to get the MySQL overview dashboard. These should be enough to identify 90 percent of the most typical issues.

We are continuing our work on usability to make Percona Monitoring and Management simpler and easier to use.

Q: Any particular reason external:service are skipped in PMM2? We use them for other datastore monitoring and we’re stuck with PMM 1 because of this.

External exporter support is planned. PMM2 was a major re-write and more than a simple addition to PMM1, and some features have not been re-implemented yet. External exporters is one of such features.

Q:  Is there any documentation on how to create custom dashboards and are there any other dashboards which are already pre-made and can be imported into PMM if not shipped out of the box

Percona Monitoring and Management uses Grafana Engine for dashboards so you can use Grafana Documentation for creating dashboards in PMM.

What You May Not Know About Random Number Generation in Sysbench

Sysbench is a well known and largely used tool to perform benchmarking. Originally written by Peter Zaitsev in early 2000, it has become a de facto standard when performing testing and benchmarking. Nowadays it is maintained by Alexey Kopytov and can be found in Github at https://github.com/akopytov/sysbench.

What I have noticed though, is that while widely-used, some aspects of sysbench are not really familiar to many. For instance, the easy way to expand/modify the MySQL tests is using the lua extension, or the embedded way it handles the random number generation. 

Why This Article? 

I wrote this article with the intent to show how easy it can be to customize sysbench to make it what you need. There are many different ways to extend sysbench use, and one of these is through proper tuning of the random IDs generation. 

By default, sysbench comes with five different methods to generate random numbers. But very often, (in fact, almost all the time), none is explicitly defined, and even more rare is seeing some parametrization when the method allows it.

If you wonder “Why should I care? Most of the time defaults are good”, well, this blog post is intended to help you understand why this may be not true.

Let us start.

What methods do we have in sysbench to generate numbers? Currently the following are implemented and you can easily check them invoking the –help option in sysbench:

• Special 
• Gaussian
• Pareto
• Zipfian 
• Uniform

Of them, Special is the default with the following parameters:

• rand-spec-iter=12   – number of iterations for the special distribution [12]
• rand-spec-pct=1    – percentage of the entire range where ‘special’ values will fall in the special distribution [1]
• rand-spec-res=75    – percentage of ‘special’ values to use for the special distribution [75]

Given I like to have simple and easy reproducible tests and scenarios, all the following data has being collected using the sysbench commands:

• sysbench ./src/lua/oltp_read.lua –mysql_storage_engine=innodb –db-driver=mysql –tables=10 –table_size=100 prepare
• sysbench ./src/lua/oltp_read_write.lua –db-driver=mysql –tables=10 –table_size=100   –skip_trx=off –report-interval=1 –mysql-ignore-errors=all –mysql_storage_engine=innodb –auto_inc=on –histogram –stats_format=csv –db-ps-mode=disable –threads=10 –time=60  –rand-type=XXX run

Feel free to play by yourself with script instruction and data here (https://github.com/Tusamarco/blogs/tree/master/sysbench_random).

What is sysbench doing with the random number generator? Well, one of the ways it is used is to generate the IDs to be used in the query generation. So for instance in our case, it will look for numbers between 1 and 100, given we have 10 tables with 100 rows each.

What will happen if I run the sysbench RUN command as above, and change only the random  –rand-type? 

I have run the script and used the general log to collect/parse the generated IDs and count their frequencies, and here we go:

Special
Uniform
Zipfian
Pareto
Gaussian

Makes a lot of sense right? Sysbench is, in the end, doing exactly what we were expecting.

Let us check one by one and do some reasoning around them.

Special

The default is Special, so whenever you DO NOT specify a random-type to use, sysbench will use special. What special does is to use a very, very limited number of IDs for the query operations. Here we can actually say it will mainly use IDs 50-51 and very sporadically a set between 44-56, and the others are practically irrelevant. Please note, the values chosen are in the middle range of the available set 1-100.

In this case, the spike is focused on two IDs representing 2 percent of the sample. If I increase the number of records to one million, the spike still exists and is focused on 7493, which is 0.74% of the sample. Given that’s even more restrictive, the number of pages will probably be more than one.  

Uniform

As declared by the name, if we use Uniform, all the values are going to be used for the IDs and the distribution will be … Uniform.

Zipfian

The Zipf distribution, sometimes referred to as the zeta distribution, is a discrete distribution commonly used in linguistics, insurance, and the modeling of rare events. In this case, sysbench will use a set of numbers starting from the lower (1) and reducing the frequency in a very fast way while moving towards bigger numbers. 

Pareto  

With Pareto that applies the rule of 80-20 (read https://en.wikipedia.org/wiki/Pareto_distribution), the IDs we will use are even less distributed and more concentrated in a small segment. 52 percent of all IDs used were using the number 1, while 73 percent of IDs used were in the first 10 numbers.  

Gaussian   

Gaussian distribution (or normal distribution) is well known and familiar (see https://en.wikipedia.org/wiki/Normal_distribution) and mostly used in statistics and prediction around a central factor. In this case, the used IDs are distributed in a bell curve starting from the mid-value and slowly decreases towards the edges. 

The Point Now Is… What For?

Each one of the above cases represents something, and if we want to group them we can say that Pareto and Special can be focused on hot-spots. In that case, an application is using the same page/data over and over. This can be fine, but we need to know what we are doing and be sure we do not end up there by mistake.

For instance, IF we are testing the efficiency of InnoDB page compression in read, we should avoid using the Special or Pareto default, which means we must change sysbench defaults. This is in case we have a dataset of 1Tb and bufferpool of 30Gb, and we query over and over the same page. That page was already read from the disk-uncompressed-available in memory.

In short, our test is a waste of time/effort. 

Same if we need to check the efficiency in writing. Writing the same page over and over is not a good way to go.

What about testing the performance? 

Well again, are we looking to identify the performance, and against what case? It is important to understand that using a different random-type WILL impact your test dramatically. So your “defaults should be good enough” may be totally wrong.

The following graphs represent differences existing when changing ONLY the rand-type value, test type, time, additional option, and the number of threads are exactly the same.

Latency differs significantly from type to type:

Here I was doing read and write, and data comes from the Performance Schema query by sys schema (sys.schema_table_statistics). As expected, Pareto and Special are taking much longer than the others given the system (MySQL-InnoDB) is artificially suffering for contention on one hot spot. 

Changing the rand-type affects not only latency but also the number of processed rows, as reported by the performance schema.

Given all the above, it is important to classify what we are trying to determine, and what we are testing. 

If my scope is to test the performance of a system, at all levels, I may prefer to use Uniform, which will equally stress the dataset/DB Server/System and will have more chances to read/load/write all over the place.

If my scope is to identify how to deal with hot-spots, then probably Pareto and Special are the right choices. 

But when doing that, do not go blind with the defaults. Defaults may be good, but they are probably recreating edge cases. That is my personal experience, and in that case, you can use the parameters to tune it properly.

For instance, you may still want to have sysbench hammering using the values in the middle, but you want to relax the interval so that it will not look like a spike (Special-default) but also not a bell curve (Gaussian). 

You can customize Special and have something like :

In this case, the IDs are still grouped and we still have possible contention, but less impact by a single hot-spot, so the range of possible contention is now on a set of IDs that can be on multiple pages, depending on the number of records by page.

Another possible test case is based on Partitioning. If, for instance, you want to test how your system will work with partitions and focus on the latest live data while archiving the old one, what can you do? 

Easy! Remember the graph of the Pareto distribution? You can modify that as well to fit your needs. 

Just tuning the –rand-pareto value, you can easily achieve exactly what you were looking for and have sysbench focus the queries on the higher values of the IDs.

Zipfian can also be tuned, and while you cannot obtain an inversion as with Pareto, you can easily get from spiking on one value to equally distributed scenarios. A good example is the following:

The last thing to keep in mind, and it looks to me that I am stating the obvious but better to say that than omit it, is that while you change the random specific parameters, the performance will also change.

See latency details:

Here you can see in green the modified values compared with the original in blue.

Conclusion

At this point, you should have realized how easy it can be to adjust the way sysbench works/handles the random generation, and how effective it can be to match your needs. Keep in mind that what I have mentioned above is valid for any call like the following, such as when we use the sysbench.rand.default call:

local function get_id()

   return sysbench.rand.default(1, sysbench.opt.table_size)

End

Given that, do not just copy and paste strings from other people’s articles, think and understand what you need and how to achieve it. 

Before running your tests, check the random method/settings to see how it comes up and if it fits your needs. To make it simpler for me, I use this simple test (https://github.com/Tusamarco/sysbench/blob/master/src/lua/test_random.lua). The test runs and will print a quite clear representation of the IDs distribution. 

My recommendation is, identify what matches your needs and do your testing/benchmarking in the right way.  

References

First and foremost, reference the great work Alexey Kopytov is doing in working on sysbench https://github.com/akopytov/sysbench

Zipfian articles:

• New distribution of random generator for sysbench – Zipf
• https://en.wikipedia.org/wiki/Zipf%27s_law

Pareto:

• https://en.wikipedia.org/wiki/Pareto_distribution

Percona article on how to extend tests in sysbench

The whole set of material I used for this article is on GitHub (https://github.com/Tusamarco/blogs/tree/master/sysbench_random)

Disclaimer: The following blog post does not try to provide a solution for split-brain situations, and the example provided is for demonstrative purposes only. Inconsistencies resulting from a split-brain scenario might possibly be more complex than the one shown, so do not use the following example as a complete guide.

What is Split-Brain?

A split-brain scenario is the result of two data sets (which were originally synced) losing the ability to sync while potentially continuing to receive DMLs over the same rows and ids on both sides. This could have consequences such as data corruption or inconsistencies where each side has data that does not exist on the other side.

For example before split-brain:

After split-brain:

Node1:

Node2:

It can be seen that after the split-brain scenario there are many differences between the nodes:

• Customer row id=2 was deleted on Node2
• There was one insert on “Customer” table on each node, but the inserted row has repeated id=3
• There was an insert on the “Customer_product” table on each node with id=2 referencing Customer with id=3

Trying to merge all the rows will result in id collision and inconsistencies.

Why Does Split-Brain Occur?

The most typical reason why split-brain occurs is when an async cluster topology loses connectivity and each node continues to give service independently, but other example cases are:

• A slave node has replication stopped and is out of sync with the master for many days; then the master fails with automatic promotion to slave. When a master server is back online it will have inconsistencies  1) master will have writes that were not replicated to slave 2) slave will have writes that are NOT on the master which occurred while it was offline.
• On a cluster, some node receives writes/deletes with replication disabled, causing data inconsistencies compared to the other nodes.
• A system migration in which old and new servers work concurrently for different customers without syncing data, and at some point the data from the old server needs to be migrated to the new server for the old to be shut down.

Why Is It Hard to Sync Split-Brain Scenarios?

Under the split-brain scenario, there are many inconsistencies that cannot be automatically merged. If restoring, information from one node to the other will lose the changes that occurred on the latter node, and if trying to merge, many issues arise, for instance, the above example:

• What happens with deleted Customer id=2 from Node2? Should it be kept or removed?
• Inserted rows with id=3 should be two different rows, or only 1 row should be kept? In which case for id=3, the correct value would be “Tom” or “Bill”?
• The referenced table has only 1 difference, but merging row id=2 will reference parent_id=3 which is ambiguous

Because of the above, fixing a split-brain scenario will depend on business rules to decide which rows to keep/discard and would result in a manual job to decide for every inconsistency.

How to Avoid Split-Brain Scenarios

When there is a network issue that results in a split-brain situation and as per the CAP theorem, it’s not possible to assure both Consistency and Availability, having to prioritize one over the other.

In the case of favoring Consistency, using Percona XtraDB Cluster allows the primary component  (if there is one) to keep serving data, while the non-primary component refuses to operate.  If using automatic failover solutions such as Orchestrator, it should be configured so that writes only go to one node, and the promotion of slave only occurs if the slave was up to date at the moment the network failed. If it’s not possible to guarantee data consistency, then nodes should refuse to work.

In case the service needs to be maintained under split-brain scenarios, inconsistencies will arise, but something which can be done to avoid having duplicate ids is using different auto_increment_increment and auto_increment_offset on each server so that, for example, server A ids are even numbers and server B ids are odd numbers.

Updates and deletes must be resolved case by case.

How Do You Fix a Split-Brain Scenario?

As previously shown, if writes occur in different nodes after a split-brain, merging the rows in a consistent way can be difficult. The procedure is manual and cumbersome as rows to be kept have to be checked one by one, and criteria for deciding inconsistencies should be based on business logic.

If only one of the nodes received writes (and provided all required variables are enabled beforehand and binary logs are available), there is a tool called MySQL rewind which can help in reverting the changes on the changed node compared to the original node. The tool has some limitations listed on the official documentation but it’s very useful provided the requirements are met.

In case two (or more) nodes have received writes after split-brain, the merge procedure will be more complex. A high-level procedure on how to fix inconsistencies would be as follows:

First, backup both databases. Then identify which rows are different on each cluster by comparing tables from both nodes. If you have tables with timestamps of creation/edition it might greatly help to identify which rows were updated/inserted after split-brain scenario started, otherwise the best would be to chose one of the databases as the source of truth, and work on the other to identify the differences, while merging the changes one by one and according to business rules.

Possible Scenarios That Can Happen:

• If a row exists on one node but not on the other, it might be a delete. Depending on business logic, decide to keep or delete it.
• If a row exists in both nodes but an update has occurred and some (but not all) column differs, use business logic and available information to choose which row to keep.
• If two rows exist with the same id but all columns differ, it might be two different inserts (on each node) and merging them would be a collision. If you decide to insert the row from Node 2 into the source of truth, you might need to adjust ids by for example adding a big number to the current id (i.e current_id + 100k).
• If a row from Node2 is going to be inserted into the source of truth and references another row, double-check if parent row id is correct, otherwise it id might have been fixed adding 100k as shown on the above example, in which case you need to fix the referenced id.

After merging the data and in case of adding ids, there might be a gap in the ids resulting in a waste of ids, in which case dumping and reloading the table might be a good idea to remove the gaps.

Conclusion

Business logic must decide in advance how to react under split-brain scenarios – whether to favor consistency or availability – by comparing the consequences of losing service vs. the cost and difficulty of data merging after the split-brain is over.

Earlier this week, Matt Yonkovit talked about how some companies are seeing an overwhelming increase in systems usage and traffic. During a crisis like the one we are experiencing, the shift in how people interact and behave has accelerated the usage of certain services and introduced people into new ways of working, living, and interacting. In many ways, it’s forcing the modernization of many jobs and people.

With this in mind, we wanted to ask – In this environment, how has your database or application traffic been affected?

We really appreciate you taking the time to do these quick polls, and we look forward to seeing your responses!

This is usually one of the first questions asked by clients when discussing the possibility of adding a TAM (Technical Account Manager) to their Percona Support contract. Although the TAM role in a general sense has been around for many years, there are often misconceptions and confusion on what exactly a TAM is, and how a TAM can prove beneficial to your company.

As a TAM at Percona, I advocate for our customers, working with them closely to understand their business and technical environment, using my industry knowledge to advise the best path forward for current issues as well as any future project planning. (Recommended reading: my colleague Michael Patrick wrote a blog post about a day in the life of a technical account manager earlier this year.)

Expertise

Percona TAMs are often SMEs (Subject Matter Experts) in many areas of database administration. Percona TAMs come from a professional DBA background, so we have years of experience managing large and complex database installations in enterprise environments as well as smaller companies and startups.

Efficiency

As a TAM, I am also involved with the customer’s planning/testing both before and after any major changes, such as database upgrades, architecture and design, and project planning. This work often includes documenting the procedures that are followed during planned events, reviewing active support tickets, and analyzing customer environments.

Research

I review the latest white papers, blogs, and newsletters to stay up to date on various database technologies, including MySQL, MongoDB, and PostgreSQL. Our clients need the latest information to stay current, compliant, and secure, and as a TAM I can help make that happen by staying involved in the industry and bringing relevant information on trends and technology to the table.

TAMs are exposed to all of Percona’s products and services with a unique internal insight. This enables us to offer expert advice to our customers to take full advantage of our products and services to optimize their database infrastructure. Percona’s wide reach also allows the TAM visibility into multiple clients, industries, and architectures that can give real-world additional insight. This information exponentially increases the ability of our customers to achieve their goals and plan successfully.

What Else?

There is, of course, much more to the TAM role than just these areas. Some of the other regular duties a Percona TAM might perform are:

• Hosting regular meetings with Percona clients to better understand the business/technical environment, infrastructure, goals, and challenges.
• Taking ownership of support tickets and retaining that ownership through to a successful resolution.
• Providing an internal escalation point for client support and consulting projects.
• Perform implementation reviews, discuss new product features, and ensure prompt and proper resolution of technical challenges.
• Staying current with any ongoing and upcoming client projects and initiatives.

I might add is that we have exposure to multiple clients/industries/architectures and that can help give additional perspective or insights to clients

TAMs are equipped to use all this information to assist customers with proactive solutions and best practices for recommended changes to production. As a TAM, our commitment to you is to be honest, empathetic, and exceed expectations, whether you’re using a Percona product or not.

Your personal Percona Support Technical Account Manager will understand you and your company better than anyone else at Percona. They are on hand to help you achieve your database business goals, ensuring you are aware of the best tools to optimize and power your company’s database infrastructure.

The adoption and market share of PostgreSQL is growing at a rapid rate. And it’s not just new installations or applications  – there are huge levels of migrations happening from other database systems as well. In this webinar, we will discuss the reasons for selection and migration to PostgreSQL, why it’s becoming a preferred open-source database, and demonstrate simple migration using Oracle_FDW.

This talk covers:

• Major Trends today
• Management, Developer, Operations, Architect, and Community reasons for migration
• PostgreSQL and Opensource ecosystem influence
• Govt and Security reasons
• Migration tools and frameworks

Please join Jobin Augustine on Wednesday, April 8th, at 2:00 pm GMT for his webinar “Why PostgreSQL is Becoming a Migration Target in Large Enterprises”.

Register Now

If you can’t attend, sign up anyway and we’ll send you the slides and recording afterward.

More and more people are nowadays thinking of cloud migration. The question of “Which cloud provider performs better for my MySQL workload?” is really common but cannot always be easily answered. However, there are ways to come up with an answer. This question also applies when thinking of moving to any provider, not necessarily a cloud one or DBaaS.

The Problem

The most reliable conclusion can be found if you have a testing environment that is fully identical and can produce the same amount of traffic compared to your production version. In this case, the comparison should be straightforward as what you have to do is point your testing environment against the under-evaluation (cloud) providers, and evaluate the differences in performance. But this is not always easy, as many organizations do not have such environments or it’s not always possible to replay all traffic.

The Idea

In this post, if you don’t already have a testing environment that can produce production-like traffic, I am going to show you one methodology that will help you decide which provider offers better performance for your workload.

First of all, what we need to take into consideration is that finding the most suitable provider to migrate to depends on a few different factors such as:

• Pricing
• Business requirements
• Features and tools provided
• Limitations that may apply
• Technologies supported
• High Availability
• And so on…

There are already many blog posts describing that, and some time ago I wrote one with a high-level overview of RDS and Aurora called “When Should I Use Amazon Aurora and When Should I use RDS MySQL?“.

Assuming that you have already evaluated everything, and came across multiple available options, you may now have to make your final decision; but this is not easy as you don’t really know which of these providers performs better for your workload.

I’m pretty sure that you already know some benchmarking tools such as sysbench, mysqlslap, tpcc-mysql, etc.

Searching around you may find out that there are multiple whitepapers and benchmarks available. And as you most probably already know, one of the major factors when performing a benchmark is the workload, and the workload for every environment is different.

For example, you may come to two totally different benchmark results if you replay different workloads. One benchmark may say that provider X is better but another benchmark may say that provider Y is better. One benchmark may say that provider X provides 20% better performance but another may say that both providers provide the same performance. And this is not strange, because assuming that hardware between the providers is similar, any actual benchmark relies on the workload.

The scenario I am going to present is really simple and based on a single host accepting full traffic, read and write. We will compare how queries collected on the current production master are behaving when replayed against different providers. The logic is similar for every topology, so you could easily do any adjustments to match your environment.

Please be aware that enabling slow logs may introduce some overhead due to the increased IO needs for writing the logs, so enable them with caution. It may also introduce increased needs for disk capacity (depending on the number of logs captured). The log file should ideally reside on a separate partition or on a partition that is not used by the main function of the server.

Prerequisites

Let’s say that we have to choose between two providers, provider A and provider B. For doing the performance evaluation, we will need the following:

1. Three hosts with identical hardware specs. One located at your current provider, another hosted at provider A, and the last one hosted at provider B. Hardware specs do not necessarily need to match your current masters but should be the ones you are attempting to move to.
2. MySQL/Percona Server for MySQL/MariaDB (depends on what you currently have) – same version and configuration to your production host – installed on all three hosts above
3. The same exact dataset for all three hosts above
4. A sample of queries ran against your master
5. A tool that will replay these queries against the three hosts above

Tasks one, two, and three should be straightforward to do on your own so I will only focus on the remaining ones.

To collect the queries performed against your master, you need to enable slog logs and set an appropriate sampling (if applicable). The suggested settings are:

• slow_query_log = ON
• long_query_time = 0
• log_output = FILE (or empty which defaults to FILE)
• log_slow_rate_limit = 100. To allow us to collect more queries this should be further lowered after initial configuration. This configuration variable applies to Percona Server for MySQL and MariaDB only.

If you are using Percona Monitoring and Management Query Analytics, the settings above may already be enabled so you may only have to temporarily adjust them. In any case, you should cross-check if this is the case and adjust as needed because your settings may not match the ones suggested.

With the settings above you are going to log every single query (long_query_time=0) with a sampling of 1/100 (log_slow_rate_limit = 100). Sampling can be further adjusted i.e. 1 so we can capture all queries.

There is no magic number for how many slow logs to collect or what the better timeframe is to collect slow logs. It depends on your workload and we usually prefer a peak time, i.e. the busiest window during a normal business day, a 24-hour window, or times when reoccurring normal jobs like end of month jobs, daily batch processing, billing, etc., so we can collect as many logs as possible.

The Benchmark

Having the slow logs, we need to replay them and collect the metrics. We will replay them using the pt-upgrade tool which is part of percona-toolkit. What you need to have in mind is that pt-upgrade does not offer a concurrency option, so all queries will be sequentially replayed.

Please be sure that, while performing the pt-upgrade operations, none of the three hosts of the above setup is taking any traffic other than the one we are going to generate, as otherwise, the results will not be reliable.

Steps to replay traffic:

We first need to run pt-query-digest on the slow logs we collected (let’s say slow.log). The file that we are going to run pt-query-digest against is the one specified by slow_query_log_file configuration variable.

time pt-query-digest --sample 50 --no-report --output slowlog slow.log > slow_log_50_samples.log

This way, we will export 50 samples per query, which is enough. The slow_log_50_samples.log will now be used for the benchmark.

On every host, we need to run pt-upgrade twice. The first execution will ensure that our InnoDB buffer pool has been warmed up while the second one will be the actual execution.

For the purposes of this blog, I’m going to share some results from a benchmarking that I recently did.

On the testing instance on our current provider:

time pt-upgrade h=localhost --max-examples=1 --save-results upc_RO_1 slow_log_50_samples.log

And again specifying a new directory,

time pt-upgrade h=localhost --max-examples=1 --save-results upc_RO_2 slow_log_50_samples.log

where slow_log_50_samples.log is the processed slow logs that we collected.

With the example above we’ve replayed all the queries and we saved the results to a folder named “upc_RO_2” (we are mostly interested in the second execution which is after warming up the InnoDB buffer pool).

We now need to do exactly the same for the two hosts that we are evaluating. Run one time to warm the InnoDB buffer pool and another for the actual benchmark. This time we won’t use the slow logs as a source, but rather the results from the first execution. For further details on how pt-upgrade works and all the available options, please check the official pt-upgrade page.

On provider A:

time pt-upgrade upc_RO_2/ h=localhost --max-examples=1 1> GCP_RO_1_results.out 2> GCP_RO_1_results.err

And again:

time pt-upgrade upc_RO_2/ h=localhost --max-examples=1 1> GCP_RO_2_results.out 2> GCP_RO_2_results.err

On provider B:

time pt-upgrade upc_RO_2/ h=localhost --max-examples=1 1> EC2_RO_1_results.out 2> EC2_RO_1_results.err

And again:

time pt-upgrade upc_RO_2/ h=localhost --max-examples=1 1> EC2_RO_2_results.out 2> EC2_RO_2_results.err

The files we should focus on are:

• GCP_RO_2_results.out which is the comparison between provider A and the current provider
• EC2_RO_2_results.out which is the comparison between provider B and the current provider

The output in each file should end with something like that. Let’s see how it looks for provider B (EC2_RO_2_results)

#-----------------------------------------------------------------------
# Stats
#-----------------------------------------------------------------------

failed_queries 60
not_select 0
queries_filtered 0
queries_no_diffs 36149
queries_read 36830
queries_with_diffs 621
queries_with_errors 0

Which shows that:

• 36830 queries were read in total
• 60 queries failed to be executed
• 36149 queries had no diffs
• 621 queries had diffs

621 queries out of 36830 queries is almost 1.6%. These queries (1.6%) include both queries that have returned a different set of rows or performing worse so actual degradation might be even less than 1.6%.

So what we know is that at least 98.4% of the queries are performing exactly the same or better than our current provider.

Please note that only queries returning a different set of data or degradation will be reported, as this is what we are actually interested in, so you won’t see something like “queries performing better” in the output above.

Queries that failed and queries with diffs should be further evaluated. These may help us uncover any incompatibilities or problems due to non-supported queries.

To evaluate these queries you should get familiar with how pt-upgrade works. Trying to give some hints only as this is not the goal of this post, the “queries_with_diffs” queries may report differences for various reason such as:

Queries are expected to report different output due to the difference in the hostname.

• monitoring queries
• performance_schema
• information_schema
• use of DATETIME functions such as NOW(), CURDATE().
• different order (missing order by) or order by column has similar values including NULL for multiple rows

Queries_with_diffs also includes the queries which are worse-performing.

Doing a grep for “increase” will help you reveal the queries performing worse, i.e.:

$ fgrep -i increase EC2_RO_2_results.out

0.002259 vs. 0.029961 seconds (13.3x increase)
0.007984 vs. 0.084002 seconds (10.5x increase)
0.005101 vs. 0.051768 seconds (10.1x increase)
0.000886 vs. 0.009347 seconds (10.5x increase)
0.010519 vs. 0.118369 seconds (11.3x increase)

So, in this case, we have only five examples of queries performing worse. These queries can be found in the EC2_RO_2_results.out file which can be further reviewed.

What you now need to do is repeat the evaluation above for the results for provider A (GCP_RO_2_results.out). After reviewing this as well, it should be straightforward for you to identify which provider offers better performance.

For the scenario above, we compared how our current provider is performing compared to provider A and provider B. Following the same logic, you can make as many combinations as you want. For example, you can compare the current provider to another single provider only, compare provider A to provider B, add another provider C, etc.

Assuming that you performed the READ ONLY testing and you are interested in WRITE performance as well, you can now do a WRITE testing. The only adjustments you’d need to do is:

• Add –no-read-only to the pt-upgrade options
• Perform only once against each node as there is no need to warm the InnoDB buffer pool
• Select a different folder to save the results and a different filename for the statistics

For your evaluation, you should follow the same strategy as you did for the READ ONLY testing.

*** Please note that WRITE testing is destructive to the dataset operation so you must be sure that you are not performing this against your master or any other valuable dataset. While doing a WRITE testing, all queries that you have captured on your master will be replayed and the subsequent write fails may fail. For example, you may have captured an “insert” query but the record may already exist in the snapshot of data you’ve used for the performance testing. So when trying to run this insert again, it may violate a PK.

Conclusion

When planning a migration you should always evaluate performance using your own workload as every workload is unique. pt-upgrade can help you identify incompatibilities, queries that may fail when replayed against different hosts, or performance degradation. It’s a great tool that can help you evaluate different providers and compare performance using your own workload.

It’s release roundup time here at Percona!

Our Release Roundups showcase the latest software updates, tools, and features to help you manage and deploy our software, with highlights and critical information, as well as links to the full release notes and direct links to the software or service itself.

Today’s post includes those which have come out since March 16, 2020.

Percona Monitoring and Management 2.4.0

On March 18, 2020, we released Percona Monitoring and Management 2.4.0. It is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance. New features include the support of Prometheus custom configuration and improvements include Time Metrics shown as “AVG per query” instead of “AVG per second”, Clickhouse queries optimized for Query Analytics, and a healthcheck indicator was implemented for the PMM Server Docker image. Several bugs were fixed as well.

Download Percona Monitoring and Management 2.4.0

Percona Server for MySQL 8.0.19-10

March 23, 2020, saw the release of Percona Server for MySQL 8.0.19-10.  It includes all the features and bug fixes available in MySQL 8.0.19 Community Edition in addition to enterprise-grade features developed by Percona. Several new features and improvements were introduced, including the addition of a server’s UUID to Percona system keys, a simplified LDAP authentication plugin, and the removal of the KEYRING_ON option from default_table_encryption. There are also many bug fixes in this release, a full list of which can be seen in the release notes.

Download Percona Server for MySQL 8.0.19-10

Percona XtraBackup 2.4.19

On March 25, 2020, Percona XtraBackup 2.4.19 was announced. It enables MySQL backups without blocking user queries, making it ideal for companies with large data sets and mission-critical applications that cannot tolerate long periods of downtime. Offered free as an open source solution, it drives down backup costs while providing unique features for MySQL backups. In this release, bug PXB-1982 was fixed, where the history table showed a wrong value for lock_time.

Download Percona XtraBackup 2.4.19

Percona Kubernetes Operator for Percona Server for MongoDB 1.4.0

Percona Kubernetes Operator for PSMDB 1.4.0 was released on March 27, 2020. Several new features and improvements were introduced.  Amazon Elastic Container Service for Kubernetes (EKS) was added to the list of the officially supported platforms, Percona Server for MongoDB 4.2 is now supported, and the operator now updates observedGeneration status message to allow better monitoring of the cluster rollout or backups/restore process. In addition, several bugs were fixed, including where setting the updateStrategy: OnDelete didn’t work if was not specified from scratch in CR

Install Percona Kubernetes Operator for Percona Server for MongoDB 1.4.0

Percona Server for MongoDB 4.0.17-10

And finally, Percona Server for MongoDB 4.0.17-10 was released on March 30, 2020. It is a free, open source drop-in replacement for MongoDB Community Edition but with enterprise-grade functionality. This version is based on MongoDB 4.0.17 and does not include any additional changes.

Download Percona Server for MongoDB 4.0.17-10

That’s it for this roundup, and be sure to follow us on Twitter to stay up-to-date on the most recent releases! Percona is a leader in providing best-of-breed enterprise-class support, consulting, managed services, training and software for MySQL, MariaDB, MongoDB, PostgreSQL, and other open source databases in on-premises and cloud environments.

We understand that choosing open source software for your business can be a potential minefield. You need to select the best available options, which fully support and adapt to your changing needs. Choosing the right open source software can allow you access to enterprise-level features, without the associated costs.

In our white paper, we discuss the key features that make open source software attractive, and why Percona’s software might be the best option for your business.

Download: When is Percona Software the Right Choice?

In my Webinar on Using Percona Monitoring and Management (PMM) for MySQL Troubleshooting, I showed how to use direct queries to ClickHouse for advanced query analysis tasks. In the followup Webinar Q&A, I promised to describe it in more detail and share some queries, so here it goes.

PMM uses ClickHouse to store query performance data which gives us great performance and a very high compression ratio. ClickHouse stores data in column-store format so it handles denormalized data very well. As a result, all query performance data is stored in one simple “metrics” table:

I provided the whole table structure here as it includes a description for many columns. Note not all columns will contain data for all database engines in all configurations, and some are not yet used at all.

Before we get to queries let me explain some general design considerations for this table.

We do not store performance information for every single query; it is not always available to begin with (for example, if using MySQL Performance Schema). Even if it was available with modern database engines capable of serving 1M+ QPS, it would still be a lot of data to store and process.

Instead, we aggregate statistics by “buckets”  which can be seen as sort key in the “metrics” table:

ORDER BY (queryid,service_name,database,schema,username,client_host,period_start)

You can think about Sort Key as similar to Clustered  Index in MySQL. Basically, for every period (1 minute by default) we store information for every queried, service_name, database, schema, username, and client_host combination.

Period_Start   is stored in the UTC timezone.

QueryID – is a  hash which identifies unique query pattern, such as “select c from sbtest1 where id=?”

Service_Name is the name of the database instance 

Database  – is the database or Catalog.  We use it in PostgreSQL terminology, not MySQL one

Schema – this is Schema, which also can be referred to as Database in MySQL 

UserName –  The Database level  User Name, which ran this given query.

Client_Host –  HostName or IP of the Client

This data storage format allows us to provide a very detailed workload analysis, for example, you can see if there is a difference in performance profile between different schemas, which is very valuable for many applications that use the “tenant per schema” approach. Or you can see specific workloads that different users generate on your database fleet. 

Another thing you may notice is that each metric for each grouping bucket stores several statistical values, such as: 

`m_query_time_cnt` Float32 COMMENT 'The statement execution time in seconds was met.',
 `m_query_time_sum` Float32 COMMENT 'The statement execution time in seconds.',
 `m_query_time_min` Float32 COMMENT 'Smallest value of query_time in bucket',
 `m_query_time_max` Float32 COMMENT 'Biggest value of query_time in bucket',
 `m_query_time_p99` Float32 COMMENT '99 percentile of value of query_time in bucket',

The  _cnt  value is the number of times this metric was reported.  Every Query should have query_time available but many other measurements may not be available for every engine and any configuration. 

The _sum value is the sum for the metric among all _cnt  queries. So if you want to compute _avg you should divide   _sum by _cnt.

_min, _max and _p99  store the minimum, maximum, and 99 percentile value.

How to Access ClickHouse

To access ClickHouse on PMM Server you should run the “clickhouse-client”  command line tool.

If you’re deploying MySQL with Docker you can just run:

docker exec -it pmm2-server clickhouse-client

Where pmm2-server is the name of the container you’re using for PMM.

Run  “use pmm”  to select the current schema to PMM.

Query Examples

ClickHouse uses SQL-like language as its query language. I call it SQL-like as it does not implement SQL standard fully, yet it has many additional and very useful extensions. You can find the complete ClickHouse Query Language reference here. 

# Number of Queries for the period

select sum(num_queries) from metrics where period_start>'2020-03-18 00:00:00';

# Average Query Execution Time for Last 6 hours 

select avg(m_query_time_sum/m_query_time_cnt)  from metrics where period_start>subtractHours(now(),6);

# What are most frequent query ids ?  Also calculate total number of queries in the same query

select queryid,sum(num_queries) cnt from metrics where period_start>subtractHours(now(),6) group by queryid with totals order by cnt desc limit 10;

# How do actual queries look for those IDs ?

select any(example),sum(num_queries) cnt from metrics where period_start>subtractHours(now(),6) group by queryid order by cnt desc limit 10 \G

# queries hitting particular host 

select any(example),sum(num_queries) cnt from metrics where period_start>subtractHours(now(),6) and node_name='mysql1' group by queryid order by cnt desc limit 10 \G

# slowest instances of the queries

select any(example),sum(num_queries) cnt, max(m_query_time_max) slowest  from metrics where period_start>subtractHours(now(),6)  group by queryid order by slowest desc limit 10 \G


# Query pattern which resulted in the largest temporary table created
select example, m_tmp_table_sizes_max  from metrics where period_start>subtractHours(now(),6) order by m_tmp_table_sizes_max desc limit 1 \G 


# Slowest Queries Containing Delete in the text 

select any(example),sum(num_queries) cnt, max(m_query_time_max) slowest  from metrics where period_start>subtractHours(now(),6) and lowerUTF8(example) like '%delete%'  group by queryid order by slowest desc limit 10 \G

I hope this gets you started!

If you create some other queries which you find particularly helpful, please feel free to leave them in the comments for others to enjoy!

Share your knowledge and experience to help inform the open source community!

In 2019, we ran our first ever Open Source Data Management Software Survey, resulting in some interesting data on the state of the open source database market.

Some notable statistics included:

• 90% of survey respondents use more than one database technology in their work environment and 85% of respondents use more than one open source database technology
• 38% of respondents use more than one cloud deployment, with single or multiple providers
• Respondents were eight times more likely to adopt an open license than the alternative
• 62% of respondents use open source software to avoid vendor lock-in
• Over 50% of survey respondents run some of their workloads in the public cloud

We found this information so interesting that we decided to run the survey annually, and compare the results year over year. Therefore, we are pleased to announce the launch of our 2020 Open Source Data Management Software Survey.

The final results will be 100% anonymous and made freely available on our website and via Creative Commons Attribution ShareAlike (CC BY-SA).

We Believe Access to Accurate Market Data Is Important

Millions of open source projects are currently running, and most are dependent on databases. Up-to-date market data helps people track the popularity of different databases, and see how and where these databases are run. This information helps us build better software and take advantage of shifting trends.

We want to help companies who are deciding which database or tool to use, and how and where to run their systems. Accurate information helps these businesses understand industry direction and make informed decisions on the software and services they choose.

How Can You Help This Survey Succeed?

First, we would love you to share your insight into current trends and new developments in open source data management software.

Second, we hope you will share this survey with other people who work with, and use open source database software, and encourage them to contribute.

The more responses we receive, the more useful this will be to the open source community. If there is anything we missed, or you would like to ask in the future, we welcome your feedback.

The survey should take around 10 minutes to complete. Have your say here:

Complete the Survey

One of the most common questions I get as a Technical Account Manager (TAM) from clients considering a migration from their current on-premise data center to AWS is “what are the biggest differences I will need to address?”  In an earlier post MySQL to the Cloud! Thoughts on Migrating Away from On-Premise I discussed some of the best practices to actually perform the migration. The goal of this post is to address some of the most common infrastructure and architectural differences I have encountered helping clients make the transition into AWS.  For the sake of this post, let’s assume you are going to spin up EC2 instances and manage your own databases.

Hardware Differences

Instance Types

When comparing AWS EC2 to a traditional on-prem deployment, the flexibility and variance of the instance types can be overwhelming.  In traditional on-premise deployments, you provision hardware based on your anticipated traffic and growth over a set time period. This often results in over-allocating hardware to the database tier.  In the case where the workload doesn’t match projections or growth isn’t as fast as expected, you can be faced with lots of idle capacity.

As a DBA, while I would hope my projections are always accurate, I’d also much rather over-provision my hardware rather than limit my performance through inadequate processing power.  I also would like the ability to tailor my hardware to the workload as needed rather than acquire several racks of identical machines and force the usage patterns onto my hardware.

With a wide variety of instance types (compute-optimized, memory-optimized, etc) available to target different workloads, you are much more likely to properly provision your servers.  You also have the ability to start with smaller instances and scale up the hardware as your workload actually increases (or downsize if you don’t hit your projections).

Networking and I/O

Depending on how you currently manage networking in your on-premise deployment, there are likely some significant differences when you migrate to AWS.  Features such as floating IPs (via commands such as ip or through a tool like keepalived) that are fairly trivial when you fully control the network are not mapped 1:1 in AWS.

While you do have control over your network to some extent when using a VPC, IP addresses are attached differently.  If you wanted to attach a specific IP address to an instance, you would need to associate an Elastic IP (which will be public).  This will need to use AWS CLI tools, reference the instance IDs of the servers, and be a very different approach.

Along the same lines as networking, there are differences in the I/O options as well.  When looking at traditional on-premise deployments, you typically have local storage for your active data directory and often some type of network storage (SAN, nfs, etc) for things like backups.  You are generally locked into a single storage class across your fleet and standardize towards the general purpose.

However, in AWS, you have numerous options for managing your database tier’s IO layer.  The primary options include:

• Ephemeral storage (fastest, but non-durable)
• Elastic Block Storage (multiple tiers of performance/price)

When using ephemeral storage, your data won’t persist after stopping an instance.  However, in some designs (such as synchronous replication with Percona XtraDB Cluster), this isn’t a huge problem as durability comes from the cluster rather than the local storage.

When using EBS volumes, you can often achieve the performance needed while gaining extra flexibility.  EBS volumes allow you to choose the storage tier (magnetic vs SSD) and even provision the IOPs capacity you need.  If you have an I/O bound workload, the flexibility in EBS can allow you to tailor instances specific to your needs.  These volumes can also be dynamically expanded so you don’t have to over-provision the size initially and hope it grows.  Rather, you can size the storage tier to what you need now and grow as your storage requirements increase.

In part two of this post, I’ll cover the biggest differences I’ve encountered from an architectural point of view.  Check back soon!

PostgreSQL has a rich set of indexing functionality, and there are many articles explaining the syntax, usage, and value of the index. In this article, I will write basic and useful queries to see the state of database indexes. People develop databases and after some time, when there is a demand to do changes in the architecture of software, they forget to do the previous indexes’ cleanup. This approach creates a mess and sometimes slows down the database because of too many indexes. Whenever we do an update or insert, the index will be updated along with the actual table, therefore there is a need for cleanup.

There is a wiki page that has some queries related to PostgreSQL Index Maintenance.

Before writing the queries I want to introduce a catalog table pg_index. The table contains information about the index. This is the basic catalog table, all the index-based views use the same table.

1 – Sometimes you need to see how many indexes your table has. This query will show the schema-qualified table name and its index names.

db=# SELECT CONCAT(n.nspname,'.', c.relname) AS table,
    i.relname AS index_name FROM pg_class c
     JOIN pg_index x ON c.oid = x.indrelid
     JOIN pg_class i ON i.oid = x.indexrelid LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
  WHERE c.relkind = ANY (ARRAY['r', 't']) AND c.relname like 'pgbench_accounts';
          table          | index_name       
-------------------------+------------------------
 public.pgbench_accounts | pgbench_accounts_pkey
 public.pgbench_accounts | pgbench_accounts_index
(2 rows)

2 – As we all know, an index is a performance feature, but along with that, it is also used to ensure uniqueness. But to ensure the uniqueness we need a separate type of index called a unique index. To check whether an index is unique or not, pg_index has a column named “indisunique” to identify the uniqueness of the index.

SELECT    i.relname AS index_name,
          indisunique is_unique
FROM      pg_class c
JOIN      pg_index x ON c.oid = x.indrelid
JOIN      pg_class i ON i.oid = x.indexrelid
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE     c.relkind = ANY (ARRAY['r', 't'])
AND       c.relname LIKE 'pgbench_accounts';
       index_name       | is_unique 
------------------------+-----------
 pgbench_accounts_pkey  | t
 pgbench_accounts_index | f
(2 rows)

3 – There is a pretty simple way to get the size of the index of PostgreSQL. Here is a query to list the PostgreSQL with size.

SELECT pg_size_pretty(pg_relation_size('pgbench_accounts_index'));
 pg_size_pretty 
----------------
 132 MB
(1 row)

4 – Here is a list of the indexes with total table size and size of the index, which is very useful to compare your table size with its corresponding indexes.  It’s very good to know the size of your table, index, and the total size of the table.

SELECT    CONCAT(n.nspname,'.', c.relname) AS table,
          i.relname AS index_name, pg_size_pretty(pg_relation_size(x.indrelid)) AS table_size,
          pg_size_pretty(pg_relation_size(x.indexrelid)) AS index_size,
          pg_size_pretty(pg_total_relation_size(x.indrelid)) AS total_size FROM pg_class c 
JOIN      pg_index x ON c.oid = x.indrelid
JOIN      pg_class i ON i.oid = x.indexrelid
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE     c.relkind = ANY (ARRAY['r', 't'])
AND       n.oid NOT IN (99, 11, 12375);
          table          | index_name       | table_size | index_size | total_size 
-------------------------+------------------------+------------+------------+------------
 public.pgbench_tellers  | pgbench_tellers_pkey   | 88 kB      | 64 kB  | 152 kB
 public.pgbench_accounts | pgbench_accounts_pkey  | 2561 MB    | 428 MB | 3122 MB
 public.pgbench_accounts | pgbench_accounts_index | 2561 MB    | 132 MB | 3122 MB
 public.pgbench_branches | pgbench_branches_pkey  | 8192 bytes | 16 kB  | 24 kB
(4 rows)

pg_relation_size: Function gives the size of relation. It is used to get the size of the table/index.

pg_total_relation_size:  This is a special function that gives the total size of the table along with its all indexes.

5 – Get the query of the index. This query will show the index creation query.

SELECT pg_get_indexdef(indexrelid) AS index_query
FROM   pg_index WHERE  indrelid = 'pgbench_accounts'::regclass;
                                     index_query
----------------------------------------------------------------------------------------
CREATE UNIQUE INDEX pgbench_accounts_pkey ON public.pgbench_accounts USING btree (aid)
CREATE INDEX pgbench_accounts_index ON public.pgbench_accounts USING btree (bid)
CREATE INDEX pgbench_accounts_index_dup ON public.pgbench_accounts USING btree (bid)
(3 rows)

6 – In case your index becomes corrupted or bloated, you need to build that index again. At the same time, you don’t want to block the operation on your table, so this REINDEX CONCURRENTLY command is your choice for that.

REINDEX INDEX CONCURRENTLY idx;
REINDEX

7 – PostgreSQL has many index methods like BTree, Hash, BRIN, GIST, and GIN. Sometimes we want to create some specific index on a column but are unable to do that. PostgreSQL has limitations that some indexes cannot be created on some data types and operators, and that makes sense too. For example, the Hash index can only be used for equal operators. Here is a query to get the list of the supported data types for a particular index.

SELECT amname,
       opfname
FROM   pg_opfamily,
       pg_am
WHERE  opfmethod = pg_am.oid
AND    amname = 'btree';

amname | opfname
--------+--------------------
btree  | array_ops
btree  | bit_ops
btree  | bool_ops
…

8 – This query will find the unused indexes. If index_scans is 0 or close to 0 then you can drop those indexes. But be careful, as maybe those indexes are for unique purposes.

SELECT s.relname AS table_name,
       indexrelname AS index_name,
       i.indisunique,
       idx_scan AS index_scans
FROM   pg_catalog.pg_stat_user_indexes s,
       pg_index i
WHERE  i.indexrelid = s.indexrelid;
table_name       | index_name            | indisunique | index_scans
------------------+-----------------------+-------------+-------------
pgbench_branches | pgbench_branches_pkey | t           | 0
pgbench_tellers  | pgbench_tellers_pkey  | t           | 0
pgbench_accounts | pgbench_accounts_pkey | t           | 0
(3 rows)

9 – Query used to find a duplicate index. In this example, pgbench_accounts has two of the same indexes. There is no need to have multiple same indexes with a different name on a table. As we already discussed, in case of update/insert, all the indexes get updated along with the actual table, which hurts the performance.

SELECT   indrelid::regclass table_name,
         att.attname column_name,
         amname index_method
FROM     pg_index i,
         pg_class c,
         pg_opclass o,
         pg_am a,
         pg_attribute att
WHERE    o.oid = ALL (indclass) 
AND      att.attnum = ANY(i.indkey)
AND      a.oid = o.opcmethod
AND      att.attrelid = c.oid
AND      c.oid = i.indrelid
GROUP BY table_name, 
         att.attname,
         indclass,
         amname, indkey
HAVING count(*) > 1;
table_name | column_name | index_method
------------+-------------+--------------
foo        | a           | btree

(1 row)

Conclusion

PostgreSQL has catalog tables to store the index information, and therefore, we can write as many queries as we need. This blog shows some basic queries and shows how to use the catalog tables to write the queries.

Working on a weekend project,  I became familiar with the SQL keyword “LATERAL”, which I had not used before because it was only introduced recently in MySQL 8.0.14,  so I wanted to share how it can be used.

Some references on this topic:

Lateral Derived Tables

LATERAL Derived Tables in MySQL 8.0

The keyword “LATERAL” comes into play when you work with derived tables. The derived tables have been available in MySQL for a long time, and schematically they look like this:

SELECT t1.foo, t2.bar FROM t1, (SELECT bar FROM table2 WHERE <condition> ) t2 WHERE t1.id=t2.id;

The table “

(SELECT bar FROM table1 WHERE <condition> ) t2

Derived tables are typically executed as materialized (ONCE per query and stored as “cached” in temporary tables) or via an outer table.

So what’s the problem with derived tables? You will find a limitation when you will try to use a condition inside a derived table that references an outside table. For example:

SELECT t1.foo, t2.bar FROM t1, (SELECT bar FROM table2 WHERE <condition> AND table2.date=t1.date ) t2 WHERE t1.id=t2.id;

This is not allowed, and to be able to use this query you need to specify a keyword LATERAL, so the query will look like:

SELECT t1.foo, t2.bar FROM t1, LATERAL (SELECT bar FROM table2 WHERE <condition> AND table2.date=t1.date ) t2 WHERE t1.id=t2.id;

What is a drawback? Well, the derived query can’t be executed now ONCE per query and will be executed FOR EACH row from table t1, so obviously, it will come with a performance penalty and should be used carefully.

You may ask, then, when do we need to use LATERAL as we were just fine before MySQL 8.0.14?

Actually I came to this query working with timeseries and forecast (prediction data). For example, for each day, we have a metric prediction for each of 30 days ahead.

E.g.

For each prediction date in the table, I need to find the metric prediction ONLY for the next day.

There are probably multiple ways to write this query, and also I was looking to handle this in the application, but with the LATERAL derived table you can write this as:

SELECT * FROM prediction_table q , 
LATERAL ( SELECT date_in_future,DATEDIFF(date_in_future, date_prediction) df  
   FROM prediction_table q1 
   WHERE q1.date_prediction=q.date_prediction ORDER BY df ASC LIMIT 1)  q2 
WHERE q2.date_in_future=q.date_in_future;

And if you need not exactly tomorrow, but rather for N days ahead, you can use WINDOW function

RANK() OVER ( ORDER BY DATEDIFF(expiration,tradedate)) r

So, the query will look like:

SELECT * FROM prediction_table q , 
LATERAL ( SELECT date_in_future,RANK() OVER ( ORDER BY  DATEDIFF(expiration,tradedate)) r 
   FROM prediction_table q1 
   WHERE q1.date_prediction=q.date_prediction)  q2 
WHERE q2.date_in_future=q.date_in_future AND r=N;

I am happy to see that MySQL 8 comes with a new rich set for SQL functions, which makes working with queries much easier.

Percona recently hosted a #PerconaChat to discuss and share some of the things we find most essential to be successful while working remotely. Even before the viral outbreak, Percona was an almost 100% remote company, so it was great to be able to impart some of our learnings.

David Busby — Information Security Analyst, Percona, and Camila Arocena — HR Specialist, Percona, were our hosts, and they kept the discussion lively, fun, and hopefully helpful to those that joined in the chat.

Here’s a recap of the questions we posed for discussion, some of the great Twitter posts received, along with our tips and tricks for remote working:

Question #1:

A: Keeping connected through chat, text, video chat, conference calling, phone, and other interactive methods ( I play online games with friends with voice chat enabled).

— Joseph Kozlowski (@PerconaJoe) March 26, 2020

How to stay in touch:

• Social distancing does not mean staying indoors. Practice staying the recommended distance from those you may see outside, but try and find a place where you can stretch, walk, or just sit outside, whenever possible.
• Keep up social connections, even if they are only digital. Reach out to friends online, or host a virtual party or meeting just to share ‘water-cooler’ topics.
• Taking regular breaks from your computer, tv, and mobile screens are imperative. It’s easy to become engrossed in what you are working on and lose track of time. Take up a craft, start a puzzle, or pick a new book to read — then share your recommendations with those on your work channels, so that you can engage in non-digital ways as well.

Question #2:

My barbershop is closed… My significant other does not appreciate this fact and has sent multiple emails claiming this is an essential service. #perconachat #2 #remotework pic.twitter.com/SM2QsiOzyD

— dim0 (@dim0) March 26, 2020

Keep Your Health Top of List:

• Set reminders to take breaks, stretch, and move around. These regular breaks help reduce stress levels. Keeping your stress manageable also ensures your teams are not unduly impacted, as your behavior will noticeably change when you are stressed.
• Having a routine is key to working from home, but the non-work items often get overlooked, such as a shower and brushing your hair. These small actions of self-care can build to bigger feelings of personal well being. So don’t skip them!

Question #3:

covid-19 scams, malware and phishing, the depravity of these individuals knows no bounds, please be extra cautious and skeptical of anything which claims to provide assistance for covid-19 at this time. #perconachat

— Percona Security (@PerconaSecurity) March 26, 2020

Practice Good Cyber Hygiene:

• Lock your screen! This is obvious for an office or public setting, but you should keep this up even at home  – if, for nothing else, it could keep the kids from accidentally sending out your next Tweet for you!
• Remember when you’re tired or stressed, you’re more susceptible to suggestions which include phishing and online scams. 
• Encryption is your ally. Tunneling VPN services such as NordVPN, Tunnelbear, Freedome are great tools, even when you are at home. 

Question #4:

I'm fortunate enough to have a dedicated home office space (with a locking door), but it just didn't feel like mine until I added a 50" tv and an Xbox (for after working hours of course!). #PerconaChat

— Travis Futas (@travisfutas) March 26, 2020

Set Up a Real Office with a Real Work Schedule:

• Set boundaries, real (physical) and virtual, for you and your family. Put your home office behind a door, if at all possible, and separate it from where you and your family spend most of your time, to keep the lines between work and home clearer.
• Use your calendar. Mark your work times clearly in your calendar so that people won’t schedule you during non-working hours or before you are ready to work. Similarly, set times in your calendar (with alerts) for breaks and lunch.
• Share your schedule with your family or roommates so that you can interact with real people during the breaks.

Question #5:

#Perconachat
Use Zoom video whenever possible to get as much non-verbal feedback as possible. Cut short lengthy text chats with a quick call – you can cover much more info quickly than prolonged chats back and forth.

— Tom Basil (@TomBasil99) March 26, 2020

Communicate With Care

• When you attend work meetings, turn the camera on! You need face time in a remote environment — it makes a huge difference.
• Over-communicate! It’s super easy to assume people know things that they don’t. Daily standups or check-ins over Slack are a great way to help keep people engaged and in the loop. 
• Say hello! When you start a chat, be sure to use those simple hellos to welcome people in, make others feel welcome, and make it feel less ‘remote’ when you are working together. 
• Recognize others! Go out of your way to recognize your co-workers or team’s accomplishments and share them widely with others.
• Be flexible — with everyone. Let people adjust meeting times to not conflict with their new family schedules, or laugh off the calls that don’t go well. Right now, life is affecting all of us in new ways, so a little kindness and understanding go a long way.
• Still looking for more tips? A few weeks ago, our own Matt Yonkovit wrote a post titled Tricks and Tips For Remote Workers which you may find helpful.

Thanks to all that joined in our chat. It was a great way to share knowledge and connect with others, which is so valuable during these times.

Want to join the next #PerconaChat? Follow us @Percona to get the details of all our upcoming events and releases.