In the world of data backup and security, physical backups play an extremely important role. Physical backup methods are faster than logical because they involve only file copying without conversion. This type of backup is suitable for large, important databases that need to be recovered quickly when problems occur.
Physical backups are the backups that consist of raw copies of the directories and files that store database contents. In addition to databases, the backup can include any related files, such as log or configuration files. Now, since backup speed and compactness are important for busy, important databases, Percona’s open source physical backup solution – Percona XtraBackup (PXB), takes into account all these aspects and benefits MySQL world with its exceptional capabilities!
This blog post will walk you through how MySQL’s InnoDB redo log archiving uses PXB to manage the redo log files on the busiest systems and look at how a new feature released in PXB version 8.0.34-29 will let you set up redo log directory while performing a backup.
InnoDB redo log archiving
Backup utilities that copy redo log records may sometimes fail to keep pace with redo log generation while a backup operation is in progress, resulting in lost redo log records due to overwritten records. This issue most often occurs when there is significant MySQL server activity during the backup operation, and the redo log file storage media operates faster than the backup storage media. The redo log archiving feature, introduced in MySQL 8.0.17, addresses this issue by sequentially writing redo log records to an archive file and the redo log files. Backup utilities can copy redo log records from the archive file as necessary, thereby avoiding the potential loss of data.
Enabling redo log archiving on the server requires setting a value for the innodb_redo_log_archive_dirs system variable. The value is specified as a semicolon-separated list of labeled redo log archive directories. The label:directory pair is separated by a colon (:). For example:
mysql> SET GLOBAL innodb_redo_log_archive_dirs='label1:directory_path1[;label2:directory_path2;…]';
Now, first, we are going to demonstrate to you how we tested the new InnoDB redo log archiving on Percona Server for MySQL 8.0.33-25 (GPL) and should be using MySQL shell (mysqlsh) for enabling the InnoDB redo log archiving.
- Create new directories for storing redo log archives.
[root@ip-xxx-xx-xx-xx ~]# mkdir -p /var/lib/mysql-redo-archive/backup1 [root@ip-xxx-xx-xx-xx ~]# chown mysql. -R /var/lib/mysql-redo-archive [root@ip-xxx-xx-xx-xx ~]# chmod -R 700 /var/lib/mysql-redo-archive/
- Login into mysqlsh by root user and check the variable innodb_redo_log_archive_dirs.
[root@ip-xxx-xx-xx-xx ~]# mysqlsh -uroot -p MySQL Shell 8.0.33 Copyright (c) 2016, 2023, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help' or '?' for help; 'quit' to exit. WARNING: Using a password on the command line interface can be insecure. Creating a session to 'root@localhost' Fetching schema names for auto-completion... Press ^C to stop. Your MySQL connection id is 10 (X protocol) Server version: 8.0.33-25 Percona Server (GPL), Release 25, Revision 60c9e2c5 No default schema selected; type use <schema> to set one. MySQL localhost:33060+ ssl SQL > MySQL localhost:33060+ ssl JS > sql Switching to SQL mode... Commands end with ; Fetching global names for auto-completion... Press ^C to stop. MySQL localhost:33060+ ssl SQL > show global variables like 'innodb_redo_log_ar%'; +------------------------------+-------+ | Variable_name | Value | +------------------------------+-------+ | innodb_redo_log_archive_dirs | | +------------------------------+-------+ 1 row in set (0.0031 sec)
- Set the value of variable innodb_redo_log_archive_dirs to the one in which we created the directories for redo log archives as below.
MySQL localhost:33060+ ssl SQL > set persist innodb_redo_log_archive_dirs='backup1:/var/lib/mysql-redo-archive/'; Query OK, 0 rows affected (0.0019 sec) MySQL localhost:33060+ ssl SQL > show global variables like 'innodb_redo_log_ar%'; +------------------------------+--------------------------------------+ | Variable_name | Value | +------------------------------+--------------------------------------+ | innodb_redo_log_archive_dirs | backup1:/var/lib/mysql-redo-archive/ | +------------------------------+--------------------------------------+ 1 row in set (0.0025 sec)
- Now it’s ready to work, but it’s not enabled. It will be enabled when the session initializes the backup and will invoke innodb_redo_log_archive_start().
MySQL localhost:33060+ ssl SQL > select innodb_redo_log_archive_start('backup1', 'backup1');
+-----------------------------------------------------+
| innodb_redo_log_archive_start('backup1', 'backup1') |
+-----------------------------------------------------+
| 0 |
+-----------------------------------------------------+
1 row in set (0.0229 sec)How to check if the redo log archiving is active
MySQL localhost:33060+ ssl SQL > select * from performance_schema.file_instances where event_name like '%::redo_log_archive_file'G *************************** 1. row *************************** FILE_NAME: /var/lib/mysql-redo-archive/backup1/archive.f255a32d-2fb4-11ee-889e-0242ac110005.000001.log EVENT_NAME: wait/io/file/innodb/meb::redo_log_archive_file OPEN_COUNT: 1 1 row in set (0.0015 sec)
So, this is not enough to ensure the redo log archiving is active. But we have the possibility also to check if the thread is active using this query:
MySQL localhost:33060+ ssl SQL > select thread_id, name, type from performance_schema.threads where name like '%redo_log_archive%'; +-----------+-----------------------------------------------------+------------+ | thread_id | name | type | +-----------+-----------------------------------------------------+------------+ | 48 | thread/innodb/meb::redo_log_archive_consumer_thread | BACKGROUND | +-----------+-----------------------------------------------------+------------+ 1 row in set (0.0021 sec)
If a row is returned, it means that the redo log archiving is enabled and active.
If, instead, you wish not to revert and to make it persist on service restart, add it to my.cnf under [mysqld]:
[mysqld] innodb_redo_log_archive_dirs = redo-archived1:/mysql/archived-redo1
How to stop redo log archiving
MySQL localhost:33060+ ssl SQL > select innodb_redo_log_archive_stop(); +--------------------------------+ | innodb_redo_log_archive_stop() | +--------------------------------+ | 0 | +--------------------------------+ 1 row in set (0.0009 sec)
[root@ip-xxx-xx-xx-xx ~]# ls -ltrh /var/lib/mysql-redo-archive/backup1/ total 24K -r--r-----. 1 mysql mysql 24K Aug 21 11:22 archive.94f4ab58-3d1c-11ee-9e4f-0af1fd7c44c9.000001.log
Revert back the changes
To revert back and disable redo log archiving:
mysql> SET GLOBAL innodb_redo_log_archive_dirs='';
How does Percona XtraBackup deal with redo log files?
PXB considers the log sequence number and starts copying away the data files when a backup is performed. The backup operation on InnoDB is a non-blocking operation, so the ibd files are allowed to change while the backup process is running. This means that while PXB is copying the files, the file reflects the state of the database at different times. PXB needs the redo log records for every change to the data files since it began execution.
PXB runs a background process that watches the transaction log files and copies changes from them. It needs to do this continually because InnoDB writes redo logs to the disk circularly and can be reused after a while. However, prior to PXB version 8.0.34-29, if InnoDB redo log archiving is not enabled and provided that redo logs are written faster, then PXB is able to copy it slower than it is written to the disk. It is possible to receive the below message:
xtrabackup: error: log block numbers mismatch: xtrabackup: error: expected log block no. X, but got no. Y from the log file. xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small. xtrabackup: Error: xtrabackup_copy_logfile() failed.
Earlier to the release of the feature redo log archiving in MySQL 8.0.17, the potential solutions to the above problem were:
- Increasing the size of the redo logs so they are not wrapped while PXB is working.
- The read speed is too slow, which usually is a sign of IO congestion.
- The write speed is too slow, usually a sign of IO congestion or network congestion.
- Taking backup off-peak time while the system saturation is lower.
However, now, with the use of the redo log archiving feature introduced in MySQL 8.0.17, you can overcome the slowness of physical backups and can efficiently perform the backup even during peak hours.
Also, PXB version 8.0.34-29 has introduced a new option –redo-log-arch-dir that can be used to set the redo log archiving directory if not already set in MySQL.
Using MySQL redo log archiving for Percona XtraBackup
So now we have a redo-log-file archived in the folder we created above demonstration.
[root@ip-xxx-xx-xx-xx ~]# ls -ltrh /var/lib/mysql-redo-archive/backup1/ total 24K -r--r-----. 1 mysql mysql 24K Aug 21 11:22 archive.94f4ab58-3d1c-11ee-9e4f-0af1fd7c44c9.000001.log
Test with Percona XtraBackup
- Run PXB as the owner of the mysqld, typically the MySQL OS user. We will be using version 8.0.34-29
[root@ip-xxx-xx-xx-xx ~]# xtrabackup --version 2023-09-11T16:07:17.696628-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql xtrabackup version 8.0.34-29 based on MySQL server 8.0.34 Linux (x86_64) (revision id: 5ba706ee)
This is required for PXB to use the InnoDB archiving feature. Otherwise, it will encounter a “Permission denied” error and will not use archiving as below:
2023-08-21T13:27:59.293520-00:00 0 [Note] [MY-011825] [Xtrabackup] xtrabackup redo_log_arch_dir is set to backup1:/var/lib/mysql-redo-archive/
2023-08-21T13:27:59.295379-00:00 0 [ERROR] [MY-011825] [Xtrabackup] failed to fetch query result select innodb_redo_log_archive_start('backup1', '1692710879293') : Cannot create redo log archive file '/var/lib/mysql-redo-archive/1692710879293/archive.94f4ab58-3d1c-11ee-9e4f-0af1fd7c44c9.000001.log' (OS errno: 13 - Permission denied)
2023-08-21T13:27:59.295429-00:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not used.PXB will create a temporary directory for the archive file. Since redo log archiving is a MySQL process, it needs to be owned by the MySQL OS user to be able to write to it.
1.1) Option 1: Login as the MySQL OS user:
shell> sudo -H -u mysql bash
Execute:
shell> xtrabackup --no-lock=1 --compress --parallel=4 --host=localhost --user=root --password='password_string' --backup=1 --target-dir=/Backup/21Aug 2> /tmp/b0-with-redo-archiving-as-mysql-os-user.log
1.2) Option 2: Directly execute as the root OS user for the MySQL user.
You must add -c on the command like:
shell> sudo -H -u mysql bash -c '<COMMAND>' 2> /tmp/backup.log
Execute:
shell> sudo -H -u mysql bash -c 'xtrabackup --no-lock=1 --compress --parallel=4 --host=localhost --user=root --password='password_string' --backup=1 --target-dir=/Backup/21Aug' 2> /tmp/b0-with-redo-archiving-as-mysql-os-user.log
4.2) Verify archiving is used:
shell> less /tmp/b0-with-redo-archiving-as-mysql-os-user.log 2023-08-22T13:36:02.773345-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql 2023-08-22T13:36:02.773543-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --no-lock=1 --compress --parallel=4 --host=localhost --user=root --password=* --backup=1 --target-dir=/Backup/22Aug xtrabackup version 8.0.34-29 based on MySQL server 8.0.34 Linux (x86_64) (revision id: 5ba706ee) 230822 13:36:02 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=localhost' as 'root' (using password: YES). 230822 13:36:02 version_check Connected to MySQL server 230822 13:36:02 version_check Executing a version check against the server... 230822 13:36:02 version_check Done. 2023-08-22T13:36:02.899397-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set 2023-08-22T13:36:02.906059-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.34 .... .... 2023-08-22T13:36:02.980089-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set 2023-08-22T13:36:02.986698-00:00 0 [Note] [MY-011825] [Xtrabackup] xtrabackup redo_log_arch_dir is set to backup1:/var/lib/mysql-redo-archive/ 2023-08-22T13:36:03.015207-00:00 0 [Note] [MY-011825] [Xtrabackup] Waiting for archive file '/var/lib/mysql-redo-archive//1692711362986/archive.94f4ab58-3d1c-11ee-9e4f-0af1fd7c44c9.000001.log' 2023-08-22T13:36:03.091572-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (19630018) .... .... 2023-08-22T13:36:04.569714-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Compressing file /Backup/22Aug/xtrabackup_info.zst 2023-08-22T13:36:05.583075-00:00 0 [Note] [MY-011825] [Xtrabackup] Transaction log of lsn (19630018) to (19630028) was copied. 2023-08-22T13:36:05.790293-00:00 0 [Note] [MY-011825] [Xtrabackup] completed OK!
How to set –redo-log-arch-dir=name in PXB command
PXB 8.0.34-29 introduces a new feature –redo-log-arch-dir=name. This option sets the redo log archive directory if this directory is not already set on the server. Here is a short example demonstrating the functioning of –redo-log-arch-dir parameter.
Suppose innodb_redo_log_archive_dirs is NULL, which means initially, the redo logs archive directory is not set in MySQL, as shown below.
mysql> select @@innodb_redo_log_archive_dirs; +--------------------------------+ | @@innodb_redo_log_archive_dirs | +--------------------------------+ | NULL | +--------------------------------+ 1 row in set (0.01 sec)
Create a redo log file directory and give the appropriate permissions
[root@ip-xxx-xx-xx-xx ~]# cd /var/lib/mysql-redo-archive/ [root@ip-xxx-xx-xx-xx ~]# mkdir redo_arch_backup ### Check mysql-redo-archive folder Permissions [root@ip-xxx-xx-xx-xx ~]# ls -ltrh /var/lib/ |grep archive drwx------. 4 mysql mysql 51 Sep 11 17:16 mysql-redo-archive
As you can see, PXB is all set to use the new redo archive directory just by passing the archive directory path to the parameter –redo-log-arch-dir
–redo-log-arch-dir=redo_arch_backup:/var/lib/mysql-redo-archive/
[root@ip-xxx-xx-xx-xx ~]# sudo -H -u mysql bash bash-5.1$ xtrabackup --no-lock=1 --compress --parallel=4 --host=localhost --user=root --password='password_string' --backup=1 --target-dir=/Backup/11Sept23_bkp --redo-log-arch-dir=redo_arch_backup:/var/lib/mysql-redo-archive/ 2> /tmp/a1-with-redo-archiving-as-mysql-os-user.log
Verifying the PXB logs:
[root@ip-xxx-xx-xx-xx ~]# cat /tmp/a1-with-redo-archiving-as-mysql-os-user.log 2023-09-11T17:16:47.836951-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql 2023-09-11T17:16:47.837105-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --no-lock=1 --compress --parallel=4 --host=localhost --user=root --password=* --backup=1 --target-dir=/Backup/11Sept23_bkp --redo_log_arch_dir=redo_arch_backup:/var/lib/mysql-redo-archive/ xtrabackup version 8.0.34-29 based on MySQL server 8.0.34 Linux (x86_64) (revision id: 5ba706ee) 230911 17:16:47 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=localhost' as 'root' (using password: YES). 230911 17:16:47 version_check Connected to MySQL server 230911 17:16:47 version_check Executing a version check against the server... 230911 17:16:47 version_check Done. 2023-09-11T17:16:47.956847-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set .... .... 2023-09-11T17:16:48.028506-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set 2023-09-11T17:16:48.034515-00:00 0 [Note] [MY-011825] [Xtrabackup] xtrabackup redo_log_arch_dir is set to redo_arch_backup:/var/lib/mysql-redo-archive/ 2023-09-11T17:16:48.056504-00:00 0 [Note] [MY-011825] [Xtrabackup] Waiting for archive file '/var/lib/mysql-redo-archive//1694452608034/archive.94f4ab58-3d1c-11ee-9e4f-0af1fd7c44c9.000001.log' 2023-09-11T17:16:48.137941-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (19816218) 2023-09-11T17:16:48.139886-00:00 0 [Note] [MY-012953] [InnoDB] Disabling background ibuf IO read threads. 2023-09-11T17:16:48.343117-00:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces .... .... 2023-09-11T17:16:49.496740-00:00 0 [Note] [MY-011825] [Xtrabackup] Compressing /Backup/11Sept23_bkp/backup-my.cnf.zst 2023-09-11T17:16:49.496789-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Compressing file /Backup/11Sept23_bkp/backup-my.cnf.zst 2023-09-11T17:16:49.498483-00:00 0 [Note] [MY-011825] [Xtrabackup] Compressing /Backup/11Sept23_bkp/xtrabackup_info.zst 2023-09-11T17:16:49.498521-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Compressing file /Backup/11Sept23_bkp/xtrabackup_info.zst 2023-09-11T17:16:50.501818-00:00 0 [Note] [MY-011825] [Xtrabackup] Transaction log of lsn (19816218) to (19816228) was copied. 2023-09-11T17:16:50.708435-00:00 0 [Note] [MY-011825] [Xtrabackup] completed OK!
mysql> select @@innodb_redo_log_archive_dirs; +-----------------------------------------------+ | @@innodb_redo_log_archive_dirs | +-----------------------------------------------+ | redo_arch_backup:/var/lib/mysql-redo-archive/ | +-----------------------------------------------+ 1 row in set (0.00 sec)
This means if innodb_redo_log_archive_dirs is not set in MySQL, PXB 8.0.34-29 will set this through –redo-log-arch-dir option and start using the redo log archive while a backup is initiated. When not set explicitly through –redo-log-arch-dir, PXB, by default, will first check innodb_redo_log_archive_dirs. If this variable is set in MySQL, as demonstrated previously, it will use the redo log arching directory assigned in innodb_redo_log_archive_dirs variable while performing the backup.
Conversely, if this MySQL variable innodb_redo_log_archive_dirs is NULL and you did not explicitly specify –redo-log-arch-dir while taking the backup, then PXB won’t use the redo log archiving feature.
Conclusion
This feature is crucial for heavy workloads when the backup storage doesn’t have the same capabilities as the production storage and cannot follow up the speed of the writes.
When enabled by the DBA, Percona XtraBackup will detect the archived redo log files and will use it automatically. To verify if the redo log archiving process was started and is still active, you can check the performance_schema.threads table as illustrated above. On the other hand, to use this feature with Percona XtraBackup, you can also explicitly use –redo-log-arch-dir=name.
Percona XtraBackup is a free, open source, complete online backup solution for all versions of Percona Server for MySQL and MySQL.
Related links:
https://docs.percona.com/percona-xtrabackup/8.0/release-notes/8.0/8.0.34-29.0.html
https://dev.mysql.com/doc/refman/8.0/en/innodb-redo-log.html#innodb-redo-log-archiving
https://www.percona.com/blog/innodb-redo-log-archiving/