2020년 9월 25일 금요일

rpl_semi_sync_master_enabled에 대하여

Semi-sync replication에서 사용되는 rpl_semi_sync_master_enabled 시스템 변수를 변경하면 MySQL이 다운되는 버그를 발견했다.

해당 현상이 일어난 버전은 MySQL 5.7.14로 마스터와 슬레이브 모두 동일하다.

서버 상황
Semi-sync replication이 활성화 되어 있었으며 문제 없이 동기가 이루어지고 있었다.
시스템 변수를 변경하기 전의 Semi-sync replication관련 설정은 다음과 같았다.

+-------------------------------------------+------------+
| Variable_name | Value |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 500 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_for_slave_count | 1 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
| rpl_semi_sync_slave_enabled | OFF |
| rpl_semi_sync_slave_trace_level | 32 |
| rpl_stop_slave_timeout | 31536000 |
+-------------------------------------------+------------+

시나리오

admin@localhost:(none) 18:35:33>SET GLOBAL rpl_semi_sync_master_enabled = 0;
ERROR 2013 (HY000): Lost connection to MySQL server during query
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql
ERROR: 
Can't connect to the server

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql

rpl_semi_sync_master_enabled를 OFF로 변경하니 갑자기 서버가 다운되었다.
해당 변수는 다이나믹으로 변경 가능하다고 공식문서에 나와있다.
https://dev.mysql.com/doc/refman/5.7/en/replication-options-master.html#sysvar_rpl_semi_sync_master_enabled

에러 로그

[Note] Semi-sync replication switched OFF.
[Note] Semi-sync replication disabled on the master.
[Note] Stopping ack receiver thread
mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=3455
max_threads=12000
thread_count=2493
connection_count=2493
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 123072893 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/app/bin/mysqld(my_print_stacktrace+0x35)[0xf27295]
/app/bin/mysqld(handle_fatal_signal+0x4a4)[0x7b90d4]
/lib64/libpthread.so.0(+0xf5d0)[0x7f8fda9cb5d0]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
mysqld_safe Number of processes running now: 0
mysqld_safe mysqld restarted

원인 조사
결론부터 말하자면, 버그인 것 같다.
검색을 해보니 비슷한 현상의 글들을 몇가지 찾을 수 있었다.
https://bugs.mysql.com/bug.php?id=65711
https://jira.percona.com/browse/PS-7113
https://jira.percona.com/browse/PS-3311

댓글 없음:

댓글 쓰기

결혼이민비자 신청방법(F-6-1 국민의 배우자)

 제가 일본인 여자친구와 결혼 후, 한국에 귀국하기 위해 신청한 결혼이민비자에 대하여 작성해보도록 하겠습니다. 필자는 일본에서 근무하고 있었으며, 한국에서의 소득은 없었습니다. 결혼이민비자를 신청한 날짜는 2021-04-21 이며, 사증이 발급된 날짜...