standby master的模擬故障與修復

?

本文檔主要演示GreenPlum數據庫的Master節點的Standby在異常后的數據同步及Master節點故障后，Standby的切換。在操作前通過gpstate命令查看Standby的狀態是正常的，數據也在同步狀態。同時將現有master（原有的standy）和原有的master再做切換，恢復原狀。

1 檢查集群gpstate -f

[gpadmin@mdw ~]$ gpstate -f 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- ? Standby address ? ? ? ? = smdw 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- ? Standby data directory ? = /greenplum/gpdata/master/gpseg-1 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- ? Standby port ? ? ? ? ? ? = 5432 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- ? Standby PID ? ? ? ? ? ? = 7593 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:- ? Standby status ? ? ? ? ? = Standby host passive 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CADFDD8 20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------

2 模擬standby 故障

mdw cat .bash_profile export PATH export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata ? sdw3 and smdw vi .bash_profile ? export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata ? vi .bashr export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1 source /usr/local/greenplum-db/greenplum_path.sh export PGPORT=5432 export PGDATABASE=archdata ? smdw： pg_ctl stop -D $MASTER_DATA_DIRECTORY [gpadmin@sdw3 ~]$ pg_ctl stop -D $MASTER_DATA_DIRECTORY waiting for server to shut down.... done server stopped [gpadmin@sdw3 ~]$

3 mdw上檢查集群狀態

gpstate -f

[gpadmin@mdw ~]$ gpstate -f 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- ? Standby address ? ? ? ? = smdw 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- ? Standby data directory ? = /greenplum/gpdata/master/gpseg-1 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:- ? Standby port ? ? ? ? ? ? = 5432 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby PID ? ? ? ? ? ? = 0 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? <<<<<<<< 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby status ? ? ? ? ? = Standby process not running ? ? ? <<<<<<<< 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-No entries found. 20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------

4 mdw上創建一個表，測試master的元數據變化

psql

psql -h 10.102.254.27 -U gpadmin -d archdata ? [gpadmin@mdw ~]$ psql -h 10.102.254.27 -U gpadmin -d archdata psql (8.3.23) Type "help" for help. ? archdata=# \c You are now connected to database "archdata" as user "gpadmin". archdata=# archdata=# \d No relations found. archdata=# create table test (id int); NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table. HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew. CREATE TABLE archdata=# \dList of relationsSchema | Name | Type | Owner | Storage --------+------+-------+---------+---------public | test | table | gpadmin | heap (1 row)

5 啟動standby節點

[gpadmin@sdw3 ~]$ pg_ctl start -D $MASTER_DATA_DIRECTORY ? server starting [gpadmin@sdw3 ~]$ 2020-04-19 17:08:36.469615 GMT,,,p29616,th-410839168,,,,0,,,seg-1,,,,,"FATAL","22023","dbid (from -b option) is not specified or is invalid. This value must be >= 0, or >= -1 in utility mode. The dbid value to pass can be determined from this server's entry in the segment configuration; it may be -1 if running in utility mode.",,,,,,,,"PostmasterMain","postmaster.c",1175,

?

6 檢查集群狀態

gpstate -f ? [gpadmin@mdw ~]$ gpstate -f 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- ? Standby address ? ? ? ? = smdw 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- ? Standby data directory ? = /greenplum/gpdata/master/gpseg-1 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:- ? Standby port ? ? ? ? ? ? = 5432 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby PID ? ? ? ? ? ? = 0 ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? <<<<<<<< 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby status ? ? ? ? ? = Standby process not running ? ? ? <<<<<<<< 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-No entries found. 20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- [gpadmin@mdw ~]$

7 嘗試重啟集群

gpstop -a ? gpstart -a ? [gpadmin@mdw ~]$ gpstop -a 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1 20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 smdw ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /greenplum/gpdata/master/gpseg-1 -m fast -w -t 120 stop"' cmd had rc=1 completed=True halted=Falsestdout=''stderr='Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 pg_ctl: PID file "/greenplum/gpdata/master/gpseg-1/postmaster.pid" does not exist Is server running? ' 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:- ? Segments stopped successfully ? ? = 12 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:- ? Segments with errors during stop ? = 0 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found 20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory [gpadmin@mdw ~]$ gpstart -a 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051' 20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Setting new master era 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Master Started... 20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Shutting down master 20200420:01:16:08:012440 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Process results... 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- ? Successful segment starts ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? = 12 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- ? Failed segment starts ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? = 0 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:- ? Skipped segment starts (segments are marked down in configuration) ? = 0 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Successfully started 12 of 12 segment instances 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /greenplum/gpdata/master/gpseg-1 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Starting standby master 20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /greenplum/gpdata/master/gpseg-1 20200420:01:16:12:012440 gpstart:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 ' 20200420:01:16:13:012440 gpstart:mdw:gpadmin-[INFO]:-Database successfully started

8 再次確認狀態

gpstat -f ? [gpadmin@mdw ~]$ gpstate -f 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- ? Standby address ? ? ? ? = smdw 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- ? Standby data directory ? = /greenplum/gpdata/master/gpseg-1 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- ? Standby port ? ? ? ? ? ? = 5432 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- ? Standby PID ? ? ? ? ? ? = 30408 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:- ? Standby status ? ? ? ? ? = Standby host passive 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB21DB0 20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- ? 狀態正常

9 如果不能同步數據的處理

如果Standby數據不能自動同步，那么就需要重新初始化Standby的數據，也就是手動全量同步數據，因為Master只存數據庫的元數據，通常都很小，這個過程很快即可完成，但是需要將Master數據庫啟動到Utility模式，否則會遇到下面的錯誤。

?

[gpadmin@mdw ~]$ gpinitstandby -n ? 20200420:01:20:23:012710 gpinitstandby:mdw:gpadmin-[INFO]:-Standy master is already up and running. ? 使用gpstart –m只將Master節點啟動到Utility模式。 ? ? [gpadmin@mdw ~]$ gpstop -a [gpadmin@mdw ~]$ gpstop -a 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1 20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait... 20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait... 20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:- ? Segments stopped successfully ? ? = 12 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:- ? Segments with errors during stop ? = 0 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:----------------------------------------------------- 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts 20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory ? ? [gpadmin@mdw ~]$ gpstart -m [gpadmin@mdw ~]$ gpstart -m 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment... 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051' 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss. 20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-**************************************************************************** ? Continue with master-only startup Yy|Nn (default=N): ? > y > 20200420:01:22:07:012908 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode > 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information > 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... > 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Setting new master era > 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Master Started... > 此處需要輸入Y確認。 ? ? 將Master啟動到Utility模式后，使用gpinitstandby來重新初始化Standby的數據。 ? ? [gpadmin@mdw ~]$ gpinitstandby -n ? [gpadmin@mdw ~]$ gpinitstandby -n 20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Starting standby master 20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /greenplum/gpdata/master/gpseg-1 20200420:01:22:36:012938 gpinitstandby:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26' Offending key for IP in /home/gpadmin/.ssh/known_hosts:1 Matching host key in /home/gpadmin/.ssh/known_hosts:9 ' 20200420:01:22:37:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully started standby master 重啟數據庫，Standby的狀態就會正常。 ? ? [gpadmin@mdw ~]$ gpstop -a ? [gpadmin@mdw ~]$ gpstart -a [gpadmin@mdw ~]$ gpstate -f ? ' 20200420:01:23:39:013123 gpstart:mdw:gpadmin-[INFO]:-Database successfully started [gpadmin@mdw ~]$ gpstate -f 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4' 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24' 20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-Standby master details 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:----------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- ? Standby address ? ? ? ? = smdw 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- ? Standby data directory ? = /greenplum/gpdata/master/gpseg-1 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- ? Standby port ? ? ? ? ? ? = 5432 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- ? Standby PID ? ? ? ? ? ? = 31874 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:- ? Standby status ? ? ? ? ? = Standby host passive 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-------------------------------------------------------------- 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB22200 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB22200 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB22010 20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------

總結

以上是生活随笔為你收集整理的Greenplum实战--standby master的模拟故障与修复的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。