2017년 9월 13일 수요일

Greenplum Master Failover


1.      Greenplum master failover


1)        Summary


§  This is for Greenplum database of HA for soft only not DCA

§  Limitation:

-          The script checks only VM/Node status of Greenplum database master as ping not gpdb process.

§  Configuration:

-          If the Greenplum database is installed on cloud as vSphere, need to disable vmotion.

2)        File list


 

File Path
Filename
Description
node
/usr/local/bin
 
gpfailover.sh
Daemon for GPDB Master HA on smdw
smdw
gpfailovershutdown.sh
Shutdown Daemon
smdw
vip_env.sh
Vip configuration
Mdw, smdw
vip_start.sh
Start vip
mdw, smdw
vip_stop.sh
Stop vip
mdw, smdw
/etc/rc.d/init.d/
vip
vip service
mdw, smdw
Gpfailover
gpfailover service
smdw
/etc/rc.d/rc3.d/
S99gpfailover
Auto Damon Start when system reboot
smdw

 

3)        Source


 

n  gpfailover.sh

#!/bin/bash
 
## get vip config
. /usr/local/bin/vip_env.sh
 
. /usr/local/greenplum-db/greenplum_path.sh
export PGPORT=5432
 
## killing previous gpfailover daemon
NO_OF_PROCESS=`ps -ef | grep gpfailover.sh | grep -v grep | grep -v status | grep -v stop | wc -l`
 
if [ $NO_OF_PROCESS -gt 2 ]; then
    exit
fi
 
while true
do
    POSTGRESCNT=`ps -ef | grep postgres | wc -l`
    if [ $POSTGRESCNT -gt 8 ]
        then
            echo "${GPMDW} was activated !!!"
            exit 0
        else
        echo "${GPSMDW} is standby !!!"
    fi
 
    CNT_A=`ping -c 6 -i 1 ${GPMDW} | grep ", 0% packet loss" | wc -l`
    if [ $CNT_A -eq 1 ]
    then
        echo "alive"
    else
        echo "run gpactivatestandby!!!"
 
        logger -i -p user.emerg "GP:WARNING : GPDB MASTER VM IS NOT AVAILABLE !!!"
        logger -i -p user.emerg "GP:INFO : GPDB STANDBY MASTER IS STARTING TO FAILOVER !!!"
 
        su - gpadmin -c "gpactivatestandby -d /data/master/gpseg-1 -a -q"
 
        ### Checking the gpactivatestandby.log
        cd /home/gpadmin/gpAdminLogs
        SUCCESS_FG=`ls -lrt gpactivatestandby_*.log | tail -1  | awk '{print $9}' | xargs tail -30 | grep "The activation of the standby master has completed successfully" | wc -l`
        if [ ${SUCCESS_FG} -eq 1 ]
        then
             logger -i -p user.emerg "GP:INFO : The activation of the standby master has completed successfully "
        else
             logger -i -p user.emerg "GP:ERROR : Failed the activation of the standby master !!!"
             exit 1
        fi
 
        ifconfig ${VIP_INTERFACE} ${VIP}  netmask 255.255.255.0 up
 
        ### Checking VIP
        VIP_FG=`ifconfig | grep '{VIP}' | wc -l`
        if [ ${VIP_FG} -eq 1 ]
        then
             logger -i -p user.emerg "GP:INFO : Virtual IP( ${VIP} ) is up  "
        else
             logger -i -p user.emerg "GP:ERROR : Failed to start Virtual IP( ${VIP} ) "
             exit 1
        fi
 
        ### arping
        arping -f -w 10 -s ${VIP}  -U ${VIP_GW} -I ${VIP_INTERFACE}
        logger -i -p user.emerg "GP:IFNO : Executed arping."
        logger -i -p user.emerg "GP:INFO : Greenplum master failover has completed successfully "
        exit 0
    fi
 
    sleep 60
done
exit 0

 

n  gpfailovershutdown.sh

ps -ef | grep -i gpfailover.sh | grep -v grep | grep -v service| awk '{print $2}' | xargs kill -11

 

 

n  vip_env.sh

GPMDW=mdw
GPSMDW=smdw
VIP_INTERFACE=eth3:0
VIP=172.16.150.128
VIP_GW=172.16.150.1

 

 

n  vip_start.sh

#!/bin/bash
 
## get vip configure
. /usr/local/bin/vip_env.sh
 
ifconfig  ${VIP_INTERFACE} ${VIP}  netmask 255.255.255.0 up
arping -f -w 10 -s ${VIP}  -U ${VIP_GW} -I ${VIP_INTERFACE}

 

n  vip_stop.sh

#!/bin/bash
 
## get vip config
. /usr/local/bin/vip_env.sh
 
ifconfig ${VIP_INTERFACE} down

 

n  gpfailover

## cat /etc/rc.d/init.d/gpfailover
#!/bin/bash
 
## get vip config
. /usr/local/bin/vip_env.sh
 
ifconfig ${VIP_INTERFACE} down
[gpadmin@smdw bin]$ cat /etc/rc.d/init.d/gpfailover
case "$1" in
    start) #
        echo -n "Starting GPDB Auto failover Daemon: "
        nohup /usr/local/bin/gpfailover.sh > /dev/null 2>&1 &
        sleep 0.1
        echo
        $0 status
        ;;
 
    stop)
        echo -n "Sutting GPDB Auto failover Daemon: "
        /usr/local/bin/gpfailovershutdown.sh > /dev/null 2>&1
        sleep 0.1
        echo
        $0 status
        ;;
 
    restart)
        $0 stop
        $0 start
        ;;
 
    status)
        NO_OF_PROCESS=`ps -ef | grep gpfailover | grep -v grep | grep -v status | grep -v stop | wc -l`
        if [ $NO_OF_PROCESS -eq 0 ]; then
            echo "gpfailover daemon is not running"
        else
            echo "gpfailover daemon is running"
        fi
        ;;
    *)
    echo "Usage : $0 {start|stop|restart|status}"
    exit 1
esac
exit 0

 

n  S99gpfailover

## cat /etc/rc.d/rc3.d/S99gpfailover
service gpfailover start

 

4)        File copy & location on Server


n   

 
[Master and Standby Master]
tar xf gpdb_master_failover.tar
cd bin
chmod +x *
vi vip_env.sh   ### editing
cp *.sh /usr/local/bin/
cp vip /etc/rc.d/init.d/
cp gpfailover /etc/rc.d/init.d/
 
[Only Standby Master]
cp S99gpfailover /etc/rc.d/rc3.d/
 

5)        Standby Master configuration of bash_profile of gpadmin


n  Bash_profile

# su - gpadmin
$ vi ~/.bash_profile
 
## added below lines
. /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/data/master/gpseg-1
export PGPORT=5432

2.      Greenplum master failback


1)        Checking Standby Master on smdw


$ ssh smdw
$ gpstate -f
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.16.0 build 1'
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.16.0 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Jul 28 2017 17:17:49'
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:-Standby master instance not configured
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:--------------------------------------------------------------
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:--pg_stat_replication
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:--------------------------------------------------------------
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:-No entries found.
20170824:04:21:41:006303 gpstate:smdw:gpadmin-[INFO]:--------------------------------------------------------------

 

2)        Checking for filespace directory /data/master/gpseg-1 on mdw


$ ssh mdw
[gpadmin@smdw ~]$ gpinitstandby -s mdw
20170824:04:24:06:006838 gpinitstandby:smdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[INFO]:-Checking for filespace directory /data/master/gpseg-1 on mdw
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[ERROR]:-Filespace directory already exists on host mdw
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[ERROR]:-Failed to create standby
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[ERROR]:-Error initializing standby master: master data directory exists
[gpadmin@smdw ~]$

 

3)        Checking for filespace directory /data/master/gpseg-1 on mdw


$ ssh smdw
[gpadmin@smdw ~]$ gpinitstandby -s mdw
20170824:04:24:06:006838 gpinitstandby:smdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[INFO]:-Checking for filespace directory /data/master/gpseg-1 on mdw
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[ERROR]:-Filespace directory already exists on host mdw
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[ERROR]:-Failed to create standby
20170824:04:24:07:006838 gpinitstandby:smdw:gpadmin-[ERROR]:-Error initializing standby master: master data directory exists
[gpadmin@smdw ~]$
 
## mdw
[gpadmin@smdw ~]$ ssh mdw
Last login: Thu Aug 24 03:52:02 2017 from smdw
[gpadmin@mdw ~]$ cd /data/master/gpseg-1
[gpadmin@mdw gpseg-1]$ cd ..
[gpadmin@mdw master]$ mv gpseg-1 gpseg-1.old
[gpadmin@mdw master]$ exit
logout
Connection to mdw closed.
 

 

4)        Initialize standby master on smdw


$ ssh smdw
[gpadmin@smdw ~]$ gpinitstandby -s mdw
…
Do you want to continue with standby master initialization? Yy|Nn (default=N):
> y
…
20170824:04:26:16:007218 gpinitstandby:smdw:gpadmin-[INFO]:-Successfully created standby master on mdw
[gpadmin@smdw ~]$

5)        Failback on mdw


·         Shutdown master instance on smdw

·         Activate master instance on mdw

## SMDW
$ ssh smdw
[gpadmin@smdw ~]$ gpstop -m
20170824:04:30:29:008291 gpstop:smdw:gpadmin-[INFO]:-Starting gpstop with args: -m
 
Continue with master-only shutdown Yy|Nn (default=N):
> y
20170824:04:30:33:008291 gpstop:smdw:gpadmin-[INFO]:-Terminating processes for segment /data/master/gpseg-1
[gpadmin@smdw ~]$
 
$ ssh mdw
[gpadmin@mdw ~]$ gpactivatestandby -d $MASTER_DATA_DIRECTORY
Do you want to continue with standby master activation? Yy|Nn (default=N):
> y
20170824:04:32:59:057251 gpactivatestandby:mdw:gpadmin-[INFO]:-found standby postmaster process
20170824:04:33:00:057251 gpactivatestandby:mdw:gpadmin-[INFO]:-mdw is now the new primary master.
20170824:04:33:00:057251 gpactivatestandby:mdw:gpadmin-[INFO]:-You will need to update your user access mechanism to reflect
…
 [gpadmin@mdw ~]$

6)        Remove old version filespace on smdw


·         Remove filespace directory /data/master/gpseg-1

## SMDW
$ ssh smdw
[gpadmin@smdw ~]$ cd $MASTER_DATA_DIRECTORY
 [gpadmin@smdw master]$ mv gpseg-1 gpseg-1.old

7)        Initilize Standby master on smdw


·         Remove filespace directory /data/master/gpseg-1

# MDW
$ ssh mdw
[gpadmin@mdw ~]$ gpinitstandby -s smdw
20170824:04:43:32:066222 gpinitstandby:mdw:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...
..
Do you want to continue with standby master initialization? Yy|Nn (default=N):
> y
20170824:04:43:52:066222 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully created standby master on smdw
[gpadmin@mdw ~]$

8)        VIP Failback


·         vip down on smdw

·         vip up on mdw

$ ssh smdw
$ su –
# service vip stop
 
# ssh mdw
# service vip start

9)        Connection test with pgadmin (client tool)

시간:
라벨:

댓글 없음:

댓글 쓰기

피드 구독하기: 댓글 (Atom)

Greenplum Disaster Recovery

Greenplum DR를 사용하면, 재해 발생 전 특정 복구 시점으로 복구 지원 Greenplum DR은 Full 백업/복구, Incremental 백업/복구, WAL 로그 기반으로 DR 기능 제공 Greenplum Disaster Recovery 지...

  • Greenplum에서 리소스 관리
    보통 분석 시스템에서 항상 나오는 이야기가 배치는 빠른데, 일반 조회성 쿼리의 성능을 맞출 수 있느냐라는 질문을 많이 받습니다. 전사에서 보는 일별 현황이 있고, 배치도 수행하고, 머신러링 형태로 분석하는 것이 동시에 수행될 때 어느 업무가 중...
  • Greenplum DDL(테이블/인덱스 생성)
    1.       테이블 & 파티션 테이블 1)         Data Type            -        Greenplum 의 Character Length 는 Byte 수가 아닌 Character ...
  • Greenplum 소개
    1. Greenplum 소개 Greenplum은 대용량 분석 작업에 최적화된 병렬처리(MPP) 기반의 오픈소스 데이터 플랫폼입니다. Greenplum 처음 프로젝트에서는 데이터웨어하우스를 목적으로 시작되었습니다. 국내 시장에서 ...