HA Resource for CTDB fails after Upgrade to SP4

SUSE Linux Enterprise Extensions SLES High Availability Extension
1

Hi folks,

I have a 2-node HA Cluster for file services (samba, nfs). After Upgrading one of them so SLES12 SP4 (HA SP4), the CTDB-Resource doesn’t start on the Upgraded host.
It’s because of the wrong ctdbd startup paramater “–logfile=/var/log/ctdb/log.ctdb” (instead of “–logging=file:/var/log/ctdb/log.ctdb”)

Any advice on solving this cluster issue?

Thanks!

Log from node with SP4:
+++ 15:04:11: ocf_start_trace:1000: echo
+++ 15:04:11: ocf_start_trace:1000: printenv
+++ 15:04:11: ocf_start_trace:1000: sort
++ 15:04:11: ocf_start_trace:1000: env=’
HA_LOGD=no
HA_LOGFACILITY=daemon
HA_cluster_type=corosync
HA_debug=0
HA_logfacility=daemon
HA_mcp=true
HA_quorum_type=corosync
HA_use_logd=off
LC_ALL=C
LC_CTYPE=de_DE.UTF-8
LRMD_MAX_CHILDREN=4
OCF_EXIT_REASON_PREFIX=ocf-exit-reason:
OCF_RA_VERSION_MAJOR=1
OCF_RA_VERSION_MINOR=0
OCF_RESKEY_CRM_meta_clone=0
OCF_RESKEY_CRM_meta_clone_max=2
OCF_RESKEY_CRM_meta_clone_node_max=1
OCF_RESKEY_CRM_meta_globally_unique=false
OCF_RESKEY_CRM_meta_name=start
OCF_RESKEY_CRM_meta_notify=false
OCF_RESKEY_CRM_meta_on_node=fileserver2t
OCF_RESKEY_CRM_meta_on_node_uuid=740214560
OCF_RESKEY_CRM_meta_timeout=90000
OCF_RESKEY_CRM_meta_trace_ra=1
OCF_RESKEY_crm_feature_set=3.0.13
OCF_RESKEY_ctdb_manages_samba=false
OCF_RESKEY_ctdb_manages_winbind=false
OCF_RESKEY_ctdb_recovery_lock=/data/ctdb/samba/ctdb.lock
OCF_RESKEY_ctdb_socket=/var/lib/ctdb/ctdb.socket
OCF_RESKEY_trace_ra=1
OCF_RESOURCE_INSTANCE=res_CTDB
OCF_RESOURCE_PROVIDER=heartbeat
OCF_RESOURCE_TYPE=CTDB
OCF_ROOT=/usr/lib/ocf
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb
PCMK_cluster_type=corosync
PCMK_debug=0
PCMK_logfacility=daemon
PCMK_mcp=true
PCMK_quorum_type=corosync
PCMK_service=lrmd
PCMK_use_logd=off
PCMK_watchdog=true
PWD=/var/lib/pacemaker/cores
SBD_DELAY_START=
SBD_DEVICE=/dev/mapper/360002ac000000000000002260001a15d-part1
SBD_OPTS=
SBD_PACEMAKER=
SBD_STARTMODE=
SBD_WATCHDOG=
SHLVL=1
_=/usr/bin/printenv
__OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/CTDB/res_CTDB.start.2019-04-15.15:04:11
__OCF_TRC_MANAGE=1’
++ 15:04:11: source:1054: ocf_is_true off
++ 15:04:11: ocf_is_true:103: case “$1” in
++ 15:04:11: ocf_is_true:103: case “$1” in
++ 15:04:11: ocf_is_true:105: false

  • 15:04:11: main:78: var_prefix=/var/lib/ctdb
  • 15:04:11: main:79: ‘[’ ‘!’ -d /var/lib/ctdb ‘]’
  • 15:04:11: main:83: run_prefix=/run
  • 15:04:11: main:84: ‘[’ ‘!’ -d /var/lib/ctdb ‘]’
  • 15:04:11: main:88: : false
  • 15:04:11: main:89: : false
  • 15:04:11: main:90: :
  • 15:04:11: main:91: :
  • 15:04:11: main:92: :
  • 15:04:11: main:93: : yes
  • 15:04:11: main:94: : 100
  • 15:04:11: main:95: : no
  • 15:04:11: main:97: : /etc/ctdb
  • 15:04:11: main:98: : /usr/bin/ctdb
  • 15:04:11: main:99: : /usr/sbin/ctdbd
  • 15:04:11: main:100: : /var/lib/ctdb
  • 15:04:11: main:101: : /var/log/ctdb/log.ctdb
  • 15:04:11: main:102: : /run/ctdb
  • 15:04:11: main:103: : /var/lib/ctdb/ctdb.socket
  • 15:04:11: main:104: : 2
  • 15:04:11: main:106: : /etc/samba/smb.conf
  • 15:04:11: main:107: : tdbsam
  • 15:04:11: main:108: : tdb2
  • 15:04:11: main:358: ‘[’ -f /etc/sysconfig/ctdb ‘]’
  • 15:04:11: main:359: CTDB_SYSCONFIG=/etc/sysconfig/ctdb
  • 15:04:11: main:369: CTDB_SYSCONFIG_BACKUP=/etc/sysconfig/ctdb.ctdb-ra-orig
  • 15:04:11: main:783: case $__OCF_ACTION in
  • 15:04:11: main:787: ctdb_start
  • 15:04:11: ctdb_start:561: ctdb_monitor
  • 15:04:11: ctdb_monitor:702: local status
    ++ 15:04:11: ctdb_monitor:708: invoke_ctdb status
    ++ 15:04:11: invoke_ctdb:373: local timeout
    ++ 15:04:11: invoke_ctdb:374: local timelimit
    ++ 15:04:11: invoke_ctdb:375: timeout=3
    ++ 15:04:11: invoke_ctdb:376: timelimit=120
    ++ 15:04:11: invoke_ctdb:378: ‘[’ -n 90000 ‘]’
    ++ 15:04:11: invoke_ctdb:379: timeout=90
    ++ 15:04:11: invoke_ctdb:380: timelimit=90
    ++ 15:04:11: invoke_ctdb:382: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:11: ctdb_monitor:708: status=‘connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:709: ‘[’ 1 -ne 0 ‘]’
  • 15:04:11: ctdb_monitor:710: echo ‘connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:710: grep -qs ‘Connection refused’
  • 15:04:11: ctdb_monitor:712: echo ‘connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:712: grep -qs ‘No such file or directory’
  • 15:04:11: ctdb_monitor:714: echo ‘connect()’ failed, errno=111 Failed to connect to CTDB daemon ‘(/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:714: grep -qs ‘connect() failed’
  • 15:04:11: ctdb_monitor:715: return 7
  • 15:04:11: ctdb_start:564: ctdb_validate
  • 15:04:11: ctdb_validate:732: for binary in pkill
  • 15:04:11: ctdb_validate:733: check_binary pkill
  • 15:04:11: check_binary:55: have_binary pkill
  • 15:04:11: have_binary:67: ‘[’ ‘’ = 1 ‘]’
    ++ 15:04:11: have_binary:70: echo pkill
    ++ 15:04:11: have_binary:70: sed -e ‘s/ -.*//’
  • 15:04:11: have_binary:70: local bin=pkill
    ++ 15:04:11: have_binary:71: which pkill
  • 15:04:11: have_binary:71: test -x /usr/bin/pkill
  • 15:04:11: ctdb_validate:736: ‘[’ -z /etc/sysconfig/ctdb ‘]’
  • 15:04:11: ctdb_validate:741: ocf_is_true false
  • 15:04:11: ocf_is_true:103: case “$1” in
  • 15:04:11: ocf_is_true:105: false
  • 15:04:11: ctdb_validate:746: ‘[’ -f /etc/ctdb/public_addresses ‘]’
  • 15:04:11: ctdb_validate:750: ‘[’ ‘!’ -f /etc/ctdb/nodes ‘]’
  • 15:04:11: ctdb_validate:755: ‘[’ -z /data/ctdb/samba/ctdb.lock ‘]’
  • 15:04:11: ctdb_validate:760: ‘[’ / == ‘!’ ‘]’
    ++ 15:04:11: ctdb_validate:770: dirname /data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:654: local a
    ++ 15:04:11: dirname:655: local b
    ++ 15:04:11: dirname:657: ‘[’ 1 = 1 ‘]’
    ++ 15:04:11: dirname:658: a=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:659: ‘[’ 1 ‘]’
    ++ 15:04:11: dirname:660: b=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:661: ‘[’ /data/ctdb/samba/ctdb.lock = /data/ctdb/samba/ctdb.lock ‘]’
    ++ 15:04:11: dirname:661: break
    ++ 15:04:11: dirname:664: b=/data/ctdb/samba
    ++ 15:04:11: dirname:665: ‘[’ -z /data/ctdb/samba -o /data/ctdb/samba/ctdb.lock = /data/ctdb/samba ‘]’
    ++ 15:04:11: dirname:667: echo /data/ctdb/samba
    ++ 15:04:11: dirname:668: return 0
  • 15:04:11: ctdb_validate:770: lock_dir=/data/ctdb/samba
  • 15:04:11: ctdb_validate:771: touch /data/ctdb/samba/11867
  • 15:04:11: ctdb_validate:772: ‘[’ 0 ‘!=’ 0 ‘]’
  • 15:04:11: ctdb_validate:776: rm /data/ctdb/samba/11867
  • 15:04:11: ctdb_validate:779: return 0
  • 15:04:11: ctdb_start:565: rv=0
  • 15:04:11: ctdb_start:566: ‘[’ 0 -ne 0 ‘]’
  • 15:04:11: ctdb_start:569: init_smb_conf
  • 15:04:11: init_smb_conf:441: ocf_is_true false
  • 15:04:11: ocf_is_true:103: case “$1” in
  • 15:04:11: ocf_is_true:105: false
  • 15:04:11: init_smb_conf:441: return 0
  • 15:04:11: ctdb_start:570: ‘[’ 0 -ne 0 ‘]’
  • 15:04:11: ctdb_start:576: generate_ctdb_sysconfig
  • 15:04:11: generate_ctdb_sysconfig:521: grep -qa ‘# CTDB-RA: Auto-generated’ /etc/sysconfig/ctdb
  • 15:04:11: generate_ctdb_sysconfig:522: ‘[’ 0 -ne 0 ‘]’
  • 15:04:11: generate_ctdb_sysconfig:526: ocf_log info ‘Generating new /etc/sysconfig/ctdb’
  • 15:04:11: ocf_log:323: ‘[’ 2 -lt 2 ‘]’
  • 15:04:11: ocf_log:327: __OCF_PRIO=info
  • 15:04:11: ocf_log:328: shift
  • 15:04:11: ocf_log:329: __OCF_MSG=‘Generating new /etc/sysconfig/ctdb’
  • 15:04:11: ocf_log:331: case “${__OCF_PRIO}” in
  • 15:04:11: ocf_log:335: __OCF_PRIO=INFO
  • 15:04:11: ocf_log:340: ‘[’ INFO = DEBUG ‘]’
  • 15:04:11: ocf_log:343: ha_log ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:11: ha_log:255: __ha_log ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:11: __ha_log:187: local ignore_stderr=false
  • 15:04:11: __ha_log:188: local loglevel
  • 15:04:11: __ha_log:190: ‘[’ ‘xINFO: Generating new /etc/sysconfig/ctdb’ = x–ignore-stderr ‘]’
  • 15:04:11: __ha_log:192: ‘[’ none = daemon ‘]’
  • 15:04:11: __ha_log:194: tty
  • 15:04:11: __ha_log:209: set_logtag
  • 15:04:11: set_logtag:177: ‘[’ -z ‘’ ‘]’
  • 15:04:11: set_logtag:178: ‘[’ -n res_CTDB ‘]’
  • 15:04:11: set_logtag:179: HA_LOGTAG=‘CTDB(res_CTDB)[11867]’
  • 15:04:11: __ha_log:211: ‘[’ xno = xyes ‘]’
  • 15:04:11: __ha_log:219: ‘[’ -n daemon ‘]’
  • 15:04:11: __ha_log:221: : logging through syslog
  • 15:04:11: __ha_log:223: loglevel=notice
  • 15:04:11: __ha_log:224: case “${*}” in
  • 15:04:11: __ha_log:227: loglevel=info
  • 15:04:11: __ha_log:229: logger -t ‘CTDB(res_CTDB)[11867]’ -p daemon.info ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:11: __ha_log:232: ‘[’ -n ‘’ ‘]’
  • 15:04:11: __ha_log:238: ‘[’ -z daemon -a -z ‘’ ‘]’
  • 15:04:11: __ha_log:244: ‘[’ -n /dev/null ‘]’
  • 15:04:11: __ha_log:246: : appending to /dev/null
  • 15:04:11: __ha_log:247: ‘[’ x ‘!=’ /dev/nullx ‘]’
    ++ 15:04:11: _ha_log:248: hadate
    ++ 15:04:11: hadate:173: date '+%Y/%m/%d    %T ’
  • 15:04:11: __ha_log:248: echo ‘CTDB(res_CTDB)[11867]: 2019/04/15_15:04:11’ ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:11: generate_ctdb_sysconfig:537: cat
    ++ 15:04:11: generate_ctdb_sysconfig:537: ocf_is_true yes
    ++ 15:04:11: ocf_is_true:103: case “$1” in
    ++ 15:04:11: ocf_is_true:103: case “$1” in
    ++ 15:04:11: ocf_is_true:104: true
    ++ 15:04:11: generate_ctdb_sysconfig:537: echo yes
    ++ 15:04:11: generate_ctdb_sysconfig:537: ocf_is_true false
    ++ 15:04:11: ocf_is_true:103: case “$1” in
    ++ 15:04:11: ocf_is_true:103: case “$1” in
    ++ 15:04:11: ocf_is_true:105: false
    ++ 15:04:11: generate_ctdb_sysconfig:537: echo no
    ++ 15:04:11: generate_ctdb_sysconfig:537: ocf_is_true false
    ++ 15:04:11: ocf_is_true:103: case “$1” in
    ++ 15:04:11: ocf_is_true:103: case “$1” in
    ++ 15:04:11: ocf_is_true:105: false
    ++ 15:04:11: generate_ctdb_sysconfig:537: echo no
  • 15:04:11: generate_ctdb_sysconfig:544: append_ctdb_sysconfig CTDB_SERVICE_SMB
  • 15:04:11: append_ctdb_sysconfig:514: ‘[’ -n ‘’ ‘]’
  • 15:04:11: generate_ctdb_sysconfig:545: append_ctdb_sysconfig CTDB_SERVICE_NMB
  • 15:04:11: append_ctdb_sysconfig:514: ‘[’ -n ‘’ ‘]’
  • 15:04:11: generate_ctdb_sysconfig:546: append_ctdb_sysconfig CTDB_SERVICE_WINBIND
  • 15:04:11: append_ctdb_sysconfig:514: ‘[’ -n ‘’ ‘]’
  • 15:04:11: ctdb_start:577: enable_event_scripts
  • 15:04:11: enable_event_scripts:392: local event_dir
  • 15:04:11: enable_event_scripts:393: event_dir=/etc/ctdb/events.d
  • 15:04:11: enable_event_scripts:395: chmod u+x /etc/ctdb/events.d/00.ctdb
  • 15:04:11: enable_event_scripts:397: ‘[’ -f /etc/ctdb/public_addresses ‘]’
  • 15:04:11: enable_event_scripts:400: chmod a-x /etc/ctdb/events.d/10.interface
  • 15:04:12: enable_event_scripts:402: ‘[’ -f /etc/ctdb/static-routes ‘]’
  • 15:04:12: enable_event_scripts:405: chmod a-x /etc/ctdb/events.d/11.routing
  • 15:04:12: enable_event_scripts:407: ocf_is_true false
  • 15:04:12: ocf_is_true:103: case “$1” in
  • 15:04:12: ocf_is_true:105: false
  • 15:04:12: enable_event_scripts:408: ocf_is_true false
  • 15:04:12: ocf_is_true:103: case “$1” in
  • 15:04:12: ocf_is_true:105: false
  • 15:04:12: enable_event_scripts:411: chmod a-x /etc/ctdb/events.d/50.samba
  • 15:04:12: ctdb_start:580: local log_option
    ++ 15:04:12: ctdb_start:583: ctdb version
    ++ 15:04:12: ctdb_start:583: awk ‘{print $NF}’
    ++ 15:04:12: ctdb_start:583: sed ‘s/\.\?[[:alpha:]].*//’
  • 15:04:12: ctdb_start:583: version=4.6.16-
  • 15:04:12: ctdb_start:584: ocf_version_cmp 4.6.16- 4.2.14
  • 15:04:12: ocf_version_cmp:606: ocf_is_ver 4.6.16-
  • 15:04:12: ocf_is_ver:576: echo 4.6.16-
  • 15:04:12: ocf_is_ver:576: grep ‘^[0-9][0-9.-]*[0-9]$’
  • 15:04:12: ocf_version_cmp:606: return 3
  • 15:04:12: ctdb_start:585: ‘[’ 3 -eq 2 ‘]’
  • 15:04:12: ctdb_start:591: log_option=–logfile=/var/log/ctdb/log.ctdb
  • 15:04:12: ctdb_start:592: ‘[’ /var/log/ctdb/log.ctdb = syslog ‘]’
    ++ 15:04:12: ctdb_start:596: dirname /var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:654: local a
    ++ 15:04:12: dirname:655: local b
    ++ 15:04:12: dirname:657: ‘[’ 1 = 1 ‘]’
    ++ 15:04:12: dirname:658: a=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:659: ‘[’ 1 ‘]’
    ++ 15:04:12: dirname:660: b=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:661: ‘[’ /var/log/ctdb/log.ctdb = /var/log/ctdb/log.ctdb ‘]’
    ++ 15:04:12: dirname:661: break
    ++ 15:04:12: dirname:664: b=/var/log/ctdb
    ++ 15:04:12: dirname:665: ‘[’ -z /var/log/ctdb -o /var/log/ctdb/log.ctdb = /var/log/ctdb ‘]’
    ++ 15:04:12: dirname:667: echo /var/log/ctdb
    ++ 15:04:12: dirname:668: return 0
  • 15:04:12: ctdb_start:596: ‘[’ ‘!’ -d /var/log/ctdb ‘]’
  • 15:04:12: ctdb_start:602: mkdir -p /run/ctdb
  • 15:04:12: ctdb_start:605: local pub_addr_option
  • 15:04:12: ctdb_start:606: pub_addr_option=
  • 15:04:12: ctdb_start:607: ‘[’ -f /etc/ctdb/public_addresses ‘]’
  • 15:04:12: ctdb_start:610: local start_as_disabled
  • 15:04:12: ctdb_start:611: start_as_disabled=–start-as-disabled
  • 15:04:12: ctdb_start:612: ocf_is_true no
  • 15:04:12: ocf_is_true:103: case “$1” in
  • 15:04:12: ocf_is_true:105: false
  • 15:04:12: ctdb_start:612: start_as_disabled=
  • 15:04:12: ctdb_start:616: /usr/sbin/ctdbd --reclock=/data/ctdb/samba/ctdb.lock --nlist=/etc/ctdb/nodes --socket=/var/lib/ctdb/ctdb.socket --dbdir=/var/lib/ctdb --dbdir-persistent=/var/lib/ctdb/persistent --event-script-dir=/etc/ctdb/events.d --notification-script=/etc/ctdb/notify.sh --transport=tcp --logfile=/var/log/ctdb/log.ctdb -d 2
  • 15:04:12: ctdb_start:626: ‘[’ 1 -ne 0 ‘]’
  • 15:04:12: ctdb_start:628: cleanup_smb_conf
  • 15:04:12: cleanup_smb_conf:505: ocf_is_true false
  • 15:04:12: ocf_is_true:103: case “$1” in
  • 15:04:12: ocf_is_true:105: false
  • 15:04:12: cleanup_smb_conf:505: return 0
  • 15:04:12: ctdb_start:630: ocf_exit_reason ‘Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: ocf_exit_reason:356: local cookie=ocf-exit-reason:
  • 15:04:12: ocf_exit_reason:357: local fmt
  • 15:04:12: ocf_exit_reason:358: local msg
  • 15:04:12: ocf_exit_reason:365: case $# in
  • 15:04:12: ocf_exit_reason:367: fmt=%s
  • 15:04:12: ocf_exit_reason:377: ‘[’ -z ocf-exit-reason: ‘]’
    ++ 15:04:12: ocf_exit_reason:382: printf %s ‘Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: ocf_exit_reason:382: msg=‘Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: ocf_exit_reason:383: printf '%s%s
    ’ ocf-exit-reason: ‘Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: ocf_exit_reason:384: __ha_log --ignore-stderr ‘ERROR: Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: __ha_log:187: local ignore_stderr=false
  • 15:04:12: __ha_log:188: local loglevel
  • 15:04:12: __ha_log:190: ‘[’ x–ignore-stderr = x–ignore-stderr ‘]’
  • 15:04:12: __ha_log:190: ignore_stderr=true
  • 15:04:12: __ha_log:190: shift
  • 15:04:12: __ha_log:192: ‘[’ none = daemon ‘]’
  • 15:04:12: __ha_log:194: tty
  • 15:04:12: __ha_log:209: set_logtag
  • 15:04:12: set_logtag:177: ‘[’ -z ‘CTDB(res_CTDB)[11867]’ ‘]’
  • 15:04:12: __ha_log:211: ‘[’ xno = xyes ‘]’
  • 15:04:12: __ha_log:219: ‘[’ -n daemon ‘]’
  • 15:04:12: __ha_log:221: : logging through syslog
  • 15:04:12: __ha_log:223: loglevel=notice
  • 15:04:12: __ha_log:224: case “${*}” in
  • 15:04:12: __ha_log:225: loglevel=err
  • 15:04:12: __ha_log:229: logger -t ‘CTDB(res_CTDB)[11867]’ -p daemon.err ‘ERROR: Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: __ha_log:232: ‘[’ -n ‘’ ‘]’
  • 15:04:12: __ha_log:238: ‘[’ -z daemon -a -z ‘’ ‘]’
  • 15:04:12: __ha_log:244: ‘[’ -n /dev/null ‘]’
  • 15:04:12: __ha_log:246: : appending to /dev/null
  • 15:04:12: __ha_log:247: ‘[’ x ‘!=’ /dev/nullx ‘]’
    ++ 15:04:12: _ha_log:248: hadate
    ++ 15:04:12: hadate:173: date '+%Y/%m/%d    %T ’
  • 15:04:12: __ha_log:248: echo ‘CTDB(res_CTDB)[11867]: 2019/04/15_15:04:12’ ‘ERROR: Failed to execute /usr/sbin/ctdbd.’
  • 15:04:12: ctdb_start:631: return 1
  • 15:04:12: main:798: rc=1
  • 15:04:12: main:799: ocf_log debug ‘res_CTDB start : 1’
  • 15:04:12: ocf_log:323: ‘[’ 2 -lt 2 ‘]’
  • 15:04:12: ocf_log:327: __OCF_PRIO=debug
  • 15:04:12: ocf_log:328: shift
  • 15:04:12: ocf_log:329: __OCF_MSG=‘res_CTDB start : 1’
  • 15:04:12: ocf_log:331: case “${__OCF_PRIO}” in
  • 15:04:12: ocf_log:336: __OCF_PRIO=DEBUG
  • 15:04:12: ocf_log:340: ‘[’ DEBUG = DEBUG ‘]’
  • 15:04:12: ocf_log:341: ha_debug ‘DEBUG: res_CTDB start : 1’
  • 15:04:12: ha_debug:260: ‘[’ x0 = x0 ‘]’
  • 15:04:12: ha_debug:261: return 0
  • 15:04:12: main:800: exit 1

Log from node with SP3:
+++ 15:04:11: ocf_start_trace:933: echo
+++ 15:04:11: ocf_start_trace:933: printenv
+++ 15:04:11: ocf_start_trace:933: sort
++ 15:04:11: ocf_start_trace:933: env=’
HA_LOGD=no
HA_LOGFACILITY=daemon
HA_cluster_type=corosync
HA_debug=0
HA_logfacility=daemon
HA_mcp=true
HA_quorum_type=corosync
HA_use_logd=off
LC_ALL=C
LC_CTYPE=de_DE.UTF-8
LRMD_MAX_CHILDREN=4
OCF_EXIT_REASON_PREFIX=ocf-exit-reason:
OCF_RA_VERSION_MAJOR=1
OCF_RA_VERSION_MINOR=0
OCF_RESKEY_CRM_meta_clone=1
OCF_RESKEY_CRM_meta_clone_max=2
OCF_RESKEY_CRM_meta_clone_node_max=1
OCF_RESKEY_CRM_meta_globally_unique=false
OCF_RESKEY_CRM_meta_name=start
OCF_RESKEY_CRM_meta_notify=false
OCF_RESKEY_CRM_meta_on_node=fileserver1t
OCF_RESKEY_CRM_meta_on_node_uuid=740214559
OCF_RESKEY_CRM_meta_timeout=90000
OCF_RESKEY_CRM_meta_trace_ra=1
OCF_RESKEY_crm_feature_set=3.0.13
OCF_RESKEY_ctdb_manages_samba=false
OCF_RESKEY_ctdb_manages_winbind=false
OCF_RESKEY_ctdb_recovery_lock=/data/ctdb/samba/ctdb.lock
OCF_RESKEY_ctdb_socket=/var/lib/ctdb/ctdb.socket
OCF_RESKEY_trace_ra=1
OCF_RESOURCE_INSTANCE=res_CTDB
OCF_RESOURCE_PROVIDER=heartbeat
OCF_RESOURCE_TYPE=CTDB
OCF_ROOT=/usr/lib/ocf
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb
PCMK_cluster_type=corosync
PCMK_debug=0
PCMK_logfacility=daemon
PCMK_mcp=true
PCMK_quorum_type=corosync
PCMK_service=lrmd
PCMK_use_logd=off
PCMK_watchdog=true
PWD=/var/lib/pacemaker/cores
SBD_DELAY_START=
SBD_DEVICE=/dev/mapper/360002ac000000000000002260001a15d-part1
SBD_OPTS=
SBD_PACEMAKER=
SBD_STARTMODE=
SBD_WATCHDOG=
SHLVL=1
_=/usr/bin/printenv
__OCF_TRC_DEST=/var/lib/heartbeat/trace_ra/CTDB/res_CTDB.start.2019-04-15.15:04:11
__OCF_TRC_MANAGE=1’
++ 15:04:11: source:945: ocf_is_true off
++ 15:04:11: ocf_is_true:102: case “$1” in
++ 15:04:11: ocf_is_true:102: case “$1” in
++ 15:04:11: ocf_is_true:104: false

  • 15:04:11: main:78: var_prefix=/var/lib/ctdb
  • 15:04:11: main:79: ‘[’ ‘!’ -d /var/lib/ctdb ‘]’
  • 15:04:11: main:83: run_prefix=/run
  • 15:04:11: main:84: ‘[’ ‘!’ -d /var/lib/ctdb ‘]’
  • 15:04:11: main:88: : false
  • 15:04:11: main:89: : false
  • 15:04:11: main:90: :
  • 15:04:11: main:91: :
  • 15:04:11: main:92: :
  • 15:04:11: main:93: : yes
  • 15:04:11: main:94: : 100
  • 15:04:11: main:95: : no
  • 15:04:11: main:97: : /etc/ctdb
  • 15:04:11: main:98: : /usr/bin/ctdb
  • 15:04:11: main:99: : /usr/sbin/ctdbd
  • 15:04:11: main:100: : /var/lib/ctdb
  • 15:04:11: main:101: : /var/log/ctdb/log.ctdb
  • 15:04:11: main:102: : /run/ctdb
  • 15:04:11: main:103: : /var/lib/ctdb/ctdb.socket
  • 15:04:11: main:104: : 2
  • 15:04:11: main:106: : /etc/samba/smb.conf
  • 15:04:11: main:107: : tdbsam
  • 15:04:11: main:108: : tdb2
  • 15:04:11: main:358: ‘[’ -f /etc/sysconfig/ctdb ‘]’
  • 15:04:11: main:359: CTDB_SYSCONFIG=/etc/sysconfig/ctdb
  • 15:04:11: main:367: CTDB_SYSCONFIG_BACKUP=/etc/sysconfig/ctdb.ctdb-ra-orig
  • 15:04:11: main:757: case $__OCF_ACTION in
  • 15:04:11: main:761: ctdb_start
  • 15:04:11: ctdb_start:547: ctdb_monitor
  • 15:04:11: ctdb_monitor:676: local status
    ++ 15:04:11: ctdb_monitor:682: invoke_ctdb status
    ++ 15:04:11: invoke_ctdb:371: local timeout
    ++ 15:04:11: invoke_ctdb:372: local timelimit
    ++ 15:04:11: invoke_ctdb:373: timeout=3
    ++ 15:04:11: invoke_ctdb:374: timelimit=120
    ++ 15:04:11: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:11: invoke_ctdb:377: timeout=90
    ++ 15:04:11: invoke_ctdb:378: timelimit=90
    ++ 15:04:11: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:11: ctdb_monitor:682: status=‘connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:683: ‘[’ 1 -ne 0 ‘]’
  • 15:04:11: ctdb_monitor:684: echo ‘connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:684: grep -qs ‘Connection refused’
  • 15:04:11: ctdb_monitor:686: echo ‘connect() failed, errno=111
    Failed to connect to CTDB daemon (/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:686: grep -qs ‘No such file or directory’
  • 15:04:11: ctdb_monitor:688: echo ‘connect()’ failed, errno=111 Failed to connect to CTDB daemon ‘(/var/lib/ctdb/ctdb.socket)’
  • 15:04:11: ctdb_monitor:688: grep -qs ‘connect() failed’
  • 15:04:11: ctdb_monitor:689: return 7
  • 15:04:11: ctdb_start:550: ctdb_validate
  • 15:04:11: ctdb_validate:706: for binary in pkill
  • 15:04:11: ctdb_validate:707: check_binary pkill
  • 15:04:11: check_binary:56: have_binary pkill
  • 15:04:11: have_binary:68: ‘[’ ‘’ = 1 ‘]’
    ++ 15:04:11: have_binary:71: echo pkill
    ++ 15:04:11: have_binary:71: sed -e ‘s/ -.*//’
  • 15:04:11: have_binary:71: local bin=pkill
    ++ 15:04:11: have_binary:72: which pkill
  • 15:04:11: have_binary:72: test -x /usr/bin/pkill
  • 15:04:11: ctdb_validate:710: ‘[’ -z /etc/sysconfig/ctdb ‘]’
  • 15:04:11: ctdb_validate:715: ocf_is_true false
  • 15:04:11: ocf_is_true:102: case “$1” in
  • 15:04:11: ocf_is_true:104: false
  • 15:04:11: ctdb_validate:720: ‘[’ -f /etc/ctdb/public_addresses ‘]’
  • 15:04:11: ctdb_validate:724: ‘[’ ‘!’ -f /etc/ctdb/nodes ‘]’
  • 15:04:11: ctdb_validate:729: ‘[’ -z /data/ctdb/samba/ctdb.lock ‘]’
  • 15:04:11: ctdb_validate:734: ‘[’ / == ‘!’ ‘]’
    ++ 15:04:11: ctdb_validate:744: dirname /data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:592: local a
    ++ 15:04:11: dirname:593: local b
    ++ 15:04:11: dirname:595: ‘[’ 1 = 1 ‘]’
    ++ 15:04:11: dirname:596: a=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:597: ‘[’ 1 ‘]’
    ++ 15:04:11: dirname:598: b=/data/ctdb/samba/ctdb.lock
    ++ 15:04:11: dirname:599: ‘[’ /data/ctdb/samba/ctdb.lock = /data/ctdb/samba/ctdb.lock ‘]’
    ++ 15:04:11: dirname:599: break
    ++ 15:04:11: dirname:602: b=/data/ctdb/samba
    ++ 15:04:11: dirname:603: ‘[’ -z /data/ctdb/samba -o /data/ctdb/samba/ctdb.lock = /data/ctdb/samba ‘]’
    ++ 15:04:11: dirname:605: echo /data/ctdb/samba
    ++ 15:04:11: dirname:606: return 0
  • 15:04:11: ctdb_validate:744: lock_dir=/data/ctdb/samba
  • 15:04:11: ctdb_validate:745: touch /data/ctdb/samba/14280
  • 15:04:11: ctdb_validate:746: ‘[’ 0 ‘!=’ 0 ‘]’
  • 15:04:11: ctdb_validate:750: rm /data/ctdb/samba/14280
  • 15:04:11: ctdb_validate:753: return 0
  • 15:04:11: ctdb_start:551: rv=0
  • 15:04:11: ctdb_start:552: ‘[’ 0 -ne 0 ‘]’
  • 15:04:11: ctdb_start:555: init_smb_conf
  • 15:04:11: init_smb_conf:439: ocf_is_true false
  • 15:04:11: ocf_is_true:102: case “$1” in
  • 15:04:11: ocf_is_true:104: false
  • 15:04:11: init_smb_conf:439: return 0
  • 15:04:11: ctdb_start:556: ‘[’ 0 -ne 0 ‘]’
  • 15:04:11: ctdb_start:562: generate_ctdb_sysconfig
  • 15:04:11: generate_ctdb_sysconfig:507: grep -qa ‘# CTDB-RA: Auto-generated’ /etc/sysconfig/ctdb
  • 15:04:11: generate_ctdb_sysconfig:508: ‘[’ 0 -ne 0 ‘]’
  • 15:04:11: generate_ctdb_sysconfig:512: ocf_log info ‘Generating new /etc/sysconfig/ctdb’
  • 15:04:11: ocf_log:322: ‘[’ 2 -lt 2 ‘]’
  • 15:04:11: ocf_log:326: __OCF_PRIO=info
  • 15:04:11: ocf_log:327: shift
  • 15:04:11: ocf_log:328: __OCF_MSG=‘Generating new /etc/sysconfig/ctdb’
  • 15:04:11: ocf_log:330: case “${__OCF_PRIO}” in
  • 15:04:11: ocf_log:334: __OCF_PRIO=INFO
  • 15:04:11: ocf_log:339: ‘[’ INFO = DEBUG ‘]’
  • 15:04:11: ocf_log:342: ha_log ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:11: ha_log:254: __ha_log ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:11: __ha_log:186: local ignore_stderr=false
  • 15:04:11: __ha_log:187: local loglevel
  • 15:04:11: __ha_log:189: ‘[’ ‘xINFO: Generating new /etc/sysconfig/ctdb’ = x–ignore-stderr ‘]’
  • 15:04:11: __ha_log:191: ‘[’ none = daemon ‘]’
  • 15:04:11: __ha_log:193: tty
  • 15:04:11: __ha_log:208: set_logtag
  • 15:04:11: set_logtag:176: ‘[’ -z ‘’ ‘]’
  • 15:04:11: set_logtag:177: ‘[’ -n res_CTDB ‘]’
  • 15:04:11: set_logtag:178: HA_LOGTAG=‘CTDB(res_CTDB)[14280]’
  • 15:04:11: __ha_log:210: ‘[’ xno = xyes ‘]’
  • 15:04:11: __ha_log:218: ‘[’ -n daemon ‘]’
  • 15:04:11: __ha_log:220: : logging through syslog
  • 15:04:11: __ha_log:222: loglevel=notice
  • 15:04:11: __ha_log:223: case “${*}” in
  • 15:04:11: __ha_log:226: loglevel=info
  • 15:04:11: __ha_log:228: logger -t ‘CTDB(res_CTDB)[14280]’ -p daemon.info ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:12: __ha_log:231: ‘[’ -n ‘’ ‘]’
  • 15:04:12: __ha_log:237: ‘[’ -z daemon -a -z ‘’ ‘]’
  • 15:04:12: __ha_log:243: ‘[’ -n /dev/null ‘]’
  • 15:04:12: __ha_log:245: : appending to /dev/null
  • 15:04:12: __ha_log:246: ‘[’ x ‘!=’ /dev/nullx ‘]’
    ++ 15:04:12: _ha_log:247: hadate
    ++ 15:04:12: hadate:172: date '+%Y/%m/%d    %T ’
  • 15:04:12: __ha_log:247: echo ‘CTDB(res_CTDB)[14280]: 2019/04/15_15:04:12’ ‘INFO: Generating new /etc/sysconfig/ctdb’
  • 15:04:12: generate_ctdb_sysconfig:523: cat
    ++ 15:04:12: generate_ctdb_sysconfig:523: ocf_is_true yes
    ++ 15:04:12: ocf_is_true:102: case “$1” in
    ++ 15:04:12: ocf_is_true:102: case “$1” in
    ++ 15:04:12: ocf_is_true:103: true
    ++ 15:04:12: generate_ctdb_sysconfig:523: echo yes
    ++ 15:04:12: generate_ctdb_sysconfig:523: ocf_is_true false
    ++ 15:04:12: ocf_is_true:102: case “$1” in
    ++ 15:04:12: ocf_is_true:102: case “$1” in
    ++ 15:04:12: ocf_is_true:104: false
    ++ 15:04:12: generate_ctdb_sysconfig:523: echo no
    ++ 15:04:12: generate_ctdb_sysconfig:523: ocf_is_true false
    ++ 15:04:12: ocf_is_true:102: case “$1” in
    ++ 15:04:12: ocf_is_true:102: case “$1” in
    ++ 15:04:12: ocf_is_true:104: false
    ++ 15:04:12: generate_ctdb_sysconfig:523: echo no
  • 15:04:12: generate_ctdb_sysconfig:530: append_ctdb_sysconfig CTDB_SERVICE_SMB
  • 15:04:12: append_ctdb_sysconfig:500: ‘[’ -n ‘’ ‘]’
  • 15:04:12: generate_ctdb_sysconfig:531: append_ctdb_sysconfig CTDB_SERVICE_NMB
  • 15:04:12: append_ctdb_sysconfig:500: ‘[’ -n ‘’ ‘]’
  • 15:04:12: generate_ctdb_sysconfig:532: append_ctdb_sysconfig CTDB_SERVICE_WINBIND
  • 15:04:12: append_ctdb_sysconfig:500: ‘[’ -n ‘’ ‘]’
  • 15:04:12: ctdb_start:563: enable_event_scripts
  • 15:04:12: enable_event_scripts:390: local event_dir
  • 15:04:12: enable_event_scripts:391: event_dir=/etc/ctdb/events.d
  • 15:04:12: enable_event_scripts:393: chmod u+x /etc/ctdb/events.d/00.ctdb
  • 15:04:12: enable_event_scripts:395: ‘[’ -f /etc/ctdb/public_addresses ‘]’
  • 15:04:12: enable_event_scripts:398: chmod a-x /etc/ctdb/events.d/10.interface
  • 15:04:12: enable_event_scripts:400: ‘[’ -f /etc/ctdb/static-routes ‘]’
  • 15:04:12: enable_event_scripts:403: chmod a-x /etc/ctdb/events.d/11.routing
  • 15:04:12: enable_event_scripts:405: ocf_is_true false
  • 15:04:12: ocf_is_true:102: case “$1” in
  • 15:04:12: ocf_is_true:104: false
  • 15:04:12: enable_event_scripts:406: ocf_is_true false
  • 15:04:12: ocf_is_true:102: case “$1” in
  • 15:04:12: ocf_is_true:104: false
  • 15:04:12: enable_event_scripts:409: chmod a-x /etc/ctdb/events.d/50.samba
  • 15:04:12: ctdb_start:566: local log_option
  • 15:04:12: ctdb_start:567: log_option=–logging=file:/var/log/ctdb/log.ctdb
  • 15:04:12: ctdb_start:568: ‘[’ /var/log/ctdb/log.ctdb = syslog ‘]’
    ++ 15:04:12: ctdb_start:570: dirname /var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:592: local a
    ++ 15:04:12: dirname:593: local b
    ++ 15:04:12: dirname:595: ‘[’ 1 = 1 ‘]’
    ++ 15:04:12: dirname:596: a=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:597: ‘[’ 1 ‘]’
    ++ 15:04:12: dirname:598: b=/var/log/ctdb/log.ctdb
    ++ 15:04:12: dirname:599: ‘[’ /var/log/ctdb/log.ctdb = /var/log/ctdb/log.ctdb ‘]’
    ++ 15:04:12: dirname:599: break
    ++ 15:04:12: dirname:602: b=/var/log/ctdb
    ++ 15:04:12: dirname:603: ‘[’ -z /var/log/ctdb -o /var/log/ctdb/log.ctdb = /var/log/ctdb ‘]’
    ++ 15:04:12: dirname:605: echo /var/log/ctdb
    ++ 15:04:12: dirname:606: return 0
  • 15:04:12: ctdb_start:570: ‘[’ ‘!’ -d /var/log/ctdb ‘]’
  • 15:04:12: ctdb_start:576: mkdir -p /run/ctdb
  • 15:04:12: ctdb_start:579: local pub_addr_option
  • 15:04:12: ctdb_start:580: pub_addr_option=
  • 15:04:12: ctdb_start:581: ‘[’ -f /etc/ctdb/public_addresses ‘]’
  • 15:04:12: ctdb_start:584: local start_as_disabled
  • 15:04:12: ctdb_start:585: start_as_disabled=–start-as-disabled
  • 15:04:12: ctdb_start:586: ocf_is_true no
  • 15:04:12: ocf_is_true:102: case “$1” in
  • 15:04:12: ocf_is_true:104: false
  • 15:04:12: ctdb_start:586: start_as_disabled=
  • 15:04:12: ctdb_start:590: /usr/sbin/ctdbd --reclock=/data/ctdb/samba/ctdb.lock --nlist=/etc/ctdb/nodes --socket=/var/lib/ctdb/ctdb.socket --dbdir=/var/lib/ctdb --dbdir-persistent=/var/lib/ctdb/persistent --event-script-dir=/etc/ctdb/events.d --notification-script=/etc/ctdb/notify.sh --transport=tcp --logging=file:/var/log/ctdb/log.ctdb -d 2
  • 15:04:12: ctdb_start:600: ‘[’ 0 -ne 0 ‘]’
  • 15:04:12: ctdb_start:609: true
  • 15:04:12: ctdb_start:613: sleep 1
    ++ 15:04:13: ctdb_start:614: invoke_ctdb status
    ++ 15:04:13: invoke_ctdb:371: local timeout
    ++ 15:04:13: invoke_ctdb:372: local timelimit
    ++ 15:04:13: invoke_ctdb:373: timeout=3
    ++ 15:04:13: invoke_ctdb:374: timelimit=120
    ++ 15:04:13: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:13: invoke_ctdb:377: timeout=90
    ++ 15:04:13: invoke_ctdb:378: timelimit=90
    ++ 15:04:13: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:13: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:-1’
  • 15:04:13: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:13: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:-1’
  • 15:04:13: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:13: ctdb_start:609: true
  • 15:04:13: ctdb_start:613: sleep 1
    ++ 15:04:14: ctdb_start:614: invoke_ctdb status
    ++ 15:04:14: invoke_ctdb:371: local timeout
    ++ 15:04:14: invoke_ctdb:372: local timelimit
    ++ 15:04:14: invoke_ctdb:373: timeout=3
    ++ 15:04:14: invoke_ctdb:374: timelimit=120
    ++ 15:04:14: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:14: invoke_ctdb:377: timeout=90
    ++ 15:04:14: invoke_ctdb:378: timelimit=90
    ++ 15:04:14: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:14: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0’
  • 15:04:14: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:14: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0’
  • 15:04:14: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:14: ctdb_start:609: true
  • 15:04:14: ctdb_start:613: sleep 1
    ++ 15:04:15: ctdb_start:614: invoke_ctdb status
    ++ 15:04:15: invoke_ctdb:371: local timeout
    ++ 15:04:15: invoke_ctdb:372: local timelimit
    ++ 15:04:15: invoke_ctdb:373: timeout=3
    ++ 15:04:15: invoke_ctdb:374: timelimit=120
    ++ 15:04:15: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:15: invoke_ctdb:377: timeout=90
    ++ 15:04:15: invoke_ctdb:378: timelimit=90
    ++ 15:04:15: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:15: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0’
  • 15:04:15: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:15: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0’
  • 15:04:15: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:15: ctdb_start:609: true
  • 15:04:15: ctdb_start:613: sleep 1
    ++ 15:04:16: ctdb_start:614: invoke_ctdb status
    ++ 15:04:16: invoke_ctdb:371: local timeout
    ++ 15:04:16: invoke_ctdb:372: local timelimit
    ++ 15:04:16: invoke_ctdb:373: timeout=3
    ++ 15:04:16: invoke_ctdb:374: timelimit=120
    ++ 15:04:16: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:16: invoke_ctdb:377: timeout=90
    ++ 15:04:16: invoke_ctdb:378: timelimit=90
    ++ 15:04:16: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:16: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0’
  • 15:04:16: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:16: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:INVALID
    Size:2
    hash:0 lmaster:0
    hash:1 lmaster:1
    Recovery mode:RECOVERY (1)
    Recovery master:0’
  • 15:04:16: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:16: ctdb_start:609: true
  • 15:04:16: ctdb_start:613: sleep 1
    ++ 15:04:17: ctdb_start:614: invoke_ctdb status
    ++ 15:04:17: invoke_ctdb:371: local timeout
    ++ 15:04:17: invoke_ctdb:372: local timelimit
    ++ 15:04:17: invoke_ctdb:373: timeout=3
    ++ 15:04:17: invoke_ctdb:374: timelimit=120
    ++ 15:04:17: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:17: invoke_ctdb:377: timeout=90
    ++ 15:04:17: invoke_ctdb:378: timelimit=90
    ++ 15:04:17: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:17: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:17: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:17: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:17: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:17: ctdb_start:609: true
  • 15:04:17: ctdb_start:613: sleep 1
    ++ 15:04:18: ctdb_start:614: invoke_ctdb status
    ++ 15:04:18: invoke_ctdb:371: local timeout
    ++ 15:04:18: invoke_ctdb:372: local timelimit
    ++ 15:04:18: invoke_ctdb:373: timeout=3
    ++ 15:04:18: invoke_ctdb:374: timelimit=120
    ++ 15:04:18: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:18: invoke_ctdb:377: timeout=90
    ++ 15:04:18: invoke_ctdb:378: timelimit=90
    ++ 15:04:18: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:18: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:18: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:18: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:18: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:19: ctdb_start:609: true
  • 15:04:19: ctdb_start:613: sleep 1
    ++ 15:04:20: ctdb_start:614: invoke_ctdb status
    ++ 15:04:20: invoke_ctdb:371: local timeout
    ++ 15:04:20: invoke_ctdb:372: local timelimit
    ++ 15:04:20: invoke_ctdb:373: timeout=3
    ++ 15:04:20: invoke_ctdb:374: timelimit=120
    ++ 15:04:20: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:20: invoke_ctdb:377: timeout=90
    ++ 15:04:20: invoke_ctdb:378: timelimit=90
    ++ 15:04:20: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:20: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:20: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:20: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:20: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:20: ctdb_start:609: true
  • 15:04:20: ctdb_start:613: sleep 1
    ++ 15:04:21: ctdb_start:614: invoke_ctdb status
    ++ 15:04:21: invoke_ctdb:371: local timeout
    ++ 15:04:21: invoke_ctdb:372: local timelimit
    ++ 15:04:21: invoke_ctdb:373: timeout=3
    ++ 15:04:21: invoke_ctdb:374: timelimit=120
    ++ 15:04:21: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:21: invoke_ctdb:377: timeout=90
    ++ 15:04:21: invoke_ctdb:378: timelimit=90
    ++ 15:04:21: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:21: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:21: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:21: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:21: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:21: ctdb_start:609: true
  • 15:04:21: ctdb_start:613: sleep 1
    ++ 15:04:22: ctdb_start:614: invoke_ctdb status
    ++ 15:04:22: invoke_ctdb:371: local timeout
    ++ 15:04:22: invoke_ctdb:372: local timelimit
    ++ 15:04:22: invoke_ctdb:373: timeout=3
    ++ 15:04:22: invoke_ctdb:374: timelimit=120
    ++ 15:04:22: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:22: invoke_ctdb:377: timeout=90
    ++ 15:04:22: invoke_ctdb:378: timelimit=90
    ++ 15:04:22: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:22: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:22: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:22: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:22: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:22: ctdb_start:609: true
  • 15:04:22: ctdb_start:613: sleep 1
    ++ 15:04:23: ctdb_start:614: invoke_ctdb status
    ++ 15:04:23: invoke_ctdb:371: local timeout
    ++ 15:04:23: invoke_ctdb:372: local timelimit
    ++ 15:04:23: invoke_ctdb:373: timeout=3
    ++ 15:04:23: invoke_ctdb:374: timelimit=120
    ++ 15:04:23: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:23: invoke_ctdb:377: timeout=90
    ++ 15:04:23: invoke_ctdb:378: timelimit=90
    ++ 15:04:23: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:23: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:23: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:23: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:23: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:23: ctdb_start:609: true
  • 15:04:23: ctdb_start:613: sleep 1
    ++ 15:04:24: ctdb_start:614: invoke_ctdb status
    ++ 15:04:24: invoke_ctdb:371: local timeout
    ++ 15:04:24: invoke_ctdb:372: local timelimit
    ++ 15:04:24: invoke_ctdb:373: timeout=3
    ++ 15:04:24: invoke_ctdb:374: timelimit=120
    ++ 15:04:24: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:24: invoke_ctdb:377: timeout=90
    ++ 15:04:24: invoke_ctdb:378: timelimit=90
    ++ 15:04:24: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:24: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:24: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:24: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:24: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:24: ctdb_start:609: true
  • 15:04:24: ctdb_start:613: sleep 1
    ++ 15:04:25: ctdb_start:614: invoke_ctdb status
    ++ 15:04:25: invoke_ctdb:371: local timeout
    ++ 15:04:25: invoke_ctdb:372: local timelimit
    ++ 15:04:25: invoke_ctdb:373: timeout=3
    ++ 15:04:25: invoke_ctdb:374: timelimit=120
    ++ 15:04:25: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:25: invoke_ctdb:377: timeout=90
    ++ 15:04:25: invoke_ctdb:378: timelimit=90
    ++ 15:04:25: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:25: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:25: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:25: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:25: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:25: ctdb_start:609: true
  • 15:04:25: ctdb_start:613: sleep 1
    ++ 15:04:26: ctdb_start:614: invoke_ctdb status
    ++ 15:04:26: invoke_ctdb:371: local timeout
    ++ 15:04:26: invoke_ctdb:372: local timelimit
    ++ 15:04:26: invoke_ctdb:373: timeout=3
    ++ 15:04:26: invoke_ctdb:374: timelimit=120
    ++ 15:04:26: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:26: invoke_ctdb:377: timeout=90
    ++ 15:04:26: invoke_ctdb:378: timelimit=90
    ++ 15:04:26: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:26: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:26: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:26: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:26: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:26: ctdb_start:609: true
  • 15:04:26: ctdb_start:613: sleep 1
    ++ 15:04:27: ctdb_start:614: invoke_ctdb status
    ++ 15:04:27: invoke_ctdb:371: local timeout
    ++ 15:04:27: invoke_ctdb:372: local timelimit
    ++ 15:04:27: invoke_ctdb:373: timeout=3
    ++ 15:04:27: invoke_ctdb:374: timelimit=120
    ++ 15:04:27: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:27: invoke_ctdb:377: timeout=90
    ++ 15:04:27: invoke_ctdb:378: timelimit=90
    ++ 15:04:27: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:27: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:27: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:27: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:27: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:27: ctdb_start:609: true
  • 15:04:27: ctdb_start:613: sleep 1
    ++ 15:04:28: ctdb_start:614: invoke_ctdb status
    ++ 15:04:28: invoke_ctdb:371: local timeout
    ++ 15:04:28: invoke_ctdb:372: local timelimit
    ++ 15:04:28: invoke_ctdb:373: timeout=3
    ++ 15:04:28: invoke_ctdb:374: timelimit=120
    ++ 15:04:28: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:28: invoke_ctdb:377: timeout=90
    ++ 15:04:28: invoke_ctdb:378: timelimit=90
    ++ 15:04:28: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:28: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:28: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:28: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:28: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:28: ctdb_start:609: true
  • 15:04:28: ctdb_start:613: sleep 1
    ++ 15:04:29: ctdb_start:614: invoke_ctdb status
    ++ 15:04:29: invoke_ctdb:371: local timeout
    ++ 15:04:29: invoke_ctdb:372: local timelimit
    ++ 15:04:29: invoke_ctdb:373: timeout=3
    ++ 15:04:29: invoke_ctdb:374: timelimit=120
    ++ 15:04:29: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:29: invoke_ctdb:377: timeout=90
    ++ 15:04:29: invoke_ctdb:378: timelimit=90
    ++ 15:04:29: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:29: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:29: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:29: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:29: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:29: ctdb_start:609: true
  • 15:04:29: ctdb_start:613: sleep 1
    ++ 15:04:30: ctdb_start:614: invoke_ctdb status
    ++ 15:04:30: invoke_ctdb:371: local timeout
    ++ 15:04:30: invoke_ctdb:372: local timelimit
    ++ 15:04:30: invoke_ctdb:373: timeout=3
    ++ 15:04:30: invoke_ctdb:374: timelimit=120
    ++ 15:04:30: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:30: invoke_ctdb:377: timeout=90
    ++ 15:04:30: invoke_ctdb:378: timelimit=90
    ++ 15:04:30: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:30: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:30: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:30: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:30: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:30: ctdb_start:609: true
  • 15:04:30: ctdb_start:613: sleep 1
    ++ 15:04:31: ctdb_start:614: invoke_ctdb status
    ++ 15:04:31: invoke_ctdb:371: local timeout
    ++ 15:04:31: invoke_ctdb:372: local timelimit
    ++ 15:04:31: invoke_ctdb:373: timeout=3
    ++ 15:04:31: invoke_ctdb:374: timelimit=120
    ++ 15:04:31: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:31: invoke_ctdb:377: timeout=90
    ++ 15:04:31: invoke_ctdb:378: timelimit=90
    ++ 15:04:31: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:31: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:31: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:31: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:31: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:31: ctdb_start:609: true
  • 15:04:31: ctdb_start:613: sleep 1
    ++ 15:04:32: ctdb_start:614: invoke_ctdb status
    ++ 15:04:32: invoke_ctdb:371: local timeout
    ++ 15:04:32: invoke_ctdb:372: local timelimit
    ++ 15:04:32: invoke_ctdb:373: timeout=3
    ++ 15:04:32: invoke_ctdb:374: timelimit=120
    ++ 15:04:32: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:32: invoke_ctdb:377: timeout=90
    ++ 15:04:32: invoke_ctdb:378: timelimit=90
    ++ 15:04:32: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:32: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:32: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:32: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 UNHEALTHY (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:32: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:32: ctdb_start:609: true
  • 15:04:32: ctdb_start:613: sleep 1
    ++ 15:04:33: ctdb_start:614: invoke_ctdb status
    ++ 15:04:33: invoke_ctdb:371: local timeout
    ++ 15:04:33: invoke_ctdb:372: local timelimit
    ++ 15:04:33: invoke_ctdb:373: timeout=3
    ++ 15:04:33: invoke_ctdb:374: timelimit=120
    ++ 15:04:33: invoke_ctdb:376: ‘[’ -n 90000 ‘]’
    ++ 15:04:33: invoke_ctdb:377: timeout=90
    ++ 15:04:33: invoke_ctdb:378: timelimit=90
    ++ 15:04:33: invoke_ctdb:380: /usr/bin/ctdb --socket=/var/lib/ctdb/ctdb.socket -t 90 -T 90 status
  • 15:04:33: ctdb_start:614: status=‘Number of nodes:2
    pnn:0 172.30.199.31 OK (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:33: ctdb_start:615: ‘[’ 0 -ne 0 ‘]’
  • 15:04:33: ctdb_start:621: echo ‘Number of nodes:2
    pnn:0 172.30.199.31 OK (THIS NODE)
    pnn:1 172.30.199.32 DISCONNECTED|UNHEALTHY|INACTIVE
    Generation:1254945127
    Size:1
    hash:0 lmaster:0
    Recovery mode:NORMAL (0)
    Recovery master:0’
  • 15:04:33: ctdb_start:621: grep -qs ‘UNHEALTHY (THIS’
  • 15:04:33: ctdb_start:625: set_ctdb_variables
  • 15:04:33: set_ctdb_variables:416: rv=0
  • 15:04:33: set_ctdb_variables:417: set
  • 15:04:33: set_ctdb_variables:417: grep ‘^CTDB_SET_’
  • 15:04:33: set_ctdb_variables:417: cut -d_ -f3-
  • 15:04:33: set_ctdb_variables:418: read v
  • 15:04:33: set_ctdb_variables:423: return 0
  • 15:04:33: ctdb_start:626: return 0
  • 15:04:33: main:772: rc=0
  • 15:04:33: main:773: ocf_log debug ‘res_CTDB start : 0’
  • 15:04:33: ocf_log:322: ‘[’ 2 -lt 2 ‘]’
  • 15:04:33: ocf_log:326: __OCF_PRIO=debug
  • 15:04:33: ocf_log:327: shift
  • 15:04:33: ocf_log:328: __OCF_MSG=‘res_CTDB start : 0’
  • 15:04:33: ocf_log:330: case “${__OCF_PRIO}” in
  • 15:04:33: ocf_log:335: __OCF_PRIO=DEBUG
  • 15:04:33: ocf_log:339: ‘[’ DEBUG = DEBUG ‘]’
  • 15:04:33: ocf_log:340: ha_debug ‘DEBUG: res_CTDB start : 0’
  • 15:04:33: ha_debug:259: ‘[’ x0 = x0 ‘]’
  • 15:04:33: ha_debug:260: return 0
  • 15:04:33: main:774: exit 0
2

geidelt,

It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.

These forums are peer-to-peer, best effort, volunteer run and that if your issue
is urgent or not getting a response, you might try one of the following options:

Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.suse.com/faq.php

If this is a reply to a duplicate posting or otherwise posted in error, please
ignore and accept our apologies and rest assured we will issue a stern reprimand
to our posting bot…

Good luck!

Your SUSE Forums Team
http://forums.suse.com