#!/bin/ksh
#
# version 7.4, 2009-04-25, Michael Wang <xw73@columbia.edu>, sponsored by
# DataBase Intelligence Group, Inc. (http://DBIGusa.com).
# * Concatenate columns in select list to ensure they are on the same line.
# * Added orapw<SID> to the backup.
# * Change "set linesize 960" after "connect / as sysdba".
#
# version 7.3, 2009-03-10, Michael Wang <xw73@columbia.edu>, sponsored by
# DataBase Intelligence Group, Inc. (http://DBIGusa.com).
# * Find spfile from v$paramter instead of $ORACLE_HOME/dbs/spfileSID.ora
# * Preserve the path for pfile, spfile.
# * Save pfile created from spfile as <spfile>.txt.
#
# version 7.2, 2008-12-28, Michael Wang <xw73@columbia.edu>, sponsored by
# DataBase Intelligence Group, Inc. (http://DBIGusa.com).
# * Added COPY_DIRECTORY_PATH option to copy files under
#   DIRECTORY_PATH of dba_directories.
# * Added non-compression option (define COMPRESS_C as /bin/cat).
#
# version 7.1, 2008-12-26, Michael Wang <xw73@columbia.edu>.
# * exported PATH in shdoc required by perldoc.
# * Check NUM_COPIES before using it.
# * define RC_ORACLE to rc.oracle instead of oracle-SID which does not work.
#
# version 7.0, 2008-05-27, Michael Wang <xw73@columbia.edu>.
# * Port to ksh88 requested and sponsored by
#   DataBase Intelligence Group, Inc. (http://DBIGusa.com).
#
#   Tested on the following platforms:
#
#     - SunOS 5.10, ksh M-11/16/88i, Oracle 10.2.0.2.0.
#     - HP-UX B.11.23, ksh 11/16/88, Oracle 10.2.0.3.0.
#     - Enterprise Linux AS release 4 (October Update 4),
#         PD KSH v5.2.14 99/07/13.2, Oracle 10.2.0.1.0.
#
# * Added Oracle 10g %r archive_log_format fix.
# * Implemented NUM_COPIES parameter.
# * Various other fixes and clean up.
#
# version 6.1, 2004-02-24, Michael Wang <xw73@columbia.edu>.
# * Look for both pfile and spfile.
#
# version 6.0, 2003-06-02, Michael Wang <xw73@columbia.edu>.
# * Add ocfs support.
#
# version 5.9, 2003-01-24, Michael Wang <xw73@columbia.edu>.
# * Added "set linesize 960/set charwidth 960".
#
# version 5.8, 2002-08-08, Michael Wang <xw73@columbia.edu>.
# * Fix a bug "*(0)@$THE_THREAD" => "*(0)$THE_THREAD"
#
# version 5.7, 2002-05-06, Michael Wang <xw73@columbia.edu>.
# * Added RESTORE section.
# * Added help=pdf.
#
# version 5.6, 2002-05-01, Michael Wang <xw73@columbia.edu>.
# * Fix a bug in archive log file processing: only file with SEQ# SEQ_A
#   is copied, not SEQ_A through SEQ_B.
# * Copy ALL controlfiles instead just 1.
#
# version 5.5, 2002-03-01, Michael Wang <xw73@columbia.edu>.
# * Check the second location for backup control file.
#
# version 5.4, 2001-12-19, Michael Wang <xw73@columbia.edu>.
# - help=y
# - clean up with mycp() and ksh93.
# - replace "set -- $(IFS=@:; print -- $i)" with "set -- ${i//[@:]/ }".
#   /usr/dt/bin/dtksh: i=a@b:c; set -- $(IFS=@:; print -- $i); print $# => 1
#
# version 5.3, 2001-11-14, Michael Wang <xw73@columbia.edu>.
# * Added support for INIT_D=NA
# * Added support for REDO_WAIT_MINS to be minus number.
#
# version 5.2, 2001-09-24, Michael Wang <xw73@columbia.edu>.
# * Made hot backup independent on oraredo.ksh except creating and releasing
#   a redo lock. It copies (not rm) the small set of archive logs itself.
# * Introduce REDO_LOCK, PRE_PROCESSING, POST_PROCESSING parameters.
#
# version 5.1, 2001-09-18, Michael Wang <xw73@columbia.edu>.
# * Added "set echo on" for all SQL statements ("set echo on" does not work in
#   interactive mode (here document, pipe).
# * Added ORA-00235 problem (create backup controlfile failure) workaround
#   (try multiple times).
# * Added Oracle 9i support: use "sqlplus" if "svrmgrl" is not available.
# * sqlplus/9i related changes. Including:
# - adding "NF == 3" in _awk 'NF == 3 {print $2}' datafile.file > tablespace_
# - "set linesize 960"/"set trimspool on" to make sure we get the correct
#   file_name and tablespace_name.
#
# version 5.0, 2001-09-17, Michael Wang <xw73@columbia.edu>.
# * Added the functionality to backup on remote filesystem by specifying
#   user@host:/filesystem in the conf file.
# * Change "*_ora_$THE_PID.trc" to "*ora_${THE_PID}*.trc".
#   Solaris: ${lower_case_sid}_ora_$spid.trc, Linux: ora_spid.trc,
#   AIX: ora_$spid_${lower_case_sid}.trc. Oracle Enhancement #: 1665035.
#
# version 4.11, 2001-06-07, Michael Wang <xw73@columbia.edu>.
# * when CDPATH is set, cd may generate output, directed to /dev/null.
#   oracle:ADEV:816$cd app/..
#   /u0/oracle
#
# version 4.1, 2001-01-06, Michael Wang <xw73@columbia.edu>.
# * Fixed the problem in the case that archive log format contains %S and
#   %T. This was found out by Ming Sun. 
#
# version 4.0, 2000-11-04, Michael Wang <xw73@columbia.edu>.
# * Copy the small set of archived logs during the hot backup from
#   redo backup dirs to the hot backup dir. This eases recovery.
# * Make SUBMIT, INIT_D configurable options.
# * Fixed the problem when running the script without typing path (as in
#   orahot.ksh SID) by using dirname.
# * Added [[ -d $ORACLE_SID ]] || mkdir $ORACLE_SID for each BKUP_DIRS.
# * Reducing mandatory configuration parameters by setting the default
#   values.
# * Change multiple sed's to single sed to increase the performance. 
# * Cleaned the obsolete ksh syntax as reported by ksh93 -n option:
#   -eq within [[...]] obsolete, use ((...))
#   '=' obsolete, use '=='
# * unset LD_LIBRARY_PATH which is not needed for properly installed
#   Oracle binaries.
# * Better handling of trap and log messages when program is killed/failed.
#
# version 3.141, 2000-09-17, Michael Wang <xw73@columbia.edu>.
# * Added options for using compress, gzip, or bzip2 as compression tool.
#
# version 3.14, 2000-08-06, Michael Wang <xw73@columbia.edu>.
# * Added "ALTER SYSTEM ARCHIVE LOG CURRENT" before backing up controlfile
#   as well per discussion with Joe Ng and Jose Garcia. To quote Joe:
#  "The reply from Oracle Support is to do a switch log first before backing
#   up the controlfile. This will ensure all the redo logs generated during
#   the hot backup are captured and stored in the control file. This will
#   cover the scenaerio of having the split blocks that may occur during the
#   hot backup process. Regards, Joe Ng"
# * Chang on ORACLE_HOME= line shown below
#   OLD => ORACLE_HOME=$(sed -n "s/^$ORACLE_SID:\(.*\):.*$/\1/p" $ORATAB)
#   NEW => ORACLE_HOME=$(sed -n "s/^$ORACLE_SID:\([^:]*\):.*$/\1/p" $ORATAB)
#   This accommodates the generalized oratab with more than 3 fields.
#
# version 3.1, 2000-07-11, Michael Wang <xw73@columbia.edu>.
# * Reverse the order of getting current sequence number and backing up
#   controlfile per suggestion from Jose Garcia.
#
# version 3.0, 2000-06-11, Michael Wang <xw73@columbia.edu>.
# * Change made in consistency with redo backup script. Multiple 
#   "ALTER SYSTEM ARCHIVE LOG CURRENT" is not necessary. The current
#    sequence# at the beginning and end of the hot backup is obtained
#    from dictionary, and verified they are backed up. 
#
# version 2.3.1, 2000-06-05, Michael Wang <xw73@columbia.edu>.
# * Change "ALTER SYSTEM SWITCH LOGFILE" to "ALTER SYSTEM ARCHIVE LOG CURRENT" 
#   per discussion with Jose Garcia. The Oracle documentaton says:
#   --
#   "Note: if you specify a redo log file group for archiving with the CHANGE or 
#    CURRENT clause, and earlier redo log file groups are not yet archived, Oracle 
#    archives all unarchived groups up to and including the specified group."
#   --
#
# version 2.3, 2000-05-06, Michael Wang <xw73@columbia.edu>.
# * Changed the way how to get the sequence number. 
#
#   OLD:
#
#   ALTER SESSION SET NLS_DATE_FORMAT='MM/DD/YY HH24:MI:SS';
#   ALTER SYSTEM SWITCH LOGFILE;
#   SELECT 'THE_SEQUENCE_NUMBER=' || sequence# SEQUENCE_NUMBER
#     FROM v$log WHERE TO_DATE(first_time, 'MM/DD/YY HH24:MI:SS') =
#       (SELECT MAX(TO_DATE(first_time, 'MM/DD/YY HH24:MI:SS'))
#           FROM v$log WHERE archived='YES' AND status='INACTIVE');
#   ALTER SYSTEM SWITCH LOGFILE;
#
#   NEW:
#
#   ALTER SYSTEM SWITCH LOGFILE;
#   SELECT 'THE_SEQUENCE_NUMBER=' || sequence# SEQUENCE_NUMBER
#     FROM v$log WHERE status='CURRENT';
#   ALTER SYSTEM SWITCH LOGFILE;
#   ALTER SYSTEM SWITCH LOGFILE;
#
# version 2.2, 2000-02-20, Michael Wang <xw73@columbia.edu>.
# * Added compatibility with Oracle 8i which has option to use either
#   archive_log_dest or archive_log_dest_n (but not both).
#
# version 2.1, 1999-08-20, Michael Wang <xw73@columbia.edu>.
# * Replace ? to $ORACLE_HOME in $THE_DUMP_DEST, and $THE_LOG_ARCHIVE_DEST
# * General code cleanup.
#
# version 2.0, 1999-03-18, Michael Wang <xw73@columbia.edu>.
# * Also backup init<SID>.ora and get_controlfile.sql per suggestion from
#   Ping Huang, and Susan Gruebel, with help again from Stephanie Shen.
# * Check the existence of hourly redo process. Wait for it to finish and
#   run redo without CHKFS option. 
# * Minor code clean up.
#
# version 1.0, 1999-01-24, Michael Wang <xw73@columbia.edu>.
# * Rewrite based on Pro*C version developped in house last modified and
#   maintained by Satyaki Chatterjee, with help from Satyaki Chatterjee, 
#   Stephanie Shen, and Sheck Cho.

OPATH=; unset OPATH
OPATH=$PATH

GETCONF=; unset GETCONF

for GETCONF in /bin/getconf /usr/bin/getconf; do
  [[ -x $GETCONF ]] && {
    PATH=$($GETCONF PATH)  # avoid ksh93 (m, l, ...) getconf
    break;                 # which has problems.
  }
done

typeset -x PATH

function my_getopts {
  typeset PATH= i
  typeset -u _I
  [[ _I = *=* ]] && eval "${_I%%?(:)=*}=; unset ${_I%%?(:)=*}"
  for i; do
    _I=$i
    eval "${_I%%?(:)=*}=; unset ${_I%%?(:)=*}"
    case $_I in
      *[!:]=*)     eval ${_I%%=*}=${_I#*=}  ;;
      *:=*)        eval ${_I%%:=*}=${i#*:=} ;;
      !(*=*))      SID=$i                   ;;
    esac
  done
}

my_getopts SID:= HELP= DEBUG= "$@"

[[ $DEBUG = Y ]] && set -x

function shdoc {
  typeset version="3.1, 2008-05-28, Michael Wang <xw73@columbia.edu>."

  typeset help doc file

  typeset    tmpdir=/tmp/shdoc_$$_$RANDOM
  typeset    base tmpf i flag=N

  eval help=y doc=pod "$@"

  [[ -n ${OPATH:-} ]] && typeset -x PATH=$OPATH

  set -- $(echo $help $doc | tr '[a-z]' '[A-Z]')
  help=$1 doc=$2

  base=${file##*/}
  base=${base%.*sh}

  case ${doc} in
    (POD) mkdir -p ${tmpdir}          && tmpf=${tmpdir}/${base}.pod        ;;  
    (MAN) mkdir -p ${tmpdir}/man/man1 && tmpf=${tmpdir}/man/man1/${base}.1 ;;
  esac

  while IFS= read -r i; do
    [[ $i = "## ${doc}_START"* ]] && { flag=Y; continue; }
    [[ $i = "## ${doc}_STOP"*  ]] && { flag=N; continue; }

    if [[ $flag = "Y" ]]; then
      i=${i#"##"}
      i=${i#" "}
      printf "%s\n" "${i}"
    fi
  done < ${file} > ${tmpf}

  case $help in
    (Y)      case ${doc} in
               (POD) perldoc ${tmpf}                   ;;
               (MAN) MANPATH=${tmpdir}/man man ${base} ;;
             esac
             ;;
    (PS|PDF) case ${doc} in
               (POD) pod2man ${tmpf} ;;
               (MAN) cat     ${tmpf} ;;
             esac |
               groff -man |
                 case ${help} in
                   (PS)  cat     >${base}.ps  ;;
                   (PDF) ps2pdf - ${base}.pdf ;;
                 esac
             ;;
    (POD)    cp ${tmpf} ${base}.pod
             ;;
    (HTML)   pod2html ${tmpf} > ${base}.html
             rm pod2htmd.tmp pod2htmi.tmp
             ;;
  esac

  rm -rf ${tmpdir}
  return 0
}

[[ $HELP = @(Y|PS|PDF|POD|HTML) ]] && {
  : ${DOC:=man}
  shdoc file=$0 doc=$DOC help=$HELP
  exit 0
}

[[ -z "$SID" ]] && {
  print "ERROR: you must specify ORACLE_SID."
  exit 1
}

typeset -x ORACLE_SID=$SID

# $0 could be ".", convert to absolute path
PDIR=$(cd $(dirname $0)/.. >/dev/null && pwd -P) 

LOGDIR=$PDIR/operlog/$ORACLE_SID
[[ -d ${LOGDIR%/*} && ! -d ${LOGDIR} ]] && mkdir ${LOGDIR}

TMPDIR=$PDIR/temp/$ORACLE_SID
[[ -d ${TMPDIR%/*} && ! -d ${TMPDIR} ]] && mkdir ${TMPDIR}

CONFILE=$PDIR/etc/orahot_$ORACLE_SID.cf

cd $TMPDIR

function chkdir {
  [[ -n "$1" ]] || {
    print "ERROR: No directory specified for chkdir function."
    return 1
  }

  typeset i=$1

  if [[ -d $i ]]
  then 
    set -- $(j=$i/foo.$$; print $j; mkdir $j; ls -d $j; rmdir $j)
  elif [[ $i = *@*:* ]]; then
    # set -- ${i//[@:]/ }
    set -- $(echo $i | sed "s/[@:]/ /g")
    set -- $(rsh -l $1 $2 "j=$3/foo.\$\$; print \$j; mkdir \$j; ls -d \$j; rmdir \$j")
  else
    print "ERROR: Can not find $i directory."
    return 1
  fi

  if (( $# == 2 )) && [[ $1 = $2 ]]; then
    return 0
  else print "ERROR: Can not write in $i directory."
    return 1
  fi
}

chkdir $LOGDIR || { print "ERROR: in checking $LOGDIR."; exit 1; }

i=; unset i

while [[ -z $i || -f $i ]]
do
  [[ -z $i ]] || sleep $(( RANDOM%10 ))
  i=$LOGDIR/orahot_$ORACLE_SID.log.$(date +%m%d.%H%M)
done

print "Please check the logfile: $i."
exec >> $i 2>&1

function get_date { date +"%Y-%m-%d %H:%M:%S"; }

job_start= job_finish= job_elapse=; unset job_start job_finish job_elapse
job_start=$(get_date)
print "$job_start"
(( job_elapse = 0 ))
(( SECONDS = 0 ))

chkdir $TMPDIR || { print "ERROR: in checking $TMPDIR."; exit 1; }

i=; unset i

for i in REDO_LOCK SUBMIT INIT_D NUM_PROCS MAX_SECS REDO_WAIT_MINS \
         PRE_PROCESSING POST_PROCESSING NUM_COPIES
do
  eval "$i=; unset $i"
done

[[ -r $CONFILE ]] || {
  print "ERROR: Can not find configuration file: $CONFILE."
  exit 1
}

. $CONFILE

: ${SUBMIT:=/usr/local/bin/submit}

[[ -x $SUBMIT ]] || {
  print "ERROR: Can not find SUBMIT=$SUBMIT."
  exit 1
}

: ${INIT_D:=/etc/init.d}

[[ $INIT_D = NA ]] || {
  i= RC_ORACLE=; unset i RC_ORACLE
  RC_ORACLE=$TMPDIR/rc.oracle
  rm -rf $RC_ORACLE

  ( cd $INIT_D && for i in oracle-$ORACLE_SID rc.oracle online; do
      [[ -f $i ]] && { ln -s $INIT_D/$i $RC_ORACLE; break; }
    done
  )

  [[ -L $RC_ORACLE ]] || {
    print "ERROR: cannot find rc.oracle|oracle-$ORACLE_SID|online in $INIT_D."
    exit 1
  }
}

COMPRESS=; unset COMPRESS
COMPRESS=${COMPRESS_C%% *}

if   [[ $COMPRESS = */compress ]]; then THEZ=".Z"
elif [[ $COMPRESS = */gzip     ]]; then THEZ=".gz"
elif [[ $COMPRESS = */bzip2    ]]; then THEZ=".bz2"
elif [[ $COMPRESS = */cat      ]]; then THEZ=""
else                                    THEZ=".Z" ; COMPRESS_C="compress -c"
fi

[[ -x $COMPRESS ]] || {
  print "ERROR: $COMPRESS is not an executable."
  exit 1
}

: ${COPY_DIRECTORY_PATH:=N}

[[ $COPY_DIRECTORY_PATH = @([YN]) ]] || {
  print "ERROR: COPY_DIRECTORY_PATH in $CONFILE must be Y or N."
  exit 1
}

[[ -n "$BKUP_DIRS" ]] || {
  print "ERROR: BKUP_DIRS not defined in $CONFILE."
  exit 1
}

: ${NUM_COPIES:=1}

[[ "$NUM_COPIES" = +([0-9]) ]] || {
  print "ERROR: NUM_COPIES in $CONFILE is not a number."
  exit 1
}

for i in $BKUP_DIRS; do
  j=; unset j
  j=$i/$ORACLE_SID

  if [[ $j = *@*:* ]]; then
    # set -- ${j//[@:]/ }
    set -- $(echo $j | sed "s/[@:]/ /g")

    {
      print "set -- ${3}_+([-0-9])_+([:0-9])"
      print
      print "[[ -d \$1 ]] && {"
      print "  (( k = 1 ))"
      print
      print "  while (( k <= $# - $NUM_COPIES + 1 )); do"
      print "    PATH=$PATH eval rm -rf \\\${\$k}"
      print "    (( k = k + 1 ))"
      print "  done"
      print "}"
      print
      print "PATH=$PATH mkdir -p $3"
    } | rsh -l $1 $2 /bin/ksh
  else
    set -- ${j}_+([-0-9])_+([:0-9])

    [[ -d $1 ]] && {
      (( k = 1 ))

      while (( k <= $# - NUM_COPIES + 1 )); do
        eval rm -rf \${$k}
        (( k = k + 1 ))
      done
    }

    mkdir -p $j
  fi

  chkdir $j || {
    print "ERROR: in checking $j."
    exit 1
  }
done

: ${NUM_PROCS:=3}

[[ "$NUM_PROCS" = +([0-9]) ]] || {
  print "ERROR: NUM_PROCS ($NUM_PROCS) in $CONFILE is not a number."
  exit 1
}

: ${MAX_SECS:=999999}

[[ "$MAX_SECS" = +([0-9]) ]] || {
  print "ERROR: MAX_SECS ($MAX_SECS) in $CONFILE is not a number."
  exit 1
}

(( MAX_SECS == 0 )) && {
  print "INFO: HOT BACKUP FOR $ORACLE_SID SKIPPED."
  exit 0
}

: ${REDO_WAIT_MINS:=60}

[[ "$REDO_WAIT_MINS" = ?([+-])+([0-9]) ]] || {
  print "ERROR: REDO_WAIT_MINS ($REDO_WAIT_MINS) in $CONFILE is not a number."
  exit 1
}  # "REDO_WAIT_MINS < 0" means don't wait

ORACLE_HOME=; unset ORACLE_HOME
ORATAB=; unset ORATAB

for i in /var/opt/oracle/oratab /etc/oratab; do
  [[ -r $i ]] && {
    ORATAB=$i
    break
  }
done

[[ -n $ORATAB ]] || {
  print "ERROR: Can not find oratab."
  exit 1
}

typeset -x ORACLE_HOME=$(awk -F: "/^$ORACLE_SID:/ {print \$2; exit}" $ORATAB)
typeset -x LD_LIBRARY_PATH=
ISQL=$ORACLE_HOME/bin/svrmgrl
[[ -x $ISQL ]] || ISQL=$ORACLE_HOME/bin/sqlplus

[[ -x $ISQL ]] || {
  print "ERROR: Can not find $ISQL. Possibly on the other node of the cluster."
  exit 1
}

[[ -z ${PRE_PROCESSING} ]] || ${PRE_PROCESSING} || {
  print "ERROR: pre-processing failed."
  exit 1
}

[[ $INIT_D = NA ]] || ksh $RC_ORACLE sid=$ORACLE_SID action=stat verbose=n || {
  print "ERROR: Database $ORACLE_SID is down before hot backup starts."
  exit 1
}

HOT_LOCK=; unset HOT_LOCK
HOT_LOCK=$LOGDIR/lock

mkdir $HOT_LOCK || {
  print "ERROR: A copy of the script is already running."
  exit 1
}

trap "rmdir $HOT_LOCK" EXIT INT

i_locked=; unset i_locked
(( i_locked = 0 ))
(( job_elapse += SECONDS ))
(( SECONDS = 0 ))

while (( SECONDS <= REDO_WAIT_MINS*60 )); do
  : ${REDO_LOCK:=$PDIR/../ora_redo/operlog/$ORACLE_SID/lock}

  mkdir $REDO_LOCK && {
    trap "rmdir $HOT_LOCK $REDO_LOCK" EXIT INT
    (( i_locked = 1 ))
    break
  }

  print "INFO: An archive log backup process is running, wait."
  sleep 60
done

(( i_locked = 1 )) || {
  print "ERROR: Could not mkdir $REDO_LOCK within $REDO_WAIT_MINS minutes."
  exit 1
}

find $LOGDIR -mtime +60 -name \*log\* -type f -print -exec rm -f {} \;
find $TMPDIR -type f -print -exec rm -f {} \;

for i in $BKUP_DIRS
do
  if [[ $i = *@*:* ]]; then
    # set -- ${i//[@:]/ }
    set -- $(echo $i | sed "s/[@:]/ /g")
    rsh -l $1 $2 "PATH=$PATH find $3/$ORACLE_SID -type f -print -exec rm -f {} \;"
  else
    find $i/$ORACLE_SID -type f -print -exec rm -f {} \;
  fi
done

isql= fsql=; unset isql fsql
typeset -Z3 isql

(( isql = 1 ))
fsql=${isql}_seq_a

{ print "spool ${fsql}.lis"
  print "set echo on"

  [[ $ISQL = *svrmgrl ]] && {
    print "connect internal"
    print "set charwidth 960"
  }

  [[ $ISQL = *sqlplus ]] && {
    print "connect / as sysdba"
    print "set linesize 960"
    print "set trimspool on"
  }

  print "SELECT 'THE_THREAD_A=' || thread# || ' ' || 'THE_SEQ_A=' || sequence# a"
  print "  FROM v\$log WHERE status='CURRENT'"
  print "/"

  print "SELECT 'THE_NUM_ACTIVE=' || TO_CHAR(COUNT(1)) b"
  print "  FROM v\$backup WHERE status='ACTIVE'"
  print "/"

  print "SELECT 'THE_CONTROLFILE=' c"
  print "  FROM DUAL"
  print "/"

  print "SELECT 'THE_CONTROLFILE=\"\$THE_CONTROLFILE '|| name ||'\"' d"
  print "  FROM v\$controlfile"
  print "/"

  print "SET SERVEROUTPUT ON SIZE 1000000"
  print
  print "DECLARE"
  print "  sp VARCHAR2(200) := NULL;"
  print "BEGIN"
  print "SELECT value INTO sp FROM v\$parameter WHERE name = 'spfile';"
  print
  print "  IF sp IS NOT NULL THEN"
  print "    EXECUTE IMMEDIATE 'CREATE PFILE = ''$TMPDIR/spfile.txt'' FROM SPFILE';"
  print "    DBMS_OUTPUT.PUT_LINE('THE_SPFILE='|| sp);"
  print "  END IF;"
  print "END;"
  print "/"

  print "spool off"
  print "exit"
} > ${fsql}.sql

[[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
[[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

i=; unset i
CONTROLFILE=; unset CONTROLFILE

i=$(grep "^THE_.*=" ${fsql}.lis)

if grep -E "ORA-|MGR-" < ${fsql}.lis ||
   (( $(print "$i" | grep -c "^") <= 2 ))
then
  print "ERROR: In getting THE_THREAD_A, THE_NUM_ACTIVE or THE_CONTROLFILE."
  exit 1
else
  eval "$i"

  [[ $THE_THREAD_A = +([0-9]) && $THE_SEQ_A = +([0-9]) ]] || {
    print "ERROR: Did not get correct thread# or sequence# at beginning."
    exit 1
  }

  (( THE_NUM_ACTIVE == 0 )) || {
    print "ERROR: One or more tablespace in backup mode before hotbackup."
    exit 1
  }

  for CONTROLFILE in $THE_CONTROLFILE; do
    [[ -f "$CONTROLFILE" ]] || {
      print "ERROR: Can not find CONTROLFILE ($CONTROLFILE)."
      exit 1
    }
  done
fi

: ${CAT:="cat"}

$CAT $(set -- $THE_CONTROLFILE; print $1) >/dev/null 2>/dev/null || {
  print 'dd o_direct=yes ${1:+if=$1} bs=8k 2>/dev/null' >$TMPDIR/cat
  chmod 755 $TMPDIR/cat
  CAT=$TMPDIR/cat
}

(( isql += 1 ))
fsql=${isql}_datafile

{ print "spool ./datafile.select"
  print "set echo on"

  [[ $ISQL = *sqlplus ]] && {
     print "connect / as sysdba"
     print "set linesize 960"
     print "set trimspool on"
  }

  [[ $ISQL = *svrmgrl ]] && {
    print "connect internal"
    print "set charwidth 960"
  }

  # Order by tablesapce according to the largest file (used byte) in the
  # table space. Under the same tablespace, order by file size (used byte).
  # This is to fully use the multiple processing. Code provided by
  # Stephanie Shen.

  print "SELECT b.file_name || ' ' || b.tablespace_name"
  print "  || ' ' || (SUM(b.bytes)/COUNT(b.bytes)-SUM(NVL(a.bytes,0))) used_bytes"
  print "FROM dba_free_space a, dba_data_files b,"
  print "(SELECT tablespace_name, MAX(bytes) bytes"
  print "  FROM (SELECT b2.tablespace_name, b2.file_id,"
  print "        SUM(b2.bytes)/COUNT(b2.bytes)-SUM(NVL(a2.bytes,0)) bytes"
  print "        FROM dba_free_space a2, dba_data_files b2"
  print "        WHERE a2.file_id(+)=b2.file_id"
  print "        GROUP BY b2.tablespace_name, b2.file_id)"
  print "  GROUP BY tablespace_name ) c"
  print "WHERE a.file_id(+)= b.file_id"
  print "AND b.tablespace_name = c.tablespace_name"
  print "GROUP BY b.tablespace_name, b.file_name, c.bytes"
  print "ORDER BY c.bytes DESC,"
  print "         b.tablespace_name,"
  print "         (SUM(b.bytes)/COUNT(b.bytes)-SUM(NVL(a.bytes,0))) DESC"
  print "/"

  [[ $COPY_DIRECTORY_PATH = Y ]] && {
    print "SELECT UNIQUE 'DIRECTORY_PATH=' || directory_path"
    print "FROM dba_directories"
    [[ -n $FILTER_DIRECTORY_PATH ]] && print $FILTER_DIRECTORY_PATH
    print "ORDER BY 1"
    print "/"
  }

  print "spool off"
  print "exit"
} > ${fsql}.sql

[[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
[[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

grep -E "ORA-|MGR-" datafile.select && {
  print "ERROR: In select datafiles and tablespaces."
  exit 1
}

grep "^/" datafile.select > datafile.file
awk 'NF == 3 {print $2}' datafile.file > tablespace
uniq tablespace > tablespace.uniq

num_file= num_tabl=; unset num_file num_tabl
(( num_file = $(wc -l < datafile.file) ))
(( num_tabl = $(wc -l < tablespace.uniq) ))

(( num_file < 1 )) && {
  print "ERROR: I have nothing to backup."
  exit 1
}

(( $(wc -l < tablespace) == num_file )) || {
  print "ERROR: Every datafile should belong to one tablespace."
  exit 1
}

typeset -Z3 x
typeset -Z3 y
(( x = 1 ))
(( y = 0 ))

while (( x <= num_tabl ))
do                                      # To make $x.*.[stat|pid|log] expandable by shell
  > $x.$y.stat; > $x.$y.pid; > $x.$y.log
  (( x+=1 ))
done

> $TMPDIR/$y.begin; > $TMPDIR/$y.end    # To make *.begin, *.end expandable

function cleanup {
  trap 'print Please wait for cleanup to finish.' EXIT INT

  (( $1 == 0 )) || print "Please wait for cleanup to finish."

  rmdir $HOT_LOCK $REDO_LOCK

  typeset i j k

  for i in +([0-9]).begin
  do
    [[ $i = +(0).begin ]] && continue

    j=${i%.begin}
    k=$(sed -n ${j}p tablespace.uniq)

    [[ -f $j.end ]] || {
      (( isql += 1 ))
      fsql=${isql}_${k}_end_cleanup

      { print "spool ${fsql}.lis"
        print "set echo on"

        [[ $ISQL = *svrmgrl ]] && print "connect internal"
        [[ $ISQL = *sqlplus ]] && print "connect / as sysdba"

        print "select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss') from dual"
        print "/"

        print "alter tablespace $k end backup"
        print "/"

        print "spool off"
        print "exit"
      } > ${fsql}.sql

      [[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
      [[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql
    }
  done

  for i in [0-9]*.[0-9]*.pid
  do
    j=${i%.pid}
    [[ -f $j.stat ]] || kill -- -$(< $i)

    [[ -s $j.log ]] && {
      print
      cat $j.log
    }
  done

  (( $1 == 0 )) || print "ERROR: HOT BACKUP FOR $ORACLE_SID FAILED."
  trap '' 0
  return 0
}

trap "print \"ERROR: Program is terminating.\";
      cleanup 1;
      exit 1" EXIT INT

typeset -Z3 the_num_file
typeset -Z3 the_num_tabl
typeset -Z3 z
(( the_num_file = 0 ))    # number of files submitted
(( the_num_tabl = 0 ))    # number of tablespace submitted
(( the_num_stat = 0 ))    # number of files finished successfully
(( the_num_end  = 0 ))    # number of tablespace finished successfully
tablespace_name_prev=""
(( job_elapse += SECONDS ))
(( SECONDS == 0 ))

while (( the_num_end < num_tabl ))
do
  (( z = 1 ))

  while (( z <= the_num_tabl ))               # go through all tablespaces
  do                                          # each time. z is an index var.
    [[ -f $z.end ]] || {
      j=$(sed -n ${z##+(0)}p tablespace.uniq) # tablespace name
      k=$(grep -c "^$j\$" tablespace)         # number of files in tablespace

      set $z.+([0-9]).stat  # how many files finished, ok and not ok.
      (( kk = $# - 1 ))

      (( kk == k )) && {    # done with tablespace
        (( isql += 1 ))
        fsql=${isql}_${j}_end

        { print "spool ${fsql}.lis"
          print "set echo on"

          [[ $ISQL = *svrmgrl ]] && print "connect internal"
          [[ $ISQL = *sqlplus ]] && print "connect / as sysdba"

          print "alter tablespace $j end backup"
          print "/"

          print "spool off"
          print "exit"
        } > ${isql}_${j}_end.sql

        [[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
        [[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

        grep -E "ORA-|MGR-"  ${fsql}.lis && {
          print "ERROR: In alter tablespace $j end backup inside while loop."
          exit 1
        }
        > $z.end
      }
    }
    (( z+=1 ))
  done

  set +([0-9]).begin
  (( the_num_tabl = $# - 1 ))

  set +([0-9]).end
  (( the_num_end = $# - 1 ))

  set +([0-9]).+([0-9]).pid
  (( the_num_file = $# - num_tabl ))

  set +([0-9]).+([0-9]).stat
  (( the_num_stat = $# - num_tabl )) 

  (( the_num_proc = the_num_file - the_num_stat ))

  print "num_file_submit:num_file_finish:num_file_allow:num_file_total=\c"
  print $the_num_file:$the_num_stat:$NUM_PROCS:$num_file

  print "num_tabl_submit:num_tabl_finish:num_tabl_allow:num_tabl_total=\c"
  print $the_num_tabl:$the_num_end:na:$num_tabl

  (( SECONDS >= MAX_SECS )) && {
    print "ERROR: exceeding maximum time ($MAX_SECS seconds)."
    (( the_num_end = num_tabl + 1 ))
    break
  }

  grep -v "^0\$" +([0-9]).+([0-9]).stat && {
    print "ERROR: one of copy sessions has a non-zero exit code."
    (( the_num_end = num_tabl + 1 ))
    break
  }

  (( the_num_file >= num_file )) && {
    print "INFO: $(date +%H:%M:%S) wait for file copying to finish."
    sleep 10
    continue
  }

  (( the_num_proc >= NUM_PROCS )) && {
    print "INFO: $(date +%H:%M:%S) maximum number of copy session reached, wait."
    sleep 10
    continue
  }

  (( the_num_file+=1 ))

  the_avail=-1

  for dir in $BKUP_DIRS
  do
    if [[ $dir = *@*:* ]]; then
      # set -- ${dir//[@:]/ }
      set -- $(echo $dir | sed "s/[@:]/ /g")
      avail=$(rsh -l $1 $2 "PATH=$PATH df -Pk $3" | awk "/^\// {print \$4}")
    else
      # avail=$(df -Pk $dir | tail -1 | awk '{print $4}')
      set -- $(df -Pk $dir)
      avail=${11}
    fi

    (( avail > the_avail )) && {
      the_avail=$avail
      THE_DIR=$dir
    }
  done
    
  tablespace_name_prev=$tablespace_name
  set $(sed -n ${the_num_file##+(0)}p datafile.file)
  path_file=$1
  tablespace_name=$2

  [[ "$tablespace_name" = "$tablespace_name_prev" ]] || {
    (( the_num_tabl += 1 ))

    (( isql += 1 ))
    fsql=${isql}_${tablespace_name}_begin

    { print "spool ${fsql}.lis"
      print "set echo on"

      [[ $ISQL = *svrmgrl ]] && print "connect internal"
      [[ $ISQL = *sqlplus ]] && print "connect / as sysdba"

      print "select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss') from dual"
      print "/"

      print "alter tablespace $tablespace_name begin backup"
      print "/"

      print "spool off"
      print "exit"
    } > ${fsql}.sql

    [[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
    [[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

    grep -E "ORA-|MGR-"  ${fsql}.lis && {
      print "ERROR: In alter tablespace $tablespace_name begin backup."
      exit 1
    }

    > $the_num_tabl.begin
  }

  file=${path_file##*/}
  path=${path_file%/*}
  relative_path=${path#/}
  THE_PATH=$THE_DIR/$ORACLE_SID/$relative_path

  if [[ $THE_PATH = *@*:* ]]; then
    # set -- ${THE_PATH//[@:]/ }
    set -- $(echo $THE_PATH | sed "s/[@:]/ /g")
    u=$1 r=$2 d=$3
    rsh -l $u $r "mkdir -p $d"

    $SUBMIT -i $the_num_tabl.$the_num_file.pid \
            -l $the_num_tabl.$the_num_file.log \
            -e $the_num_tabl.$the_num_file.stat \
            "$CAT $path_file | $COMPRESS_C | rsh -l $u $r \"PATH=$PATH cat > $d/$file$THEZ\""
  else
    mkdir -p $THE_PATH

    $SUBMIT -i $the_num_tabl.$the_num_file.pid \
            -l $the_num_tabl.$the_num_file.log \
            -e $the_num_tabl.$the_num_file.stat \
            "$CAT $path_file | $COMPRESS_C | cat > $THE_PATH/$file$THEZ"
  fi  # not UUOC
done

(( the_num_end == num_tabl )) || {
  print "ERROR: In copying datafile."
  exit 1
}

[[ $INIT_D = NA ]] || ksh $RC_ORACLE sid=$ORACLE_SID action=stat verbose=n || {
  print "ERROR: Database $ORACLE_SID is down after hot backup."
  exit 1
}

[[ -z "$POST_PROCESSING" ]] || $POST_PROCESSING || {
  print "ERROR: POST_PROCESSING $POST_PROCESSING failed."
  exit 1
}

BACKUP_CONTROLFILE=$TMPDIR/controlfile

(( isql += 1 ))
fsql=${isql}_archive

{ print "spool ${fsql}.lis"
  print "set echo on"

  [[ $ISQL = *svrmgrl ]] && print "connect internal"
  [[ $ISQL = *sqlplus ]] && print "connect / as sysdba"

  print "alter system archive log current"
  print "/"

  print "spool off"
  print "exit"
} > ${fsql}.sql

[[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
[[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

grep -E "ORA-|MGR-" ${fsql}.lis && {
  print "ERROR: in alter system archive log current."
  exit 1
}

i= imax=; unset i imax
(( imax = 10 ))
(( i = 1 ))

while (( i <= imax )); do
  (( i += 1 ))
  (( isql += 1 ))
  fsql=${isql}_backup_controlfile
  
  { print "spool ${fsql}.lis"
    print "set echo on"

    [[ $ISQL = *svrmgrl ]] && print "connect internal"
    [[ $ISQL = *sqlplus ]] && print "connect / as sysdba"

    print "ALTER DATABASE BACKUP CONTROLFILE TO '$BACKUP_CONTROLFILE' REUSE"
    print "/"

    print "spool off"
    print "exit"
  } > ${fsql}.sql
  
  [[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
  [[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql
  
  grep -E "ORA-|MGR-" ${fsql}.lis && {
    print "WARNING: in alter system archive log current, try again."
    sleep 60
    continue
  }

  break
done

(( i == imax + 1 )) && {
  print "ERROR: in alter system archive log current, exit."
  exit 1
}

(( isql += 1 ))
fsql=${isql}_seq_b

{ print "spool ${fsql}.lis"
  print "set echo on"

  [[ $ISQL = *svrmgrl ]] && {
    print "connect internal"
    print "set charwidth 960"
  }

  [[ $ISQL = *sqlplus ]] && {
    print "connect / as sysdba"
    print "set linesize 960"
    print "set trimspool on"
  }

  print "SELECT 'THE_THREAD_B=' || thread# || ' ' || 'THE_SEQ_B=' || sequence# a"
  print "  FROM v\$log WHERE status='CURRENT'"
  print "/"

  print "ALTER SYSTEM ARCHIVE LOG CURRENT"
  print "/"

  print "ALTER DATABASE BACKUP CONTROLFILE TO TRACE"
  print "/"

  print "SELECT 'THE_NUM_ACTIVE=' || TO_CHAR(COUNT(1)) b"
  print "  FROM v\$backup WHERE status='ACTIVE'" 
  print "/"

  print "SELECT 'THE_LOG_ARCHIVE_DEST=' || value c"
  print "  FROM v\$parameter WHERE name='log_archive_dest'" 
  print "/"

  print "SELECT 'THE_LOG_ARCHIVE_FORMAT=' || value d"
  print "  FROM v\$parameter WHERE name='log_archive_format'"
  print "/"

  print "SELECT 'THE_DUMP_DEST=' || value e"
  print "  FROM v\$parameter WHERE name='user_dump_dest'"
  print "/"

  print "SELECT 'THE_PID=' || p.spid f"
  print "  FROM v\$process p, v\$session s, v\$mystat stat"
  print " WHERE s.sid=stat.sid AND p.addr=s.paddr AND rownum=1"
  print "/"

  print "spool off"
  print "exit"
} > ${fsql}.sql

[[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
[[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

i=; unset i
i=$(grep "^THE_.*=" ${fsql}.lis)

grep -E "ORA-|MGR-" ${fsql}.lis || (( $(print "$i" | grep -c "^") != 6 )) && {
  print "ERROR: in SEQ_B statements."
  exit 1
}

eval "$i"

[[ -z $THE_LOG_ARCHIVE_DEST ]] && {
  (( isql += 1 ))
  fsql=${isql}_log_dest

  { print "spool ${fsql}.lis"
    print "set echo on"

    [[ $ISQL = *svrmgrl ]] && {
      print "connect internal"
      print "set charwidth 960"
    }

    [[ $ISQL = *sqlplus ]] && {
      print "connect / as sysdba"
      print "set linesize 960"
      print "set trimspool on"
    }

    print "SELECT 'THE_LOG_ARCHIVE_DEST=' || destination d"
    print "  FROM v\$archive_dest"
    print " WHERE target='PRIMARY'"
    print "   AND destination LIKE '%/%'"
    print "   AND rownum <=1"
    print "/"

    print "spool off"
    print "exit"
  } > ${fsql}.sql

  [[ $ISQL = *svrmgrl ]] && $ISQL command="@${fsql}.sql"
  [[ $ISQL = *sqlplus ]] && $ISQL /nolog   @${fsql}.sql

  i=; unset i
  i=$(grep "^THE_.*=" ${fsql}.lis)

  grep -E "ORA-|MGR-" ${fsql}.lis || [[ $(print "$i" | grep -c "^THE_.*=") != 1 ]] && {
    print "ERROR: in select from v\$archive_dest."
    exit 1
  }

  eval "$i"
}

(( THE_NUM_ACTIVE == 0 )) || {
  print "ERROR: one or more tablespaces in backup mode after hot backup."
  exit 1
}

[[ $THE_THREAD_B = +([0-9]) && $THE_SEQ_B = +([0-9]) ]] &&
(( THE_THREAD_B = THE_THREAD_A && THE_SEQ_B >= THE_SEQ_A + 1 )) || {
  print "ERROR: Did not get correct thread# or sequence# at the end."
  exit 1
}

(( THE_THREAD = THE_THREAD_A ))

# THE_DUMP_DEST=${THE_DUMP_DEST/#\?/$ORACLE_HOME}

[[ $THE_DUMP_DEST = [?]* ]] && {
  THE_DUMP_DEST=$(echo ${THE_DUMP_DEST} | sed "s:^\?:$ORACLE_HOME:")
}

[[  -d "$THE_DUMP_DEST" ]] || {
  print "ERROR: Can not find THE_DUMP_DEST=$THE_DUMP_DEST."
  exit 1
}

print "INFO: THE_DUMP_DEST = $THE_DUMP_DEST."

[[ "$THE_PID" = +([0-9]) ]] || {
  print "ERROR: THE_PID=$THE_PID is not a number."
  exit 1
}

function mycp {
  typeset file_i=$1
  typeset file_j=$2
  typeset cksum_i=$(cksum < $file_i)
  typeset cksum_j

  [[ -f $file_i ]] || {
    print "ERROR: Can not find the file: $file_i"
    exit 1
  }

  if [[ $file_j = *@*:* ]]; then
    # set -- ${file_j//[@:]/ }
    set -- $(echo $file_j | sed "s/[@:]/ /g")
    rsh -l $1 $2 "PATH=$PATH mkdir -p ${3%/*}"
    rcp $file_i $file_j
    cksum_j=$(rsh -l $1 $2 "PATH=$PATH cksum < $3")
  else
    mkdir -p ${file_j%/*}
    cp $file_i $file_j
    cksum_j=$(cksum < $file_j)
  fi

  if [[ "$cksum_i" = "$cksum_j" ]]
  then print "INFO: cp $file_i $file_j ok."
  else print "ERROR: cp $file_i $file_j failed."
       exit 1
  fi
}

[[ -f $BACKUP_CONTROLFILE ]] || {
  print "ERROR: backup controlfile to $BACKUP_CONTROLFILE failed."
  exit 1
}

for CONTROLFILE in $THE_CONTROLFILE; do
  mycp $BACKUP_CONTROLFILE $THE_DIR/$ORACLE_SID/${CONTROLFILE#/}
done

# THE_LOG_ARCHIVE_DEST=${THE_LOG_ARCHIVE_DEST/\#?/$ORACLE_HOME}
THE_LOG_ARCHIVE_DEST=$(echo ${THE_LOG_ARCHIVE_DEST} | sed "s:^\?:$ORACLE_HOME:")

THE_DESTMAT=; unset THE_DESTMAT
if [[ -d "$THE_LOG_ARCHIVE_DEST" ]]
then        #/* log_archive_dest=/path; log_archive_format=arch_SID_%s */
  THE_DESTMAT=${THE_LOG_ARCHIVE_DEST%/}/${THE_LOG_ARCHIVE_FORMAT}
elif [[ -d "${THE_LOG_ARCHIVE_DEST%/*}" ]]
then        #/* log_archive_dest=/path/arch; log_archive_format=_SID_%s */
  THE_DESTMAT=${THE_LOG_ARCHIVE_DEST}${THE_LOG_ARCHIVE_FORMAT}
else
  print "ERROR: Did not get THE_LOG_ARCHIVE_DEST=$THE_LOG_ARCHIVE_DEST ok."
  exit 1
fi

[[ $THE_DESTMAT = *%[stSTr]* ]] || {
  print "ERROR: Did not get log_archive_format correctly."
  exit 1
}

i_redo=; unset i_redo
DESTMAT=; unset DESTMAT
(( i_redo = THE_SEQ_A ))

while (( i_redo <= THE_SEQ_B )); do

  DESTMAT=$THE_DESTMAT

  # DESTMAT=${DESTMAT//%s/@($i_redo)}          # %s = sequence#
  # DESTMAT=${DESTMAT//%S/*(0)$i_redo}         # %S = sequence# (padded with 0s)
  # DESTMAT=${DESTMAT//%t/@($THE_THREAD)}      # %t = thread#
  # DESTMAT=${DESTMAT//%T/*(0)$THE_THREAD}     # %T = thread# (padded with 0s)
  # DESTMAT=${DESTMAT//%r/+([0-9])}            # %r = logical incarnation

  DESTMAT=$(echo ${DESTMAT} | sed "s:%s:@($i_redo):g")
  DESTMAT=$(echo ${DESTMAT} | sed "s:%S:*(0)$i_redo}:g")
  DESTMAT=$(echo ${DESTMAT} | sed "s:%t:@($THE_THREAD):g")
  DESTMAT=$(echo ${DESTMAT} | sed "s:%T:*(0)$THE_THREAD:g")
  DESTMAT=$(echo ${DESTMAT} | sed "s:%r:+([0-9]):g")

  eval set -- "$DESTMAT"  # @(...)/*(0): $DESTMAT does not match itself.

  (( $# == 1 )) && eval [[ $1 = $DESTMAT ]] || {
    print "ERROR: Can not find archive log matched by $DESTMAT (sequence# $i_redo)."
    exit 1
  }

  DESTMAT=$1
  mycp $DESTMAT $THE_DIR/$ORACLE_SID/${DESTMAT#/}
  (( i_redo += 1 ))
done

i= j=; unset i j
for i in $ORACLE_HOME/dbs/@(init$ORACLE_SID.ora|orapw$ORACLE_SID); do
  j=${i#/}
  mycp $i $THE_DIR/$ORACLE_SID/$j
done

[[ -n $THE_SPFILE && -r $THE_SPFILE ]] && {
  mycp $THE_SPFILE $THE_DIR/$ORACLE_SID/${THE_SPFILE#/}
  mycp $TMPDIR/spfile.txt $THE_DIR/$ORACLE_SID/${THE_SPFILE#/}.txt
}

set -- $(ls -t $THE_DUMP_DEST/*ora_${THE_PID}@([._])*trc)

if (( $# >= 1 )); then
  mycp $1 $THE_DIR/$ORACLE_SID/gen_controlfile.sql
else
  set -- $(ls -t $ORACLE_HOME/rdbms/log/*ora_${THE_PID}@([._])*trc)

  if (( $# >= 1 )); then
    mycp $1 $THE_DIR/$ORACLE_SID/gen_controlfile.sql
  else
    print "ERROR: Can not find backup controlfile (*ora_${THE_PID}@([._])*trc)."
    exit 1
  fi
fi

[[ $COPY_DIRECTORY_PATH = Y ]] && {
  dir= file= rel_file=; unset dir file rel_file

  for dir in $(awk -F= "/^DIRECTORY_PATH=/ {print \$2}" datafile.select); do
    set -- $dir/.* $dir/*

    for file in "$@"; do
      [[ -d $file ]] && continue

      rel_file=$file
      rel_file=${rel_file##+(/)}

      [[ ( -f $file || -L $file ) && -r $file ]] && {
        mycp $file $THE_DIR/$ORACLE_SID/$rel_file
      }
    done
  done
}

cleanup 0

FINISH_DATE= i= j= k=; unset FINISH_DATE i j k
FINISH_DATE=$(date +%Y-%m-%d_%H:%M:%S)

for i in $BKUP_DIRS; do
  j=$i/$ORACLE_SID
  if [[ $j = *@*:* ]]; then
    set -- $(echo $j | sed "s/[@:]/ /g")
    rsh -l $1 $2 "PATH=$PATH mv $3 ${3}_${FINISH_DATE}; echo \$?" | read k
    (( k == 0 )) || {
      print "ERROR: Can not rename $j to ${3}_${FINISH_DATE}."
      exit 1
    }
  else
    mv ${j} ${j}_${FINISH_DATE} || {
      print "ERROR: Can not rename $j to ${j}_${FINISH_DATE}."
      exit 1
    }
  fi
done

job_finish=$(get_date)
(( job_elapse += SECONDS ))
JOB="${0##*/} for $ORACLE_SID"

print
print "$JOB started:  $job_start."
print "$JOB finished: $job_finish."
print "$JOB duration: $(( job_elapse/60 )) minutes."

print "HOT BACKUP FOR $ORACLE_SID FINISHED SUCCESSFULLY."
trap - EXIT
exit 0

## POD_START
## =head1 NAME
##
## orahot.ksh - make Oracle database hot (online) backup.
##
## =head1 SYNOPSIS
##
## oracold.ksh help=y
##
## oracold.ksh MySID|sid:=MySid [debug=y]
##
## =head1 DESCRIPTION
##
## B<orahot.ksh> first generates a list of I<file_names> and I<tablespace
## names> ordered by I<used space>. Put one or more tablespace in I<begin
## backup> mode as needed so that I<NUM_PROCS> (specified in configuration
## file) of files can be compressed simutaneously to I<multiple backup
## directories> (I<BKUP_DIRS>, specified in configuration file). I<As soon
## as> all of the files within a tablespace are backed up, the tablespace
## is put back to I<end backup> mode.
## 
## I<ALTER SYSTEM SWITCH LOGFILE> statement is issued and the archive log
## file up to the last inactive logfile is compressed to the backup
## directories.  The original archive log files are I<not> removed.
## Backing up and removing the archive log files is the job of
## oraredo.ksh, not orahot.ksh.
## 
## initI<SID>.ora file is backed to one of the backup directories with
## unix copy.
## 
## The I<control file> is backed to one of the backup directories with
## I<ALTER DATABASE CONTROLFILE TO <file>> statement, and unix copy.
## 
## The I<text copy of the control file> is generated with I<ALTER DATABASE
## CONTROLFILE TO TRACE> statement in I<user_dump_dest> directory. This
## file is backed up to one of the backup directories with unix copy, and
## the file is renamed I<gen_controlfile.sql>. After verification of
## successful copying, the original file is removed.
## 
## The backup file systems are expected to be backed up via data center
## backup service.
## 
## Features include
## 
## =over 4
## 
## =item *
## 
## Support paralell backup processes (I<NUM_PROCS>). Tablespaces will be
## put "begin backup" mode and "end backup" mode as needed.
##
## =item *
##
## Choice of compress, gzip, bzip2.
##
## =item *
##
## Large file support (it is not required to have a large file capable
## compress, gzip, bzip2).
##
## =item *
## 
## Support multiple backup directories (I<BKUP_DIRS>). It can be used for I/O and
## capacity load balancing. Backup files are dynamically allocated to
## directories with largest free sapce.
##
## =item *
##
## Remote backup directory support. (Currently coded for rsh, it can be expanded
## to support a choice of rsh, or ssh).
## 
## =item *
##
## Manage the number of backup copies online (I<NUM_COPIES>).
##
## =item *
## 
## Put tablespace in I<end backup> mode and clean up processes when the
## backup program encouters an error.
## 
## =item *
## 
## Program terminates itself after maximum allowed time (I<MAX_SECS> in
## configuration file). If I<MAX_SECS> is 0, the backup is skipped.
## 
## =item *
## 
## Prevent duplicate hot backup processes from running.
## 
## =item *
## 
## Cooperate with oraredo.ksh backup. orahot.ksh and oraredo.ksh wait for
## each other.
## 
## =item *
##
## Backup files under DIRECTORY_PATH.
##
## =back
##
## =head1 INSTALLATION
##
## To install, follow these easy 1-2-3 steps:
##
## =over 4
##
## =item 1.
## 
## Create an install directory for hot backup <job_dir>/ora_hot,
## and under ora_hot, create following sub-directories: 
##
##  cd <job_dir>/ora_hot &&
##  mkdir -p script etc operlog/<SID> temp/<SID> ../ora_redo/operlog/<SID>
##
## Where SID is the actual ORACLE_SID name. If you have multiple
## SIDs, then just create them under operlog and temp.
##
## Create ../ora_redo/operlog/<SID> even if you do not use ora_redo.ksh
## as ora_hot.ksh places a lock there to cooperate with redo backup.
##
## =item 2. 
##
##   cp orahot.ksh script/orahot.ksh
##   chmod 755 script/orahot.ksh
##
## =item 3.
##
## Edit etc/orahot_SID.cf from the sample configuration file shown below. If
## you have multiple SIDs, then just create multiple files. 
##
## =back
##
##   # *
##   # * Where do you want the hot backup files to go?
##   # * It can be a directory on a remote server.
##   # *
##    
##   BKUP_DIRS="/bkup01/hot oracle@castor:/bkup02/hot"
##   
##   # *
##   # * How many simutaneous compress processes you want to run?
##   # * Rule of thumb is to set the number to be the same as the n of processors. 
##   # * Default is 3.
##   # *
##    
##   NUM_PROCS=3
##   
##   # *
##   # * How many copies you want to keep online?
##   # * ( If you keep only 1 copy, make sure the file system backup
##   # * occurs after the Oracle backup finishes. )
##   # *
##
##   NUM_COPIES=2
##
##   # *
##   # * What is the maximum time you allow the backup to run?
##   # * Backup will kill itself exceeding the maximum allowed time. 
##   # * Default is 999999.
##   # *
##
##   MAX_SECS=21600
##   
##   # *
##   # * Where is the redo backup script? (Hot backup and redo backup work together.)
##   # * If you use the standard naming conventions (ie <install_dir>/ora_redo,
##   # * <install_dir>/ora_hot), then it will derived.
##   # *
##   # * New version (5.0+) does not rely on redo backup script anymore.
##   # * REDO_LOCK directory is used to prevent redo backup from running during
##   # * the hot backup. If you use the standard naming conventions, this
##   # * variable is derived (<install_dir>/ora_redo/operlog/$ORACLE_SID/lock).
##   # *
##   # * If you are sure redo backup is not run and no lock is needed, then you can
##   # * define REDO_WAIT_MINS (see below) as a negative number to bypass the locking
##   # * mechanism.
##   # *
##   
##   # REDO_PROG="/ds1-export/apps/jobs/ora_redo/script/oraredo.ksh" (depreciated)
##   # REDO_LOCK="/ds1-export/apps/jobs/ora_redo/operlog/$ORACLE_SID/lock" (derivable)
##   
##   # *
##   # * Where does redo backup go?
##   # * If you use the standard naming conventions (ie <install_dir>/ora_redo,
##   # * <install_dir>/ora_hot), then it will derived.
##   # *
##   # * This variable is not used in new version (5.0+).
##   # *
##   
##   # REDO_BKUP_DIRS="/bkup01/redo /bkup02/redo" (depreciated)
##   
##   # *
##   # * How long do you want to wait when another process is running redo backup?
##   # * Default is 60.
##   # *
##   
##   REDO_WAIT_MINS=60
##   
##   # *
##   # * What cat command do you use to cat the data files (pipe to
##   # * compress utility). Default is "cat", and if "cat" can not
##   # * open first controlfile due to restricted permission ocfs,
##   # * then cat is defined as $TMPDIR/cat which is:
##   # * "dd o_direct=yes ${1:+if=$1} bs=8k 2>/dev/null". If this logic
##   # * does not work, define CAT.
##   # *
##
##   CAT="cat"
##   # CAT="/path/to/special/cat" where cat is something like:
##   # "dd o_direct=yes ${1:+if=$1} bs=8k 2>/dev/null"
##
##   # *
##   # * What command do you use to compress a file to standard output?
##   # * Default is "compress -c".
##   # *
##   
##   COMPRESS_C="/usr/bin/compress -c"
##   # COMPRESS_C="/usr/local/bin/gzip -c"
##   # COMPRESS_C="/usr/local/bin/bzip2 -c"
##    
##   # *
##   # * Whether to copy files under each DIRECTORY_PATH
##   # * from dba_directories.
##   # *
##
##   COPY_DIRECTORY_PATH=Y
##
##   # *
##   # * FILTER_DIRECTORY_PATH: to include or exclude certain
##   # * OWNER, DIRECTORY_NAME, or DIRECTORY_PATH from DBA_DIRECTORIES.
##   # *
##
##   FILTER_DIRECTORY_PATH="where DIRECTORY_NAME not in ('AUDIT_DIR')"
##
##   # *
##   # * Where can you find submit?
##   # * /usr/local/bin/submit is the default.
##   # *
##   
##   SUBMIT=<job_dir>/common/submit
##   # SUBMIT=/usr/local/bin/submit
##   
##   # *
##   # * What is the directory where you can find oracle-SID?
##   # * /etc/init.d is the default.
##   # *
##   # * The oracle-SID is used to verify if Oracle is up before and after
##   # * the backup. If you want to bypass the tests, then define INIT_D as
##   # * "NA" (Not Apply).
##   # *
##   
##   INIT_D=<job_dir>/common
##   # INIT_D=/etc/init.d
##   
##   # *
##   # * You can define pre and post processing command. 
##   # *
##   
##   # PRE_PROCESSING="/bin/true"
##   # POST_PROCESSING="/bin/true"
## 
## =head1 RESTORE
##
## The ora_hot.ksh copies Oracle data files, control files,
## and archive log files generated I<during> the hot backup.
## The text copy of the control file gen_controlfile.sql, and
## parameter file initI<SID>.ora are also backed up gratuitously.
## 
## A sample hot backup file set is shown below: 
## 
##   /bkup01/hot/ora901/ora10/oradata/ora901/arch/1_63.dbf
##   /bkup01/hot/ora901/ora10/oradata/ora901/arch/1_64.dbf
##   /bkup01/hot/ora901/ora10/oradata/ora901/control01.ctl
##   /bkup01/hot/ora901/ora10/oradata/ora901/control02.ctl
##   /bkup01/hot/ora901/ora10/oradata/ora901/control03.ctl
##   /bkup01/hot/ora901/ora10/oradata/ora901/cwmlite01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/drsys01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/example01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/indx01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/system01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/tools01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/undotbs01.dbf.Z
##   /bkup01/hot/ora901/ora10/oradata/ora901/users01.dbf.Z
##   /bkup01/hot/ora901/gen_controlfile.sql
##   /bkup01/hot/ora901/initora901.ora
## 
## To restore, first copy Oracle data files, control files,
## and archive log files to where they were. As the original
## file path are recorded in the backup set, it is very easy to
## automate the task. The following I<ad hoc> script
## does this.
## 
##   #!/bin/ksh
##   
##   find /bkup01/hot/ora901/ora* -type f | while read i; do
##     j=${i#*ora901}
##     dir=${j%/*}
##     file=${j%.Z}
##   
##     CAT=cat
##     [[ $i = *.Z ]] && CAT=zcat
##   
##     mkdir -p $dir
##     $CAT $i > $file
##   done
## 
## What it does is something like,
## 
##   mkdir -p /ora10/oradata/ora901
##   zcat /bkup01/hot/ora901/ora10/oradata/ora901/system01.dbf.Z \
##     > /ora10/oradata/ora901/system01.dbf
## 
## Once all the files are in place, we can go ahead to
## recover and open the database. A sample session is shown below,
## 
##   SQL> startup mount
##   
##   ORACLE instance started.
##   Total System Global Area   63734856 bytes
##   Fixed Size                   279624 bytes
##   Variable Size              46137344 bytes
##   Database Buffers           16777216 bytes
##   Redo Buffers                 540672 bytes
##   Database mounted.
##   
##   SQL> recover database using backup controlfile until cancel
##   
##   ORA-00279: change 1329616 generated at 05/02/2002 13:30:39
##   + needed for thread 1
##   ORA-00289: suggestion : /ora10/oradata/ora901/arch/1_63.dbf
##   ORA-00280: change 1329616 for thread 1 is in sequence #63
##   
##   Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
##   
##   ORA-00279: change 1329687 generated at 05/02/2002 13:40:42
##   + needed for thread 1
##   ORA-00289: suggestion : /ora10/oradata/ora901/arch/1_64.dbf
##   ORA-00280: change 1329687 for thread 1 is in sequence #64
##   ORA-00278: log file '/ora10/oradata/ora901/arch/1_63.dbf'
##   + no longer needed for this recovery
##   
##   Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
##   
##   ORA-00279: change 1329692 generated at 05/02/2002 13:40:46
##   + needed for thread 1
##   ORA-00289: suggestion : /ora10/oradata/ora901/arch/1_65.dbf
##   ORA-00280: change 1329692 for thread 1 is in sequence #65
##   ORA-00278: log file '/ora10/oradata/ora901/arch/1_64.dbf'
##   + no longer needed for this recovery
##   
##   Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
##   CANCEL
##   Media recovery cancelled.
##   
##   SQL> alter database open RESETLOGS
##     2  /
##   
##   Database altered.
##   
## That is it, happy restore!
##
## =head1 SEE ALSO
##
## submit help=y, rc.oracle help=y, oraredo.ksh help=y.
##
## =head1 AUTHOR
##
## Michael Wang <F<xw73@columbia.edu>> with help from many others
## acknowledged in version notes.
##
## This is free software. You may copy or redistribute it under the same
## terms as Perl. However, if you modify it, you need to send the
## modification to the author via email.
## POD_STOP
## MAN_START
## .\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32
## .\"
## .\" Standard preamble:
## .\" ========================================================================
## .de Sh \" Subsection heading
## .br
## .if t .Sp
## .ne 5
## .PP
## \fB\\$1\fR
## .PP
## ..
## .de Sp \" Vertical space (when we can't use .PP)
## .if t .sp .5v
## .if n .sp
## ..
## .de Vb \" Begin verbatim text
## .ft CW
## .nf
## .ne \\$1
## ..
## .de Ve \" End verbatim text
## .ft R
## .fi
## ..
## .\" Set up some character translations and predefined strings.  \*(-- will
## .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
## .\" double quote, and \*(R" will give a right double quote.  | will give a
## .\" real vertical bar.  \*(C+ will give a nicer C++.  Capital omega is used to
## .\" do unbreakable dashes and therefore won't be available.  \*(C` and \*(C'
## .\" expand to `' in nroff, nothing in troff, for use with C<>.
## .tr \(*W-|\(bv\*(Tr
## .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
## .ie n \{\
## .    ds -- \(*W-
## .    ds PI pi
## .    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
## .    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
## .    ds L" ""
## .    ds R" ""
## .    ds C` ""
## .    ds C' ""
## 'br\}
## .el\{\
## .    ds -- \|\(em\|
## .    ds PI \(*p
## .    ds L" ``
## .    ds R" ''
## 'br\}
## .\"
## .\" If the F register is turned on, we'll generate index entries on stderr for
## .\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index
## .\" entries marked with X<> in POD.  Of course, you'll have to process the
## .\" output yourself in some meaningful fashion.
## .if \nF \{\
## .    de IX
## .    tm Index:\\$1\t\\n%\t"\\$2"
## ..
## .    nr % 0
## .    rr F
## .\}
## .\"
## .\" For nroff, turn off justification.  Always turn off hyphenation; it makes
## .\" way too many mistakes in technical documents.
## .hy 0
## .if n .na
## .\"
## .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
## .\" Fear.  Run.  Save yourself.  No user-serviceable parts.
## .    \" fudge factors for nroff and troff
## .if n \{\
## .    ds #H 0
## .    ds #V .8m
## .    ds #F .3m
## .    ds #[ \f1
## .    ds #] \fP
## .\}
## .if t \{\
## .    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
## .    ds #V .6m
## .    ds #F 0
## .    ds #[ \&
## .    ds #] \&
## .\}
## .    \" simple accents for nroff and troff
## .if n \{\
## .    ds ' \&
## .    ds ` \&
## .    ds ^ \&
## .    ds , \&
## .    ds ~ ~
## .    ds /
## .\}
## .if t \{\
## .    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
## .    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
## .    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
## .    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
## .    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
## .    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
## .\}
## .    \" troff and (daisy-wheel) nroff accents
## .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
## .ds 8 \h'\*(#H'\(*b\h'-\*(#H'
## .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
## .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
## .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
## .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
## .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
## .ds ae a\h'-(\w'a'u*4/10)'e
## .ds Ae A\h'-(\w'A'u*4/10)'E
## .    \" corrections for vroff
## .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
## .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
## .    \" for low resolution devices (crt and lpr)
## .if \n(.H>23 .if \n(.V>19 \
## \{\
## .    ds : e
## .    ds 8 ss
## .    ds o a
## .    ds d- d\h'-1'\(ga
## .    ds D- D\h'-1'\(hy
## .    ds th \o'bp'
## .    ds Th \o'LP'
## .    ds ae ae
## .    ds Ae AE
## .\}
## .rm #[ #] #H #V #F C
## .\" ========================================================================
## .\"
## .IX Title "ORAHOT.KSH.TXT 1"
## .TH ORAHOT.KSH.TXT 1 "2008-12-28" "perl v5.8.8" "User Contributed Perl Documentation"
## .SH "NAME"
## orahot.ksh \- make Oracle database hot (online) backup.
## .SH "SYNOPSIS"
## .IX Header "SYNOPSIS"
## oracold.ksh help=y
## .PP
## oracold.ksh MySID|sid:=MySid [debug=y]
## .SH "DESCRIPTION"
## .IX Header "DESCRIPTION"
## \&\fBorahot.ksh\fR first generates a list of \fIfile_names\fR and \fItablespace
## names\fR ordered by \fIused space\fR. Put one or more tablespace in \fIbegin
## backup\fR mode as needed so that \fI\s-1NUM_PROCS\s0\fR (specified in configuration
## file) of files can be compressed simutaneously to \fImultiple backup
## directories\fR (\fI\s-1BKUP_DIRS\s0\fR, specified in configuration file). \fIAs soon
## as\fR all of the files within a tablespace are backed up, the tablespace
## is put back to \fIend backup\fR mode.
## .PP
## \&\fI\s-1ALTER\s0 \s-1SYSTEM\s0 \s-1SWITCH\s0 \s-1LOGFILE\s0\fR statement is issued and the archive log
## file up to the last inactive logfile is compressed to the backup
## directories.  The original archive log files are \fInot\fR removed.
## Backing up and removing the archive log files is the job of
## oraredo.ksh, not orahot.ksh.
## .PP
## init\fI\s-1SID\s0\fR.ora file is backed to one of the backup directories with
## unix copy.
## .PP
## The \fIcontrol file\fR is backed to one of the backup directories with
## \&\fI\s-1ALTER\s0 \s-1DATABASE\s0 \s-1CONTROLFILE\s0 \s-1TO\s0 <file\fR> statement, and unix copy.
## .PP
## The \fItext copy of the control file\fR is generated with \fI\s-1ALTER\s0 \s-1DATABASE\s0
## \&\s-1CONTROLFILE\s0 \s-1TO\s0 \s-1TRACE\s0\fR statement in \fIuser_dump_dest\fR directory. This
## file is backed up to one of the backup directories with unix copy, and
## the file is renamed \fIgen_controlfile.sql\fR. After verification of
## successful copying, the original file is removed.
## .PP
## The backup file systems are expected to be backed up via data center
## backup service.
## .PP
## Features include
## .IP "\(bu" 4
## Support paralell backup processes (\fI\s-1NUM_PROCS\s0\fR). Tablespaces will be
## put \*(L"begin backup\*(R" mode and \*(L"end backup\*(R" mode as needed.
## .IP "\(bu" 4
## Choice of compress, gzip, bzip2.
## .IP "\(bu" 4
## Large file support (it is not required to have a large file capable
## compress, gzip, bzip2).
## .IP "\(bu" 4
## Support multiple backup directories (\fI\s-1BKUP_DIRS\s0\fR). It can be used for I/O and
## capacity load balancing. Backup files are dynamically allocated to
## directories with largest free sapce.
## .IP "\(bu" 4
## Remote backup directory support. (Currently coded for rsh, it can be expanded
## to support a choice of rsh, or ssh).
## .IP "\(bu" 4
## Manage the number of backup copies online (\fI\s-1NUM_COPIES\s0\fR).
## .IP "\(bu" 4
## Put tablespace in \fIend backup\fR mode and clean up processes when the
## backup program encouters an error.
## .IP "\(bu" 4
## Program terminates itself after maximum allowed time (\fI\s-1MAX_SECS\s0\fR in
## configuration file). If \fI\s-1MAX_SECS\s0\fR is 0, the backup is skipped.
## .IP "\(bu" 4
## Prevent duplicate hot backup processes from running.
## .IP "\(bu" 4
## Cooperate with oraredo.ksh backup. orahot.ksh and oraredo.ksh wait for
## each other.
## .IP "\(bu" 4
## Backup files under \s-1DIRECTORY_PATH\s0.
## .SH "INSTALLATION"
## .IX Header "INSTALLATION"
## To install, follow these easy 1\-2\-3 steps:
## .IP "1." 4
## Create an install directory for hot backup <job_dir>/ora_hot,
## and under ora_hot, create following sub\-directories: 
## .Sp
## .Vb 2
## \& cd <job_dir>/ora_hot &&
## \& mkdir -p script etc operlog/<SID> temp/<SID> ../ora_redo/operlog/<SID>
## .Ve
## .Sp
## Where \s-1SID\s0 is the actual \s-1ORACLE_SID\s0 name. If you have multiple
## SIDs, then just create them under operlog and temp.
## .Sp
## Create ../ora_redo/operlog/<\s-1SID\s0> even if you do not use ora_redo.ksh
## as ora_hot.ksh places a lock there to cooperate with redo backup.
## .IP "2." 4
## .Vb 2
## \&  cp orahot.ksh script/orahot.ksh
## \&  chmod 755 script/orahot.ksh
## .Ve
## .IP "3." 4
## Edit etc/orahot_SID.cf from the sample configuration file shown below. If
## you have multiple SIDs, then just create multiple files. 
## .PP
## .Vb 4
## \&  # *
## \&  # * Where do you want the hot backup files to go?
## \&  # * It can be a directory on a remote server.
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  BKUP_DIRS="/bkup01/hot oracle@castor:/bkup02/hot"
## .Ve
## .PP
## .Vb 5
## \&  # *
## \&  # * How many simutaneous compress processes you want to run?
## \&  # * Rule of thumb is to set the number to be the same as the n of processors. 
## \&  # * Default is 3.
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  NUM_PROCS=3
## .Ve
## .PP
## .Vb 5
## \&  # *
## \&  # * How many copies you want to keep online?
## \&  # * ( If you keep only 1 copy, make sure the file system backup
## \&  # * occurs after the Oracle backup finishes. )
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  NUM_COPIES=2
## .Ve
## .PP
## .Vb 5
## \&  # *
## \&  # * What is the maximum time you allow the backup to run?
## \&  # * Backup will kill itself exceeding the maximum allowed time. 
## \&  # * Default is 999999.
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  MAX_SECS=21600
## .Ve
## .PP
## .Vb 14
## \&  # *
## \&  # * Where is the redo backup script? (Hot backup and redo backup work together.)
## \&  # * If you use the standard naming conventions (ie <install_dir>/ora_redo,
## \&  # * <install_dir>/ora_hot), then it will derived.
## \&  # *
## \&  # * New version (5.0+) does not rely on redo backup script anymore.
## \&  # * REDO_LOCK directory is used to prevent redo backup from running during
## \&  # * the hot backup. If you use the standard naming conventions, this
## \&  # * variable is derived (<install_dir>/ora_redo/operlog/$ORACLE_SID/lock).
## \&  # *
## \&  # * If you are sure redo backup is not run and no lock is needed, then you can
## \&  # * define REDO_WAIT_MINS (see below) as a negative number to bypass the locking
## \&  # * mechanism.
## \&  # *
## .Ve
## .PP
## .Vb 2
## \&  # REDO_PROG="/ds1-export/apps/jobs/ora_redo/script/oraredo.ksh" (depreciated)
## \&  # REDO_LOCK="/ds1-export/apps/jobs/ora_redo/operlog/$ORACLE_SID/lock" (derivable)
## .Ve
## .PP
## .Vb 7
## \&  # *
## \&  # * Where does redo backup go?
## \&  # * If you use the standard naming conventions (ie <install_dir>/ora_redo,
## \&  # * <install_dir>/ora_hot), then it will derived.
## \&  # *
## \&  # * This variable is not used in new version (5.0+).
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  # REDO_BKUP_DIRS="/bkup01/redo /bkup02/redo" (depreciated)
## .Ve
## .PP
## .Vb 4
## \&  # *
## \&  # * How long do you want to wait when another process is running redo backup?
## \&  # * Default is 60.
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  REDO_WAIT_MINS=60
## .Ve
## .PP
## .Vb 8
## \&  # *
## \&  # * What cat command do you use to cat the data files (pipe to
## \&  # * compress utility). Default is "cat", and if "cat" can not
## \&  # * open first controlfile due to restricted permission ocfs,
## \&  # * then cat is defined as $TMPDIR/cat which is:
## \&  # * "dd o_direct=yes ${1:+if=$1} bs=8k 2>/dev/null". If this logic
## \&  # * does not work, define CAT.
## \&  # *
## .Ve
## .PP
## .Vb 3
## \&  CAT="cat"
## \&  # CAT="/path/to/special/cat" where cat is something like:
## \&  # "dd o_direct=yes ${1:+if=$1} bs=8k 2>/dev/null"
## .Ve
## .PP
## .Vb 4
## \&  # *
## \&  # * What command do you use to compress a file to standard output?
## \&  # * Default is "compress -c".
## \&  # *
## .Ve
## .PP
## .Vb 3
## \&  COMPRESS_C="/usr/bin/compress -c"
## \&  # COMPRESS_C="/usr/local/bin/gzip -c"
## \&  # COMPRESS_C="/usr/local/bin/bzip2 -c"
## .Ve
## .PP
## .Vb 4
## \&  # *
## \&  # * Whether to copy files under each DIRECTORY_PATH
## \&  # * from dba_directories.
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  COPY_DIRECTORY_PATH=Y
## .Ve
## .PP
## .Vb 4
## \&  # *
## \&  # * FILTER_DIRECTORY_PATH: to include or exclude certain
## \&  # * OWNER, DIRECTORY_NAME, or DIRECTORY_PATH from DBA_DIRECTORIES.
## \&  # *
## .Ve
## .PP
## .Vb 1
## \&  FILTER_DIRECTORY_PATH="where DIRECTORY_NAME not in ('AUDIT_DIR')"
## .Ve
## .PP
## .Vb 4
## \&  # *
## \&  # * Where can you find submit?
## \&  # * /usr/local/bin/submit is the default.
## \&  # *
## .Ve
## .PP
## .Vb 2
## \&  SUBMIT=<job_dir>/common/submit
## \&  # SUBMIT=/usr/local/bin/submit
## .Ve
## .PP
## .Vb 8
## \&  # *
## \&  # * What is the directory where you can find oracle-SID?
## \&  # * /etc/init.d is the default.
## \&  # *
## \&  # * The oracle-SID is used to verify if Oracle is up before and after
## \&  # * the backup. If you want to bypass the tests, then define INIT_D as
## \&  # * "NA" (Not Apply).
## \&  # *
## .Ve
## .PP
## .Vb 2
## \&  INIT_D=<job_dir>/common
## \&  # INIT_D=/etc/init.d
## .Ve
## .PP
## .Vb 3
## \&  # *
## \&  # * You can define pre and post processing command. 
## \&  # *
## .Ve
## .PP
## .Vb 2
## \&  # PRE_PROCESSING="/bin/true"
## \&  # POST_PROCESSING="/bin/true"
## .Ve
## .SH "RESTORE"
## .IX Header "RESTORE"
## The ora_hot.ksh copies Oracle data files, control files,
## and archive log files generated \fIduring\fR the hot backup.
## The text copy of the control file gen_controlfile.sql, and
## parameter file init\fI\s-1SID\s0\fR.ora are also backed up gratuitously.
## .PP
## A sample hot backup file set is shown below: 
## .PP
## .Vb 15
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/arch/1_63.dbf
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/arch/1_64.dbf
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/control01.ctl
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/control02.ctl
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/control03.ctl
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/cwmlite01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/drsys01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/example01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/indx01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/system01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/tools01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/undotbs01.dbf.Z
## \&  /bkup01/hot/ora901/ora10/oradata/ora901/users01.dbf.Z
## \&  /bkup01/hot/ora901/gen_controlfile.sql
## \&  /bkup01/hot/ora901/initora901.ora
## .Ve
## .PP
## To restore, first copy Oracle data files, control files,
## and archive log files to where they were. As the original
## file path are recorded in the backup set, it is very easy to
## automate the task. The following \fIad hoc\fR script
## does this.
## .PP
## .Vb 1
## \&  #!/bin/ksh
## .Ve
## .PP
## .Vb 4
## \&  find /bkup01/hot/ora901/ora* -type f | while read i; do
## \&    j=${i#*ora901}
## \&    dir=${j%/*}
## \&    file=${j%.Z}
## .Ve
## .PP
## .Vb 2
## \&    CAT=cat
## \&    [[ $i = *.Z ]] && CAT=zcat
## .Ve
## .PP
## .Vb 3
## \&    mkdir -p $dir
## \&    $CAT $i > $file
## \&  done
## .Ve
## .PP
## What it does is something like,
## .PP
## .Vb 3
## \&  mkdir -p /ora10/oradata/ora901
## \&  zcat /bkup01/hot/ora901/ora10/oradata/ora901/system01.dbf.Z \e
## \&    > /ora10/oradata/ora901/system01.dbf
## .Ve
## .PP
## Once all the files are in place, we can go ahead to
## recover and open the database. A sample session is shown below,
## .PP
## .Vb 1
## \&  SQL> startup mount
## .Ve
## .PP
## .Vb 7
## \&  ORACLE instance started.
## \&  Total System Global Area   63734856 bytes
## \&  Fixed Size                   279624 bytes
## \&  Variable Size              46137344 bytes
## \&  Database Buffers           16777216 bytes
## \&  Redo Buffers                 540672 bytes
## \&  Database mounted.
## .Ve
## .PP
## .Vb 1
## \&  SQL> recover database using backup controlfile until cancel
## .Ve
## .PP
## .Vb 4
## \&  ORA-00279: change 1329616 generated at 05/02/2002 13:30:39
## \&  + needed for thread 1
## \&  ORA-00289: suggestion : /ora10/oradata/ora901/arch/1_63.dbf
## \&  ORA-00280: change 1329616 for thread 1 is in sequence #63
## .Ve
## .PP
## .Vb 1
## \&  Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
## .Ve
## .PP
## .Vb 6
## \&  ORA-00279: change 1329687 generated at 05/02/2002 13:40:42
## \&  + needed for thread 1
## \&  ORA-00289: suggestion : /ora10/oradata/ora901/arch/1_64.dbf
## \&  ORA-00280: change 1329687 for thread 1 is in sequence #64
## \&  ORA-00278: log file '/ora10/oradata/ora901/arch/1_63.dbf'
## \&  + no longer needed for this recovery
## .Ve
## .PP
## .Vb 1
## \&  Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
## .Ve
## .PP
## .Vb 6
## \&  ORA-00279: change 1329692 generated at 05/02/2002 13:40:46
## \&  + needed for thread 1
## \&  ORA-00289: suggestion : /ora10/oradata/ora901/arch/1_65.dbf
## \&  ORA-00280: change 1329692 for thread 1 is in sequence #65
## \&  ORA-00278: log file '/ora10/oradata/ora901/arch/1_64.dbf'
## \&  + no longer needed for this recovery
## .Ve
## .PP
## .Vb 3
## \&  Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
## \&  CANCEL
## \&  Media recovery cancelled.
## .Ve
## .PP
## .Vb 2
## \&  SQL> alter database open RESETLOGS
## \&    2  /
## .Ve
## .PP
## .Vb 1
## \&  Database altered.
## .Ve
## .PP
## That is it, happy restore!
## .SH "SEE ALSO"
## .IX Header "SEE ALSO"
## submit help=y, rc.oracle help=y, oraredo.ksh help=y.
## .SH "AUTHOR"
## .IX Header "AUTHOR"
## Michael Wang <\fIxw73@columbia.edu\fR> with help from many others
## acknowledged in version notes.
## .PP
## This is free software. You may copy or redistribute it under the same
## terms as Perl. However, if you modify it, you need to send the
## modification to the author via email.
## MAN_STOP