After applying a patch with adop: (prepare/apply/finalize), the cutover failed.
The services on primary node possibly partially up, but services down on the secondary node.
Summary report for current adop session:
Node acedisupplier: Failed
- Cleanup status: Failed
- Apply status: Completed successfully
- Finalize status: Completed successfully
- Cutover status: FORCE_SHUTDOWN_COMPLETED
Node prdsvr: Failed
- Cleanup status: Failed
- Apply status: Completed successfully
- Finalize status: Completed successfully
- Cutover status: PRE_CUTOVER_CHECKS_COMPLETED
For more details, run the command: adop -status -detail
If we see CUTOVER_STATUS column the below will determine the code and status.
-->='Y' 'COMPLETED'
--> not in ('N','Y','X') and status='F' 'FAILED'
-->='0' 'CUTOVER STARTED'
-->='1' 'SERVICES SHUTDOWN COMPLETED'
-->='3' 'DB CUTOVER COMPLETED'
-->='D' 'FLIP SNAPSHOTS COMPLETED'
-->='4' 'FS CUTOVER COMPLETED'
-->='5' 'ADMIN STARTUP COMPLETED'
-->='6' 'SERVICES STARTUP COMPLETED'
-->='N' 'NOT STARTED'
-->='X' 'NOT APPLICABLE'
Summary report for current adop session:
Node acedisupplier: Failed
- Cleanup status: Failed
- Apply status: Completed successfully
- Finalize status: Completed successfully
- Cutover status: FORCE_SHUTDOWN_COMPLETED
Node prdsvr: Failed
- Cleanup status: Failed
- Apply status: Completed successfully
- Finalize status: Completed successfully
- Cutover status: PRE_CUTOVER_CHECKS_COMPLETED
For more details, run the command: adop -status -detail
To check the details information setup analyzer.
ADOP Analyzer: Please upload the ICM ADOP Application Tier Analyzer output zip-file by using the LATEST version of the analyzer available in
EBS Installation Configuration Management (ICM) Application DBA Online Patching (ADOP) Analyzer (Doc ID 2379656.1).
SQL>select ADOP_SESSION_ID, EDITION_NAME, NODE_TYPE, NODE_NAME, APPLTOP_ID, PREPARE_STATUS,APPLY_STATUS, CUTOVER_STATUS, CLEANUP_STATUS, ABORT_STATUS, STATUS from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;
ODM Data Collection:-
35.00 V_20211201_2323 slave acedisupplier 193.00 Y Y 1 N N F
35.00 V_20211201_2323 master prdsvr 193.00 Y Y 0 N N F
For the slave completed the shutdown but for the master is just started.
Checking for existing adop sessions.
Continuing with existing session [Session ID: 35].
Session Id : 35
Prepare phase status : COMPLETED
Apply phase status : COMPLETED
Cutover phase status : NOT COMPLETED
Abort phase status : NOT COMPLETED
Session status : RUNNING
cat /oracle/aced/EBSFS/fs1/inst/apps/aced_prdsvr/logs/appl/admin/log/adapcctl.txt
12/01/21-23:51:35 :: adapcctl.sh version 120.0.12020000.6
12/01/21-23:51:35 :: adapcctl.sh: stopping OPMN managed OHS instance
opmnctl stopproc: stopping opmn managed processes...
================================================================================
opmn id=prdsvr.aced-cranes.com:6203
Response: 0 of 1 processes stopped.
ias-instance id=EBS_web_OHS1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
EBS_web/OHS/OHS/
Error
--> Process (index=1,uid=1243434072,pid=34408)
time out while waiting for a managed process to stop
second stop attempted
time out while waiting for a managed process to stop
Log:
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log
12/01/21-23:53:39 :: adapcctl.sh: exiting with status 204
Checking console~OHS~1.log.
--------
21/12/01 23:55:43 Stop process
--------
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
21/12/01 23:56:45 Stop process
--------
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/apachectl hardstop: httpd (no pid file) not running
--------
21/12/01 23:57:47 Stop process
--------
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/apachectl hardstop: httpd (no pid file) not running
FileName
----------------
Attachment: <console_OHS_1.log>
Okay, we can try to stop the service manually by running the
/oracle/aced/EBSFS/fs1/inst/apps/aced_prdsvr/admin/scripts/adapcctl.sh stop
If it fails check below:
1. Please confirm the httpd.pid file configuration in httpd.conf file.
2. Please switch to the directory /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/ and run the following command,
Check the output
$ grep PIDFILE apachectl
Below are output
[applmgr@prdsvr bin]$ grep PIDFILE apachectl
PIDFILE=${ORACLE_CONFIG_HOME}/logs/httpd.pid
PIDFILE=${COMPONENT_LOG_PATH}/httpd.pid
if [ -f $PIDFILE ]
PID=`cat $PIDFILE`
#PIDFILE=`grep PidFile $1 | grep -v "#" | cut -d" " -f 2`
PIDFILE=`grep "^PidFile" $1| cut -d" " -f 2`
PIDFILE=`eval echo $PIDFILE`
if [ "x$PIDFILE" = "x" ]
We will check if the PID does exist :
[applmgr@prdsvr scripts]$ cat /oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/httpd.pid
cat: /oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/httpd.pid: No such file or directory
Some processes are running as below.
[applmgr@prdsvr scripts]$ ps -ef|grep applmgr
applmgr 3580 1 0 Oct23 ? 00:00:05 frmweb server webfile=HTTP-0,0,1,default
applmgr 9514 34408 0 Dec01 ? 00:00:20 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 10168 34408 0 Dec01 ? 00:00:21 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 16163 1 0 Nov16 ? 00:00:01 frmweb server webfile=HTTP-0,0,1,default
root 19601 1270 0 04:32 pts/1 00:00:00 su - applmgr
applmgr 19603 19601 0 04:32 pts/1 00:00:00 -bash
root 19781 19717 0 Dec01 pts/0 00:00:00 su - applmgr
applmgr 19783 19781 0 Dec01 pts/0 00:00:00 -bash
lp 21624 35206 0 04:38 ? 00:00:00 smb://192.168.11.209/P6SPDDM1 226243 applmgr COMM-P6SPD.57030008 1 cpi=12 finishings=3 fitplot lpi=6
media=na_letter_8.5x11in number-up=1 page-bottom=86 page-left=57 page-right=57 page-top=30 scaling=100 wrap job-uuid=urn:uuid:a7311f94-7621-
3e6e-5a1b-8e6322635064 job-originating-host-name=localhost
applmgr 21707 19603 0 04:40 pts/1 00:00:00 ps -ef
applmgr 21708 19603 0 04:40 pts/1 00:00:00 grep applmgr
applmgr 34408 1 0 Oct16 ? 00:03:04 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 34506 34408 0 Oct16 ? 00:00:02 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/odl_rotatelogs -l
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web-%Y%m%d%H%M%S.log 10M 70M
applmgr 34507 34408 0 Oct16 ? 00:24:08 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/odl_rotatelogs
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/access_log 43200
applmgr 34508 34408 0 Oct16 ? 00:00:00 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/rotatelogs
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/access_log 86400
applmgr 34509 34408 0 Oct16 ? 00:00:00 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/rotatelogs
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/ssl_request_log 86400
applmgr 34519 34408 0 Oct16 ? 00:00:00 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/odl_rotatelogs -l -h:/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/component_events.xml_EBS_web
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/auditlogs/OHS/EBS_web/audit-pid34408-%Y%m%d%H%M%S.log 1M 4M
applmgr 34520 34408 0 Oct16 ? 00:00:47 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 39310 34408 0 Dec01 ? 00:00:23 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
It seems the opmn is already down. Confirm that :
Then check and confirm is started and PID does exist :
$ps -ef | grep -i opm
[applmgr@prdsvr scripts]$ ./adopmnctl.sh status
You are running adopmnctl.sh version 120.0.12020000.2
Checking status of OPMN managed processes...
Processes in Instance: EBS_web_OHS1
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
EBS_web | OHS | 34408 | Stop
After kill pid 34408. it started now.
$kill -9 34408
We will go to the master node and start the opmn ?
If it's down then run cutover again.
Now Executed adop phase=cutover.
In adopmon frequently showing as below.
2021/12/02 05:05:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
2021/12/02 05:05:35 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:06:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
2021/12/02 05:06:35 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:07:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
2021/12/02 05:07:35 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:08:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
2021/12/02 05:08:35 acedisupplier EVENT Will wait for another minute and check again.
It seems cutover is still trying to stop it regardless the opmn is down already. Since PID does not exist is constantly failing.
cat /oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/httpd.pid
Then if everything looks good run the cutover again.
2021/12/02 05:33:44 prdsvr EVENT Cutover System
2021/12/02 05:33:44 prdsvr EVENT Switching to Patch Edition
2021/12/02 05:33:44 prdsvr EVENT Killing old sessions
2021/12/02 05:33:46 prdsvr EVENT Flipping snapshots in run and patch editions.
2021/12/02 05:33:47 prdsvr EVENT Performing file system cutover.
2021/12/02 05:34:38 acedisupplier EVENT Waiting for flipping of snapshots to complete on Admin node.
2021/12/02 05:34:39 acedisupplier EVENT Performing file system cutover.
2021/12/02 05:34:43 prdsvr EVENT Checking if file system cutover completed.
2021/12/02 05:34:43 prdsvr EVENT File System Cutover did not complete on node(s): <acedisupplier>
2021/12/02 05:34:43 prdsvr EVENT Will wait for another minute and check again.
2021/12/02 05:35:43 prdsvr EVENT File System Cutover did not complete on node(s): <acedisupplier>
2021/12/02 05:35:43 prdsvr EVENT Will wait for another minute and check again.
2021/12/02 05:36:43 prdsvr EVENT File System Cutover did not complete on node(s): <acedisupplier>
2021/12/02 05:36:43 prdsvr EVENT Will wait for another minute and check again.
2021/12/02 05:36:55 acedisupplier EVENT Checking if file system cutover completed.
2021/12/02 05:36:55 acedisupplier EVENT Waiting for Admin Server to start.
2021/12/02 05:36:55 acedisupplier EVENT Starting Admin Server did not complete on Admin node: <prdsvr>
2021/12/02 05:36:55 acedisupplier EVENT Will wait for another minute and check again.
Broadcast message from applmgr@prdsvr.aced-cranes.com (Thu Dec 2 05:37:43 2021):
aced environment has changed.
All users must re-source the environment.
2021/12/02 05:37:43 prdsvr EVENT Starting run edition Admin Server.
2021/12/02 05:37:43 prdsvrdb EVENT Compile Edition: V_20211201_2323, In a Loop
2021/12/02 05:37:44 prdsvrdb EVENT Compile Edition: V_20211201_2323
2021/12/02 05:37:52 prdsvrdb EVENT Compile Edition: V_20211201_2323
2021/12/02 05:37:55 acedisupplier EVENT Starting Admin Server did not complete on Admin node: <prdsvr>
2021/12/02 05:37:55 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:38:55 acedisupplier EVENT Starting Admin Server did not complete on Admin node: <prdsvr>
2021/12/02 05:38:55 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:38:59 prdsvr EVENT Starting application tier services.
2021/12/02 05:39:56 acedisupplier EVENT Starting application tier services.
Summary report for current adop session:
Node acedisupplier: Completed successfully
- Cutover status: Completed successfully
Node prdsvr: Completed successfully
- Cutover status: Completed successfully
For more details, run the command: adop -status -detail
[applmgr@prdsvr scripts]$ adop -status -detail
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 35
Command: status
Output: /oracle/aced/EBSFS/fs_ne/EBSapps/log/adop/35/20211202_054604/adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
prdsvr master PREPARE COMPLETED 2021/12/01 23:23:23 2021/12/01 23:31:07 0:07:44
APPLY COMPLETED 2021/12/01 23:39:26 2021/12/01 23:41:34 0:02:08
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:48:16 2021/12/02 05:44:14 5:55:58
CLEANUP NOT STARTED
acedisupplier slave PREPARE COMPLETED 2021/12/01 23:25:25 2021/12/01 23:38:21 0:12:56
APPLY COMPLETED 2021/12/01 23:44:14 2021/12/01 23:44:15 0:00:01
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:50:19 2021/12/02 05:45:47 5:55:28
CLEANUP NOT STARTED
After successfully complted cutover on master and slave node. Now we will run adop phase=cleanup.
I have checked that all services are started. Now cleanup is completed.
Summary report for current adop session:
Node acedisupplier: Completed successfully
- Cleanup status: Completed successfully
Node prdsvr: Completed successfully
- Cleanup status: Completed successfully
For more details, run the command: adop -status -detail
The cleanup phase completed successfully.
adop exiting with status = 0 (Success)
You have new mail in /var/spool/mail/applmgr
[applmgr@prdsvr scripts]$
You have new mail in /var/spool/mail/applmgr
[applmgr@prdsvr scripts]$
[applmgr@prdsvr scripts]$
[applmgr@prdsvr scripts]$
[applmgr@prdsvr scripts]$ adop -status -detail
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 35
Command: status
Output: /oracle/aced/EBSFS/fs_ne/EBSapps/log/adop/35/20211202_060018/adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
--------------- ---------- --------------- --------------- -------------------- -------------------- ------------
prdsvr master PREPARE COMPLETED 2021/12/01 23:23:23 2021/12/01 23:31:07 0:07:44
APPLY COMPLETED 2021/12/01 23:39:26 2021/12/01 23:41:34 0:02:08
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:48:16 2021/12/02 05:44:14 5:55:58
CLEANUP COMPLETED 2021/12/02 05:51:51 2021/12/02 05:54:15 0:02:24
acedisupplier slave PREPARE COMPLETED 2021/12/01 23:25:25 2021/12/01 23:38:21 0:12:56
APPLY COMPLETED 2021/12/01 23:44:14 2021/12/01 23:44:15 0:00:01
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:50:19 2021/12/02 05:45:47 5:55:28
CLEANUP COMPLETED 2021/12/02 05:51:51 2021/12/02 05:54:15 0:02:24
File System Synchronization Type: Full
Generating full adop status report: /oracle/aced/EBSFS/fs_ne/EBSapps/log/adop/35/20211202_060018/adzdshowstatus.out
Cutover failed to stop the web since the PID was already killed
As there was not PID to kill cutover failed trying to stop the web
