IAS component OC4JGroup:default_group (OC4J:oacore) stopped/failed automatically in EBS

In our production instance R12.1.X.X  IAS component "oacore" frequently

down/failed.

Check the services in case of oacore failure..

$sh adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance.

Processes in Instance: PROD.xxxxxxxxxx
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------

OC4JGroup:default_group          | OC4J:oafm           |   19410 | Alive

OC4JGroup:default_group          | OC4J:forms          |   19319 | Alive

OC4JGroup:default_group          | OC4J:oacore        |   19140 | Failed

HTTP_Server                             | HTTP_Server       |   19069 | Alive

OC4JGroup:default_group          | OC4J:oacore    |  23108 | failed

To resolve the above issue  need to be modify/increase the JVM processes as below in context file: 

<oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-

server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M -

XX:NewRatio=2 

To 

<oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-

server -verbose:gc -Xmx1024M -Xms512M -XX:MaxPermSize=256M -

XX:NewRatio=2 

=> Stop the services

 
=> Run autoconfig 

=> Restart the services 


Check it again:-

$sh adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2

Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: PROD.xxxxxxxxx
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------

OC4JGroup:default_group          | OC4J:oafm          |   19410 | Alive

OC4JGroup:default_group          | OC4J:forms         |   19319 | Alive

OC4JGroup:default_group          | OC4J:oacore       |   19140 | Alive

HTTP_Server                             | HTTP_Server     |   19069 | Alive

In my case issue was resolved successfully ......😁

adop phase=prepare ORA-20008: No Concurrent Manager is running that can run concurrent program

Below error coming while executed adop phase=prepare.

Submitting ADZDPATCH concurrent program.
    Waiting for conflicting requests to complete.
    [ERROR]     Failed to execute sql statement:
select AD_ZD_ADOP.WAIT_FOR_CP_TO_RUN(35360925) from dual
    [ERROR]     SQLPLUS error: buffer=
SQL*Plus: Release 10.1.0.5.0 - Production on Sat Sep 8 14:01:00 2018
Copyright (c) 1982, 2005, Oracle.  All rights reserved.

SQL> SQL> Connected.

SQL> select AD_ZD_ADOP.WAIT_FOR_CP_TO_RUN(35360925) from dual
       *
ERROR at line 1:
ORA-20008: No Concurrent Manager is running that can run concurrent program
ADZDPATCH
ORA-06512: at "APPS.AD_ZD_ADOP", line 282
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

    [UNEXPECTED]Error calling runConcurrentProgram subroutine.
    Stopping services on patch file system.
    Stopping admin server.
    Stopping node manager.
    [ERROR]     Prepare phase has failed.

Solution:-

Set the Run filesystem / environment and run command:


FNDLOAD apps/apps 0 Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct $AD_TOP/patch/115/import/US/adzdpatch.ldt - CUSTOM_MODE=FORCE

If still not resolve the issue. Pls follow the below steps.

First of all truncate the following tables.

truncate table applsys.fnd_nodes;

truncate table applsys.fnd_oam_context_files;

truncate table applsys.adop_valid_nodes;

 If on AD/TXK 8 or higher:

truncate table applsys.ad_nodes_config_status;

run autoconfig on DB and appsTier.

To check whether its working or not.

The syntax to run an empty patch cycle is:

Quick Version:

$ cd <EBS_ROOT>

$ source EBSapps.env run

$ adop phase=prepare,finalize,cutover

$ source EBSapps.env run

$ adop phase=cleanup

Complete Version:

$ cd <EBS_ROOT>

$ source EBSapps.env run

$ adop phase=prepare

$ adop phase=actualize_all

$ adop phase=finalize finalize_mode=full

$ adop phase=cutover

$ source EBSapps.env run

$ adop phase=cleanup cleanup_mode=full

How To Purge E-Mail workflow notifications From The EBS workflow queue.

Sometimes we need to purge the huge notification queue in ebs workflow, because of mailer services got failed and giving an error while starting or while refreshing/cloning new instance. 

As we know that old email notification not needed in cloned instance. In this scenario we have to purge all those workflow notification queue from cloned ebs instance.

Below script will verify the current status in table WF_NOTIFICATIONS that has to be sent when mailer services gets started.

SQL>select notification_id, begin_date, recipient_role, message_type, message_name, status, mail_status from wf_notifications Where Status In ('OPEN', 'CANCELED') And Mail_Status In ('MAIL', 'INVALID') --and begin_date < sysdate-30  --> List only emails older than 30 days ago order by notification_id;

Output:-

NOTIFICATION_ID BEGIN_DATE RECIPIENT_ROLE MESSAGE_TYPE

 

MESSAGE_NAME STATUS MAIL_STATUS

6273813 4/26/2016 4:01:17 PM SYSADMIN OMERROR OMERROR_MSG 

OPEN MAIL

6273814 4/26/2016 4:01:18 PM SYSADMIN OMERROR OMERROR_MSG 

OPEN MAIL

6273815 4/26/2016 4:01:18 PM SYSADMIN OMERROR OMERROR_MSG 

OPEN MAIL

6273816 4/26/2016 4:01:19 PM SYSADMIN OMERROR OMERROR_MSG 

OPEN MAIL

Update the mail_status to sent , if you have not requirement for old workflow email notification. It will not en-queue the messages again.

SQL>update WF_NOTIFICATIONS set mail_status = 'SENT' where mail_status in ('MAIL','INVALID') and Status In ('OPEN', 'CANCELED');

Purge the WF_NOTIFICATION_OUT queue and rebuild it with data currently in the WF_NOTIFICATIONS table.

SQL> @$FND_TOP/patch/115/sql/wfntfqup.sql apps apps applsys

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.01

Commit complete.

Elapsed: 00:00:00.00

**** TEMPORARY TABLES / AQs created ****

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.43

**** Messages backed up ****

Commit complete.

Elapsed: 00:00:00.00

**** wfaqback.sql completed,  executing SQLs from wfntfqup

PL/SQL procedure successfully completed.

Elapsed: 00:00:17.37

***  invoking wfaqrenq.sql ******

*** Re-enqueing messages

PL/SQL procedure successfully completed.

Elapsed: 00:00:00.26

Commit complete.

Elapsed: 00:00:00.00

***** Objects related to wf_queue_temp_evt_table and 

wf_queue_temp_jms_table are dropped *****

***** Re-enqueue OF Alerts completed *****

Disconnected from Oracle Database 11g Enterprise Edition Release 

11.2.0.4.0 - 64 bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options

[applmgr@prdsrv scripts]$

After that need to be start the workflow mailer services.