Sajid's Oracle DBA Blog: December 2020

START: Precloning the OHS Config txkSetOHSConfig.pl ERROR: Script failed, exit code 1

$perl adpreclone.pl appsTier

$adop phase=fs_clone

Redwood Shores, California, USA

Oracle E-Business Suite Rapid Clone

Version 12.2

adpreclone Version 120.31.12020000.25

Enter the APPS User Password:

Enter the Weblogic AdminServer password :

Checking the status of the Oracle WebLogic Administration Server....

Running perl /data05/EBS_FS_12.2.5/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_acetest/appl/admin/PHASE1_xxxtest.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

wlsDomainName: EBS_domain

WLS Domain Name is VALID.

Running:

perl /data05/EBS_FS_12.2.5/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/util/jdk64 mode=stage stage=/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone component=appsTier method= appctx=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/appl/admin/PHASE1_xxxtest.xml showProgress

Setting the wls environment

Beginning application tier Stage - Mon Dec 28 13:52:57 2020

/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/util/jdk64/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/lib/xmlparserv2.jar:/data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/java/classes:/data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/oui/jlib/ewt3.jar:/data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/oui/jlib/share.jar:/data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/jlib/ojmisc.jar:/acedata05/EBS_FS_12.2.5/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/data05/EBS_FS_12.2.5/fs2/FMW_Home/oracle_common/jlib/obfuscatepassword.jar oracle.apps.ad.clone.StageAppsTier -e /data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_acetest/appl/admin/PHASE1_acetest.xml -stage /data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone -tmp /tmp -method CUSTOM -showProgress -nopromptmsg

Log file located at /data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/admin/log/clone/StageAppsTier_12281352.log

- 20% completed

ERROR while running Stage...

Mon Dec 28 13:58:46 2020

ERROR while running perl /data05/EBS_FS_12.2.5/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/util/jdk64 mode=stage stage=/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone component=appsTier method= appctx=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/appl/admin/PHASE1_xxxtest.xml showProgress ...

Mon Dec 28 13:58:46 2020

Log file located at :-

/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/admin/log/clone/StageAppsTier_12281229.log

START: Staging OHS

Evaluated internal cloning stage: /data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone

START: Precloning the OHS Config.

Running /data05/EBS_FS_12.2.5/fs2/FMW_Home/webtier/perl/bin/perl /data05/EBS_FS_12.2.5/fs2/EBSapps/appl/fnd/12.0.0/patch/115/bin/txkSetOHSConfig.pl preCloneInstance -ctxfile=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/appl/admin/PHASE1_xxxtest.xml -t2plogloc=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/admin/log/clone/ohsT2PStage -logdir=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/admin/log/clone -console=off

Script Executed in 1401 milliseconds, returning status 1

ERROR: Script failed, exit code 1

cd /data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_xxxtest/admin/log/clone/ohsT2PStage

$CLONE2020-12-28_12-35-20_2049110780.error CLONE2020-12-28_12-35-20_2049110780.log

[applmgr@erpbackup ohsT2PStage]$ cat CLONE2020-12-28_12-35-20_2049110780.error

SEVERE : Dec 28, 2020 12:35:20 - ERROR - CLONE-20218 Cloning is not successful.

SEVERE : Dec 28, 2020 12:35:20 - CAUSE - CLONE-20218 An internal operation failed.

SEVERE : Dec 28, 2020 12:35:20 - ACTION - CLONE-20218 Provide the clone log and error file for investigation.

oracle.as.t2p.exceptions.FMWT2PCopyConfigException: java.lang.NullPointerException

at oracle.as.clone.ohs.util.OHSCopyConfig.includeCustomisableDirectiveFiles(OHSCopyConfig.java:150)

at oracle.as.clone.ohs.impl.OHSComponentCreateClonerImpl.doCopyConfig(OHSComponentCreateClonerImpl.java:108)

at oracle.as.clone.cloner.component.ComponentCreateCloner.doClone(ComponentCreateCloner.java:122)

at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)

at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:162)

at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:94)

at oracle.as.clone.request.CreateCloneRequest._clone(CreateCloneRequest.java:69)

at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)

at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)

at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:236)

at oracle.as.clone.client.CloningClient.main(CloningClient.java:124)

Caused by: java.lang.NullPointerException

at java.io.File.<init>(File.java:277)

at oracle.as.clone.ohs.util.OHSCopyConfig.addFilesToJarForDirectiveType(OHSCopyConfig.java:184)

at oracle.as.clone.ohs.util.OHSCopyConfig.includeCustomisableDirectiveFiles(OHSCopyConfig.java:137)

... 10 more

Root Cause:-

SSL is not configured, however strong cipher configuration is implemented as documented in the following:

Fix:-

To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Revert the strong cipher steps in the fileopmn.xml ($IAS_ORACLE_HOME/instances/EBS_web_<sid>_ohs#/config/OPMN/opmn/opmn.xml) from 'Section 5: Strong Cipher Configuration Instructions'.

2. Bounce FMW 11.1.1.9 WebTier server

3. Re-test the adop phase=fs_clone command and confirm it completes successfully

-----------------------------------------------------------

12.2 E-Business Suite adop phase=fs_clone Fails On /txkSetOHSConfig.pl With Error 'FATAL ERROR ERRORMSG: ...adclone.pl did not go through successfully' (Doc ID 2179658.1)

adpreclone.pl Fails on Applications Tier With Error Message "Edit session is enabled in domain

adpreclone.pl failed On Patch File System with following error:-

$perl adpreclone.pl appsTier

Redwood Shores, California, USA

Oracle E-Business Suite Rapid Clone

Version 12.2

adpreclone Version 120.31.12020000.25

Enter the APPS User Password:

Enter the Weblogic AdminServer password :

Checking the status of the Oracle WebLogic Administration Server....

Running perl /data05/EBS_FS_12.2.5/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/adProvisionEBS.pl ebs-get-serverstatus -contextfile=/data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_xxxtest/appl/admin/PHASE1_xxxtest.xml -servername=AdminServer -promptmsg=hide

The Oracle WebLogic Administration Server is up.

wlsDomainName: EBS_domain

WLS Domain Name is VALID.

Running:

perl /data05/EBS_FS_12.2.5/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/data05/EBS_FS_12.2.5/fs1/EBSapps/comn/util/jdk64 mode=stage stage=/data05/EBS_FS_12.2.5/fs1/EBSapps/comn/clone component=appsTier method= appctx=/data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_xxxtest/appl/admin/PHASE1_xxxtest.xml showProgress

Setting the wls environment

- 20% completed

ERROR while running Stage...

Mon Dec 28 10:47:57 2020

ERROR while running perl /data05/EBS_FS_12.2.5/fs2/EBSapps/appl/ad/12.0.0/bin/adclone.pl java=/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/util/jdk64 mode=stage stage=/data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone component=appsTier method= appctx=/data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_acetest/appl/admin/PHASE1_acetest.xml showProgress ...

Mon Dec 28 10:47:57 2020

StageAppsTier_12281046.log

START: Creating FMW archive.

Running /data05/EBS_FS_12.2.5/fs2/FMW_Home/oracle_common/bin/copyBinary.sh -javaHome /data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone/FMW/t2pjdk -al /data05/EBS_FS_12.2.5/fs2/EBSapps/comn/clone/FMW/FMW_Home.jar -smw /data05/EBS_FS_12.2.5/fs2/FMW_Home -ldl /data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_acetest/admin/log/clone/fmwT2PStage -invPtrLoc /etc/oraInst.loc -silent true -debug true

Script Executed in 25233 milliseconds, returning status 255

ERROR: Script failed, exit code 255

In the Weblogic Console, Edit session is enabled in domain. Hence, adpreclone.pl is unable to complete successfully.

Workaround:-

Execute the following steps to fix the issue:

1) Login to the WLS Console

2) Review the "Change Center" and click the "Release Configuration"

3) You should see the domain in "Lock & Edit" mode.

4) Re-run adpreclone on the Applications Tier.

The Server is not able to service this request: [Socket:000445]Connection rejected, filter blocked Socket, weblogic.security.net.FilterException: [Security:090220]rule 3

If you are unable to access weblogic Admin console then find config.xml under domain and config directory for e.g

$FMW_HOME/user_projects/domains/EBS_domain_PROD/config

and stop application server or Admin Server -> make backup file of config.xml

once you replace deny with allow start application server or Admin Server.

After start Admin Server , able to open login page.

http://xxxx.xxx-xxx.com:7026/console

Serial number in context file contains lower value than that of database copy.

$ adop phase=prepare

ERROR: InDbCtxFile.uploadCtx() : Exception : Error executing BEGIN

fnd_gsm_util.upload_context_file(:1,:2,:3,:4,:5); END;: 1; Serial number in context file contains lower value than that of database copy.

Cause: Context file editing through Oracle Applications Manager did not complete file system synchronization. Please correct the errors that caused during editing and then run this program again. (FILE={CONTEXT_FILE}.xml)

oracle.apps.ad.autoconfig.oam.InDbCtxFileException: Error executing BEGIN fnd_gsm_util.upload_context_file(:1,:2,:3,:4,:5); END;: 1; Serial number in context file contains lower value than that of database copy.

This error will occur when the value for s_contextserial in the context file doesn't match the value stored in the FND_OAM_CONTEXT_FILES table. The context files with the issue looked as follows:

RUN filesystem context file:

$ . ./EBSapps.env run

$ grep s_contextserial $CONTEXT_FILE

--------------------------

<oa_context_serial oa_var="s_contextserial">1790</oa_context_serial>

PATCH filesystem context file:

$ . ./EBSapps.env patch

$ grep s_contextserial $CONTEXT_FILE

<oa_context_serial oa_var="s_contextserial">1789</oa_context_serial>

So in the above case, the context file on the PATCH filesystem has s_contextserial set to 1789.

In the database, we see the following value for s_contextserial for the PATCH filesystem:

SQL>

SELECT extractValue(XMLType(TEXT),'//oa_context_serial') SERIAL_NUMBER, extractValue(XMLType(TEXT),'//file_edition_type') FILE_EDITION from fnd_oam_context_files where name not in ('TEMPLATE','METADATA') and (status !='H') and (CTX_TYPE !='D') ;

Value from FND_OAM_CONTEXT_FILES table:

SERIAL_NUMBER = 1789 for the PATCH FILE_EDITION.

1. Take a backup of the following file:

$. ./EBSapps.env patch

$ cp $CONTEXT_FILE $CONTEXT_FILE.bak

2. Update the following entry in the context file:

From:

<oa_context_serial oa_var="s_contextserial">1789</oa_context_serial>

To:

<oa_context_serial oa_var="s_contextserial">1790</oa_context_serial>

3. Run the failed adop phase=prepare again.

How to create the patch file system in oracle apps R12.2

Here we are discussing 'how to recreate the patch file system in R12.2'. Execute perl adcfgclone.pl appsTier to recreate the patch file system in r12.2.

Follow the below step's to configure patch file system.

1) Run adpreclone:

On the RUN File-System, source RUN File-System:

$ source EBSapps.env run

# RUN Admin server MUST be up for this step

# check status:

$ $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh status

# start if needed:

$ $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start

# execute adpreclone:

$ cd $INST_TOP/admin/scripts

$ perl adpreclone.pl appsTier

2) Detach PATCH homes:

This step is performed from the RUN file-system detaching oracle_homes on the PATCH file-system:

$ cd $FMW_HOME/oracle_common/oui/bin

$ ./runInstaller -detachhome ORACLE_HOME={oracle home} -silent

# for each PATCH oracle_home

./runInstaller -detachhome ORACLE_HOME=/u01/app/fs2/FMW_Home/webtier -silent

./runInstaller -detachhome ORACLE_HOME=/u01/app/fs2/FMW_Home/Oracle_EBS-app1 -silent

./runInstaller -detachhome ORACLE_HOME=/u01/app/fs2/FMW_Home/oracle_common -silent

# The 10.1.2 Oracle Home needs a different syntax

$ ./runInstaller -removeHome ORACLE_HOME=<s_tools_oh> -silent

$ ./runInstaller -removeHome ORACLE_HOME=/u01/app/fs2/EBSapps/10.1.2 -silent

3) Copy RUN file system to PATCH file system:

# it may be helpful for RUN environment to be DOWN for this step

# copy each of these locations from RUN to PATCH

<APPL_TOP> [ /oracle/VIS/fs1/EBSapps/appl ]

<COMMON_TOP> [ /oracle/VIS/fs1/EBSapps/comn ]

<OracleAS Tools 10.1.2 ORACLE_HOME> [/oracle/VIS/fs1/EBSapps/10.1.2 ]

4) Remove/rename PATCH FMW_HOME directory:

/oracle/VIS/fs2/FMW_Home

5) Disable ebs_login:

Linux#> sqlplus system/<pwd>

SQLPlus#> alter trigger ebs_logon disable;

6) Execute adcfgclone:

$ cd <PATCH COMMON_TOP>/clone/bin$ perl adcfgclone.pl appsTier

For the prompts in this step:

Do you want to add a node (yes/no) [no] : no

Target System File Edition type [run] : patch

Enter the full path of Run File System Context file : <absolute path to the Run File System Context file>

RC-00217: Warning: Configuration home directory (s_config_home) evaluates to <PATCH s_config_home>. A directory with this name already exists and is not empty.

Do you want to continue (y/n) : y

This is expected as part of the rebuild.

Target System Port Pool [0-99] : <desired PATCH File System Port Pool>

This will be the PATCH File System port pool. Would suggest that the RUN and PATCH port pools be different by approx twice number of managed servers on this node.

7) Verify work:

In the new terminal - source PATCH file-system

$ EBSapps.env patch

$ echo $FILE_EDITION

In new terminal - source RUN file-system

# Run Admin server MUST be up for this step

$ EBSapps.env run

$ adop phase=fs_clone force=yes

8) Enable ebs_login:

Linux#> sqlplus system/<pwd>

SQLPlus#> alter trigger ebs_logon enable;

Adop Fails With Error: ssh is not enabled for the following nodes

Getting the following error applying a patch running Adop:

adop phase=prepare

Enter the APPS password:

Enter the SYSTEM password:

Enter the WLSADMIN password:

Validating credentials.

Initializing.

Run Edition context : /data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_***test/appl/admin/PHASE1_***test.xml

Patch edition context: /data05/EBS_FS_12.2.5/fs2/inst/apps/PHASE1_***test/appl/admin/PHASE1_***test.xml

Patch file system free space: 284.47 GB

Validating system setup.

Node registry is valid.

Log: /data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_***test/logs/appl/rgf/TXK/verifyssh.log

Output: /data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_***test/logs/appl/rgf/TXK/out.xml

ssh is not enabled for the following nodes

- - - - - - - - - - - - - - - - - - - - - -

erpbackup

[UNEXPECTED]ssh is not enabled.

[UNEXPECTED]Available options are as follows.

[UNEXPECTED]- To enable ssh, run the following command:

perl /data05/EBS_FS_12.2.5/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh \

-contextfile=/data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_***test/appl/admin/PHASE1_***test.xml \

-hosts=***test,erpbackup

Then try the adop command again.

[UNEXPECTED]- Otherwise, with ssh disabled you must run adop on each node:

On the admin node, run: adop phase=<desired phase> allnodes=no action=db

On the other nodes, run: adop phase=<desired phase> allndoes=no action=nodb

[UNEXPECTED]Error verifying ssh setup.

[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.

adop exiting with status = 2 (Fail)

Use the txkRunSSHSetup.pl command to enable SSH:

[applmgr@acetest ~]$ perl /data05/EBS_FS_12.2.5/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkRunSSHSetup.pl enablessh \

> -contextfile=/data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_***test/appl/admin/PHASE1_***test.xml \

> -hosts=***test,erpbackup

Enter SSH User password for the OS user applmgr:

Log: /data05/EBS_FS_12.2.5/fs1/inst/apps/PHASE1_***test/logs/appl/rgf/TXK/enablessh.log

SSH equivalence setup successful.

Sajid's Oracle DBA Blog