Resolve: kkjcre1p unable to spawn jobq slave process, slot 0, error 1089 in oracle database.

Root Cause: -

This situation occurs while we fire “shutdown immediate” in oracle database. Continuously error message are coming in alert log file.

kkjcre1p unable to spawn jobq slave process, slot 0, error 1089 in oracle database.

Error message: Linux-x86_64 Error: 23: Too many open files in system

Additional information: 1

Writing to the above trace file is disabled for now on..

Process J000 died, see its trace file

Process m001 died, see its trace file

Process m002 died, see its trace file

Process m001 died, see its trace file

Process m003 died, see its trace file

Most probably this will happened because of following reasons.

       👉     If you have fired “SHUTDOWN IMMEDIATE” Command and press cntrl+C to cancel the session due to taking long time to shut it down.

Suppose database unable to shutdown and taking long time to shutdown  then try as below.

Check the process id as ps –ef|grep oracle and kill the session LOCAL=YES only.

[oracle@xxxprod ~]$ ps -ef|grep oracle

54323     5771     1  0 10:31 ?        00:00:00 oracleSAVIOR (LOCAL=NO)
54323      721     1  0 09:30 ?         00:00:06 oracleSAVIOR (LOCAL=NO)
54323     2079     1  1 09:47 ?        00:00:38 oracleSAVIOR (LOCAL=NO)
oracle    4266     1  0 Sep24 ?        00:00:13 ora_pmon_cdb1
oracle    4270     1  0 Sep24 ?        00:00:12 ora_psp0_cdb1

oracle    4274     1  0 Sep24 ?        00:00:00 ora_vktm_cdb1

oratest 31935 31930  0 09:33    00:00:27oraclePROD (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

$kill -9 31935

$ps –ef|grep oracle -à make sure no oracle process, before start it again.

$sqlplus / as sysdba

SQL> startup;

OR

    👉    If you have memory issue and you are unable to start/stop the database.

Workaround:-

In case of memory issue, start it as force.

$sqlplus / as sysdba

SQL> startup force;

This will cause your database will be forcefully down and start it again. Shutdown abort will not work in this situation.

Other way.

If your are frequently getting kkjcre1p: unable to spawn jobq slave process and Linux-x86_64 Error: 23: Too many open files in system  and don't want to shutdown APPS Tier and DB Tier. Pls follow the below step to resolve this issue.

cd $ADMIN_SCRIPTS_HOME

[applmgr@***** scripts]$ sh adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2

Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: ACE_samarth.samarth.ace-cranes.com

---------------------------------+--------------------+---------+---------

ias-component                          | process-type       |     pid    | status

---------------------------------+--------------------+---------+---------

OC4JGroup:default_group          | OC4J:oafm          |    6632  | Alive

OC4JGroup:default_group          | OC4J:forms         |    6519  | Alive

OC4JGroup:default_group          | OC4J:oacore        |   34150 | Alive

OC4JGroup:default_group          | OC4J:oacore        |   33584 | Alive

OC4JGroup:default_group          | OC4J:oacore        |   33443 | Alive

HTTP_Server                             | HTTP_Server       |    6155  | Alive

adapcctl.sh: exiting with status 0

Kill the oacore pid. Automatically start , if not doing same. Need to be start it manually.

$ kill -9 34150 

$ kill -9 33584 

$ sh adapcctl.sh status

Here Status should be Alive.