ORA-03113 on Unix - What Information to Collect [ID 17613.1]

An ORA-3113 "end of file on communication channel" error is a general error usually reported by a client process connected to an Oracle database. The error basically means 'I cannot communicate with the Oracle shadow process'. As it is such a general error more information must be collected to help determine what has happened - this error by itself does not indicate the cause of the problem. For example, ORA-3113 could be signalled for any of these scenarios:

    Server machine crashed
    Your server process was killed at O/S level
    Network problems
    Oracle internal errors / aborts on the server
    Client incorrectly handling multiple connections
    etc.. etc.. etc.. - a lot of possible causes !!

This short article describes what information to collect for an ORA-3113 error. It is common for this error to be accompanied by other errors such as:

    ORA-1041 internal error. hostdef extension doesn't exist
    ORA-3114 not connected to ORACLE
    ORA-1012 not logged on

These are all symptomatic of being disconnected from Oracle.

Please collect as much information as possible from the items listed below and submit this information to Oracle support. Where a step produces an output file / trace file this may be needed by Oracle Support to help determine the cause of the error. The more information you can present in one go the better your chance of a speedy solution. Note that some sections may not be applicable.

If viewing this article on a Web Browser you can follow the links, otherwise manually go to the relevant section.

 
What Scenario does the ORA-3113 occur in ?

  A. When attempting to startup Oracle ?                    -> Section A
  B. When attempting to make a connection to the database ? -> Section B
  C. Client gets the error running SQL / PLSQL ?            -> Section C
  D. Server trace file reports ORA-3113 ?                   -> Section D

You may find it useful to scan the checklist in Section E at the end of this article. This covers some questions / issues relevant to all problem sections.

NOTE: References to SVRMGR are outdated with Oracle9i and Oracle 10g. If you are on an earlier version of the database, you will use 'svrmgrl' instead of SQL*Plus to connect as the internal user.

e.g. svrmgrl
SVRMGR> connect / as sysdba (or connect internal)
(A) ORA-3113 when attempting to STARTUP Oracle

  
  There are several phases involved in starting up a database. If ORA-3113
  occurs during startup then abort the instance and start up using the
  sequence below. If an error occurs at any step then see the related notes
  below.

    a. Start any required services.        On error see A1
       Eg: On NT start the OracleServiceSID

    b. Connect as a SYSDBA user.          On error see A1
       Eg:  sqlplus /nolog
        SQL> connect / as sysdba

    c. Startup nomount.              On error see A1
       Eg:
        SQL> startup nomount

    d. Mount the database.             On error see A1 and A2
       Eg:
        SQL> alter database mount;

    e. Recover the database         On error see A3
       Eg:
        SQL> recover database

    f. Open the database             On error see A4
       Eg:    
        SQL> alter database open;

    g. Wait 3 minutes then issue a select.   On error see A4
       Eg:    
        SQL> select count(*) from DBA_OBJECTS;



A1) Errors connecting as SYSDBA / Internal OR on startup nomount
   
    There is something fundamental wrong with the software / environment
    if you cannot connect to Server Manager as a DBA user.
    The steps here cover errors such as ORA-3113, ORA-12547: TNS:lost contact
    or similar errors connecting to Oracle or starting the instance NOMOUNT. 
    Check the following items:

    A1.1)    If possible reboot the server disabling any automatic
        startup of Oracle before you do so. This may seem drastic
        but helps make sure you are working from a consistent
        starting point.

    A1.2)     Check your environment points at the expected ORACLE_HOME
        and ORACLE_SID and that TWO_TASK is not set (Unix) or
        LOCAL is not set (NT registry).
                Check the USER_DUMP_DEST and BACKGROUND_DUMP_DEST and default
                trace directories under this environment for any user trace
                files or alert log entries generated. These may help indicate
                the cause of the problem.
                Eg: ORA-600[SKGMINVALID] may indicate a problem with the
                    shared memory Unix parameters on Unix systems.
                Try to show that any trace file / alert log entry you
                find is truely related to the "CONNECT" command by re-issuing
                the "connect" and checking for a new trace file / alert entry
                at the time of the error.

        A1.3)   Unix only:
        Some Unix platforms need LD_LIBRARY_PATH to be set
                correctly to resolve any dynamically linked libraries.
                As the user with the problem:

                        % script /tmp/ldd.out
                        % id
                        % cd $ORACLE_HOME/bin
                        % ldd oracle
                        % exit

                If the 'ldd' command does not exist go to the next step below.
                Check that all lines listed show a full library file. If there
                are any 'not found' lines reported contact Oracle support
                with the output of /tmp/ldd.out .


        A1.4)   Unix only:
        Your 'oracle' executable may be corrupt. Relink it thus:

                        Log in as the 'oracle' user.
                        % script /tmp/relink.out
                        % cd $ORACLE_HOME/rdbms/lib
            % mv $ORACLE_HOME/bin/oracle $ORACLE_HOME/bin/oracle.dd.mon.yy
                        % rm -f ./oracle       
                        % make -f ins_rdbms.mk ioracle
                        % exit

        Prior to Oracle7.3 the relink command was:
            make -f oracle.mk ioracle

                If this reports any errors Oracle support will need to see
                the contents of the file /tmp/relink.out .

   
    A1.5)   Have you installed the Parallel Server Option ?
                ORA-3113 can occur if you have installed the Parallel
        Server Option but do NOT have a Distributed Lock Manager
        installed or running correctly.

                Unix:
          If the Parallel Server Option was installed by accident
          then it can be de-installed by relinking.
          Eg:
                        Shut down any Oracle instances
                        % script /tmp/relink.out
                        % cd $ORACLE_HOME/rdbms/lib
                        # 'oracle' should not exist so delete it if it present
                        % rm -f oracle
                        % make -f ins_rdbms.mk no_parropt ioracle
                        % exit

          NB: Do NOT deinstall the Parallel Server Option if the
              database is using Parallel Server unless both nodes
              are shut down otherwise database corruption could occur.

    A1.6)   If the error is on STARTUP NOMOUNT:

            Check the init.ora file used to start the database.
            This provides the configuration details used
            configure the instance. To help isolate the problem
            it may be useful use a very basic init.ora file
            when starting the instance. If this works then
            parameters can be increased / introduced one at a
            time to see if there is a problem with a particular
            setting.

    A1.7)     Check for server side trace files which may give more
        indication what the underlying problem is.
        See section C for details on how to check
        for server trace files.
       

    A1.8)     Ensure there is free disk space in:

          a. Your USER_DUMP_DEST and BACKGROUND_DUMP_DEST locations

          b. Your AUDIT destination (Unix)
            The default is $ORACLE_HOME/rdbms/audit

          c. Your Oracle Trace directory if Oracle Trace is enabled
            See Note:45482.1

        A1.9) Windows 2000 - Only
              If the Server's sqlnet.ora file contains Authnetication services
              which are NOT reachable by Oracle, then an ORA-3113 error will
              result.
              For example, if the sqlnet.ora file contains the parameter:
              SQLNET.AUTHENTICATION_SERVICES = (NTS) and the Oracle database
              is moved from a Windows NT Domain to an Active Directory one,
              or if a Domain Controller is introduced, then an error will
              result trying to start the database.
              Remove the sqlnet.authentication_services line so that Oracle
              does not look for a non-existent KDC (Kerberos Domain Controller).


           
A2) Errors Mounting the database

    Check all the items in A1 first.

    If an error occurs when mounting the database there may be problems
    with the control-files or data files, or with resources required to
    open these files.

    A2.1)     The location of the control files are specified in the
        init.ora file.  Try mounting using each controlfile in
        turn.
        eg: "Shutdown abort",
            edit the init.ora to refer to ONE of the controlfiles only,
            "startup nomount",
            "alter database mount"
        Repeat for each controlfile to see if any controlfile works.
       
    A2.2)    It is possible to re-create the controlfiles if you know the
        location of all datafiles and online logs, or to restore an old
        backup controlfile. Always back up the current controlfiles before
        restoring any backup copies or issuing a CREATE CONTROLFILE
        command.
        The steps for this are not documented here.

        A2.3)   Unix:
                Some unix platforms have a 'truss' command (or 'tusc').
        If available this can be used to help trace how far Oracle
        gets before the error occurs.
                Eg: 
                        % truss -o /tmp/truss.out -f svrmgrl

                Keep the file /tmp/truss.out safe - Oracle Support MAY need to see it.


A3) Errors on RECOVER DATABASE

      ORA-3113 during recover database is often related to a corruption on the
    database or redo stream which causes the shadow process to die. There should
    be a server side trace file produced for this sort of problem.
    See Section C for details on how to locate any trace files
    from both USER_DUMP_DEST and BACKGROUND_DUMP_DEST.

    A3.1)    If the "recover database" fails fairly quickly then it
        may help to collect the redo up to the point of failure as this
        may help identify where the problem is.

        Use the following commands just prior to the RECOVER DATABASE
        command:

          SQL> alter session set max_dump_file_size=unlimited;
          SQL> alter session set events
          2> '10228 trace name context forever, level 10';
          SQL> RECOVER DATABASE
       
        This causes redo information to be written to the user trace
        file. The last items of redo may help determine which file
        has problems.

    A3.2)    If you do not have many datafiles in the database it may be
        as quick to recover each file in turn to see if this narrows
        down the problem.
        Eg:
          SQL> select name from v$datafile;
       
        and then for each file:

          SQL> RECOVER DATAFILE 'full_file_name'

        If this gets to a problem file then back up the file and
        use standard recovery options as if the file was lost.
       

A4) Errors on ALTER DATABASE OPEN

    Database open performs very many operations and so it is necessary
    to collect any trace information before determining the next steps.
    However, the following may help isolate the problem more quickly:

    A4.1)     Move files out of your USER_DUMP_DEST and BACKGROUND_DUMP_DEST
        directory as these steps will generate a lot of trace.

    A4.2)     Edit the init.ora file and add the lines:

            event="10046 trace name context forever, level 12"
            event="10015 trace name context forever, level 1"
            event="10228 trace name context forever, level 1"

        If you already have "EVENT=" lines in the init.ora file
        this MUST directly below the other "Event=" lines.
        These lines will trace:

            SQL and BIND activity during startup
            REDO applied
            Information about transactional rollback required

    A4.3)     Startup the instance as described at the top of this section.
       
        As soon as the error occurs REMOVE the above events from the init.ora
        file and shutdown. Collect together the trace files and alert logs
        as described in Section C

       


(B) ORA-3113 attempting to connect to the database

  
   Which SQL*Net layer are you using ?

    a. SQL*Net2 or Net8  -> Goto B1
       This is the default for Oracle 7.0.15 onwards

    b. SQL*Net 1         -> Goto B2
       Note: This can only be used with Oracle releases before 7.3


B1) Connecting to Oracle using SQL*Net V2 / Net8

   Net8 or SQL*Net2 should report network related errors if a problem
   occurs whilst establishing a connection to a remote Oracle shadow
   process. An ORA-3113 implies that the connection itself has been established
   but then is lost. As such follow the steps in Section C

B2) Connecting to Oracle using SQL*Net V1

   This section is included for historical reasons.


   Is the tool you are trying to connect from

    a. A local connection ?    -> Goto BL
       ie: The client is on the same machine as the database and you are
        not trying to connect over a network connection.

    b. A remote connection ? -> Goto BR


   BL) Local SQL*Net V1 Connections

      For local connections check the following:

        BL1)    Have you installed the Parallel Server Option ?
                ORA-3113 will occur if you have installed the Parallel
                Server Option but do NOT have a Distributed Lock Manager
                installed or running.

                To deinstall the Parallel Server Option:

                        Shut down any Oracle instances
                        % script /tmp/relink.out
                        % cd $ORACLE_HOME/rdbms/lib
                        # 'oracle' should not exist so delete it if it present
                        % rm -f oracle
                        % make -f oracle.mk no_parropt ioracle
                        % exit

                If this reports any errors Oracle support will need to see
                the contents of the file /tmp/relink.out .


        BL2)    Try using the SQL*Net V1 driver for local connections:

                        setenv TWO_TASK P:

                Then try the client tool. If this now works you may have a
                problem with the default SQL*Net driver.


        BL3)    Your 'oracle' executable may be corrupt. Relink it thus:

                        Log in as the 'oracle' user.
                        % script /tmp/relink.out
                        % cd $ORACLE_HOME/rdbms/lib
                        % rm -f oracle       
                        % make -f oracle.mk ioracle
                        % exit

                If this reports any errors Oracle support will need to see
                the contents of the file /tmp/relink.out .


        BL4)    Some Unix platforms need LD_LIBRARY_PATH to be set
                correctly to resolve any dynamically linked libraries.
                As the user with the problem:

                        % script /tmp/ldd.out
                        % id
                        % cd $ORACLE_HOME/bin
                        % ldd oracle
                        % exit

                If the 'ldd' command does not exist go to the next step below.
                Check that all lines listed show a full library file. If there
                are any 'not found' lines reported contact Oracle support
                with the output of /tmp/ldd.out .


        BL5)    If you cannot connect as the Oracle user AND your system has
                the 'truss' command try the following when logged in as
                'oracle' (using the relevant client tool):
               
                        % truss -o /tmp/truss.out -f sqlplus user/password
                        Exit from sqlplus (or the problem tool)

                Keep the file /tmp/truss.out safe - Oracle MAY need to see it.


   BR) Remote SQL*Net v1 Connections

      For remote connections check the following:

        BR1)    Check if you can make LOCAL connections. If not then follow
                the steps above for LOCAL connection problems.


        BR2)    Which SQL*Net protocol are you using ?

                Table B.1
                ~~~~~~~~~
                SYMBOL     SQL*Net V1 Layer  Prefix  Debug       Listener
                ------     ----------------  ------  -----       --------
                osnptt      PIPE Two Task     P:     OSNTTD      none
                osnasy      ASYNC             A:     OSNADBUG    none
                osnttt      TCP/IP Two Task   T:     OSNTDBUG    orasrv
                osntlitcp   TCP/TLI Two Task  TT:    OSNTLIDBUG  tcptlisrv
                osntlispx   SPX Two Task      X:     OSNTLIDBUG  spxsrv
                osndnt      DECNET Two Task   D:     OSNDDBUG    none


                For the protocol you are using check that the 'oracle'
                executable has this linked in thus:

                        Log in as oracle on the server
                        % script /tmp/drivers.out
                        % cd $ORACLE_HOME/bin
                        % drivers oracle
                        % exit

                        Eg: If you are using TCP/IP it should list TCP/IP.

                If the 'drivers' command does not exist on your machine check
                the 'oracle' executable as below substituting the relevant
                symbol from Table C.1 for the word 'SYMBOL'. If you get *NO*
                output the driver is probably *NOT* installed.

                        % script /tmp/symbols.out
                        % cd $ORACLE_HOME/bin
                        % nm oracle | grep -i SYMBOL    # Use relevant SYMBOL
                        % exit

                        Eg: For SQL*Net TCP/IP you would use the command:

                                % nm oracle | grep -i osnttt

                If the required driver is not installed you should:

                        a) Relink Oracle (See step (B3) above).
                        b) Re-check the 'oracle' executable for the
                           relevant driver. If it is still missing then
                           the relevant SQL*Net driver has probably not
                           been installed. Reinstall the required SQL*Net
                           driver.


        BR3)    Check your /etc/oratab or /var/opt/oracle/oratab file
                is of the form:

                        # Comments begin with a HASH
                        SID:/path/to/oracle/home:N

                And confirm:
               
                        [ ] There are no blank lines.
                        [ ] The PATH to ORACLE_HOME is correct and contains
                             no environment variables.
                        [ ] There are no ':'s in the ORACLE_HOME path.
                        [ ] There is NOTHING at the end of the line.
                            The last character on a line should be Y or N.
                            There should NOT be a fourth field.


        BR4)    Unix only:
        If you have 'truss' available try to truss the Oracle
                connection. You will normally need 'root' privilege to do this
                and should use truss on the relevant listener process (see
                Table B1)
                Eg: For TCP/IP the listener is 'orasrv' so enter these commands
                    as 'root':

                        % truss -o /tmp/truss.out -f -eall -p <PID_of_orasrv>
               
                        Attempt the connection to reproduce the ORA-3113 then
                        interrupt this 'truss' session.

                This will produce a LOT of output so keep it but do not send it
                to Oracle initially - just make a note that it is available.


(C) Client sees ORA-3113 running SQL / PLSQL

  
   If the ORA-3113 error occurs AFTER you have connected to Oracle then
   it is most likely that the 'oracle' executable has terminated unexpectedly.

        C1)     Determine which database you were connected to and
                obtain the following init.ora parameter values:
       
                        Parameter               Default
                        ~~~~~~~~~               ~~~~~~~
                        USER_DUMP_DEST          $ORACLE_HOME/rdbms/log
                        BACKGROUND_DUMP_DEST    $ORACLE_HOME/rdbms/log
                        CORE_DUMP_DEST          $ORACLE_HOME/dbs

                Eg: To find these log into Server Manager and:

                        SQL> show parameter dump


        C2)     Check in your 'USER_DUMP_DEST' for any Oracle trace file.
                It is important to find the correct trace file.
        On Unix:
          Use the command 'ls -ltr' to list files in time order with the
                  latest trace files appearing LAST. The trace file will typically
          be of the form '<SID>_ora_<PID>.trc'.
        On NT:
          Click on the "Modified" column in Windows Explorer to sort the
          files by their modified date. Files will typically be of the form
          'ORA<PID>.TRC'.
       
        If you are not sure which trace file may be relevant MOVE all
        the current trace files to a different directory and reproduce
        the problem.

        C3)     Check in your 'BACKGROUND_DUMP_DEST' for your alert log and
        any other trace files produced close to the time of the error.
        On Unix this should be named 'alert_<SID>.log'.
        On NT this should be named '<SID>ALRT.LOG'

        C4)     Unix Only:
        If there is no trace file check for a 'core' dump in the
                CORE_DUMP_DEST. Check thus:

                        % cd $ORACLE_HOME/dbs   # Or your CORE_DUMP_DEST
                        % ls -l core*

                If there is a file called 'core' check its time matches the
                time of the problem. If there are directories called
                'core_<PID>' check for core files in each of these. It is
                IMPORTANT to get the correct core file. Now obtain a stack
                trace from this 'core' file. Check each of the sequences below
                for how to do this - one of these should work for your
                platform:


                    If you have dbx:
                        % script /tmp/core.stack
                        % dbx $ORACLE_HOME/bin/oracle core
                        (dbx) where
                        ...
                        (dbx) quit
                        % exit
                       
                    If you have sdb:
                        % script /tmp/core.stack
                        % sdb $ORACLE_HOME/bin/oracle core
                        * t
                        ...
                        * q
                        % exit
                       
                    If you have xdb:
                        % script /tmp/core.stack
                        % xdb $ORACLE_HOME/bin/oracle core
                        (xdb) t
                        ...
                        (xdb) q
                        % exit


                    If you have adb:
                        % script /tmp/core.stack
                        % adb $ORACLE_HOME/bin/oracle core
                        $c
                        ...
                        $q
                        % exit


        C5)     Try to isolate the SQL command that is executing when
                the error occurs. Eg: Is it a particular SQL statement
                or PL/SQL block that causes the error ?
        In many cases this will be listed in the trace file
        produced under the heading "Current SQL statement", or
         near the middle of the trace file under the cursor referred
        to by the "Current cursor NN" line.

        If the trace does not show the failing statement then
        then SQL_TRACE may be used to help determine this provided
          the problem reproduces. SQL_TRACE can be enabled in most
        client tools:

                Eg: Product     Action
                    ~~~~~~~     ~~~~~~
                    SQL*Plus    Issue 'ALTER SESSION SET SQL_TRACE TRUE;'
                    Pro*        EXEC SQL ALTER SESSION SET SQL_TRACE TRUE;

                This should force a server side SQL trace file as detailed
                in C2 above. The trace file should give a clue as to what
                SQL was being executed.


        C6)     If no trace file can be found and the problem is reproducible
        then SQL*Net trace may help to show what the latest operation sent
        to the 'oracle' process was.
                For SQL*Net V2 / V8 tracing see the article Note:16564.1
   
        Historical note:
                     For SQL*Net V1 check which SQL*Net protocol you are using
                    and note the 'Debug' environment variable from table B1 below.
                    Then catch SQL*Net trace from the client. Eg: For SQL*Net
                    TCP/IP and sqlplus:
   
                            % setenv OSNTDBUG -1    # Use correct OSN*DBUG variable
                            % sh
                            % sqlplus scott/tiger@t:hostname:sid 2>/tmp/net1.out
       

        C7)     Based on information collected above try to determine a small
                test case which will reproduce the problem. This is important
                for two reasons:

                        a) It gives Oracle support a small test case if the
                           problem does not look like a known problem.
               
                        b) It gives you a simple way to check if any patch
                           supplied will fix the problem.



    C8)    If a statement can be isolated which consistently raised an
        ORA-3113 error then it is worth spending some time collecting
        additional information such as:
           
            - An execution plan for the statement
            - Table definitions, column definitions
            - Information on constraints, triggers etc..

        ie: Any additional information about the statement which fails.
            eg: If a SELECT fails then it may succeed if run under a
            different optimizer mode.



    C9)     Check if your server administrator has any scripts which abort
        long running or CPU intensive processes. An ORA-3113 process
        can occur if someone kills your Oracle shadow process at O/S
        level (Eg: kill -9 on Unix).


(D) Server trace reports ORA-3113

    D1)     It is unusual for a server trace to report ORA-3113.
        However, this can occur if the server loses contact with
        the client OR a database link connection.
        Treat this the same as an ORA-3113 in a client process
        and follow the steps in Section C.

    D2)     The following line may be added to the init.ora file
        to help collect maximum information when the error occurs:

            event="3113 trace name errorstack level 3"

        If you already have "EVENT=" lines in the init.ora file
        this MUST directly below the other "Event=" lines.

(E) Additional Checks / Information

        E1)     Is it just this one tool that encounters the error or
                do you get ORA-3113 from any tool doing a similar operation ?
                If the problem reproduces in SQL*Plus use this in all tests
                below.

        E2)     Unix only:
        Check if the problem is just restricted to:

                        [ ] One particular UNIX user,
                        [ ] Any UNIX user
                    or  [ ] Any UNIX user EXCEPT as the Oracle user.


        E3)     Check if the problem is just restricted to:

                        [ ] One particular ORACLE logon
                    or  [ ] Any ORACLE logon that has access to the
                                relevant tables.

        E4)     If this is a client-server set up does this occur from:

                        [ ] Any client
                        [ ] Just one particular client
                    or  [ ] Just one group of clients ?
                            If so what do these clients have in common ?
                            Eg: Software release .
               
        E5)     Do you have a second server or database version where the
                same operation works correctly ?


    E6)     Ensure there is free disk space in:

          a. Your USER_DUMP_DEST and BACKGROUND_DUMP_DEST locations

          b. Your AUDIT destination (Unix)
            The default is $ORACLE_HOME/rdbms/audit

          c. Your Oracle Trace directory if Oracle Trace is enabled
            See Note:45482.1