StackOverflowError is usually an indication of an infrastructure error, as the error does not appear tied to any specific composite. Based on the oracle.soa.bpel.engine.dispatch logger, it is thrown by the BPEL dispatch engine, as shown in the following example:

<Aug 5, 2015 12:00:02 AM EDT> <Error> <oracle.soa.bpel.engine.dispatch> <BEA-000000> <failed to handle message
javax.ejb.EJBException: EJB Exception: : java.lang.StackOverflowError

For an error like this, when you have little or no information even after you throttle all your loggers to full, it is wise to search My Oracle Support (https://support.oracle.com) or engage Oracle Support.

Here, the Mediator Service Engine (as shown by the oracle.soa.mediator.serviceEngine logger) appears to have thrown SOAPFaultException, which in itself is not very useful. However, the details of the error are quite clear. This is a business exception thrown by an external service or resource. Though there is no information here to tie it back to a specific composite or instance, the same error, in more detail, can be found in the wls_soa1.log file, which will provide the specific information. Rest assured, this exception will also appear in the composite instance fault on the console. Refer to the following log entry:

<Aug 6, 2015 10:10:33 AM EDT> <Error> <oracle.soa.mediator.serviceEngine> <BEA-000000> <Got an exception: oracle.fabric.common.FabricInvocationException: javax.xml.ws.soap.SOAPFaultException: CreateOrder failed with Message: Cannot insert the value NULL into column 'OrderID', table '@Orders'; column does not allow nulls. INSERT fails.

The error is obvious; a table column does not allow NULL and therefore insertion is rejected by the database.

Some infrastructure exceptions are related to the backend components that are unavailable or configured improperly. Some of them may be categorized by a warning instead of an error, but this is still a problem. Refer to the error in the following log entry that appears to be related to a WebLogic JMS Server. In this case, the JMS Server named CustomJMSServer was referencing the persistent file store CustomFileStore that did not exist:

<Aug 25, 2015 7:30:41 PM EST> <Warning> <Management> <BEA-141197> <The deployment of CustomJMSServer failed.weblogic.management.DeploymentException: Internalrror activating the JMS Server CustomJMSServer: weblogic.management.DeploymentException: The persistent store "CustomFileStore" does not exist

Such exceptions can be resolved by making sure that the required backbone infrastructure is present. This specific issue is resolved by creating the required file store and restarting the targeted managed servers.

FabricInvocationException is thrown by the Mediator Service Engine. The Unable to access the following endpoint error is somewhat misleading, as it could be related to any number of exceptions:

<Aug 29, 2015 11:45:06 AM EST> <Error> <oracle.soa.mediator.serviceEngine> <BEA-000000> <Got an exception: oracle.fabric.common.FabricInvocationException: Unable to access the following endpoint(s): http://payment-processing-server-dev:7777/proc/servlet/createCustomer

By reviewing the nested exception later on (not shown here), you will find that this particular error was due to a timeout of the external service. Though technically this is not a coding error since it is something that occurred only at runtime, it is not a SOA infrastructure error either, as it is related to an external service on which we have no control. From a development standpoint, the developer should be able to handle these cases by enabling some type of retry and/or error logging. You can also enable more detailed logging for these issues by enabling debugging while starting the server as follows:

-Djavax.net.debug=all

This Unable to allocate additional threads error appears to be quite serious, but it really is not. However, this is an infrastructure-related issue and should be remedied.

Let's refer to the following log snippet:

<Sep 30, 2015 11:30:04 PM EDT> <Warning> <oracle.integration.platform.instance.store.async> <BEA-000000> <Unable to allocate additional threads, as all the threads [10] are in use. Threads distribution : Fabric Instance Activity = 1,Fabric-Instance-Manager = 9,>

This error is apparently due to a failed transaction that continues to retry. The Oracle WebLogic Server is unable to process more messages in parallel and has run out of threads that would eventually clear themselves. It is possible to increase the number of threads if this becomes a persistent problem.

The following exception is due to the fact that at runtime, the XSD was not available:

java.io.IOException: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/apps/Fault/Common/XSD/SalesOrderHeader.xsd does not exist.

This actually could either be a coding error or an infrastructure error. Therefore, further investigation is required. If the soa-mds data source is unavailable, this error would be attributed to an infrastructure problem. If the data source is available though, then it is likely due to the code referring to a resource that does not exist in the MDS.

The following exception is returned in the wls_osb1.log file. From the looks of it, a particular OSB business service is responsible for this error:

####<Aug 16, 2015 10:57:11 AM EDT> <Info> <OSB Kernel> <soahost1> <wls_osb1> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <c3674305f67d3622:-7bff816b:136a9bbdd80:-8000-0000000000008a10> <1334588231098> <BEA-398102> <
 [OSB Tracing] Exiting route node with fault: 
<con:fault xmlns:con="http://www.bea.com/wli/sb/context">
  <con:errorCode>BEA-380000</con:errorCode>
  <con:reason>Request Entity Too Large</con:reason>
  <con:location>
    <con:node>CustomerBusSvc</con:node>
    <con:path>response-pipeline</con:path>
  </con:location>
</con:fault>>

Since this error appears to be specific to a single OSB service, a few online searches reveal the root cause of the Request Entity Too Large error, which is that authentication cannot be handled automatically in chunked mode, and this particular service was using authentication. A simple request to the developer to disable the Use Chunked Streaming Mode setting can remedy this, or the administrator can make this change dynamically at runtime.

By now, you must know that the engine that is responsible for executing composite instances in your infrastructure is soa-infra, which in itself is a J2EE application. The soa-infra application, in some cases, may fail to load in all the managed servers across a cluster and is partially available due to the following exception:

Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster

The preceding issue is due to the fact that the coherence multicast channel in use by the domain in a clustered mode is overloaded. The coherence cluster address for the domains is set to a unique multicast address and port in the setDomainEnv.sh file. To confirm whether the multicast channel in use is not overloaded, use a simple multicast test by following the steps in the code:

source setAntEnv.sh
export CLASSPATH=${ORACLE_HOME}/wlserver/server/lib/weblogic.jar
java utils.MulticastTest -N [managedServerName] -A [multicastHost] -P [multicastPort] -T 10 –S

The issue can be resolved by untargeting the soa-infra application from all servers in the cluster, bouncing back all managed servers, and retargeting it back to the cluster. This is an infrastructure problem.

Not all errors are directly due to issues within your SOA Infrastructure. All running and completed instances in Oracle SOA Suite 12c are saved in a backend data store. Lack of space in the database may prevent the composite instances to complete their processing. For instance, you are most likely to see the following exception if your database runs out of disk space:

oracle.toplink.exceptions.DatabaseException
Internal Exception: java.sql.BatchUpdateException: ORA-01691: unable to extend lob segment DEV_SOAINFRA.SYS_LOB$$ by 128 in tablespace DEV_SOAINFRA

These messages typically indicate space issues in the database that will likely require you to add more data files or more space to the existing data files. Additionally, completed instances can be purged and disk space can be reclaimed. This is discussed in Chapter 8, Managing the Database. Another useful guide to dealing with databases can be found in Managing Database Growth of the Oracle® Fusion Middleware Administering Oracle SOA Suite and Oracle Business Process Management Suite, which can be found at https://docs.oracle.com/middleware/1213/soasuite/administer/soa-database-growth-manage.htm#SOAAG97257.

We have covered several types of exceptions that can be encountered to give you an idea of the types of errors that you might expect. It is impossible to provide a comprehensive list of errors, and our intent is to show you why we have classified some errors as infrastructure errors and others as instance errors. Most of the time, the type of error is obvious, but other times, it is not. Differentiating between both helps you narrow your troubleshooting efforts to a particular area.