User 325f2762fd
06-09-2008 08:16:50
Hi,
Oracle environment:
Oracle Database 10g Release 10.2.0.3.0 - Production
PL/SQL Release 10.2.0.3.0 - Production
CORE 10.2.0.3.0 Production
TNS for Linux: Version 10.2.0.3.0 - Production
NLSRTL Version 10.2.0.3.0 - Production
JChem Server environment:
Java VM vendor: Sun Microsystems Inc.
Java VM version: 10.0-b22
JChem version: 5.1.0
JDBC driver version: 10.2.0.3.0
We are migrating 1.2 million compounds. This is running as 10 dbms_jobs in oracle. We are getting many rmi server exeception in each of the log file of the jobs. I have removed the smiles from the log file snippets (below) due to confidentiality. Any idea? Could this be configuration setup or jobs are requesting data from jchem server too quick?
21:26:51:Opened
21:51:16:100 compounds migrated
21:53:56:100 compounds migrated
21:56:30:100 compounds migrated
21:59:03:100 compounds migrated
22:01:51:100 compounds migrated
22:04:17:100 compounds migrated
22:07:00:100 compounds migrated
22:09:39:100 compounds migrated
22:12:07:100 compounds migrated
22:14:27:100 compounds migrated
22:16:37:100 compounds migrated
22:18:56:100 compounds migrated
22:19:38:Unable to migrate the compound for the notebook : OD1434/060/91 , smiles = <smile_string>, Error = ORA-29532: Java call terminated by uncaught Java exception:
ORA-06512: at line 1
ORA-06512: at "CCDIII.MAINTAIN_CONSTANTS", line 83
ORA-06512: at "CCDIII.MAINTAIN_COMPOUNDS", line 3976
ORA-29532: Java call terminated by uncaught Java exception:
ORA-06512: at line 1
ORA-06512: at "CCDIII.MAINTAIN_CONSTANTS", line 83
ORA-06512: at "CCDIII.MAINTAIN_COMPOUNDS", line 1561
ORA-29532: Java call terminated by uncaught Java exception: java.rmi.ServerException: RemoteException occurred in server thread; nested ex
22:20:53:Unable to migrate the compound for the notebook : OD1434/093/39 , smiles = <smile_string>, Error = ORA-29532: Java call terminated by uncaught Java exception:
ORA-06512: at line 1
ORA-06512: at "CCDIII.MAINTAIN_CONSTANTS", line 83
ORA-06512: at "CCDIII.MAINTAIN_COMPOUNDS", line 3976
ORA-29532: Java call terminated by uncaught Java exception:
ORA-06512: at line 1
ORA-06512: at "CCDIII.MAINTAIN_CONSTANTS", line 83
ORA-06512: at "CCDIII.MAINTAIN_COMPOUNDS", line 1561
23:17:01:100 compounds migrated
23:19:30:100 compounds migrated
23:21:56:100 compounds migrated
Many Thanks
Rajeev
User 325f2762fd
06-09-2008 08:44:13
I am attaching the jchem server log file and also below are the latest log file entries
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:01:20 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:01:27 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:01:46 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:01:46 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:01:56 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:02:00 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:12:08 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:12:09 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:12:09 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:12:29 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:13:21 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:13:29 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:13:36 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:15:00 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:15:42 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:15:43 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:15:55 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:16:01 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
Sep 6, 2008 9:16:02 AM chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil handleError
SEVERE: Error in RMI call
java.lang.NullPointerException
ChemAxon aa7c50abf8
06-09-2008 10:00:30
The lack of Java stack traces is rather unexpected. (This is bad, because I don't see anything specific about the errors -- which I should.) When seeing unusual behavior like this, my first suspects are out of memory errors. How much memory has been allocated to JChemServer? What kind of operations are involved during migration? (It would be helpful, if the error messages generated by the cartridge operator calls on the client side could be seen.) What other activities are run concurrently on this cartridge instance?
Does the thee errors with stack trace at the beginning of the log file look like familiar? Do you know where they came from and whether their causes have been clarified?
Thanks
Peter
User 325f2762fd
06-09-2008 19:04:47
700M has been allocated to JChemServer.
We are inserting structures into a structure table which has two columns with domain index as below.
CREATE INDEX CCDIII.IXJC_STRUC_DIS_SMILES ON CCDIII.STRUCTURES
(DISPLAYED_SMILES)
INDEXTYPE IS JCHEM.JC_IDXTYPE
PARAMETERS('TABLESPACE=Indx_CCDIII,tableType=molecules')
NOPARALLEL;
CREATE INDEX CCDIII.IXJC_STRUC_SMILES ON CCDIII.STRUCTURES
(OVERLAP_SMILES)
INDEXTYPE IS JCHEM.JC_IDXTYPE
PARAMETERS('TABLESPACE=Indx_CCDIII,tableType=molecules')
NOPARALLEL;
We also split the compound structure into molecules and unique molecule is inserted into another table called molecule. Before inserting into molecules table we do a jchem search to check whether exact molecule already exists or not. If not then we insert it into the molecule table.
I would say we are inserting structure data concurrently into tables by 10 sessions and at the same we are also performing exact search using jchem operator. No other activities were performed in the database server and jchem server (both are on same server).
Could we change some setting to get the full error messages?
ChemAxon aa7c50abf8
06-09-2008 20:48:20
The following step alone may help getting more info:
In the server.sh file, insert after line#66 the following line:
| Code: |
| -XX:-OmitStackTraceInFastThrow \ |
(mind the backslash at the end). You need to restart JChemServer for this change to take effect.
If the above doesn't help, we have the option to increase logging verbosity:
1. Back-up the conf/log.properties file, so you can restore the original.
2. Change line#29 from
to
3. Put line#41 into comment (insert a # sign at the beginnig)
4. Insert the following line after line#41:
| Code: |
| java.util.logging.FileHandler.formatter = chemaxon.util.logging.SimpleFormatterTh |
For these changes in the log.properties file to take effect, you need either to restart JChemServer or call .
The second set of changes will result in a large log file, which you may want to send me by mail or ftp rather than posting it here. For the same reason, it would probably make sense to stop the migration process after the first couple of errors appear in order to first see how useful the increased log details are.
Thanks
Peter
ChemAxon aa7c50abf8
06-09-2008 20:57:49
It would make sense to first see if the problem is input-specific -- id est: to see if it can be consistently reproduced with the structures alone whose migration appears fail (eg: OD1434/060/91). If the problem is input-specific, using the problematic structures alone would make the test case much easier to manage.
Thanks
Peter
User 325f2762fd
08-09-2008 14:35:05
We will try your suggestion and let you know. I am unable to change the logging parameters now since migration is running since last 3 days and it has done 50 % migration. So I wait for the migration to finish and then try your suggestion.
User 325f2762fd
09-09-2008 22:32:38
I have noticed oracle dbms jobs have created trace files in the server and contain the following error message for the smiles that have failed to migrate. Could that be the full error message we are looking for?
java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:336)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:133)
at chemaxon.jchem.cartridge.rmi.impl.MiscellaniousImpl_Stub.standardize(Unknown Source)
at chemaxon.jchem.cartridge.JFunctions.standardize(JFunctions.java:1700)
Caused by: java.rmi.RemoteException
at chemaxon.jchem.cartridge.rmi.impl.RmiImplUtil.handleError(RmiImplUtil.java:60)
at chemaxon.jchem.cartridge.rmi.impl.MiscellaniousImpl.standardize(MiscellaniousImpl.java:65)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
ChemAxon aa7c50abf8
10-09-2008 07:45:03
Even though it stops short of indicating what the error is, this message tells us at least in which operation the error occurs: standardization.
Would it be possible to pick a few of the smiles that have failed to migrate and standardizing them with the Standardizer configuration you are using during migration? Or (if standardizing alone doesn't reproduce the problem) would it be possible to replay a mini migration using theses few structures as inputs?
(If the smiles are not highly confidential, you could send some of them to us along with the standardizer configuration by e-mail, so we can look at them for ourselves.)
Thanks
Peter
ChemAxon aa7c50abf8
10-09-2008 20:01:05
This is a bug in Standardizer which has been fixed in JChem 5.1.1:
| Quote: |
| Standardizer threw NullPointerException in some cases when "clearstereo" was performed. |
Thanks
Peter