Upgrade to latest version, problems with table recalculation

User 93ffa33d02

27-02-2015 12:50:28

I am trying to upgrade to latest chemaxon version (15.2.16) and need to recalculate our molecule table. I am doing this from Windows workstation (32GB RAM) using JChem manager connected to oracle database running on linux. In molecule table there are about 35 million records. First of all process is very slow and no progress is displayed. After some 12 hours of standstill I get error log (submitted later). What causes the OutOfMemoryError and is there a solution for this problem? Also is there something I can do to speed up the recalculation process?

========================================================================================

Error at molecule with cd_id: 5756

Error at molecule with cd_id: 5762

Error at molecule with cd_id: 5757

Error at molecule with cd_id: 5760

java.lang.OutOfMemoryError: Java heap space

at chemaxon.struc.MoleculeGraph.createCHtab(MoleculeGraph.java:6056)

at chemaxon.calculations.Charge.initMolArrays(Charge.java:1691)

at chemaxon.calculations.Charge.initArrays(Charge.java:1679)

at chemaxon.calculations.Charge.initChargeCalc(Charge.java:338)

at chemaxon.calculations.Tautomerization.calcDonorAcceptorSites(Tautomer

ization.java:1666)

at chemaxon.calculations.Tautomerization.initCalc(Tautomerization.java:2

217)

at chemaxon.calculations.Tautomerization.createGenericTautomerMain(Tauto

merization.java:702)

at chemaxon.calculations.Tautomerization.createGenericTautomer(Tautomeri

zation.java:662)

at chemaxon.marvin.calculations.TautomerizationPlugin.run(Tautomerizatio

nPlugin.java:830)

at chemaxon.sss.search.tautomer.TautomerUtil$GenericTautomerizer.createG

enericTautomer(TautomerUtil.java:97)

at chemaxon.sss.search.tautomer.TautomerUtil$GenericTautomerizer.<init>(

TautomerUtil.java:72)

at chemaxon.sss.search.tautomer.TautomerUtil.createGenericTautomer(Tauto

merUtil.java:203)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.setTa

utHashValues(StructureRecordMaker.java:779)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.init(

StructureRecordMaker.java:537)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.initR

ecord(StructureRecordMaker.java:468)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.initS

tructureAndMolecule(StructureRecordMaker.java:344)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.updat

eRecord(StructureRecordMaker.java:387)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.updat

eRecord(StructureRecordMaker.java:381)

at chemaxon.jchem.db.UpdateHandler.execute(UpdateHandler.java:934)

at chemaxon.jchem.db.UpdateHandler.execute(UpdateHandler.java:903)

at chemaxon.jchem.db.Regenerator$RegenThread.process(Regenerator.java:17

2)

at chemaxon.jchem.db.Regenerator$RegenThread.run(Regenerator.java:131)

java.lang.OutOfMemoryError: Java heap space

at java.util.HashMap.values(HashMap.java:953)

at chemaxon.formats.recognizer.RecognitionSubsystem.getPossibleFormats(R

ecognitionSubsystem.java:333)

at chemaxon.formats.recognizer.RecognitionSubsystem.processLine(Recognit

ionSubsystem.java:199)

at chemaxon.formats.MolInputStream.processLine(MolInputStream.java:335)

java.lang.OutOfMemoryError: Java heap space

at java.lang.StringCoding.decode(StringCoding.java:187)

at java.lang.StringCoding.decode(StringCoding.java:254)

at java.lang.String.<init>(String.java:534)

at chemaxon.marvin.io.PositionedInputStream.readLine(PositionedInputStre

am.java:319)

at chemaxon.marvin.io.formats.AbstractMRecordReader.readLine(AbstractMRe

cordReader.java:139)

at chemaxon.marvin.io.formats.mdl.MolRecordReader.readLine(MolRecordRead

er.java:178)

at chemaxon.marvin.io.formats.mdl.MolRecordReader.readCtab(MolRecordRead

er.java:617)

at chemaxon.marvin.io.formats.mdl.MolRecordReader.readMol0(MolRecordRead

er.java:199)

at chemaxon.marvin.io.formats.mdl.MolRecordReader.nextRecord(MolRecordRe

ader.java:141)

at chemaxon.marvin.io.formats.mdl.MolRecordReader.nextRecord(MolRecordRe

ader.java:78)

at chemaxon.marvin.io.MRecordImporter.startReadingNext(MRecordImporter.j

ava:1028)

at chemaxon.marvin.io.MRecordImporter.readRecord(MRecordImporter.java:67

2)

at chemaxon.marvin.io.MRecordImporter.readMol0(MRecordImporter.java:587)

at chemaxon.marvin.io.MRecordImporter.readMol(MRecordImporter.java:511)

at chemaxon.formats.MolImporter.readMol(MolImporter.java:883)

at chemaxon.formats.MolImporter.read(MolImporter.java:766)

at chemaxon.formats.MolImporter.read(MolImporter.java:736)

at chemaxon.util.MolHandlerBase.importMol(MolHandlerBase.java:97)

at chemaxon.util.MolHandler.setMolecule(MolHandler.java:173)

at chemaxon.util.MolHandler.<init>(MolHandler.java:127)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.initR

ecord(StructureRecordMaker.java:462)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.initS

tructureAndMolecule(StructureRecordMaker.java:344)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.updat

eRecord(StructureRecordMaker.java:387)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.updat

eRecord(StructureRecordMaker.java:381)

at chemaxon.jchem.db.UpdateHandler.execute(UpdateHandler.java:934)

at chemaxon.jchem.db.UpdateHandler.execute(UpdateHandler.java:903)

at chemaxon.jchem.db.Regenerator$RegenThread.process(Regenerator.java:17

2)

at chemaxon.jchem.db.Regenerator$RegenThread.run(Regenerator.java:131)

java.lang.OutOfMemoryError: Java heap space

Error at molecule with cd_id: 5763

Error at molecule with cd_id: 5753

Error at molecule with cd_id: 5754

Error at molecule with cd_id: 5761

Error at molecule with cd_id: 5755

java.lang.OutOfMemoryError: Java heap space

at chemaxon.formats.recognizer.RecognitionSubsystem.initializeRecognizer

s(RecognitionSubsystem.java:165)

at chemaxon.formats.recognizer.RecognitionSubsystem.<init>(RecognitionSu

bsystem.java:156)

at chemaxon.formats.MolInputStream.initTextFormat(MolInputStream.java:22

1)

at chemaxon.formats.MolInputStream.init(MolInputStream.java:135)

at chemaxon.formats.MolInputStream.<init>(MolInputStream.java:115)

at chemaxon.formats.MolInputStream.<init>(MolInputStream.java:57)

at chemaxon.util.MolHandlerBase.importMol(MolHandlerBase.java:92)

at chemaxon.util.MolHandler.setMolecule(MolHandler.java:173)

at chemaxon.util.MolHandler.<init>(MolHandler.java:127)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.initR

ecord(StructureRecordMaker.java:462)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.initS

tructureAndMolecule(StructureRecordMaker.java:344)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.updat

eRecord(StructureRecordMaker.java:387)

at chemaxon.jchem.base.search.structurerecord.StructureRecordMaker.updat

eRecord(StructureRecordMaker.java:381)

at chemaxon.jchem.db.UpdateHandler.execute(UpdateHandler.java:934)

at chemaxon.jchem.db.UpdateHandler.execute(UpdateHandler.java:903)

at chemaxon.jchem.db.Regenerator$RegenThread.process(Regenerator.java:17

2)

at chemaxon.jchem.db.Regenerator$RegenThread.run(Regenerator.java:131)

java.lang.OutOfMemoryError: Java heap space

Feb 26, 2015 9:47:29 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.lang.OutOfMemoryError: Java heap space

Feb 26, 2015 9:50:36 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [7, 1]

Feb 26, 2015 9:50:31 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [5]

Feb 26, 2015 9:50:28 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [77]

Feb 26, 2015 9:50:24 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [28]

Feb 26, 2015 9:50:12 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [10]

Exception in thread "Thread-9" Exception in thread "Thread-4" Exception in threa

d "Thread-7" Exception in thread "Thread-5" Exception in thread "Thread-10" java

.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

Exception in thread "AWT-EventQueue-0" Feb 26, 2015 9:50:01 PM chemaxon.jchem.db

.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [196]

java.lang.OutOfMemoryError: Java heap space

Feb 26, 2015 9:49:31 PM chemaxon.jchem.db.Regenerator$RegenThread run

SEVERE: java.sql.SQLException: Protocol violation: [5]

ChemAxon a9ded07333

27-02-2015 15:26:21

Hi,

It is possible that you have a huge structure in your database that runs out of heap during process. You may try to increase the useable heap size of your JVM.

Unfortunately, increasing heap won't speed up the process - 35M structures are 35M structures ... this may take a whole day.

Best regards,
Tamás

User 93ffa33d02

27-02-2015 16:06:46

I have allready set HEAP_LIMIT=20000 in jcman.bat. Do I need to change something else?

ChemAxon a9ded07333

02-03-2015 10:56:05

Could you send us your modified jcman.bat file?
Please also attach the whole command you use to start JChem Manager.

Regards,

Tamás

User 93ffa33d02

02-03-2015 11:14:07

Attached please find archived jcman.bat

I don't run JChem manager from command line, I run JChemManager64.exe

ChemAxon a9ded07333

03-03-2015 14:29:24

Hi,

Probably there is a bug in our JChemManager64.exe program. Please try to run jcman.bat directly from command line.

Regards,
Tamás

User 93ffa33d02

05-03-2015 11:59:51

We could not resolve regeneration problem so we created a new table and moved data from old table to new table with UpdateHandler. Right now we have a molecule table with valid data, but... we have some more problems.

First of all, we are using JChem Base 15.2.16.0 in our web application. Application server is Tomcat 7. All ChemAxon JChem libs are added to web application (WEB-INF\lib) like it was done with JChem Base 6.2 version. When first chemical search is made, we get error: Invalid XPath expression: '//MarkushDescriptorConfiguration/Parameters'

I found the same error mentioned https://www.chemaxon.com/forum/ftopic12483.html but not sure what must be done to resolve the issue.

Error stack trace:

on.jchem.db.InterruptSQLHandler$1.call(InterruptSQLHandler.java:60)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
    at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
    at chemaxon.jchem.db.JChemSearch.search(JChemSearch.java:1443)
    at chemaxon.jchem.db.JChemSearch.searchLimitConcurrency(JChemSearch.java:1391)
    at chemaxon.jchem.db.JChemSearch.run(JChemSearch.java:1348)
    at webapp.services.portal.chemicalsearch.searchers.ChemicalSearcher.Search(ChemicalSearcher.java:360)
    at webapp.services.portal.chemicalsearch.actions.test.doGETAndPOST(test.java:43)
    at webapp.services.core.actions.generic.GenericAction.InvokeAction(GenericAction.java:385)
    at webapp.services.servlets.Dispatcher.ProcessRequest(Dispatcher.java:74)
    at webapp.services.core.servlets.GenericServlet.processGenericRequest(GenericServlet.java:176)
    at webapp.services.core.servlets.GenericServlet.doGet(GenericServlet.java:156)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:157)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:131)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:107)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:76)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:934)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:90)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1015)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:646)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1576)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1534)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.RuntimeException: Failed to create default XML configuration.
    at chemaxon.descriptors.MDParameters.initParameters(MDParameters.java:228)
    at chemaxon.descriptors.MarkushDescriptorParameters.<init>(MarkushDescriptorParameters.java:61)
    at chemaxon.jchem.db.MarkushTableInfo$DefaultConfigSingleTable.getMDTableInfoArray(MarkushTableInfo.java:253)
    at chemaxon.jchem.db.MarkushTableInfo.<clinit>(MarkushTableInfo.java:100)
    ... 50 more
Caused by: org.dom4j.InvalidXPathException: Invalid XPath expression: '//MarkushDescriptorConfiguration/Parameters'. Caused by: org.jaxen.dom4j.DocumentNavigator.getInstance()Lorg/jaxen/Navigator;
    at org.dom4j.xpath.DefaultXPath.parse(DefaultXPath.java:362)
    at org.dom4j.xpath.DefaultXPath.<init>(DefaultXPath.java:59)
    at org.dom4j.DocumentFactory.createXPath(DocumentFactory.java:230)
    at org.dom4j.tree.AbstractNode.createXPath(AbstractNode.java:207)
    at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:183)
    at chemaxon.descriptors.MarkushDescriptorParameters.processDocument(MarkushDescriptorParameters.java:131)
    at chemaxon.descriptors.MDParameters.initParameters(MDParameters.java:220)
    ... 53 more

ChemAxon efa1591b5a

10-03-2015 10:49:35

Apologies for not being able to respond to your enquiry, and thanks for your patience. We will deal with it soon.

Regards,

Miklos

ChemAxon a9ded07333

11-03-2015 10:28:13

Hi,

Is it possible that some of your old (JChem 6.2) jars are still present in your Tomcat folder?

Please try to delete ALL JChem related jars from Tomcat than copy 15.2.16 jars again.

Regards,
Tamás