Problem Initializing Cache?

Hi,





We use the JChem Searcher in combination with MarvinSketch for retrieving results form a database. When the database was smaller (about 800.000 rows) everything worked fine. We recognized that the table caching process after a server reboot took some seconds, but it was still tolerable.





Right now we have more than 6.000.000 entities in our database and we were not able to accomplish a db search. I think it is because of the caching process. Is it possible?








Is it possible to force the searcher to cache the tables before the first search request?








Settings: JChem and Marvin 5.2.0

Hi,





could you post me the the correct failure report, with stack trace if it is available?


Which Java version and which database server do yo use?





Regards, Edvard

We use a MySQL DB and Java 1.6.0_11.

Code:

15:32:11,599 ERROR [STDERR] java.lang.OutOfMemoryError: Java heap space 15:32:11,599 ERROR [STDERR]     at chemaxon.jchem.db.StructureCache.ensureCapacity(StructureCache.java:816) 15:32:11,599 ERROR [STDERR]     at chemaxon.jchem.db.StructureCache.addStructure(StructureCache.java:600) 15:32:11,599 ERROR [STDERR]     at chemaxon.jchem.db.StructureCache.loadMySQL(StructureCache.java:765) 15:32:11,599 ERROR [STDERR]     at chemaxon.jchem.db.StructureCache.load(StructureCache.java:424) 15:32:11,599 ERROR [STDERR]     at chemaxon.jchem.db.StructureCache.load(StructureCache.java:226) 15:32:11,599 ERROR [STDERR]     at chemaxon.jchem.db.StructureCache.loadIfNeeded(StructureCache.java:847) 15:32:11,600 ERROR [STDERR]     at chemaxon.jchem.db.JChemSearch.loadCacheIfNeeded(JChemSearch.java:2763) 15:32:11,600 ERROR [STDERR]     at chemaxon.jchem.db.JChemSearch.search1(JChemSearch.java:2502) 15:32:11,600 ERROR [STDERR]     at chemaxon.jchem.db.JChemSearch.search(JChemSearch.java:2340) 15:32:11,600 ERROR [STDERR]     at chemaxon.jchem.db.JChemSearch.setRunning(JChemSearch.java:2172) 15:32:11,600 ERROR [STDERR]     at chemaxon.jchem.db.JChemSearch.run(JChemSearch.java:2199) 15:32:11,600 ERROR [STDERR]     at de.l3s.vifachem.portal.dbsearch.structure.JChemSearcher.dbSearch(JChemSearcher.java:219) 15:32:11,600 ERROR [STDERR]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 15:32:11,600 ERROR [STDERR]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 15:32:11,600 ERROR [STDERR]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 15:32:11,600 ERROR [STDERR]     at java.lang.reflect.Method.invoke(Method.java:597) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.util.Reflections.invoke(Reflections.java:22) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:31) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:77) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke(SynchronizationInterceptor.java:32) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:185) 15:32:11,600 ERROR [STDERR]     at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:103) 15:32:11,600 ERROR [STDERR]     at de.l3s.vifachem.portal.dbsearch.structure.JChemSearcher_$$_javassist_2.dbSearch(JChemSearcher_$$_javassist_2.java) 15:32:11,600 ERROR [STDERR]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 15:32:11,611 ERROR [STDERR] 31.03.2009 15:32:11 chemaxon.jchem.db.JChemSearch loadCacheIfNeeded SCHWERWIEGEND: Java heap space: java.lang.OutOfMemoryError: Java heap space     at chemaxon.jchem.db.StructureCache.ensureCapacity(StructureCache.java:816)     at chemaxon.jchem.db.StructureCache.addStructure(StructureCache.java:600)     at chemaxon.jchem.db.StructureCache.loadMySQL(StructureCache.java:765)     at chemaxon.jchem.db.StructureCache.load(StructureCache.java:424)     at chemaxon.jchem.db.StructureCache.load(StructureCache.java:226)     at chemaxon.jchem.db.StructureCache.loadIfNeeded(StructureCache.java:847)     at chemaxon.jchem.db.JChemSearch.loadCacheIfNeeded(JChemSearch.java:2763)     at chemaxon.jchem.db.JChemSearch.search1(JChemSearch.java:2502)     at chemaxon.jchem.db.JChemSearch.search(JChemSearch.java:2340)     at chemaxon.jchem.db.JChemSearch.setRunning(JChemSearch.java:2172)     at chemaxon.jchem.db.JChemSearch.run(JChemSearch.java:2199)     at de.l3s.vifachem.portal.dbsearch.structure.JChemSearcher.dbSearch(JChemSearcher.java:219)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)     at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:31)     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)     at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)     at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:77)     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)     at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)     at org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke(SynchronizationInterceptor.java:32)     at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)     at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)     at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:185)     at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:103)     at de.l3s.vifachem.portal.dbsearch.structure.JChemSearcher_$$_javassist_2.dbSearch(JChemSearcher_$$_javassist_2.java)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 15:32:11,611 ERROR [STDERR] 31.03.2009 15:32:11 chemaxon.jchem.db.JChemSearch loadCacheIfNeeded

The Error message is clear. It is a Web application which I try to run from Eclipse. What can I do if the table does not fit into the cache?





By the way, I already saw the benchmarks regarding structure cache initialization time. It should take around 300 s for 6.000.000 structures. Is there a way to cache the table independent from the first search?

Please try to increase the usable memory size in the JVM with the -XmX parameter to the correct value, and try to search again in the huge database.


You can find some information about the memory increasing and memory need of JChem Search in the JChem FAQ: http://chemaxon.com/jchem/FAQ.html#outofmemory, http://chemaxon.com/jchem/FAQ.html#outofmemory








Regards, Edvard

Quote:

Is there a way to cache the table independent from the first search?

Currently there is not, though we pan to provide an API call in the future.








Until then you may run a simple search (e.g. with empty query) after server startup to load the cache.





Best regards,





Szilard

Hi again,

Now our DB has more than 23 Million Entities and needs around 26 GB space. I still cannot do a caching of this table.

Is there any trick to get it started? Or is it always necessary to cache all fingerprints and structures?

 

Hi,

Sorry for answering a bit late, but we are having our European User Group Meeting right now in Budapest.

 

The cache for 23 million typical drug-like structures should fit into approximately 2.3 GB of RAM (1 million structures = 100 MB).

So 26 GB RAM (?) you mention seems to be unproportional.

(In case you mentioned database file size please note that the size of the structure cache is not diretly related to the size of your database files, we only cache from structure tables ands only the relevant information in a compact form.)

An empirical formula is available here to roughly estimate the required memory need for the cache:

http://chemaxon.com/jchem/FAQ.html#cacheSize

Please note that this is only the extra memory by the cache, some processing memory is also needed for other parts of our code, your own code and the application server.

How much heap size is alliwed currently for the JVM ?

Do you still get the same (out of memory) error ?

 

Best regards,

 

Szilard

First, thanks for your reply.

It is not 26 GB Ram needed. This is the size of the db table. I just mentioned it to get an idea of the dimension.

I know that only the fingerprints and structures are cached. Right now I am working on a smaller subset. I will switch back to the huge tabel when the development phase is done. A lot of time is needed to do the caching. Since I have to restart the server quite often it is not practicable.

Then I will deploy the application on a server that will hopefully have enough memory free for the JVM. Otherwise I will let you know ;)

 

Regards,

Ben

A lot of time is needed to do the caching

Fortunately live systems are not restarted so often.

I agree it is a good idea to use a smaller data set for development.

Let us know if the cache loading is much slower than in this benchmark (<1 minute for 1 milion structures)

http://www.chemaxon.com/jchem/FAQ.html#cacheSize

 

Best regards,

Szilard