User a7e5cba6fc
13-05-2007 19:00:36
hello,
My problem is that I must deal with a large Database (about 5 Million rows). the JVM(ver. 64-bit) is running on 64-bit machine (intel dual core) with 2.5 Gb physical memory and 5 Gb swap with the following arguments: -server -Xms 6000m -Xmx 6000m.
after the connection to the DB (with Tomcat ver. 6.0), I wait for the checking procedure for about 30 mn., the java proc take about 5 Gb of memory but I can't see nothing!! the checking reach the number 680
My question is, how could I resolve this problem?
Any help would be greatly appreciated.
Thanks
ChemAxon a3d59b832c
13-05-2007 20:27:36
Hi,
At first sight, "-Xms 6000m" seems too much. 5 million cached molecules will not likely to consume that much memory, and the "-Xms" option sets the initial memory consumption. I suggest to omit "-Xms 6000m" from the command line.
Let us know if it does not help.
Szabolcs
ChemAxon 9c0afc9aaf
14-05-2007 09:27:54
Hi,
5 million drug-like structures consume about 500 MB memory, so -Xmx800m should be enough including temporary allocations.
There is no need to specify -Xms, in that case you should see much less memory consumption.
Now back to the essence of the problem:
During the first search the table is loaded into the structure cache(displaying the contents of the table is also a "blank" search).
This can take some time depending on the speed of the database connection and other factors.
It seems in this case the cache loading time is excessively slow due to some unknown reason.
Please let us know
- the JChem version you are using
- the type and version of your RDBMS
- the type of storage engine / table type used if relevant
- the version of the JDBC driver
Since the cache loading time is proportional to the table size, you can also try creating a smaller table (e.g. 100,000 rows), and see how long it takes to load it.
You can check this information on the standard error output log (usually catalina.out) of Tomcat. You should see something like this:
| Code: |
Mon May 14 11:25:27 CEST 2007
Search mode: SUBSTRUCTURE
Structure table: structures
Query: <Empty query>
Screened: 102417
Hits: 102417
Cache loading: 4985 ms
Cache size (this table / total): 9.15 / 9.15 MBytes
Total time: 796 ms Screening: 16 ms
Processing threads: 2 |
Best regards,
Szilard
User a7e5cba6fc
18-05-2007 14:37:12
hier is ther required informations:
JCHEM: JChemManager 3.2.4, (C) 2000-2006 ChemAxon Ltd.
RDBMS: postgresql 8.1
JDBC: postgresql-8.1-408.jdbc3.jar
I didn't understand what you mean with "type of storage engine / table type used"!
hier is an example from the catalina.out:
| Quote: |
Fri May 18 16:17:24 CEST 2007
Search mode: SUBSTRUCTURE
Structure table: public.mdr
Query: <Empty query>
Screened: 149189
Hits: 149189
Cache loading: 7778 ms
Cache size (this table / total): 17.44 / 17.46 MBytes
Total time: 53 ms Screening: 2 ms
Processing threads: 2 |
But with 700000 structures a heap error is generated.
The Appache-tomcat start options:"apache-tomcat-6.0.10/bin/startup.sh -server -Xmx800M"
hier is the stack trace error:
| Code: |
org.postgresql.util.PSQLException: Ran out of memory retrieving query results.
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1298)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:191)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:452)
May 18, 2007 4:19:28 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: Java heap space
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:337)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:236)
at chemaxon.jchem.db.StructureCache.load(StructureCache.java:421)
at chemaxon.jchem.db.StructureCache.load(StructureCache.java:193)
at chemaxon.jchem.db.StructureCache.loadIfNeeded(StructureCache.java:769)
at chemaxon.jchem.db.JChemSearch.loadCacheIfNeeded(JChemSearch.java:2651)
at chemaxon.jchem.db.JChemSearch.search1(JChemSearch.java:2396)
at chemaxon.jchem.db.JChemSearch.search(JChemSearch.java:2240)
at chemaxon.jchem.db.JChemSearch.access$1200(JChemSearch.java:79)
at chemaxon.jchem.db.JChemSearch$SearchThread.run(JChemSearch.java:427)
Caused by: java.lang.OutOfMemoryError: Java heap space
usage: java org.apache.catalina.startup.Catalina [ -config {pathname} ] [ -nonaming ] { start | stop }
May 18, 2007 4:24:11 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-6-sun-1.6.0.00/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.00/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.00/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
May 18, 2007 4:24:11 PM org.apache.coyote.http11.Http11Protocol init
SEVERE: Error initializing endpoint
java.net.BindException: Address already in use:8080
at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:499)
at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:175)
at org.apache.catalina.connector.Connector.initialize(Connector.java:1059)
at org.apache.catalina.core.StandardService.initialize(StandardService.java:578)
at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:792)
at org.apache.catalina.startup.Catalina.load(Catalina.java:504)
at org.apache.catalina.startup.Catalina.start(Catalina.java:544)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
May 18, 2007 4:24:11 PM org.apache.catalina.startup.Catalina load
SEVERE: Catalina.start
LifecycleException: Protocol handler initialization failed: java.net.BindException: Address already in use:8080
at org.apache.catalina.connector.Connector.initialize(Connector.java:1061)
at org.apache.catalina.core.StandardService.initialize(StandardService.java:578)
at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:792)
at org.apache.catalina.startup.Catalina.load(Catalina.java:504)
at org.apache.catalina.startup.Catalina.start(Catalina.java:544)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
May 18, 2007 4:24:11 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 664 ms
May 18, 2007 4:24:11 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
May 18, 2007 4:24:11 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.10
May 18, 2007 4:24:13 PM org.apache.coyote.http11.Http11Protocol start
SEVERE: Error starting endpoint
java.net.BindException: Address already in use:8080
at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:499)
at org.apache.tomcat.util.net.JIoEndpoint.start(JIoEndpoint.java:513)
at org.apache.coyote.http11.Http11Protocol.start(Http11Protocol.java:202)
at org.apache.catalina.connector.Connector.start(Connector.java:1132)
at org.apache.catalina.core.StandardService.start(StandardService.java:457)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
May 18, 2007 4:24:13 PM org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1139)
at org.apache.catalina.core.StandardService.start(StandardService.java:457)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
May 18, 2007 4:24:13 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1789 ms
May 18, 2007 4:24:13 PM org.apache.catalina.core.StandardServer await
SEVERE: StandardServer.await: create[8005]:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
at java.net.ServerSocket.bind(ServerSocket.java:319)
at java.net.ServerSocket.<init>(ServerSocket.java:185)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:373)
at org.apache.catalina.startup.Catalina.await(Catalina.java:616)
at org.apache.catalina.startup.Catalina.start(Catalina.java:576)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
May 18, 2007 4:24:13 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
May 18, 2007 4:24:13 PM org.apache.catalina.connector.Connector pause
SEVERE: Protocol handler pause failed
java.lang.NullPointerException
at org.apache.jk.server.JkMain.pause(JkMain.java:679)
at org.apache.jk.server.JkCoyoteHandler.pause(JkCoyoteHandler.java:153)
at org.apache.catalina.connector.Connector.pause(Connector.java:1074)
at org.apache.catalina.core.StandardService.stop(StandardService.java:489)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:602)
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:645)
May 18, 2007 4:24:14 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
May 18, 2007 4:24:14 PM org.apache.catalina.connector.MapperListener destroy
WARNING: Error unregistering MBeanServerDelegate
java.lang.NullPointerException
at org.apache.catalina.connector.MapperListener.destroy(MapperListener.java:162)
at org.apache.catalina.connector.Connector.stop(Connector.java:1180)
at org.apache.catalina.core.StandardService.stop(StandardService.java:519)
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:744)
at org.apache.catalina.startup.Catalina.stop(Catalina.java:602)
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:645)
May 18, 2007 4:24:14 PM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
May 18, 2007 4:24:14 PM org.apache.catalina.connector.Connector stop
SEVERE: Coyote connector has not been started
May 18, 2007 4:25:01 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-8080-7" java.lang.OutOfMemoryError: Java heap space
Exception in thread "JMX server connection timeout 49" java.lang.OutOfMemoryError: Java heap space
chemaxon.jchem.db.DatabaseSearchException:
ERROR: JChem has run out of memory when trying to cache chemical hashed fingerprint information.
Please visit http://www.chemaxon.com/jchem/FAQ.html#outofmemory
at chemaxon.jchem.db.JChemSearch.loadCacheIfNeeded(JChemSearch.java:2683)
at chemaxon.jchem.db.JChemSearch.search1(JChemSearch.java:2396)
at chemaxon.jchem.db.JChemSearch.search(JChemSearch.java:2240)
at chemaxon.jchem.db.JChemSearch.access$1200(JChemSearch.java:79)
at chemaxon.jchem.db.JChemSearch$SearchThread.run(JChemSearch.java:427)
java.lang.OutOfMemoryError: Java heap space
|
User a7e5cba6fc
18-05-2007 15:03:51
hi,
when I change the tomcat start options to "-Xmx 1700m" the Jchem can load all the structures and I get them in the browser.
hier is the catalina output:
| Code: |
May 18, 2007 4:51:13 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
May 18, 2007 4:51:13 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
May 18, 2007 4:51:13 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/77 config=null
May 18, 2007 4:51:13 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3641 ms
Fri May 18 16:51:35 CEST 2007
Search mode: SUBSTRUCTURE
Structure table: public.zinc
Query: <Empty query>
Screened: 737468
Hits: 737468
Cache loading: 53178 ms
Cache size (this table / total): 76.69 / 76.69 MBytes
Total time: 931 ms Screening: 29 ms
Processing threads: 2
|
Is there any solution to load the 5Mio structures?
is should i change any configuration or it's a Driver's problem?
Nejd
User a7e5cba6fc
18-05-2007 15:16:18
hi,
when I try with the same tomcat option to load the 5mio structure, i get this error in catalina out:
| Quote: |
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-8080-2" chemaxon.jchem.db.DatabaseSearchException:
ERROR: JChem has run out of memory when trying to cache chemical hashed fingerprint information.
Please visit http://www.chemaxon.com/jchem/FAQ.html#outofmemory
at chemaxon.jchem.db.JChemSearch.loadCacheIfNeeded(JChemSearch.java:2683)
at chemaxon.jchem.db.JChemSearch.search1(JChemSearch.java:2396)
at chemaxon.jchem.db.JChemSearch.search(JChemSearch.java:2240)
at chemaxon.jchem.db.JChemSearch.access$1200(JChemSearch.java:79)
at chemaxon.jchem.db.JChemSearch$SearchThread.run(JChemSearch.java:427)
java.lang.OutOfMemoryError: Java heap space
|
you can ask me for any further informations.
thanks for your help.
Nejd.
ChemAxon 9c0afc9aaf
21-05-2007 11:51:12
Hi,
Since the next JChem version is about a month away (there was a release just before fixing this bug) I have complied a jchem.jar which is identical to version 3.2.6, except this bugfix.
You can download it from here:
http://www.chemaxon.com/download.php?d=/data/download/jchem/3_2_6_mod1/
Szilard
PS : please remember to restart your (web) application after replacing the jar.
User a7e5cba6fc
22-05-2007 08:56:13
Thank you very much, I will test it and give you the feedback.
Nejd