Importer - large SD-File

I try import large SD-File (1.2GB, 400k records). I use import-features from jchem\examples\db_search with tomcat 6.0.14. I get such error:

Warning: check limit is 2000000000 lines for stream.
java.lang.OutOfMemoryError
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:199)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at chemaxon.marvin.io.PositionedInputStream.readLine8bit(PositionedInputStream.java:468)
        at chemaxon.marvin.io.PositionedInputStream.readLine(PositionedInputStream.java:301)
        at chemaxon.marvin.io.formats.AbstractMRecordReader.readLine(AbstractMRecordReader.java:127)
        at chemaxon.marvin.io.formats.mdl.MolRecordReader.readLine(MolRecordReader.java:177)
        at chemaxon.marvin.io.formats.mdl.MolRecordReader.readCtab(MolRecordReader.java:582)
        at chemaxon.marvin.io.formats.mdl.MolRecordReader.readMol0(MolRecordReader.java:198)
        at chemaxon.marvin.io.formats.mdl.MolRecordReader.nextRecord(MolRecordReader.java:140)
        at chemaxon.marvin.io.formats.mdl.MolRecordReader.nextRecord(MolRecordReader.java:79)
        at chemaxon.formats.MolFileHandler.collectFileInfo(MolFileHandler.java:48)
        at chemaxon.jchem.file.ImportFileHandler.collectFileInfo(ImportFileHandler.java:52)
        at chemaxon.jchem.db.Importer.getFileInfo(Importer.java:667)
        at chemaxon.jchem.db.Importer.init(Importer.java:605)
        at chemaxon.jchem.db.Importer.run(Importer.java:370)

Have Java enough memory if i use  -Xms1400m

Or is that global limitation?

Hi,

Coul dyou please send us your exact JChem version ? ("About" button) This is required for all support requests.

Is this the full stack trac eor there is more ?

If you do not check 

"Check whole file for field names in selected file"

then the probably you will not get this error.

Please note the "number of lines" actually mes number of records - we will fix this. 

Please specify a reasonable amount.

Do you get the error right after this screen, or do you see a dialog for connecting fields ?

This simple example is not prepared for such large imports: JChemManager (jcman) and Instant JChem are better suited for such large imports an should not have a memory problem.

(here a Stream seems to be buffered, which is not needed normally for files)

So this is not a global limitation.

Have Java enough memory if i use  -Xms1400m

I assume you mean this fixes your problem. Is your -Xmx setting the same ?

I have recently noticed that if -Xms is also specified for the same -Xmx, certaing JVMs can utilize more memory (esp. 64 bit)

Best,

 

Szilard 

JChem version: 5.1.3_2
Table version: 50103

It's full stack trace.

>Do you get the error right after this screen, or do you see a dialog for connecting fields ?

I see dialog for connecting fields, i have error on export start.

Is your -Xmx setting the same ?

yes, it's  -Xms1400m

 

Hi,

Looking at the code this should only happen if an InputStream is specified for Importer.setInput(), which in theory shouldn't be the case in the unmodified JSP source in db_search/transfer/import.jsp.

File file = new File(filePath);

[...]

ithread.setInput(file);

 

Anyhow, in general we do not support such an old version, I'm glad the workaround works for you.

Best,

Szilard