Working on a local database of several SDF's (total of ~1M compounds), I get the error above.
Closing IJC and re-opening it lets the query run without the error.
Making the query more general (retrieves more results) causes it to crash again with the same error.
What is the cause of the problem and how can I fix it?
JChem searches use a in-memory cache of the chemcial hashed fingerprints to allow rapid searching. The amount of memory this consumes is proportional to the size of the structure tables that JChem is searching.
Instant JChem in defualt configuration is set up to allow searching of several hundreds of thousands of structures, but when you go above this level you will need to allocate extra memory to Instant JChem.
Do this using Tools -> Options. Go to the Miscellanous section and then the startup settings. Increase the Maximumn suze property. See here for more information: