i am very intersesting by the evaluation of distance between molecules used in libMCS to cluster the compounds.
Is there any paper which explain the concepts used ?
Samuel, I am not sure I do properly understand your question.
Are you interested in distance calculations applied in libMCS, or you are interested in using the maximum common substructure of molecules to define their distance?
In fact, i am intersting bye those 2 topics.
1. LibMCS uses the Tanimoto similarity of topological fingerprints to estimate wether two molecules to be compared can share a non-trivial common subgraph. If two molecular structures do not exhihit Tanimoto similarity above a given threshold (see -t command line parameter) they are unlikely to share an interesting common core (ie. carbon-carbon single bond is not considered as 'interesting common core').
2. There are various ways to define a similarity measure based on the MCS of two compounds. See for instance: http://magix.fri.uni-lj.si/idamap/idamap2001/papers/cuissart.pdf