tomcat "pausing" after several days of no activity

After a few days (e.g. 3-4) of no activity/usage with JChemSearchWS (or others), the tomcat seems to go into hibernation or "pause" - not allowing any SOAP calls. A tail of the catalina.out is below.

Is there some parameter that can be set to avoid this hibernation?

It seems the only way to recover is to restart catalina. I hope we don't have to cron a job to check for "INFO: Pausing Coyote HTTP/1.1" in the catalina.out and then force a restart.

Please confirm.

 

--- from catalina.out -----

Thu Mar 11 02:10:56 MST 2010

Search mode: SUBSTRUCTURE

Structure table: jchemmol

Query: [H][C@]12C[C@](O)(CO)[C@](C)(O1)n1c3ccccc3c3c4CNC(=O)c4c4c5ccccc5n2c4c13

Screened: 64

Hits: 30

Total time: 94 ms  Screening: 5 ms

Processing threads: 8

Current / peak / maximum searches per minute: 2 / 2 / Unlimited

Creating idsearchID428040106 attempt=0

 cd_ids has 30 results.                 

Mar 15, 2010 8:14:08 PM org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-8180 

Hi Steve,

We haven't experienced this problem yet with our own servers, and we are still researching this issue.

 

Does the tomcat server accept any non-soap requests?  For example, does opening a browser to either of the following urls produce a response?

http://<yourhost>:<your port>/

http://<yourhost>:<your port>/axis2

 

Jon

Steve,

Here are some questions and ideas that might get us closer to a solution.

 

Questions:

Has this happened more than once or on a repetitive basis?

Is the Pause message the last message found in catalina.out?

What type of system (e.g. Windows/Linux) are you running on? 

Are you running the server as a Windows service?

 

Ideas:

Pausing of Coyote (the Http Coonector in Tomcat) happens as part of a
normal shutdown. Is it possible something is calling shutdown.bat or
shutdown.sh?

Have you added an other (automated) additions to the web server?  Perhaps rolling over log files which might pause the web server.

Inappropriately calling a system.exit() might causes some issues with the JVM.  http://www.mail-archive.com/[email protected]/msg35414.html

Hi Roland -

 Yes, this happens frequently - especially when there are stretches of inactivity. Sometimes there is no pause message at all.  I am hoping it is not related to "dirty" application  usage. I thought we resolved that per your suggestions in a previous post.

 This is running on a linux environment:

% uname -a

Linux ip-173-201-23-121.ip.secureserver.net 2.6.18-028stab059.6 #1 SMP Fri Nov 14 14:01:22 MSK 2008 i686 i686 i386 GNU/Linux            

  To try to reproduce, restart the app server, issue a SOAP request or two, then make sure no requests are made for a couple days. Then try again.

  I can wait until it dies again and then check to see if any response to the app server through the urls you suggest (i.e. simple + ajax).

  The restart process (after is appears dead), I use is as follows:

 

cd /home/kkb/ChemAxon/JChemWebServices/bin

echo "Shutting down catalina...."

./shutdown.sh

echo "Killing all java...."

killall java

echo "Starting things back up...."

./startup.sh

 

If there are log rollover options that you suggest, please let me know where to alter them.         

-Steve

 

No-go on these after noting the "PAUSING.." in the logfile:

http://<yourhost>:<your port>/

http://<yourhost>:<your port>/axis2

 

Indeed, "ps -ef | fgrep java" in this scenario does NOT show the java process corresponding to the tomcat process - which indicates it has crashed: i.e. This is NOT a process:

/usr/java/jre1.5.0_14/bin/java -Xmx256M -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=../tomcat/conf/logging.properties -Djava.endorsed.dirs=../tomcat/endorsed -classpath :../tomcat/bin/bootstrap.jar -Dcatalina.base=../tomcat -Dcatalina.home=../tomcat -Djava.io.tmpdir=../tomcat/temp org.apache.catalina.startup.Bootstrap start     

 

Could it be that the "cleanup" you recommended in the previous post is seeding something for failure after some time-out?

 

Again, the workflow is as below for reference - the second new SoapClient still troubles me as it does not intuitively cleanup the first "new SoapClient":

$maxhits=20;

$soap = new SoapClient('http://myserver.com:8180/axis2/services/JChemSearchWS?wsdl'); 

$conn_param = array( 'driver' => 'com.mysql.jdbc.Driver',

                     'url' => 'jdbc:mysql://myserver.com/mysqlDb',

                     'username' => 'myusername',

                     'password' => 'mypassword',

                     'propertyTable' => 'JChemProperties');  

 

 $result= strip_tags($soap->getConnectionAndTableInfo($conn_param)->return);

$id=substr($result,0,strpos($result,"\n"));

 $search_param = array(  'connectionHandlerId' => $id, 

                        'tableName' => 'jchemmol',

                        'queryMolecule' => $smi,

                        'queryOptions' => 'sep=! t:s!maxHitCount:0!maxTime:60000!returnNonHits:n!',

                        'beginIndex' => '0',

                        'count' => $maxhits,

                        'outputFormat' => 'smiles',

                        'dataFieldNames' => 'mr_id',

                        'hitColorAndAlignmentOptions' => ''

); 

 

$search_result = $soap->runCompleteSearch($search_param)->return;

$soap = new SoapClient('http://myserver.com:8180/axis2/services/ConnectionWS?wsdl');

$soap->close(array('connectionHandlerId' => $id));  

 

 

Did you guys reproduce this? 

I am thinking this is possibly memory related. Please confirm the minimum memory configuration required to run the webservices.

Possibly Tomcat handles the op. system signals in a wrong way. Try the -Xrs VM argument during startup, which may help you. You can see the documentation about it here:

http://java.sun.com/javase/6/docs/technotes/tools/windows/java.html

Cheers,

Roland

Steve,

I also found this webpage about tomcat shutdown "features" possibly related to your issue.

http://marc.info/?l=tomcat-user&m=103177045008593&w=2

 

There seems to be a few ways that the server could be shutting down tomcat (or the java virtual machine running tomcat) externally. 

For example, sending a SHUTDOWN keyword to a specific port from the localhost.

To avoid this, please ensure that you change the port and/or shutdown keyword in the <jchem web services dir>/tomcat/conf/server.xml file.  

 

Many of these issues are dependent on the operations of your specific server.  If you can further identify the cause of the shutdown we can help you further.