mysql - Exception thrown when try to add documents to the lucene index continuously inside the for loop

Question

I’m using “compass-2.2.0” to create a lucene index in MySql database table. This is part of my code to index documents, following Exception thrown when try to add documents to the lucene index continuously inside the for loop. Any workaround to overcome this error?

My hosting server is WSo2 Stratoes tomcat based server and Wso2 Stratoes data service server. My program works fime in local tomcat/ mySql servers.

This is the sample blog post that I have followed- http://mprabhat.wordpress.com/2012/08/13/create-lucene-index-in-database-using-jdbcdirectory/

 private void addIndex(IndexWriter indexWriter) throws CorruptIndexException, IOException {
        try {
            for (int i = 0; i < docNames.length; i++) {
                StringReader strRdElt = new StringReader(new DefaultTokenizer().processText(filesInText[i]));
                StringReader docId = new StringReader(Integer.toString(i));

                Document doc = new Document();

                doc.add(new Field("doccontent", strRdElt, Field.TermVector.YES));
                doc.add(new Field("docid", docId, Field.TermVector.YES));


                indexWriter.addDocument(doc);

            }
          //  indexWriter.close();

        } catch (Exception e) {
            e.printStackTrace();
        }
    }

This is the new error i'm getting, the server that i'm using dosen't allow creating temp files. It's somehow creating temp files when trying to write the content in to the database. I debuged inside above loop, around 5 iterations it goes successfully then occurs the following error. any workaround to overcome this error ?

java.lang.SecurityException: Unable to create temporary file
        at java.io.File.checkAndCreate(File.java:1701)
        at java.io.File.createTempFile(File.java:1792)
        at java.io.File.createTempFile(File.java:1828)
        at org.apache.lucene.store.jdbc.index.FileJdbcIndexOutput.configure(File
JdbcIndexOutput.java:46)
        at org.apache.lucene.store.jdbc.index.RAMAndFileJdbcIndexOutput.switchIf
Needed(RAMAndFileJdbcIndexOutput.java:113)
        at org.apache.lucene.store.jdbc.index.RAMAndFileJdbcIndexOutput.writeByt
es(RAMAndFileJdbcIndexOutput.java:73)
        at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:43)
        at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:
65)
        at org.apache.lucene.index.TermVectorsTermsWriter.finishDocument(TermVec
torsTermsWriter.java:170)
        at org.apache.lucene.index.TermVectorsTermsWriter$PerDoc.finish(TermVect
orsTermsWriter.java:258)
        at org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(Docum
entsWriter.java:1404)
        at org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter
.java:1424)
        at org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWrite
r.java:1043)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWrite
r.java:772)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060
)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2034
)
        at org.karsha.base.DocIndexer_test.addIndex(DocIndexer_test.java:168)
        at org.karsha.base.DocIndexer_test.indexToDB(DocIndexer_test.java:115)
        at org.karsha.base.DocIndexer_test.topKFiboTerms(DocIndexer_test.java:35
9)
        at org.karsha.controler.RecommondTermsServelet.doGet(RecommondTermsServe
let.java:379)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274
)
        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:271
)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:3
06)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.
java:166)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:299)
        at org.apache.catalina.core.ApplicationFilterChain.access$000(Applicatio
nFilterChain.java:57)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilt
erChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilt
erChain.java:189)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:225)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:123)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authentica
torBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:98)
        at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValv
e.java:172)
        at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.inv
oke(CarbonStuckThreadDetectionValve.java:156)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
927)
        at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(Ca
rbonContextCreatorValve.java:52)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp
11Processor.java:1001)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(
AbstractProtocol.java:579)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoin
t.java:1653)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
        at java.lang.Thread.run(Thread.java:619)

Answer 1

There are many reasons why you can get ThreadDeath exception from your DB.

1. DB URL is incorrect
2. Port number of database has changed, similar issue here, see the last comment in discussion
3. You have too many sessions open and your DB cannot provide you with new session.

EDIT As per updated stacktrace

Compass gives you two things:

1. IndexInput and 2. IndexOutput

By default IndexOutput is set to RAMAndFileJDBCIndexOutput, which after a certain limit switches to FileJdbcIndexOutput and your temporary file is getting created.

To Fix this either you can change this default value or change the default IndexOutput in your JDBCDirectorySettings.