Error starting splice machine in region server

Splice Machine Community Forum Getting Started with Splice Error starting splice machine in region server

This topic contains 7 replies, has 2 voices, and was last updated by  rakeshj 1 month, 3 weeks ago.

Viewing 8 posts - 1 through 8 (of 8 total)
  • Author
    Posts
  • #766236

    rakeshj
    Participant

    I followed the steps as per below reference URL, to install splice machine on out 4-node HDP 2.4.2 version cluster, buit getting below error in region server

    
    2016-12-01 16:20:23,768 ERROR [pool-14-thread-1] impl.SkeletonTxnNetworkLayer:
    2016-12-01 16:20:23,782 ERROR [pool-14-thread-1] lifecycle.DatabaseLifecycleManager: Error during during startup of service com.splicemachine.derby.lifecycle.EngineLifecycleService@499988c:
    java.sql.SQLException: Failed to start database 'splicedb' with class loader sun.misc.Launcher$AppClassLoader@18b4aac2, see the next exception for details.
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:102)
            at com.splicemachine.db.impl.jdbc.Util.newEmbedSQLException(Util.java:170)
            at com.splicemachine.db.impl.jdbc.Util.seeNextException(Util.java:306)
            at com.splicemachine.db.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2326)
            at com.splicemachine.db.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:302)
            at com.splicemachine.db.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:72)
            at com.splicemachine.db.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:57)
            at com.splicemachine.db.jdbc.Driver40.getNewEmbedConnection(Driver40.java:69)
            at com.splicemachine.db.jdbc.InternalDriver.connect(InternalDriver.java:256)
            at com.splicemachine.db.jdbc.EmbeddedDriver.connect(EmbeddedDriver.java:125)
            at com.splicemachine.tools.EmbedConnectionMaker.createFirstNew(EmbedConnectionMaker.java:56)
            at com.splicemachine.derby.lifecycle.EngineLifecycleService.start(EngineLifecycleService.java:95)
            at com.splicemachine.lifecycle.DatabaseLifecycleManager$Startup.bootServices(DatabaseLifecycleManager.java:229)
            at com.splicemachine.lifecycle.DatabaseLifecycleManager$Startup.run(DatabaseLifecycleManager.java:220)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
    Caused by: java.sql.SQLException: Failed to start database 'splicedb' with class loader sun.misc.Launcher$AppClassLoader@18b4aac2, see the next exception for details.
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:46)
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:126)
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:75)
            ... 16 more
    Caused by: java.sql.SQLException: Java exception: ': java.lang.NullPointerException'.
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:46)
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:126)
            at com.splicemachine.db.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:75)
            at com.splicemachine.db.impl.jdbc.Util.newEmbedSQLException(Util.java:170)
            at com.splicemachine.db.impl.jdbc.Util.javaException(Util.java:327)
            at com.splicemachine.db.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2322)
            ... 13 more
    Caused by: java.lang.NullPointerException
            at com.splicemachine.pipeline.Exceptions.parseException(Exceptions.java:37)
            at com.splicemachine.derby.impl.store.access.SpliceTransaction.doElevate(SpliceTransaction.java:386)
            at com.splicemachine.derby.impl.store.access.SpliceTransaction.elevate(SpliceTransaction.java:319)
            at com.splicemachine.derby.impl.store.access.SpliceAccessManager.boot(SpliceAccessManager.java:683)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1996)
            at com.splicemachine.db.impl.services.monitor.TopService.bootModule(TopService.java:337)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:545)
            at com.splicemachine.db.impl.services.monitor.FileMonitor.startModule(FileMonitor.java:51)
            at com.splicemachine.db.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:430)
            at com.splicemachine.derby.impl.db.SpliceDatabase.bootStore(SpliceDatabase.java:447)
            at com.splicemachine.db.impl.db.BasicDatabase.boot(BasicDatabase.java:164)
            at com.splicemachine.derby.impl.db.SpliceDatabase.boot(SpliceDatabase.java:115)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1996)
            at com.splicemachine.db.impl.services.monitor.TopService.bootModule(TopService.java:337)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1830)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1696)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1574)
            at com.splicemachine.db.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:993)
            at com.splicemachine.db.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:553)
            at com.splicemachine.db.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2287)
            ... 13 more
    

    http://doc.splicemachine.com/Administrators/InstallingSpliceMachine/HDPInstall.html

    #766237

    dgomezferro
    Moderator

    Hi rakeshj,

    Can you paste the logs for the region server and the HMaster? You can use an online pasting service or join us on our Slack channel: http://community.splicemachine.com/slack-channel-signup/

    If you join Slack you can send me the logs directly @daniel if you don’t want to share them publicly.

    Looking forward to helping you with your issue,
    Daniel

    #766238

    rakeshj
    Participant

    Hi Danial,

    Thanks for your reply.
    Here are logs
    1. splice-derby.log over hbase master node
    https://justpaste.it/10ysm
    2. HBase master log
    https://justpaste.it/10yt6
    3. region server log
    https://justpaste.it/10yt4

    Added reference links to content. I hope these links would be accessible from your end.
    Please let me know if you need anything else.
    Cluster details: HDP-2.4.2.0-258 , 5 node cluster ( 4- data nodes, 1 Master/management node)

    #766240

    dgomezferro
    Moderator

    Hi rakeshj,

    Are you trying to run Splice Machine’s community edition on a secure cluster? That’s an enterprise only feature, and we don’t support it on the community edition. Please have a look at: http://doc.splicemachine.com/Administrators/InstallingSpliceMachine/Authentication.html?Highlight=secure and http://doc.splicemachine.com/GettingStarted/SpliceEditions.html

    Please let me know if that’s the case.

    Cheers,
    Daniel

    #766241

    rakeshj
    Participant

    Hi Danial,

    Yes, our cluster is kerberized; I forgot to mention it. Thanks for the details.
    I am trying now with unkerberized cluster, hoping it will work smoothly.

    Thanks,
    Rakesh J

    #766242

    rakeshj
    Participant

    Hi Daniel,

    I did try with unkerberized cluster, but no luck.
    I received same error, but with additional exception inside the all region servers log and then all the region servers got crashed/aborted:

    2016-12-02 19:14:15,518 ERROR [RS_OPEN_REGION-hscale-dev1-dn1:16020-2] coprocessor.CoprocessorHost: The coprocessor com.splicemachine.derby.hbase.SpliceIndexObserver threw java.io.IOException: java.lang.IllegalStateException: Unable to register service, startup failed
    java.io.IOException: java.lang.IllegalStateException: Unable to register service, startup failed
    at com.splicemachine.derby.hbase.SpliceIndexObserver.start(SpliceIndexObserver.java:141)
    at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:411)

    Here is the full log from one of the region servers
    https://justpaste.it/10zvm

    Thanks,
    Rakesh J

    #766284

    dgomezferro
    Moderator

    Hi rakeshj,

    Sorry for the late reply, I missed your post.

    Are you still having this issue? It looks like some misconfiguration, are you able to start HBase alone (without SpliceMachine) after you unkerberized your cluster?

    #766285

    rakeshj
    Participant

    Hi dgomezferro
    Yeah, past this issue, below are details you may find useful

    Exception in brief:
    Splice machine services checks for master initialization while being initialized with limited (configurable) retry attempts, but fails once retry count finishes/exhausts.
    Solution
    Increase retry count in hbse configuration from Ambari console to 30
    hbase.client.retries.number = 30

    Thanks,
    Rakesh J

    • This reply was modified 2 weeks, 6 days ago by  Splice Admin.
Viewing 8 posts - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.