架構使用spark streaming 消費kafka的數據，并通過impala來插入到kudu中，但是通過對比發現落地到kudu表中的數據比kafka消息數要少，通過后臺日志發現，偶發性的出現java.sql.SQLException: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Syntax error in line 1 原因是調用過程中使用了數據庫連接池,會合并多行sql執行，實現如下：

import com.alibaba.druid.pool.DruidDataSource; import com.alibaba.druid.pool.DruidDataSourceFactory; import com.alibaba.druid.pool.DruidPooledConnection; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory;import java.sql.SQLException; import java.util.Properties; public class ImapalConnPool {private static Log logger = LogFactory.getLog(ImapalConnPool.class);private static ImapalConnPool imapalConnPool = null;private static DruidDataSource druidDataSource = null;static {Properties properties = new Properties();properties.setProperty("driverClassName","com.cloudera.impala.jdbc41.Driver");properties.setProperty("url","jdbc:impala://127.0.0.1:21050"); properties.setProperty("username","");properties.setProperty("password","");properties.setProperty("initialSize","50");properties.setProperty("maxActive","100");properties.setProperty("maxWait","60000");properties.setProperty("timeBetweenEvictionRunsMillis","60000");properties.setProperty("minEvictableIdleTimeMillis","300000");properties.setProperty("validationQuery","SELECT 1");properties.setProperty("testWhileIdle","true");properties.setProperty("testOnBorrow","false");properties.setProperty("testOnReturn","false");properties.setProperty("poolPreparedStatements","false");//當該值大于0時，啟用pool，poolPreparedStatements為trueproperties.setProperty("maxPoolPreparedStatementPerConnectionSize","-1");try {druidDataSource = (DruidDataSource) DruidDataSourceFactory.createDataSource(properties); //DruidDataSrouce工廠模式} catch (Exception e) {logger.error(e);}}public static ImapalConnPool getInstance(){if (null == imapalConnPool){synchronized(ImapalConnPool.class) {if(null == imapalConnPool) {imapalConnPool = new ImapalConnPool();}}}return imapalConnPool;}public DruidPooledConnection getConnection() throws SQLException {return druidDataSource.getConnection();}}

?

調整上述參數poolPreparedStatements和maxPoolPreparedStatementPerConnectionSize任然不能解決同一個connection合并多行sql的問題 后去掉連接池，改為jdbc直連問題解決。有解決過上面數據庫連接池問題的麻煩告知我一下 Connection connectionn = null; Statement statement = null; try {Class.forName("com.cloudera.impala.jdbc41.Driver");connectionn = DriverManager.getConnection("jdbc:impala://127.0.0.1:21050");statement = connectionn.createStatement();for (String item : execSql) {statement.execute(item);}processResult = true; }catch (Exception ex){logger.error(this,ex); }finally {if(null != statement){try {statement.close();}catch (Exception ex){logger.error(this,ex);}}if(null != connectionn) {try {connectionn.close();}catch (Exception ex){logger.error(this,ex);}} }

?

?

轉載于:https://www.cnblogs.com/qizhelongdeyang/p/10339029.html

總結

以上是生活随笔為你收集整理的impala jdbc驱动执行impala sql的一个坑（不支持多行sql）的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。