51Testing软件测试论坛

标题: 记一次mycat的问题检查-栈溢出 [打印本页]
作者: TimiZheng    时间: 2019-10-18 16:27
标题: 记一次mycat的问题检查-栈溢出
本帖最后由 TimiZheng 于 2019-10-18 16:58 编辑

以下问题为去年的问题，记录在自己的为知笔记中，拿出来分享下。
现象：运行了两年的mycat，突然变得十分不稳定。
4个mycat实例，在运行一段时间（有时几个小时，有时1-2天）总会出现偶尔无法连接等问题，监控程序经常会报错。

日志：查看mycat.log日志中，没有任何的信息。
在console.log中找到stackoverflowerror，但是只有1条这样的错误，和检测不能连接的时间并不相同。所以当时没有在意。
对jvm进行分析，也没有发现问题，没有长时间的gc。
找java开发的大牛，对jvm参数一通调整，各种分析，还是无济于事。
经过一个星期的折腾，慢慢发现了规律。
规律：重启后一段时间内会很正常，但是一旦出现stackoverflowerror错误，后面就会陆续出现不能连接的情况，随着时间推移，不能连接会越来越严重，直到守护进程认为mycat不可用，就会自动重启程序。
于是开始仔细检查这个错误：

  1. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">Exception in thread "$_NIOREACTOR-3-RW"
  2. 2018-03-27T02:16:14.612+0800: 196587.704: Total time for which application
  3. threads were stopped: 0.0019378 seconds, Stopping threads took: 0.0001489
  4. seconds</p>
  5. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">java.lang.StackOverflowError</p>
  6. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  7. com.alibaba.druid.sql.ast.SQLObjectImpl.accept(SQLObjectImpl.</p>
  8. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  9. com.alibaba.druid.sql.visitor.functions.Concat.eval_r(Concat.</p>
  10. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  11. com.alibaba.druid.sql.visitor.SQLEvalVisitorUtils.visit(SQLEvalVisitorUtils.</p>
  12. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  13. com.alibaba.druid.sql.visitor.SQLEvalVisitorImpl.visit(SQLEvalVisitorImpl.</p>
  14. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  15. com.alibaba.druid.sql.ast.expr.SQLMethodInvokeExpr.accept0(SQLMethodInvokeExpr.</p>
  16. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  17. com.alibaba.druid.sql.ast.SQLObjectImpl.accept(SQLObjectImpl.</p>
  18. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  19. com.alibaba.druid.sql.visitor.SQLEvalVisitorUtils.visit(SQLEvalVisitorUtils.</p>
  20. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  21. com.alibaba.druid.sql.visitor.SQLEvalVisitorImpl.visit(SQLEvalVisitorImpl.</p>
  22. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  23. com.alibaba.druid.sql.ast.expr.SQLBinaryOpExpr.accept0(SQLBinaryOpExpr.</p>
  24. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  25. com.alibaba.druid.sql.ast.SQLObjectImpl.accept(SQLObjectImpl.</p>
  26. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  27. com.alibaba.druid.wall.spi.WallVisitorUtils.eval_r(WallVisitorUtils.</p>
  28. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  29. com.alibaba.druid.wall.spi.WallVisitorUtils.getValue(WallVisitorUtils.</p>
  30. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  31. com.alibaba.druid.wall.spi.WallVisitorUtils.getValue(WallVisitorUtils.</p>
  32. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  33. com.alibaba.druid.wall.spi.WallVisitorUtils.getValue(WallVisitorUtils.</p>
  34. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  35. org.opencloudb.route.util.RouterUtil.isConditionAlwaysFalse(RouterUtil.</p>
  36. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  37. org.opencloudb.parser.druid.MycatSchemaStatVisitor.addExprIfNotFalse(MycatSchemaStatVisitor.</p>
  38. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  39. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  40. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  41. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  42. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  43. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  44. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  45. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  46. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  47. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  48. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  49. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  50. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  51. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  52. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  53. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  54. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  55. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  56. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  57. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  58. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  59. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  60. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  61. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  62. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  63. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  64. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  65. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  66. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  67. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  68. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  69. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  70. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  71. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  72. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  73. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  74. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  75. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  76. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei=""> <wbr>  <wbr>  <wbr>  <wbr> at
  77. org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor.</p>
  78. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">...........................</p>
  79. <p style="margin: 8px 0px; font-family: Helvetica,;" hiragino="" sans="" yahei="">org.opencloudb.parser.druid.MycatSchemaStatVisitor.splitUntilNoOr(MycatSchemaStatVisitor. </p>
复制代码




文章来源：沧海大声啸的博客           作者：Kervin（博为峰网校讲师）
讲师作品： MySQL实战校园业务数据库：http://www.atstudy.com/course/2001
作者: TimiZheng    时间: 2019-10-18 16:59
这个就是栈溢出的问题所在，没有办法只能去找源代码来看看，这是什么鬼。

找到源代码如下：

  
  1.   splitUntilNoOr(WhereUnit whereUnit) {       (whereUnit.isFinishedParse()) {          (whereUnit.getSubWhereUnit().size() > ) {             (i = ; i < whereUnit.getSubWhereUnit().size(); i++) {                splitUntilNoOr(whereUnit.getSubWhereUnit().get(i));             }          }        } {          SQLBinaryOpExpr expr = whereUnit.getCanSplitExpr();          (expr.getOperator() == SQLBinaryOperator.BooleanOr) { addExprIfNotFalse(whereUnit, expr.getRight());             (expr.getLeft() SQLBinaryOpExpr) {                whereUnit.setCanSplitExpr((SQLBinaryOpExpr)expr.getLeft());                splitUntilNoOr(whereUnit);             } {                addExprIfNotFalse(whereUnit, expr.getLeft());             }          } {             addExprIfNotFalse(whereUnit, expr);             whereUnit.setFinishedParse();          }       }     }        
复制代码
原来是递归解析sql语句（注：新版本中已经没有这段代码）。  大致推测是sql语句引起，找开发问了一圈，都说没有改过程序。     
这么大一个错误也不给我打印一个sql语句出来，于是找了其中一台打开debug模式，并降低这台mycat的负载，但是debug模式日志量仍然很大，1分钟就能打出50M，又不知道什么时间才会出来，所以做了一个脚本去监控console.log ，一旦发现有错，立即保存日志。   
最后终于找到了sql语句，是一个有程序拼出来的sql，发现是用户数据有大量的重复导致拼出来的sql有非常多的or组成。  处理：数据去重，并设置表的唯一索引防止类似事件发生。




欢迎光临 51Testing软件测试论坛 (http://bbs.51testing.com/) Powered by Discuz! X3.2