druid.io - Not enough direct memory

发表于

今天在新的机器上部署 druid , 起来后发现有 task 错误, 提示是 direct memory 不够.

日志

错误日志如下:

2017-03-23T16:49:43,161 ERROR [main] io.druid.cli.CliPeon - Error when starting up.  Failing.
com.google.inject.ProvisionException: Guice provision errors:

1) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,908,932,608], memoryNeeded[2,684,354,560] = druid.processing.buffer.sizeBytes[536,870,912] * ( druid.processing.numThreads[4] + 1 )
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:79)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:85)
  while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:64)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:82)
  while locating io.druid.query.groupby.GroupByQueryRunnerFactory
  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=28, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:38)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36)
  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:55)
  while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:95)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:153)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:96)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:180)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.indexing.overlord.TaskRunner
    for parameter 3 at io.druid.indexing.worker.executor.ExecutorLifecycle.<init>(ExecutorLifecycle.java:78)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:170)
  while locating io.druid.indexing.worker.executor.ExecutorLifecycle

2) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,908,932,608], memoryNeeded[2,684,354,560] = druid.processing.buffer.sizeBytes[536,870,912] * ( druid.processing.numThreads[4] + 1 )
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:79)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:85)
  while locating io.druid.query.groupby.GroupByQueryEngine
    for parameter 2 at io.druid.query.groupby.GroupByQueryQueryToolChest.<init>(GroupByQueryQueryToolChest.java:113)
  at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:74)
  while locating io.druid.query.groupby.GroupByQueryQueryToolChest
    for parameter 3 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:64)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:82)
  while locating io.druid.query.groupby.GroupByQueryRunnerFactory
  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=28, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:38)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36)
  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:55)
  while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:95)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:153)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:96)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:180)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.indexing.overlord.TaskRunner
    for parameter 3 at io.druid.indexing.worker.executor.ExecutorLifecycle.<init>(ExecutorLifecycle.java:78)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:170)
  while locating io.druid.indexing.worker.executor.ExecutorLifecycle

3) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,908,932,608], memoryNeeded[2,684,354,560] = druid.processing.buffer.sizeBytes[536,870,912] * ( druid.processing.numThreads[4] + 1 )
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 3 at io.druid.query.groupby.GroupByQueryQueryToolChest.<init>(GroupByQueryQueryToolChest.java:113)
  at io.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:74)
  while locating io.druid.query.groupby.GroupByQueryQueryToolChest
    for parameter 3 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:64)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:82)
  while locating io.druid.query.groupby.GroupByQueryRunnerFactory
  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=28, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:38)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36)
  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:55)
  while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:95)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:153)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:96)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:180)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.indexing.overlord.TaskRunner
    for parameter 3 at io.druid.indexing.worker.executor.ExecutorLifecycle.<init>(ExecutorLifecycle.java:78)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:170)
  while locating io.druid.indexing.worker.executor.ExecutorLifecycle

4) Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[1,908,932,608], memoryNeeded[2,684,354,560] = druid.processing.buffer.sizeBytes[536,870,912] * ( druid.processing.numThreads[4] + 1 )
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:108)
  while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()
    for parameter 4 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:64)
  at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:82)
  while locating io.druid.query.groupby.GroupByQueryRunnerFactory
  while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=28, type=MAPBINDER)
  at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:38)
  while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>
    for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36)
  while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate
  at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:55)
  while locating io.druid.query.QueryRunnerFactoryConglomerate
    for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:95)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:153)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:96)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:180)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.indexing.overlord.TaskRunner
    for parameter 3 at io.druid.indexing.worker.executor.ExecutorLifecycle.<init>(ExecutorLifecycle.java:78)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:170)
  while locating io.druid.indexing.worker.executor.ExecutorLifecycle

4 errors
	at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014) ~[guice-4.0-beta.jar:?]
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036) ~[guice-4.0-beta.jar:?]
	at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:153) ~[druid-api-0.9.1.1.jar:0.9.1.1]
	at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:91) [druid-services-0.9.1.1.jar:0.9.1.1]
	at io.druid.cli.CliPeon.run(CliPeon.java:274) [druid-services-0.9.1.1.jar:0.9.1.1]
	at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.1.1.jar:0.9.1.1]

如上日志所示, 当前需要 memoryNeeded[2,684,354,560] 的 direct memory,
但是当前只有 maxDirectMemory[1,908,932,608] 这么多.
因为 memoryNeeded 的值是这么计算出来的:
druid.processing.buffer.sizeBytes * ( druid.processing.numThreads + 1 ),
因此可以调整 -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes,
或者 druid.processing.numThreads 这三者之一.

druid.processing 下的 buffer 和 numThreads 设置

druid.processing.buffer.sizeBytes 设置是在

middleManager/runtime.properties:15:druid.processing.buffer.sizeBytes=..

druid.processing.numThreads 设置是在

middleManager/runtime.properties:16:druid.processing.numThreads=

具体的值要根据 -XX:MaxDirectMemorySize 来选择设置, 计算后不能超过 -XX:MaxDirectMemorySize,
而默认的 -XX:MaxDirectMemorySize 值,
可以参考这篇博文 : Default HotSpot Maximum Direct Memory Size - DZone Java.

-XX:MaxDirectMemorySize

如果的确有需求, 而 druid.processing.buffer.sizeBytes 和 druid.processing.numThreads 都需要保留,
那么只有改 -XX:MaxDirectMemorySize 了,
可以把设置加到这个地方 :

druid/middleManager/runtime.properties:8:druid.indexer.runner.javaOpts=..