scalaer
V2EX  ›  问与答

有 akka 大佬吗, 请教一个问题

  •  
  •   scalaer · Nov 15, 2019 · 2028 views
    This topic created in 2370 days ago, the information mentioned may be changed or developed.

    用 slick 流式读取 mysql 一个大表 700w, 在 map 的时候计数,发现到 360w 就停了, 请问下这是什么原因?

    val source: Source[Event, NotUsed] = Slick.source(query1)
    
    val sink: Sink[ProducerRecord[String, Event], Future[Done]] = Producer.plainSink(p)
    
    var counter = 0;
    
    val done: Future[Done] =
            source
              .map(
                e =>
                  {
                    counter += 1
                    println(counter)
                    new ProducerRecord(topic, e.sid, EventFactory.build_event(e))
                  }
    ).runWith(sink)
    
    4 replies    2019-11-16 22:26:33 +08:00
    woodfish
        1
    woodfish  
       Nov 15, 2019
    是不是 mysql keepalive 超时了
    scalaer
        2
    scalaer  
    OP
       Nov 15, 2019
    @woodfish 现在定位到是 kafka producer 超时了, 这个有啥参数可以设置的吗?
    scalaer
        4
    scalaer  
    OP
       Nov 16, 2019
    @woodfish close-timeout 设置了很大, 不起作用..
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Solana   ·   1172 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 51ms · UTC 23:10 · PVG 07:10 · LAX 16:10 · JFK 19:10
    ♥ Do have faith in what you're doing.