• Petr Špaček's avatar
    orchestrator: fix progress report · 0da94645
    Petr Špaček authored
    For some reason, the code reading answers was using imap() function to
    read them _in order_ they were sent. This could cause answer buildup
    in an internal multiprocessing queue, and eventually the program logged
    multiple 'Received xxx answers' lines at once, which made the status
    reporting useless.
    
    It was like that since d422292d, and
    this commit claims it reduced storage requirements by factor 25.
    The problem is that the commit did three changes at once:
    - it moved txn.put() into central place in the main thread
    - it coalesced all writes into one huge transactoin
    - it changed imap_unordered() to imap() call
    
    In my tests today, centralized txn.put() in the main thread in
    combination with one huge transaction and imap_unordered() increases
    storage size by 5 %, and that does not seem worth incorrect status
    reporting.
    
    Tested with LMDB version 0.9.29.
    0da94645