Fix a hang in JobSystem

The hang was caused by a subtle race. When a job is completed, its 
thread must signal all the threads that might be waiting on this job.
The signaling code was attempting to signal only the minimum number
of threads -- this was important especially in the case where no threads
were waiting, then the call to notify() could be avoided.

Unfortunately, for performance reasons we're not calling notify() with
the condition lock held, this meant that between the time the number of 
waiting threads was latched and the time of the notify() call, more
threads could enter their condition variable wait(), and it would
then be possible for these threads to wake up, instead of the thread
we were trying to wake up (the one waiting on the job).

It would then get stuck forever.

This bug was introduced in 2df639133b


Also add some debugging code for this kind of failure (disabled)
This commit is contained in:
Mathias Agopian
2021-03-12 11:47:10 -08:00
committed by Mathias Agopian
parent 24121a4734
commit e8b16d600e
2 changed files with 45 additions and 9 deletions

View File

@@ -378,13 +378,12 @@ private:
return !index ? nullptr : &mJobStorageBase[index - 1];
}
void wait(std::unique_lock<Mutex>& lock) noexcept;
void wait(std::unique_lock<Mutex>& lock, Job* job = nullptr) noexcept;
void wake() noexcept;
// these have thread contention, keep them together
utils::Mutex mWaiterLock;
utils::Condition mWaiterCondition;
uint32_t mWaiterCount = 0;
std::atomic<uint32_t> mActiveJobs = { 0 };
utils::Arena<utils::ThreadSafeObjectPoolAllocator<Job>, LockingPolicy::NoLock> mJobPool;