MessageManager::Lock: Release the entry mutex whenever it was gained

[yumasansansan]

MessageManager::Lock gains its entry mutex in exclusiveTryAcquire(), and of that mutex the header says: "If multiple threads call enter() simultaneously, only one will succeed in gaining this mutex. The mutex is released again in exit()." It is not always released there. exit() releases it only when the lock was acquired, and there is one way to succeed without acquiring anything: tryAcquire() returns true straight away when the calling thread already has exclusive access to the MessageManager. Then acquired stays false, exit() returns before it makes the guard that releases the mutex, and the mutex is still held when the Lock is destroyed:

  WARNING: ThreadSanitizer: destroy of a locked mutex
      #0 pthread_mutex_destroy
      #1 juce::CriticalSection::~CriticalSection() juce_SharedCode_posix.h:50
      #2 juce::MessageManager::Lock::~Lock() juce_MessageManager.cpp:280
      #3 juce::MessageManagerLock::~MessageManagerLock() juce_MessageManager.cpp:449
      #4 juce::JuceVST3Component::LockedVSTComSmartPtr<juce::JuceAudioProcessor>::~LockedVSTComSmartPtr() juce_audio_plugin_client_VST3.cpp:3831
      #5 juce::JuceVST3Component::JuceVST3Component(...) juce_audio_plugin_client_VST3.cpp:2573

Destroying a locked mutex is undefined, and pthread_mutex_destroy() is allowed to answer EBUSY and leave the mutex as it was. The same invariant is broken the other way round as well: the blocking message can set acquired after tryAcquire() has given up waiting for it, and exit() would then release a mutex the object never gained.

What the entry mutex needs is its own answer to "was it gained", which is not the same question as "was the MessageManager locked". The Lock remembers that it gained the mutex and releases it on that; acquired keeps its meaning, and the cleanup that belongs to it is unchanged. Only the thread holding the entry mutex touches the new member, so it needs no mutex of its own.

Every VST3 plug-in built with JUCE goes through it: JuceVST3Component's constructor makes a LockedVSTComSmartPtr, and that smart pointer's destructor takes a MessageManagerLock on a thread which, in a host, already holds the message manager lock. ADLplug-Next found it with the thread sanitizer, in its own renderer, which loads the VST3 the build has just made; the report is the first thing that happens after the plug-in is created.

It is in 9.0.2 and in develop as it stands, and it is independent of the commits of ours that are already open (#1749, #1750).