RegisterClassA/W with hbrBackground:=GetSysColorBrush(...), leads to errors

[WeberAndre]

In our styled APP a external DLL (QT) registers a own WndClass with RegisterClass and used GetSysColorBrush( ... ) to fill the hbrBackground member of this record, this is wrong - according to the MS documentation.

The brush will be destroyed durring UnregisterClass, but the brush handle is still in the "VCLStylesBrush" dictionary stored and later calls to GetSysColorBrush() will return an invalid brush handle. With leads to errors in serveral other places.

We can't change that library - so the fix is to detour also RegisterClassA/W and check there if the hbrBackground is in the VCLStylesBrush dictionary if so - replace the HBrush with a newly created one.

function Detour_RegisterClassW(wndClass: PWndClassW): ATOM; stdcall;
var
  LCurrentStyleBrush: TListStyleBrush;
  enum: TListStyleBrush.TPairEnumerator;
  LColor: TColor;
begin
  if not(ExecutingInMainThread) then
     Exit(Trampoline_user32_RegisterClassW(wndClass^));

  VCLStylesLock.Enter;
  try
    if StyleServices.IsSystemStyle or not TSysStyleManager.Enabled then
       Exit(Trampoline_user32_RegisterClassW(wndClass^));

    if VCLStylesBrush.TryGetValue(StyleServices.Name, LCurrentStyleBrush) then
    begin
      enum := LCurrentStyleBrush.GetEnumerator;
      while enum.MoveNext do
      begin
        if enum.Current.Value = wndClass^.hbrBackground  then
        begin
          // never Register a WindowClass with a Detour_GetSysColorBrush
          // because UnregisterClass will destroy the HBRUSH ... and also the
          // HBRUSH cached in <VCLStylesBrush> ...
          if enum.Current.Key = COLOR_HOTLIGHT then
             LColor := StyleServices.GetSystemColor(clHighlight)
          else
             LColor := StyleServices.GetSystemColor(TColor(enum.Current.Key or Integer($FF000000)));
          // create the same colored brush again ... so windows can destroy it
          wndClass^.hbrBackground := CreateSolidBrush(LColor);
          break;
        end;
      end;
    end;
  finally
    VCLStylesLock.Leave;
  end;
  result := Trampoline_user32_RegisterClassW(wndClass^);
end;

the same fix must be done for Detour_RegisterClassA