Preventing duplication of game rating logs

[BElluu]

Closed #433

[L-e-x-o-n]

We should check how many logs are affected.

This is a good change, but it doesn't address the cause, why is the same match having duplicate logs in the first place?

[geekingfrog]

What is the SQL for this? I'm pretty sure this is going to acquire an exclusive lock on the entire table for the duration of the operation. This table has 13M rows at time of writing.
How long would the index creation take? If it's more than a couple of seconds, we need to change it and make it non blocking.

with postgres, you can create an index concurrently, and then create a unique constraint using the existing index.

[BElluu]

CREATE UNIQUE INDEX teiserver_game_rating_logs_match_id_user_id_rating_type_id_index
ON teiserver_game_rating_logs (match_id, user_id, rating_type_id);

Creating index for that many rows, can take some minutes.

So maybe we should firstly manualy delete duplicates

DO $$
DECLARE deleted integer;
BEGIN
  LOOP
    WITH dupes AS (
      SELECT id FROM (
        SELECT id, ROW_NUMBER() OVER (
          PARTITION BY match_id, user_id, rating_type_id
          ORDER BY id ASC
        ) rn
        FROM teiserver_game_rating_logs
      ) s WHERE s.rn > 1
      LIMIT 10000
    )
    DELETE FROM teiserver_game_rating_logs t USING dupes WHERE t.id = dupes.id;
    GET DIAGNOSTICS deleted = ROW_COUNT;
    EXIT WHEN deleted = 0;
    PERFORM pg_sleep(0.1);
  END LOOP;
END $$;

and then create index concurently out of transaction without lock

defmodule Teiserver.Repo.Migrations.UniqueIndexRatingLogs do
  use Ecto.Migration

  @disable_ddl_transaction true
  @disable_migration_lock true

  def up do
    create unique_index(
             :teiserver_game_rating_logs,
             [:match_id, :user_id, :rating_type_id],
             concurrently: true
           )
  end

  def down do
    drop unique_index(
           :teiserver_game_rating_logs,
           [:match_id, :user_id, :rating_type_id],
           concurrently: true
         )
  end
end

What do you think ?