Fix measuring time during work item migration

[satano]

A have already fixed this in PR #2706 but it was broken (basically reverted) back in #2712. Now the processing of 10 000 items says, it will finish in 4 minutes and this time rapidly decreases – after processing 1000 items, the remainig time is only only 1 minute.

There are two main problems.

The duration of activity is not measured, but set to fixed time of 10 seconds using activity?.SetEndTime(activity.StartTimeUtc.AddSeconds(10));. I assume, that you did it because the Duration was zero when you accessed it when calculating average. This is because the Duration is not dynamic – it will not retur actual duration of activity. It is possible to set it directly to some fixed time with SetEndTime. But in our case when we need real duration of activity, we must stop it. When activity.Stop() is called, Duration is set to real time it has taken and so we need to stop the activity before we access Duration.

The second problem si calculating average and remaining time. In description of #2712 you wrote:

Removed the ProgressTimer class and associated logic, replacing it with simpler calculations for average and remaining time during work item processing.

It is not possible to count average and remaining time during work item processing, because that is processing of just single one work item – that means, the duration of activity you count with is just the duration of processing 1 item. There is no information about how long all the processing already took.

So say the processing is stable and to process one item always takes 1 second. This value is activity.Duration. Now you calculate average as:

var average = new TimeSpan(0, 0, 0, 0, (int)(activity.Duration.TotalMilliseconds / _current));

So if first item was processed, average time is 1 / 1 that means 1 s. If second item is processed, average time is 1 / 2 = 0,5 s. After 10 processed item the average is 1 / 10 = 0,1 s. So every processed item decreases average time even if the processing takes always the same time.

We really need something above the one processed item and that is the ProgressTimer class. Because we need to track cumulative processing time of every item and calculate average as this cumulative time divided by processed items. And this class also handles retrying, so if the item is processed three times (two times failed), it is taken into account.