Better handling of images which are gone

[benoit74]

As StackExchange is getting older, it also becomes plagued with missing images (images which are really gone from where they were).

This true for all domains, but is of course very visible on stackoverflow.com tasks. For instance on 3dprinting.stackexchange.com we have 9 images (out of 6504) which are gone.

This means the scraper is spending time to retrieve images we somehow "know" are gone. And we fill logs which warnings which are somehow "normal". Images might reappear at some point, but there is probably very negligible chance it will happen, especially at the same URL.

If feel like we should better handle this, two-fold:

• first, to enhance scraper behavior, we should have a mechanism for negative cache in S3 optimization cache: store there images we know are gone
• second, to enhance final ZIM, we should have a mechanism to replace missing images with some image of our own indicating that image is gone from StackExchange (instead of a broken image)

I'm not really sure how to implement both, but at least now we have an issue opened about the problem.

[MVoloshin]

P.S. Can we somehow fetch such images from older snapshots, ZIMs, etc (that were taken when these images were still available)?

[benoit74]

P.S. Can we somehow fetch such images from older snapshots, ZIMs, etc (that were taken when these images were still available)?

While a good idea, this is typically the kind of move we tend to avoid because way too complex to maintain in the long run. This is the kind of move we prefer to push "upstream" to website so it benefits everybody (not only offline readers) and we have nothing to maintain on our side. StackExchange could indeed use old ZIMs, wayback machine, ... to retrieve these missing images and re-upload them somewhere else.