Language preference not working

[marco-calautti]

I have setup the language preference to some foreign languages. Looking at prowlarr's search historty, I can see that comet-uncached is able to find some matches, however, I do not see any of them in stremio.

[Zaarrg]

@marco-calautti
This is probably because the found results, in those different languages, are not cached.
You can check in your logs what exactly is happening. You will see how many hashes where found and how many of those where cached and ultimately shown.
If you want the uncached ones of that found language to still show you will need to check what indexer the results are from and select that indexer in the config for uncached indexer.

Also this might be caused if you only have the Language Preference set. You might have to try to set the Language Search option aswell to allow for the title matching to match those other languages properly.

If the above does not help i would appreciate if you could post the logs of comet after the issue happened.

[marco-calautti]

All my indexers are also added to the uncached list, so it is not a problem of uncached files. I also tried the language search option, but it did not work. The point is that by simply searching for "<movie/series name> SXX <lang tag>" in prowlarr I find much more results than the ones that comet is able to find.

Maybe the search strings that comet uses can be improved? As an example, I can see from the prowlarr's logs of comet's requests, that the queries are either extremely specific, like:
"<series name> SXXEYY",
and when these fail, it immediately falls back to just
"<series name>"

[marco-calautti]

I think adding the language tag in the search query directly would be a better way to retrieve content in foreign language.

[Zaarrg]

All my indexers are also added to the uncached list, so it is not a problem of uncached files. I also tried the language search option, but it did not work. The point is that by simply searching for "<movie/series name> SXX " in prowlarr I find much more results than the ones that comet is able to find.

Maybe the search strings that comet uses can be improved? As an example, I can see from the prowlarr's logs of comet's requests, that the queries are either extremely specific, like: " SXXEYY", and when these fail, it immediately falls back to just ""

I think adding the language tag in the search query directly would be a better way to retrieve content in foreign language.

I see. Will take a look and add the language tag thing.
I only really use jackett and there with the indexers i tried " translated series name + SXXEYY " worked fine.

Didnt really consider the language tag. As language preference only sorts the selected languages to the top.

And Language Search adds to the default search "series name + SXXEYY" the search for "translated series name + SXXEYY" and then also uses that translated name as a possible title so it can pass the title match check.

[marco-calautti]

How do you get the translated name?

[Zaarrg]

Its the Language Search option. Any language selected there will also be searched in. Meaning if for example u select German there it will search for spongebob squarepants and spongebob schwammkopf.
You can see also what titles it uses in the logs.

Maybe the search strings that comet uses can be improved? As an example, I can see from the prowlarr's logs of comet's requests, that the queries are either extremely specific, like: " SXXEYY", and when these fail, it immediately falls back to just ""

Also if i understand this correctly, u want it to search for "series name SXXEYY" and for example if German is selected for
"series name German SXXEYY" ?

[marco-calautti]

What I mean is: where do you get the official series translated name from? Still from IMDb?

I would say that I got the best results in general by trying first:
Translatedseriesname SXXEYY

If this does not work, probably the problem is the string SXXEYY, so the next best option is:
Translatedseriesname SXX (this is useful of course if you search inside the pack for the episode)

If also that does not work, then:
Originalseriesname SXXEYY (or just SXX) langtag

[Zaarrg]

Yeah the translated titles are from imdb. Uses the imdb_id that is provided by stremio and is basically used for everything other then anime kitsu.

I would say that I got the best results in general by trying first: Translatedseriesname SXXEYY

If this does not work, probably the problem is the string SXXEYY, so the next best option is: Translatedseriesname SXX (this is useful of course if you search inside the pack for the episode)

If also that does not work, then: Originalseriesname SXXEYY (or just SXX) langtag

Seems good enough. Will test and maybe add that.
But not sure if it will make a big difference.
As currently if it searches when a language for searching is selected it would search those three:

seriesname
seriesname SXXEYY
Translatedseriesname
Translatedseriesname SXXEYY

Only a langtag would be missing.
Idk also about the Translatedseriesname SXX with just the season because when searching for Translatedseriesname the results probably also include anything with Translatedseriesname SXX or Translatedseriesname SXXEYY

Maybe the issue is this:
You could check what prowlarr returned when comet was searching. I think the history includes also the result.

And then compare in the logs how many torrents, infohashes it found. Will be like this
stream.stream - 267 torrents found for Interstellar with jackett, Zilean, Torrentio

And then check the infohash log, as those are the torrents that had a hash returned and will be checked if are cached.
stream.stream - 128 info hashes found for Interstellar

Lastly check what it says for found cached/uncached torrents:
stream.stream - 22 cached files found on debridlink for Interstellar
general.add_uncached_files - 28 uncached files found on torrentio for Interstellar

Cause i know there is some issue in the step from torrents to infohash using jackett or prowlarr. Cause i often times had a similar issue where the search query on jackett with the translated name would return the correct torrents but because there was no infoHash returned comet basically throws them away.

[victorgveloso]

Hey there, just to add my 2 cents.

I recently implemented a bypass for this kind of tag search in another add-on (stremio-jackett) because it's way worse than just using the translated title in most of the indexers I tested. In my experience, indexers' support for advanced search is inconsistent and unreliable.

On the other hand, I've noticed most add-ons rely on tmdb for resolving translated titles while yours is the only one relying on IMDb. I wonder if TMDB supports more language options. For instance, it supports country code (e.g., en-GB, en-US, pt-PT, pt-BR, es-ES). What languages do imdb support?

[Zaarrg]

I see, well if you have any recommendation im happy to hear it. As im still thinking about it and what the best way to fix this would be.

Well it supports every language listed under the Language tab on the web ui with both language code and Country Code.
These are current tables in use:

    lang_to_country = {
        "en": "US", "ja": "JP", "zh": "CN", "ru": "RU", "ar": "SA", "pt": "PT",
        "es": "ES", "fr": "FR", "de": "DE", "it": "IT", "ko": "KR", "hi": "IN",
        "bn": "BD", "pa": "IN", "mr": "IN", "gu": "IN", "ta": "IN", "te": "IN",
        "kn": "IN", "ml": "IN", "th": "TH", "vi": "VN", "id": "ID", "tr": "TR",
        "he": "IL", "fa": "IR", "uk": "UA", "el": "GR", "lt": "LT", "lv": "LV",
        "et": "EE", "pl": "PL", "cs": "CZ", "sk": "SK", "hu": "HU", "ro": "RO",
        "bg": "BG", "sr": "RS", "hr": "HR", "sl": "SI", "nl": "NL", "da": "DK",
        "fi": "FI", "sv": "SE", "no": "NO", "ms": "MY"
    }
    
    # Or this table:
    LANGUAGES_TRANSLATION_TABLE = {
    "en": "English", "ja": "Japanese", "zh": "Chinese", "ru": "Russian", "ar": "Arabic", "pt": "Portuguese",
    "es": "Spanish", "fr": "French", "de": "German", "it": "Italian", "ko": "Korean", "hi": "Hindi", "bn": "Bengali",
    "pa": "Punjabi", "mr": "Marathi", "gu": "Gujarati", "ta": "Tamil", "te": "Telugu", "kn": "Kannada", "ml": "Malayalam",
    "th": "Thai", "vi": "Vietnamese", "id": "Indonesian", "tr": "Turkish", "he": "Hebrew", "fa": "Persian", "uk": "Ukrainian",
    "el": "Greek", "lt": "Lithuanian", "lv": "Latvian", "et": "Estonian", "pl": "Polish", "cs": "Czech", "sk": "Slovak",
    "hu": "Hungarian", "ro": "Romanian", "bg": "Bulgarian", "sr": "Serbian", "hr": "Croatian", "sl": "Slovenian", "nl": "Dutch",
    "da": "Danish", "fi": "Finnish", "sv": "Swedish", "no": "Norwegian", "ms": "Malay", "la": "Latino"
}
    

I dont think there is any difference between imdb and tmdb. Only difference possible is imdb has less alternative titles then tmdb. But would be weird if imdb would be missing some that tmdb has.

I personally only really use private trackers for one specific language. Meaning all names listed there are the translated ones. Which is basically also why i added this feature as searching with the original name on jacket didnt return any results.

But like already said atm it only searches jackett/prowlarr with those:
seriesname
seriesname SXXEYY
Translatedseriesname
Translatedseriesname SXXEYY

Meaning there is no language tags added at all.

Edit:
Okay so checked out the tmdb api and what alternative titles it returns. And tmdb actually returns much less translated titles. For fight club 1999 i got from tmdb 16 and on imdb 55.
Also checked trakt api as this is what is used in the original comet repo for aliases and it also returns less with 47.

[victorgveloso]

Looks like you do a language to country mapping (as in "pt": "PT", "en": "US", etc.). The opposite would make more sense. As there are more countries than languages, e.g., we could map en to US, GB, CA, and AU, es to several countries, fr to FR and CA, and pt to PT and BR. As you can see here there are many titles that differ from Portugal to Brazil. As there might be similar differences in countries speaking other languages.

seriesname
seriesname SXXEYY
Translatedseriesname
Translatedseriesname SXXEYY

Honestly, besides the limited support for title translations, the implementation you described is optimal. Adding either advanced search filters or language tag in the search would worsen the existing behavior.