Localization Support

[MohabCodeX]

✅ tasks

• fix tinygettext for mac/linux.
• add global translation support.
• set/getCurrentTranslationLanguage maybe need to be compatible with getLocalization.
• overview documentation.

This PR adds localization support for MTA resources using .po files. Resources can include translation files directly in their meta.xml and use functions to get translated text.

Addresses Issue

Resolves the localization framework request in [mtasa-resources#361] (Thanks to jlillis for the suggestion), enabling multi-language support instead of a single language.

Features

• Add <translation src="locales/en.po" primary="true" /> tags in meta.xml
• Fallback to primary language, then first imported language when translations are missing

New Functions

Server-Side:

• getTranslation(msgid [, lang]) → string
• getAvailableTranslations() → table

Client-Side:

• getTranslation(msgid [, lang]) → string
• getCurrentTranslationLanguage() → string
• setCurrentTranslationLanguage(lang) → boolean
• getAvailableTranslations() → table

How it works

Resources place .po files in a locales/ folder (or any folder inside the resource) and declare them in meta.xml. The system loads translations on resource start. Missing translations fallback to the primary language, then to the first imported language, or finally the original message ID. The primary language is set with primary="true"; otherwise, the first translation file becomes the primary.

Details

• Built on MTA's existing tinygettext library
• Separate dictionaries prevent language data corruption
• Zero network overhead; translations cached locally on both client and server

Testing

test_localization.zip

Additional translation functions will be added in future updates to expand functionality.

[samr46]

Great job! This solution should be more efficient than Lua based options.

I'm wondering if it's feasible to add this functionality on client level (so translations could be shared across all resources). This way complex projects can use a single resource to load translation files and avoid using exports / call.
Language key collisions can be handled in the order of loading (so the keys from last loaded file overwrite the existing ones).
Basically a global="true" scope option for .po files.

[ghost]

Personally, I'd rather implement this in Lua instead of directly in the project. It's easy to do there, and not every server actually needs it. Even if some do, the PO file based setup feels outdated now and probably wouldn't match what a lot of folks prefer.

[ghost]

Great job! This solution should be more efficient than Lua based options.

I'm wondering if it's feasible to add this functionality on client level (so translations could be shared across all resources). This way complex projects can use a single resource to load translation files and avoid using exports / call. Language key collisions can be handled in the order of loading (so the keys from last loaded file overwrite the existing ones). Basically a global="true" scope option for .po files.

The overhead of doing this in Lua is actually pretty minimal, especially for most use cases. You can load translations once in a resource and share them with others using call or even loadstring if needed.

[MohabCodeX]

The overhead of doing this in Lua is actually pretty minimal, especially for most use cases. You can load translations once in a resource and share them with others using call or even loadstring if needed.

What about exploring another solution instead of using Lua? I still believe there might be an option that satisfies both.

[FileEX]

Personally, I think this is a very good idea. It’s true that this can be achieved using regular Lua tables, but using .po files seems like a better, more appropriate, and professional solution, as it allows integration with automated platforms like Crowdin.

Additionally, it’s much simpler and easier than building an entire translation system from scratch. If someone prefers their own Lua-based solution, they can still stick with it—nobody is forced to use .po files. Using .po files is by no means outdated; the format is still widely used in the translation industry, and I see no reason why it couldn’t be used for translations on your server. It’s fully sufficient and straightforward.

As we can see from the reactions, this PR has been received positively.

The only thing I personally miss is some kind of automatic scaling option. Unfortunately, even MTA doesn’t currently have this feature. Specifically, I mean something like what’s used in GTA code—different languages have different characters; a short word in one language might be long in another, not to mention entire sentences. As a result, content in different languages is often cut off or not fully visible. For example, buttons may be too narrow to fit the same text in various languages (like in the admin panel). I’m not sure if this should be a separate PR or part of this one, or maybe it’s something that should be handled independently in Lua.

[MohabCodeX]

Server-Side Functions

1. getTranslation(msgid, language)

Side: Server
Parameters:

• msgid (string): The message ID to translate
• language (string, optional): Target language code (e.g., "en_US", "es_ES"). If empty, uses primary language

Returns: string - Translated message or original msgid if translation not found

Usage:

local translated = getTranslation("Welcome to the server", "es_ES")
-- Returns: "Bienvenido al servidor" (if Spanish translation exists)

2. getAvailableTranslations()

Side: Server
Parameters: None

Returns: table - Array of available language codes for the current resource

Usage:

local languages = getAvailableTranslations()
-- Returns: {"en_US", "es_ES", "fr_FR", "de_DE"}

3. getGlobalTranslationProviders()

Side: Server
Parameters: None

Returns: table - Array of resource names that provide global translations

Usage:

local providers = getGlobalTranslationProviders()
-- Returns: {"common-translations", "ui-translations"}

4. isResourceGlobalTranslationProvider(resource)

Side: Server
Parameters:

• resource (resource): The resource to check

Returns: boolean - True if the resource is a global translation provider

Usage:

local isProvider = isResourceGlobalTranslationProvider(getResourceFromName("common-translations"))
-- Returns: true if the resource provides global translations

5. getResourceGlobalTranslationProviders(resource)

Side: Server
Parameters:

• resource (resource, optional): The resource to check. If not provided, uses current resource

Returns: table - Array of global translation providers for the specified resource

Usage:

local providers = getResourceGlobalTranslationProviders()
-- Returns: {"common-translations", "ui-translations"}

Client-Side Functions

1. setCurrentTranslationLanguage(language)

Side: Client
Parameters:

• language (string): Language code to set as current (e.g., "en_US", "es_ES")

Returns: boolean - True if language was successfully set

Usage:

local success = setCurrentTranslationLanguage("es_ES")
if success then
    -- Language changed successfully
    -- onClientTranslationLanguageChange event will be triggered
end

2. getCurrentTranslationLanguage()

Side: Client
Parameters: None

Returns: string - Current language code

Usage:

local currentLang = getCurrentTranslationLanguage()
-- Returns: "es_ES" (current language)

3. getTranslation(msgid, language)

Side: Client
Parameters:

• msgid (string): The message ID to translate
• language (string, optional): Target language code. If empty, uses current client language

Returns: string - Translated message or original msgid if translation not found

Usage:

local translated = getTranslation("Hello world")
-- Uses current client language
-- Returns: "Hola mundo" (if current language is Spanish)

4. getAvailableTranslations()

Side: Client
Parameters: None

Returns: table - Array of available language codes for the current resource

Usage:

local languages = getAvailableTranslations()
-- Returns: {"en_US", "es_ES", "fr_FR"}

---

Fallback Cases

1. Translation Not Found

• Behavior: Returns original msgid
• Logging: No warning (normal behavior)
• Example: getTranslation("Unknown message") → "Unknown message"

2. Language Not Available

• Behavior: Falls back to primary language, then to original msgid
• Logging: Warning logged with fallback information
• Example: Requesting "de_DE" but only "en_US" available → uses "en_US" (if it's the primary)

3. Invalid Language Code

• Behavior: Falls back to primary language
• Logging: Error logged with validation failure
• Example: "invalid-lang" → falls back to primary language

4. Translation System Not Initialized

• Behavior: Returns original msgid
• Logging: Warning logged with resource name
• Example: Resource without translation manager → returns original text

5. Empty Message ID

• Behavior: Returns empty string
• Logging: No logging (normal behavior)
• Example: getTranslation("") → ""

6. Global Translation Fallback

• Behavior: Tries local translation first, then global providers in order
• Logging: No logging (normal behavior)
• Example: Local not found → tries global providers → returns first match

7. Complete Fallback Hierarchy

The system follows this specific fallback order:

1. Requested language (if valid and available)
2. Primary language (if requested language not available)
3. Global providers (if local translation not found)
4. Primary language via global providers (if requested language differs from primary)
5. Original msgid (if no translation found anywhere)

---

Debug/Error Cases

1. Translation File Loading Errors

File Not Found

• Error: "Translation file not found: /path/to/file.po"
• Severity: Error
• Action: Resource fails to start
• Debug Info: Full file path, resource name

Invalid PO File Format

• Error: "Exception loading translation file '/path/to/file.po': [exception details]"
• Severity: Error
• Action: Resource fails to start
• Debug Info: File path, exception message

File Validation Failure

• Error: "Could not open translation file: /path/to/file.po"
• Severity: Error
• Action: Resource fails to start
• Debug Info: File path, resource name

2. Language Validation Errors

Invalid Language Code

• Error: "Invalid language 'invalid-lang' - use standard locale format (e.g., en_US, es_ES)"
• Severity: Error
• Action: Falls back to primary language
• Debug Info: Invalid language code, resource name

Language Not Available

• Warning: "Language 'de_DE' not available, falling back to primary language 'en_US'"
• Severity: Warning
• Action: Uses primary language
• Debug Info: Requested language, fallback language

3. System Initialization Errors

Translation Manager Not Available

• Warning: "Translation system not initialized for resource 'my-resource'"
• Severity: Warning
• Action: Returns original text
• Debug Info: Resource name, function context

Global Provider Registration Failure

• Error: "Global translation: Translation manager not available for resource 'my-resource' when adding provider 'common-translations'"
• Severity: Error
• Action: Global provider not registered
• Debug Info: Resource name, provider name

4. Resource Configuration Errors

Empty Global Translation Provider

• Error: "Global translation: Empty provider resource name in meta.xml for resource 'my-resource'. Check your <global-translation src=\"...\"/> tag."
• Severity: Error
• Action: Global translation not loaded
• Debug Info: Resource name, XML context

Duplicate Global Provider Registration

• Warning: "Global translation provider 'common-translations' is already registered, replacing with new instance. This might indicate a resource restart or duplicate provider registration."
• Severity: Warning
• Action: Replaces existing provider
• Debug Info: Provider name, context

5. Client-Side Errors

Language Change Failure

• Warning: "Translation system not initialized for resource 'my-resource'"
• Severity: Warning
• Action: Returns false
• Debug Info: Resource name

Empty Language Code

• Warning: "setCurrentTranslationLanguage called with empty language code"
• Severity: Warning
• Action: No change made
• Debug Info: Function context

6. TinyGetText Library Errors

Dictionary Parsing Errors

• Error: "TinyGetText Error: [error message]"
• Severity: Error
• Action: Translation loading fails
• Debug Info: Error message, file path

Dictionary Warnings

• Warning: "TinyGetText Warning: [warning message]"
• Severity: Warning
• Action: Continues with warnings
• Debug Info: Warning message, file path

---

Resource Configuration

Meta.xml Configuration

Translation Files

<file src="translations/en_US.po" type="translation" />
<file src="translations/es_ES.po" type="translation" primary="true" />
<file src="translations/fr_FR.po" type="translation" />

Global Translation Provider

<global-translation-provider />

Global Translation Consumer

<file src="common-translations" type="global-translation" />

File Structure

my-resource/
├── meta.xml
├── script.lua
└── translations/
    ├── en_US.po
    ├── es_ES.po
    └── fr_FR.po

---

File Naming Conventions and Rules

Translation File Naming

Language Code Extraction

• Rule: Language code is extracted from the filename stem (filename without extension)
• Method: Uses std::filesystem::path::stem() to extract language code
• Examples:
  • en_US.po → Language: en_US
  • translations/es_ES.po → Language: es_ES
  • lang/fr_FR.po → Language: fr_FR

Supported File Extensions

• Primary: .po (GNU gettext Portable Object files)
• Template: .pot (GNU gettext Portable Object Template files)
• Note: Only .po files are loaded at runtime, .pot files are for translation templates

File Validation Rules

1. File must exist on the filesystem
2. File must not be empty (size > 0 bytes)
3. File must end with newline (\n) character
4. File must be valid PO format (parsed by tinygettext library)

Language Code Rules

Standard Format

• Format: {language}_{COUNTRY} (ISO 639-1 + ISO 3166-1)
• Examples: en_US, es_ES, fr_FR, de_DE, ja_JP

Legacy Support

The system automatically converts old 2-letter codes to full locale format:

• en → en_US
• fi → fi_FI
• az → az_AZ
• ka → ka_GE

Validation Process

1. Empty check: Empty language codes are rejected
2. Legacy conversion: 2-letter codes converted to full format
3. TinyGetText validation: Uses tinygettext::Language::from_name() for validation
4. Validation fallback: If validation fails, returns empty string (no fallback to en_US)

Note: The en_US fallback in the validation function is only for internal TinyGetText validation - if even en_US validation fails, the function returns empty string, which means the language code is invalid.

Primary Language Rules

Primary Language Selection

• XML Attribute: primary="true" in meta.xml
• Auto-selection: If no primary specified, first loaded language becomes primary
• Single primary: Only one language can be marked as primary per resource
• Fallback behavior: Primary language used when requested language not available

Meta.xml Configuration

<!-- Primary language (fallback) -->
<file src="translations/en_US.po" type="translation" primary="true" />

<!-- Secondary languages -->
<file src="translations/es_ES.po" type="translation" />
<file src="translations/fr_FR.po" type="translation" />

Global Translation Rules

Provider Resource Naming

• Format: Resource name (as specified in src attribute)
• Validation: Must be valid resource name (no path separators)
• Examples: common-translations, ui-translations, core-messages

Consumer Configuration

<!-- Reference to provider resource -->
<file src="common-translations" type="global-translation" />
<file src="ui-translations" type="global-translation" />

File Encoding and Charset

Default Encoding

• Charset: UTF-8 (default)
• Constructor: CResourceTranslationManager(resourceName, "UTF-8")
• PO Files: Must be UTF-8 encoded
• Validation: TinyGetText library handles encoding validation

Character Support

• Full Unicode: Supports all Unicode characters
• Special Characters: Accents, umlauts, Cyrillic, Arabic, Chinese, etc.
• Line Endings: Unix-style (\n) required

Resource File Types

Translation File Type

• Enum: RESOURCE_FILE_TYPE_TRANSLATION
• Class: CResourceTranslationItem
• Purpose: Local resource translation files

Global Translation File Type

• Enum: RESOURCE_FILE_TYPE_GLOBAL_TRANSLATION
• Class: CGlobalTranslationItem
• Purpose: References to global translation providers

Directory Structure Best Practices

Recommended Structure

my-resource/
├── meta.xml
├── script.lua
├── translations/
│   ├── en_US.po          # Primary (fallback)
│   ├── es_ES.po          # Spanish
│   ├── fr_FR.po          # French
│   └── de_DE.po          # German
└── other-files/

Alternative Structures

# Flat structure
my-resource/
├── meta.xml
├── en_US.po
├── es_ES.po
└── script.lua

# Nested structure
my-resource/
├── meta.xml
├── lang/
│   ├── translations/
│   │   ├── en_US.po
│   │   └── es_ES.po
└── script.lua

Error Handling for File Issues

File Not Found

• Error: "Translation file not found: /path/to/file.po"
• Action: Resource fails to start
• Debug: Full file path provided

Invalid Language Code

• Error: "Invalid language code 'invalid-lang'"
• Action: File not loaded
• Debug: Extracted language code shown

PO File Validation Failure

• Error: "Exception loading translation file '/path/to/file.po': [exception details]"
• Action: Resource fails to start
• Debug: Exception message from TinyGetText parser

Missing Primary Language

• Behavior: First loaded language becomes primary
• Warning: No explicit warning (normal behavior)
• Fallback: Uses first available language

---

Events

Client-Side Events

onClientTranslationLanguageChange

Triggered: When client language is changed via setCurrentTranslationLanguage()
Parameters:

• oldLanguage (string): Previous language code
• newLanguage (string): New language code

Usage:

addEventHandler("onClientTranslationLanguageChange", root, function(oldLang, newLang)
    outputChatBox("Language changed from " .. oldLang .. " to " .. newLang)
end)

---

Performance Considerations

Memory Usage

• Each loaded translation file consumes memory for dictionary storage
• Global translation providers are cached in memory
• Language switching is lightweight (no file reloading)

Thread Safety

• All translation operations are thread-safe
• Global provider registration uses mutex protection
• Client-side operations are safe for concurrent access

File I/O

• Translation files are loaded once during resource startup
• No file I/O during translation lookups
• PO files are parsed using efficient streaming parser