============================================= Internationalization & Localization Framework ============================================= Internationalization (also known as i18n) is the process of designing software so that it can be easily (i.e. without any source code modifications) adapted to various languages and regions. Localization (also known as L10n) is the process of adapting internationalized software for a specific language or region (e.g. by translating text, formatting date or time). Basics ====== Locale class ------------ Instances of ``\Neos\Flow\I18n\Locale`` class are fundamental for the whole i18n and L10n functionality. They are used to specify what language should be used for translation, how date and time should be formatted, and so on. They can be treated as simple wrappers for locale identifiers (like *de* or *pl_PL*). Many methods from the i18n framework accept Locale objects as a optional parameter - if not provided, the default ``Locale`` instance for a Flow installation will be used. You can create a ``Locale`` object for any valid locale identifier (specified by RFC 4646), even if it is not explicitly meant to be supported by the current Flow installation (i.e. there are no localized resources for this locale). This can be useful, because Flow uses the `Common Locale Data Repository`_ (CLDR), so each Flow installation knows how to localize numbers, date, time and so on to almost any language and region on the world. Additionally Flow creates a special collection of available ``Locale`` objects. Those can either be configured explicitly in settings via ``Neos.Flow.i18n.availableLocales`` or they are automatically generated by scanning the filesystem for any localized resources. You can use the i18n service API to obtain these verified ``Locale`` objects. .. note:: You can configure which folders Flow should scan for finding available locales through the ``Neos.Flow.i18n.scan.includePaths`` setting. This is useful to restrict the scanning to specific paths when you have a big file structure in your package ``Resources``. You can also exclude folders through ``Neos.Flow.i18n.scan.excludePatterns``. By default the ``Public`` and ``Private/Translations`` folders, except 'node_modules', 'bower_components' and any folder starting with a dot will be scanned. Locales are organized in a hierarchy. For example, *en* is a parent of *en_US* which is a parent of *en_US_POSIX*. Thanks to the hierarchical relation resources can be automatically shared between related resources. For example, when you request a *foobar* item for *en_US* locale, and it does not exist, but the item does exist for the *en* locale, it will be used. Common Locale Data Repository ----------------------------- Flow comes bundled with the CLDR (*Common Locale Data Repository*). It's an Unicode project with the aim to provide a systematic representation of data used for the localization process (like formatting numbers or date and time). The i18n framework provides a convenient API to access this data. .. note:: For now Flow covers only a subset of the CLDR data. For example, only the Gregorian calendar is supported for date and time formatting or parsing. Detecting user locale --------------------- The ``Detector`` class can be used for matching one of the available locales with locales accepted by the user. For example, you can provide the *AcceptLanguage* HTTP header to the ``detectLocaleFromHttpHeader()`` method, which will analyze the header and return the best matching ``Locale`` object. Also methods exist which accept a locale identifier or template ``Locale`` object as a parameter and will return a best match. Translating text ================ Translator class ---------------- The ``\Neos\Flow\I18n\Translator`` class is the central place for the translation related functionality. Two translation modes can be used: translating by original label or by ID. ``Translator`` also supports plural forms and placeholders. For ``translateByOriginalLabel()`` you need to provide the original (untranslated, source) message to be used for searching the translated message. It makes view templates more readable. ``translateById()`` expects you to provide the systematic ID (like *user.notRegistered*) of a message. Both methods accept the following optional arguments: * ``arguments`` - array of values which will replace corresponding placeholders * ``quantity`` - integer or decimal number used for finding the correct plural form * ``sourceName`` - name of source catalog to read the translation from. * ``packageKey`` of the package the source catalog is contained in. .. hint:: Translation by label is very easy and readable, but if you ever want to change the original text, you are in trouble. The use of IDs gives you more flexibility in that respect. Another issue: some labels do not contain their context, like "Name". What is meant here, a person's name or a category label? This can be solved by using IDs that convey the context (note that both could be "Name" in the final output): * party.person.fullName * blog.category.name We therefore recommend to use ``translationById()`` in your code. Plural forms ------------ The ``Translator`` supports plural forms. English has only two plural forms: *singular* and *plurals* but the CLDR defines six plural forms: *zero*, *one*, *two*, *few*, *many*, *other*. Though english only uses *one* and *other*, different languages use more forms (like *one*, *few*, and *other* for Polish) or less forms (like only *other* for Japanese). Sets of rules exist for every language defining which plural form should be used for a particular quantity of a noun. If no rules match, the implicit *other* rule is assumed. This is the only form existing in every language. If the catalogs with translated messages define different translations for particular plural forms, the correct form can be obtained by the ``Translator`` class. You just need to provide the ``quantity`` parameter - an integer or decimal number which specifies the quantity of a noun in the sentence being translated. Placeholders ------------ Translated messages (labels) can contain placeholders - special markers denoting he place where to insert a particular value and optional configuration on how to format it. The syntax of placeholders is very simple:: {id[,formatter[,attribute1[,attribute2...]]]} where: * *id* is an integer used to index the arguments to insert * *formatter* (optional) is a name of one of the Formatters_ to use for formatting the argument (if no formatter is given the provided argument will be cast to string) * *attributes* (optional) are strings directly passed to the ``Formatter``. What they do depends on the concrete ``Formatter`` which is being used, but generally they are used to specify formatting more precisely. Some examples: .. code-block:: none {0} {0,number,decimal} {1,datetime,time,full} 1. The first example would output the first argument (indexing starts with 0), simply string-casted. 2. The second example would use ``NumberFormatter`` (which would receive one attribute: *decimal*) to format first argument. 3. The third example would output the second argument formatted by the ``DatetimeFormatter``, which would receive two attributes: *time* and *full* (they stand for format *type* and *length*, accordingly). Formatters ---------- A ``Formatter`` is a class implementing the ``\Neos\Flow\I18n\Formatter\FormatterInterface``. A formatter can be used to format a value of particular type: to convert it to string in locale-aware manner. For example, the number *1234.567* would be formatted for French locale as *1 234,567*. It is possible to define more elements than just the position and symbols of separators. Together with placeholders, formatters provide robust and easy way to place formatted values in strings. But formatters can be used directly (i.e. not in placeholder, but in your class by injection), providing you more control over the results of formatting. The following formatters are available in Flow by default: ``\Neos\Flow\I18n\Formatter\NumberFormatter`` Formats integers or floats in order to display them as strings in localized manner. Uses patterns obtained from CLDR for specified locale (pattern defines such elements like minimal and maximal size of decimal part, symbol for decimal and group separator, etc.). You can indirectly define a pattern by providing format type (first additional attribute in placeholder) as *decimal* or *percent*. You can also manually set the pattern if you use this class directly (i.e. not in placeholder, but in your class by injection). ``\Neos\Flow\I18n\Formatter\DatetimeFormatter`` Formats date and / or time part of PHP ``\DateTime`` object. Supports most of very extensive pattern syntax from CLDR. Has three format types: *date*, *time*, and *datetime*. You can also manually set the pattern if you use this class directly. The following parameters are generally accepted by Formatters' methods: * ``locale`` - formatting result depends on the localization, which is defined by provided ``Locale`` object * ``formatLength`` (optional) - CLDR provides different formats for *full*, *long*, *medium*, *short*, and *default* length Every formatter provides few methods, one for each format type. For example, ``NumberFormatter`` has methods ``formatDecimalNumber()`` - for formatting decimals and integers - and ``formatPercentNumber()`` - for percentage (parsed value is automatically multiplied by 100). You can create your own formatter class which will be available for use in placeholders. Just make sure your class implements the ``\Neos\Flow\I18n\Formatter\FormatterInterface``. Use the fully qualified class name, without the leading backslash, as formatter name:: {0,Acme\Foobar\Formatter\SampleFormatter} Translation Providers --------------------- Translation providers are classes implementing the ``TranslationProviderInterface``. They are used by the ``Translator`` class for accessing actual data from translation files (message catalogs). A ``TranslationProvider``'s task is to read (understand) the concrete format of catalogs. Flow comes with one translation provider by default: the ``XliffTranslationProvider``. It supports translations stored in `XLIFF message catalogs`_, supports plural forms, and both translation modes. You can create and use your own translation provider which reads the file format you need, like *PO*, *YAML* or even *PHP* arrays. Just implement the interface mentioned earlier and use the *Objects.yaml* configuration file to set your translation provider to be injected into the ``Translator``. Please keep in mind that you have to take care of overrides yourself as this is within the responsibilities of the translation provider. Fluid ViewHelper ---------------- There is a ``TranslateViewHelper`` for Fluid. It covers all ``Translator`` features: it supports both translation modes, plural forms, and placeholders. In the simplest case, the ``TranslateViewHelper`` can be used like this: .. code-block:: xml It will output the translation with the ID "label.id" (corresponding to the trans-unit id in XLIFF files). The ``TranslateViewHelper`` also accepts all optional parameters the ``Translator`` does. .. code-block:: xml It will translate the label using *someLabelsCatalog*. Then it will insert string casted value "*foo*" in place of *{0}* and localized formatted *99.9* in place of *{1,number}*. Translation by label is also possible: .. code-block:: xml Unregistered User It will output the translation assigned to *user.unregistered* key. When the translation for particular label or ID is not found, value placed between ```` and ```` tags will be displayed. Localizing validation error messages ==================================== Flow comes with a bundle of translations for all basic validator error messages. To make use of these translations, you have to adjust your templates to make use of the ``TranslateViewHelper``. .. code-block:: xml {error -> f:translate(id: error.code, arguments: error.arguments, package: 'Neos.Flow', source: 'ValidationErrors')} If you want to change the validation messages, you can use your own package and override the labels there. See the "XLIFF file overrides" section below. .. tip:: If you want to have different messages depending on the property, for example if you want to be more elaborate about specific validation errors depending on context, you could add the property to the translate key and provide your own translations. Localizing resources ==================== Resources can be localized easily in Flow. The only thing you need to do is to put a locale identifier just before the extension. For example, *foobar.png* can be localized as *foobar.en.png*, *foobar.de_DE.png*, and so on. This works with any resource type when working with the Flow ResourceManagement. Just use the ``getLocalizedFilename()`` of the i18n ``Service`` singleton to obtain a localized resource path by providing a path to the non-localized file and a ``Locale`` object. The method will return a path to the best matching localized version of the file. Fluid ViewHelper ---------------- The ``ResourceViewHelper`` will by default use locale-specific versions of any resources you ask for. If you want to avoid that you can disable that: .. code-block:: xml {f:uri.resource(path: 'header.png', localize: 0)} Validating and parsing input ============================ Validators ---------- A validator is a class implementing ``ValidatorInterface`` and is used by the Flow Validation Framework for assuring correctness of user input. Flow provides two validators that utilize i18n functionality: ``\Neos\Flow\Validation\Validator\NumberValidator`` Validates decimal and integer numbers provided as strings (e.g. from user's input). ``\Neos\Flow\Validation\Validator\DateTimeValidator`` Validates date, time, or both date and time provided as strings. Both validators accept the following options: *locale*, *strictMode*, *formatType*, *formatLength*. These validators are working on top of the parsers API. Please refer to the Parsers_ documentation for details about functionality and accepted options. Parsers ------- A Parsers' task is to read user input of particular type (e.g. number, date, time), with respect to the localization used and return it in a form that can be further processed. The following parsers are available in Flow: ``\Neos\Flow\I18n\Parser\NumberParser`` Accepts strings with integer or decimal number and converts it to a float. ``\Neos\Flow\I18n\Parser\DatetimeParser`` Accepts strings with date, time or both date and time and returns an array with date / time elements (like day, hour, timezone, etc.) which were successfully recognized. The following parameters are generally accepted by parsers' methods: * *locale* - formatting results depend on the localization, which is defined by the provided ``Locale`` object * *formatLength* - CLDR provides different formats for *full*, *long*, *medium*, *short*, and *default* length * *strictMode* - whether to work in *strict* or *lenient* mode Parsers are complement to Formatters_. Every parser provides a few methods, one for each format type. Additionally each parser has a method which accepts a custom format (pattern). You can provide your own pattern and it will be used for matching input. The syntax of patterns depends on particular parser and is the same for a corresponding formatter (e.g. ``NumberParser`` and ``NumberFormatter`` support the same pattern syntax). Parsers can work in two modes: *strict* and *lenient*. In *strict* mode, the parsed value has to conform the pattern exactly (even literals are important). In *lenient* mode, the pattern is only a "base". Everything that can be ignored will be ignored, some simplifications in the pattern are done. The parser tries to do it's best to read the value. XLIFF message catalogs ====================== The primary source of translations in Flow are XLIFF message catalogs. `XLIFF `_, the *XML Localisation Interchange File Format* is an `OASIS-blessed `_ standard format for translations. .. note:: In a nutshell an XLIFF document contains one or more ```` elements. Each file element usually corresponds to a source (file or database table) and contains the source of the localizable data. Once translated, the corresponding localized data for one, and only one, locale is added. Localizable data are stored in ```` elements. The ```` contains a ```` element to store the source text and a (non-mandatory) ```` element to store the translated text. File locations and naming ------------------------- Each Flow package may contain any number of XLIFF files. The location for these files is the *Resources/Private/Translations* folder. The files there can be named at will, but keep in mind that *Main* is the default catalog name. The target locale is then added as a directory hierarchy in between. The minimum needed to provide message catalogs for the *en* and *de* locales thus would be: .. code-block:: text Resources/ Private/ Translations/ en/ Main.xlf de/ Main.xlf XLIFF file creation ------------------- It is possible to create initial translation files for a given language. With Flow command .. code-block:: bash ./flow kickstart:translation --package-key Some.Package --source-language-key en --target-language-keys "de,fr" the files for the default language *english* in the package *Some.Package* will be created as well as the translation files for *german* and *french*. Already existing files will not be overwritten. Translations that do not yet exist are generated based on the default language. A minimal XLIFF file looks like this: .. code-block:: xml Skarhøj Sarkosh If possible you should set up your editor to use the XLIFF 1.2 strict schema to validate the files you are working on. .. note:: When using ``translationById()`` the framework will check the catalog's source language against the currently needed locale and use the ```` element if no ```` element is found. This eliminates the need to duplicate messages in catalogs where source and target language are the same. But you may still ask yourself *do I really need to duplicate all the strings in XLIFF files?* The answer is *you should*. Using target allows to fix typos or change wording without breaking translation by label for all other languages. .. admonition:: How to create meaningful XLIFF ids When using the recommended way of translating by id, it is even more important to use meaningful identifiers. Our suggestion is to group identifiers and use dot notation to build a hierarchy that is meaningful and intuitive:: settings.account.keepLoggedIn settings.display.compactControls book.title book.author … Labels may contain placeholders to be replaced with given arguments during output. Earlier we saw an example use of the TranslateViewHelper: .. code-block:: xml The corresponding XLIFF files will contain placeholders in the source and target strings: .. code-block:: xml Untranslated {0} and {1,number} Übersetzung mit {1,number} und {0} As you can see, placeholders may be reordered in translations if needed. Plural forms in XLIFF files --------------------------- Plural forms are also supported in XLIFF. The following example defines a string in two forms that will be used depending on the count: .. code-block:: xml This is only {0} item. Dies ist nur {0} Element. These are {0} items. Dies sind {0} Elemente. Please be aware that the number of the available plural forms depends on the language! If you want to find out which plural forms are available for a locale you can have a look at *Neos.Flow/Resources/Private/I18n/CLDR/Sources/supplemental/plurals.xml* XLIFF file translation ---------------------- To translate XLIFF files you can use any text editor, but translation is a lot easier using one the available translation tools. To name two of them: Virtaal is a free and open-source tool for offline use and Pootle (both from the `Translate Toolkit `_ project) is a web-based translation server. XLIFF can also easily be converted to *PO* file format, edited by well known *PO* editors (like *Poedit*, which supports plural forms), and converted back to *XLIFF* format. The *xliff2po* and *po2xliff* tools from the *Translate Toolkit* project can convert without information loss. XLIFF file overrides -------------------- As of Flow 4.2, XLIFF files are no longer solely identified by their location in the file system. Instead, the ````'s ``product-name`` and ``original`` attributes are evaluated to the known ``package`` and ``source`` properties, if given. The actual location in the file system is only taken into account if this information is missing and mainly for backwards compatibility. This allows for an override mechanism, which comes in two levels: Package-based overrides ^^^^^^^^^^^^^^^^^^^^^^^ Translation files are assembled by collecting labels along the composer dependency graph. This means that as long a package depends (directly or indirectly) on another package, it can override or enrich the other package's XLIFF files by using the other package's ``product-name`` and ``original`` values. .. note:: If you have trouble overriding another package's translations, please check your ``composer.json`` if you correctly declared that package as a dependency. Global translations overrides ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In case translations are provided by another source than packages (e.g. via import from a third party system), a global translation path can be declared and is evaluated with highest priority in that it overrides all translations provided by packages. The default value for this is ``Data/Translations`` and can be changed via the configuration parameter .. code-block:: yaml Neos: Flow: i18n: globalTranslationPath: '%FLOW_PATH_DATA%Translations/' **Example** ``Packages/Framework/Neos.Flow/Resources/Private/Translations/en/ValidationErrors.xlf`` .. code-block:: xml Only regular characters (a to z, umlauts, ...) and numbers are allowed. ``Packages/Application/Acme.Package/Resources/Private/Translations/en/ValidationErrors.xlf`` .. code-block:: xml Whatever translation more appropriate to your domain comes to your mind. .. note:: In case of undetected labels, please make sure the ``original`` and ``product-name`` attributes are properly set (or not at all, if the file resides in the matching directory). Since these fields are used to detect overrides, they are now meaningful and cannot be filled arbitrarily any more.