Internationalization & Localization Framework

Internationalization (also known as i18n) is the process of designing software so that it can be easily (i.e. without any source code modifications) adapted to various languages and regions. Localization (also known as L10n) is the process of adapting internationalized software for a specific language or region (e.g. by translating text, formatting date or time).

Basics

Locale class

Instances of \Neos\Flow\I18n\Locale class are fundamental for the whole i18n and L10n functionality. They are used to specify what language should be used for translation, how date and time should be formatted, and so on. They can be treated as simple wrappers for locale identifiers (like de or pl_PL). Many methods from the i18n framework accept Locale objects as a optional parameter - if not provided, the default Locale instance for a Flow installation will be used.

You can create a Locale object for any valid locale identifier (specified by RFC 4646), even if it is not explicitly meant to be supported by the current Flow installation (i.e. there are no localized resources for this locale). This can be useful, because Flow uses the Common Locale Data Repository (CLDR), so each Flow installation knows how to localize numbers, date, time and so on to almost any language and region on the world.

Additionally Flow creates a special collection of available Locale objects. Those can either be configured explicitly in settings via Neos.Flow.i18n.availableLocales or they are automatically generated by scanning the filesystem for any localized resources. You can use the i18n service API to obtain these verified Locale objects.

Note

You can configure which folders Flow should scan for finding available locales through the Neos.Flow.i18n.scan.includePaths setting. This is useful to restrict the scanning to specific paths when you have a big file structure in your package Resources. You can also exclude folders through Neos.Flow.i18n.scan.excludePatterns. By default the Public and Private/Translations folders, except ‘node_modules’, ‘bower_components’ and any folder starting with a dot will be scanned.

Locales are organized in a hierarchy. For example, en is a parent of en_US which is a parent of en_US_POSIX. Thanks to the hierarchical relation resources can be automatically shared between related resources. For example, when you request a foobar item for en_US locale, and it does not exist, but the item does exist for the en locale, it will be used.

Common Locale Data Repository

Flow comes bundled with the CLDR (Common Locale Data Repository). It’s an Unicode project with the aim to provide a systematic representation of data used for the localization process (like formatting numbers or date and time). The i18n framework provides a convenient API to access this data.

Note

For now Flow covers only a subset of the CLDR data. For example, only the Gregorian calendar is supported for date and time formatting or parsing.

Detecting user locale

The Detector class can be used for matching one of the available locales with locales accepted by the user. For example, you can provide the AcceptLanguage HTTP header to the detectLocaleFromHttpHeader() method, which will analyze the header and return the best matching Locale object. Also methods exist which accept a locale identifier or template Locale object as a parameter and will return a best match.

Translating text

Translator class

The \Neos\Flow\I18n\Translator class is the central place for the translation related functionality. Two translation modes can be used: translating by original label or by ID. Translator also supports plural forms and placeholders.

For translateByOriginalLabel() you need to provide the original (untranslated, source) message to be used for searching the translated message. It makes view templates more readable.

translateById() expects you to provide the systematic ID (like user.notRegistered) of a message.

Both methods accept the following optional arguments:

arguments - array of values which will replace corresponding placeholders
quantity - integer or decimal number used for finding the correct plural form
sourceName - name of source catalog to read the translation from.
packageKey of the package the source catalog is contained in.

Hint

Translation by label is very easy and readable, but if you ever want to change the original text, you are in trouble. The use of IDs gives you more flexibility in that respect.

Another issue: some labels do not contain their context, like “Name”. What is meant here, a person’s name or a category label? This can be solved by using IDs that convey the context (note that both could be “Name” in the final output):

party.person.fullName
blog.category.name

We therefore recommend to use translationById() in your code.

Plural forms

The Translator supports plural forms. English has only two plural forms: singular and plurals but the CLDR defines six plural forms: zero, one, two, few, many, other. Though english only uses one and other, different languages use more forms (like one, few, and other for Polish) or less forms (like only other for Japanese).

Sets of rules exist for every language defining which plural form should be used for a particular quantity of a noun. If no rules match, the implicit other rule is assumed. This is the only form existing in every language.

If the catalogs with translated messages define different translations for particular plural forms, the correct form can be obtained by the Translator class. You just need to provide the quantity parameter - an integer or decimal number which specifies the quantity of a noun in the sentence being translated.

Placeholders

Translated messages (labels) can contain placeholders - special markers denoting he place where to insert a particular value and optional configuration on how to format it.

The syntax of placeholders is very simple:

{id[,formatter[,attribute1[,attribute2...]]]}

where:

id is an integer used to index the arguments to insert
formatter (optional) is a name of one of the Formatters to use for formatting the argument (if no formatter is given the provided argument will be cast to string)
attributes (optional) are strings directly passed to the Formatter. What they do depends on the concrete Formatter which is being used, but generally they are used to specify formatting more precisely.

Some examples:

{0}
{0,number,decimal}
{1,datetime,time,full}

The first example would output the first argument (indexing starts with 0), simply string-casted.
The second example would use NumberFormatter (which would receive one attribute: decimal) to format first argument.
The third example would output the second argument formatted by the DatetimeFormatter, which would receive two attributes: time and full (they stand for format type and length, accordingly).

Formatters

A Formatter is a class implementing the \Neos\Flow\I18n\Formatter\FormatterInterface. A formatter can be used to format a value of particular type: to convert it to string in locale-aware manner. For example, the number 1234.567 would be formatted for French locale as 1 234,567. It is possible to define more elements than just the position and symbols of separators.

Together with placeholders, formatters provide robust and easy way to place formatted values in strings. But formatters can be used directly (i.e. not in placeholder, but in your class by injection), providing you more control over the results of formatting.

The following formatters are available in Flow by default:

\Neos\Flow\I18n\Formatter\NumberFormatter: Formats integers or floats in order to display them as strings in localized manner. Uses patterns obtained from CLDR for specified locale (pattern defines such elements like minimal and maximal size of decimal part, symbol for decimal and group separator, etc.). You can indirectly define a pattern by providing format type (first additional attribute in placeholder) as decimal or percent. You can also manually set the pattern if you use this class directly (i.e. not in placeholder, but in your class by injection).
\Neos\Flow\I18n\Formatter\DatetimeFormatter: Formats date and / or time part of PHP \DateTime object. Supports most of very extensive pattern syntax from CLDR. Has three format types: date, time, and datetime. You can also manually set the pattern if you use this class directly.

The following parameters are generally accepted by Formatters’ methods:

locale - formatting result depends on the localization, which is defined by provided Locale object
formatLength (optional) - CLDR provides different formats for full, long, medium, short, and default length

Every formatter provides few methods, one for each format type. For example, NumberFormatter has methods formatDecimalNumber() - for formatting decimals and integers - and formatPercentNumber() - for percentage (parsed value is automatically multiplied by 100).

You can create your own formatter class which will be available for use in placeholders. Just make sure your class implements the \Neos\Flow\I18n\Formatter\FormatterInterface. Use the fully qualified class name, without the leading backslash, as formatter name:

{0,Acme\Foobar\Formatter\SampleFormatter}

Translation Providers

Translation providers are classes implementing the TranslationProviderInterface. They are used by the Translator class for accessing actual data from translation files (message catalogs).

A TranslationProvider’s task is to read (understand) the concrete format of catalogs. Flow comes with one translation provider by default: the XliffTranslationProvider. It supports translations stored in XLIFF message catalogs, supports plural forms, and both translation modes.

You can create and use your own translation provider which reads the file format you need, like PO, YAML or even PHP arrays. Just implement the interface mentioned earlier and use the Objects.yaml configuration file to set your translation provider to be injected into the Translator. Please keep in mind that you have to take care of overrides yourself as this is within the responsibilities of the translation provider.

Fluid ViewHelper

There is a TranslateViewHelper for Fluid. It covers all Translator features: it supports both translation modes, plural forms, and placeholders. In the simplest case, the TranslateViewHelper can be used like this:

<f:translate id="label.id"/>

It will output the translation with the ID “label.id” (corresponding to the trans-unit id in XLIFF files).

The TranslateViewHelper also accepts all optional parameters the Translator does.

<f:translate id="label.id" source="someLabelsCatalog" arguments="{0: 'foo', 1: '99.9'}"/>

It will translate the label using someLabelsCatalog. Then it will insert string casted value “foo” in place of {0} and localized formatted 99.9 in place of {1,number}.

Translation by label is also possible:

<f:translate>Unregistered User</f:translate>

It will output the translation assigned to user.unregistered key.

When the translation for particular label or ID is not found, value placed between <f:translate> and </f:translate> tags will be displayed.

Localizing validation error messages

Flow comes with a bundle of translations for all basic validator error messages. To make use of these translations, you have to adjust your templates to make use of the TranslateViewHelper.

<f:validation.results for="{property}">
      <f:for each="{validationResults.errors}" as="error">
              {error -> f:translate(id: error.code, arguments: error.arguments, package: 'Neos.Flow', source: 'ValidationErrors')}
      </f:for>
</f:validation.results>

If you want to change the validation messages, you can use your own package and override the labels there. See the “XLIFF file overrides” section below.

Tip

If you want to have different messages depending on the property, for example if you want to be more elaborate about specific validation errors depending on context, you could add the property to the translate key and provide your own translations.

Localizing resources

Resources can be localized easily in Flow. The only thing you need to do is to put a locale identifier just before the extension. For example, foobar.png can be localized as foobar.en.png, foobar.de_DE.png, and so on. This works with any resource type when working with the Flow ResourceManagement.

Just use the getLocalizedFilename() of the i18n Service singleton to obtain a localized resource path by providing a path to the non-localized file and a Locale object. The method will return a path to the best matching localized version of the file.

Fluid ViewHelper

The ResourceViewHelper will by default use locale-specific versions of any resources you ask for. If you want to avoid that you can disable that:

{f:uri.resource(path: 'header.png', localize: 0)}

Validating and parsing input

Validators

A validator is a class implementing ValidatorInterface and is used by the Flow Validation Framework for assuring correctness of user input. Flow provides two validators that utilize i18n functionality:

\Neos\Flow\Validation\Validator\NumberValidator: Validates decimal and integer numbers provided as strings (e.g. from user’s input).
\Neos\Flow\Validation\Validator\DateTimeValidator: Validates date, time, or both date and time provided as strings.

Both validators accept the following options: locale, strictMode, formatType, formatLength.

These validators are working on top of the parsers API. Please refer to the Parsers documentation for details about functionality and accepted options.

Parsers

A Parsers’ task is to read user input of particular type (e.g. number, date, time), with respect to the localization used and return it in a form that can be further processed. The following parsers are available in Flow:

\Neos\Flow\I18n\Parser\NumberParser: Accepts strings with integer or decimal number and converts it to a float.
\Neos\Flow\I18n\Parser\DatetimeParser: Accepts strings with date, time or both date and time and returns an array with date / time elements (like day, hour, timezone, etc.) which were successfully recognized.

The following parameters are generally accepted by parsers’ methods:

locale - formatting results depend on the localization, which is defined by the provided Locale object
formatLength - CLDR provides different formats for full, long, medium, short, and default length
strictMode - whether to work in strict or lenient mode

Parsers are complement to Formatters. Every parser provides a few methods, one for each format type. Additionally each parser has a method which accepts a custom format (pattern). You can provide your own pattern and it will be used for matching input. The syntax of patterns depends on particular parser and is the same for a corresponding formatter (e.g. NumberParser and NumberFormatter support the same pattern syntax).

Parsers can work in two modes: strict and lenient. In strict mode, the parsed value has to conform the pattern exactly (even literals are important). In lenient mode, the pattern is only a “base”. Everything that can be ignored will be ignored, some simplifications in the pattern are done. The parser tries to do it’s best to read the value.

XLIFF message catalogs

The primary source of translations in Flow are XLIFF message catalogs. XLIFF, the XML Localisation Interchange File Format is an OASIS-blessed standard format for translations.

Note

In a nutshell an XLIFF document contains one or more <file> elements. Each file element usually corresponds to a source (file or database table) and contains the source of the localizable data. Once translated, the corresponding localized data for one, and only one, locale is added.

Localizable data are stored in <trans-unit> elements. The <trans-unit> contains a <source> element to store the source text and a (non-mandatory) <target> element to store the translated text.

File locations and naming

Each Flow package may contain any number of XLIFF files. The location for these files is the Resources/Private/Translations folder. The files there can be named at will, but keep in mind that Main is the default catalog name. The target locale is then added as a directory hierarchy in between. The minimum needed to provide message catalogs for the en and de locales thus would be:

Resources/
  Private/
    Translations/
      en/
        Main.xlf
      de/
        Main.xlf

XLIFF file creation

It is possible to create initial translation files for a given language. With Flow command

./flow kickstart:translation --package-key Some.Package --source-language-key en --target-language-keys "de,fr"

the files for the default language english in the package Some.Package will be created as well as the translation files for german and french. Already existing files will not be overwritten. Translations that do not yet exist are generated based on the default language.

A minimal XLIFF file looks like this:

<?xml version="1.0"?>
<xliff version="1.2" xmlns="urn:oasis:names:tc:xliff:document:1.2">
        <file original="" source-language="da" target-language="fr" datatype="plaintext">
                <body>
                        <trans-unit id="danish.celebrity">
                                <source>Skarhøj</source>
                                <target>Sarkosh</target>
                        </trans-unit>
                </body>
        </file>
</xliff>

If possible you should set up your editor to use the XLIFF 1.2 strict schema to validate the files you are working on.

Note

When using translationById() the framework will check the catalog’s source language against the currently needed locale and use the <source> element if no <target> element is found. This eliminates the need to duplicate messages in catalogs where source and target language are the same.

But you may still ask yourself do I really need to duplicate all the strings in XLIFF files? The answer is you should. Using target allows to fix typos or change wording without breaking translation by label for all other languages.

How to create meaningful XLIFF ids

When using the recommended way of translating by id, it is even more important to use meaningful identifiers. Our suggestion is to group identifiers and use dot notation to build a hierarchy that is meaningful and intuitive:

settings.account.keepLoggedIn
settings.display.compactControls
book.title
book.author
…

Labels may contain placeholders to be replaced with given arguments during output. Earlier we saw an example use of the TranslateViewHelper:

<f:translate id="label.id" arguments="{0: 'foo', 1: '99.9'}"/>

The corresponding XLIFF files will contain placeholders in the source and target strings:

<trans-unit id="some.label">
        <source>Untranslated {0} and {1,number}</source>
        <target>Übersetzung mit {1,number} und {0}</target>
</trans-unit>

As you can see, placeholders may be reordered in translations if needed.

Plural forms in XLIFF files

Plural forms are also supported in XLIFF. The following example defines a string in two forms that will be used depending on the count:

<group id="some.label" restype="x-gettext-plurals">
        <trans-unit id="some.label[0]">
                <source>This is only {0} item.</source>
                <target>Dies ist nur {0} Element.</target>
        </trans-unit>
        <trans-unit  id="some.label[1]">
                <source>These are {0} items.</source>
                <target>Dies sind {0} Elemente.</target>
        </trans-unit>
</group>

Please be aware that the number of the available plural forms depends on the language! If you want to find out which plural forms are available for a locale you can have a look at Neos.Flow/Resources/Private/I18n/CLDR/Sources/supplemental/plurals.xml

XLIFF file translation

To translate XLIFF files you can use any text editor, but translation is a lot easier using one the available translation tools. To name two of them: Virtaal is a free and open-source tool for offline use and Pootle (both from the Translate Toolkit project) is a web-based translation server.

XLIFF can also easily be converted to PO file format, edited by well known PO editors (like Poedit, which supports plural forms), and converted back to XLIFF format. The xliff2po and po2xliff tools from the Translate Toolkit project can convert without information loss.

XLIFF file overrides

As of Flow 4.2, XLIFF files are no longer solely identified by their location in the file system. Instead, the <file>’s product-name and original attributes are evaluated to the known package and source properties, if given. The actual location in the file system is only taken into account if this information is missing and mainly for backwards compatibility.

This allows for an override mechanism, which comes in two levels:

Package-based overrides

Translation files are assembled by collecting labels along the composer dependency graph. This means that as long a package depends (directly or indirectly) on another package, it can override or enrich the other package’s XLIFF files by using the other package’s product-name and original values.

Note

If you have trouble overriding another package’s translations, please check your composer.json if you correctly declared that package as a dependency.

Global translations overrides

In case translations are provided by another source than packages (e.g. via import from a third party system), a global translation path can be declared and is evaluated with highest priority in that it overrides all translations provided by packages. The default value for this is Data/Translations and can be changed via the configuration parameter

Neos:
  Flow:
    i18n:
      globalTranslationPath: '%FLOW_PATH_DATA%Translations/'

Example

Packages/Framework/Neos.Flow/Resources/Private/Translations/en/ValidationErrors.xlf

<file original="" product-name="Neos.Flow" source-language="en" datatype="plaintext">
  <body>
    <trans-unit id="1221551320" xml:space="preserve">
      <source>Only regular characters (a to z, umlauts, ...) and numbers are allowed.</source>
    </trans-unit>
  </body>
</file>

Packages/Application/Acme.Package/Resources/Private/Translations/en/ValidationErrors.xlf

<file original="ValidationErrors" product-name="Neos.Flow" source-language="en" datatype="plaintext">
  <body>
    <trans-unit id="1221551320" xml:space="preserve">
      <source>Whatever translation more appropriate to your domain comes to your mind.</source>
    </trans-unit>
  </body>
</file>

Note

In case of undetected labels, please make sure the original and product-name attributes are properly set (or not at all, if the file resides in the matching directory). Since these fields are used to detect overrides, they are now meaningful and cannot be filled arbitrarily any more.