Localization Tips, Part 3: Sorting and Collating
This is the third entry in a multi-part series about localization.
Sorting and collating data is something everyone does on a frequent basis and typically won’t spend a lot of time thinking about. If you’ve ever seen a physical rolodex before, this is the essence of sorting and collating: putting a list of contacts in a particular order, collated (or grouped) in a manner which everyone agrees upon. For example:
A Aaron Burr Alexander Hamilton B Benjamin Franklin C Carter Braxton Charles Pickney D Daniel Carroll S Samuel Adams
Unless you happen to be Danish:
A Alexander Hamilton B Benjamin Franklin C Charles Pickney Carter Braxton D Daniel Carroll S Samuel Adams Å Aaron Burr
A Aaron Burr Alexander Hamilton B Benjamin Franklin C Carter Braxton D Daniel Carroll CH Charles Pickney S Samuel Adams
So what’s going on here? There are a few different considerations on how these characters get sorted.
Each language has a set of “exemplar characters” which contain the commonly used letters fo a given modern form of a language. More simply, you can think of them as the “alphabet” or the “native characters” for a given language. Here are a few examples of these characters:
English: a b c d e f g h i j k l m n o p q r s t u v w x y z Danish: a b c d e f g h i j k l m n o p q r s t u v w x y z æ ø å Czech: a á b c č d ď e é ě f g h ch i í j k l m n ň o ó p q r ř s š t ť u ú ů v w x y ý z ž
Sometimes a sequence of Latin characters will map to a particular native character. In Danish, “AA” is treated as “Å” which sorts after “Z.” Sometimes an exemplar character is actually expressed by multiple characters. In Czech, for example, strings starting with “CH” sort after “H” but strings starting with just “C” sort after “B.”
The Unicode collation algorithm defines the default sorting applied to the characters supported by Unicode. The order defined here is important because it serves as the basis for how other languages and locales customize collation. However, collating data based on this standard isn’t particularly useful to end users who expect data sorted in a manner that is specific to their own conventions.
Different languages will define different rules for how non-native characters are sorted and collated. Frequently they will appear after the native characters in order defined by the Unicode collation algorithm. However, Japanese’s default collation will show latin characters first, then Japanese characters, then non-native.
Here is a brief list of things to keep in mind when dealing with collation in your application:
- If your application’s user interface displays data in a collated or “grouped” view, ensure it does so in a locale-aware manner. For example, if your application displays a list of contacts with a header row for each letter, the letters should be appropriate for that locale. Define that set of exemplar characters for each language in the appropriate localization files. There typically should also be an “other” section after the last exemplar character for text starting with “non-native” characters.
- Ensure your application sorts data according to the user’s selected locale. Frequently this will involve passing the proper locale to the API’s “sort” function. Using the Unicode collation might seem like a fair compromise but will lead to a result that is displeasing to most non-English speakers.
- If one of the tiers doesn’t provide support for locale-aware sorting, remove sorting code from that tier and rely on another tier to perform all sorting.
- If the application relies on caching of data, ensure that a locale-aware sort is performed after the data is retrieved from cache. This issue can manifest in many different places, including caching of JSON or XML responses over HTTP.