Not accepting Accept-Language is one of my major pet peeves. What makes it worse is that many multilingual websites translate their language-switching buttons and the list of languages to the current language .... which is beyond fucking stupid and defeats the purpose. Wikipedia does this right. The button to switch languages is clear, using a universal multilanguage icon, and a list of languages (using the name of the language in that same language) in alphabetical order, with the most likely candidates on the top (presumably based on geoip).
E.g. an English Wikipedia page will present me with the following language suggestions:
Suggested languages
Deutsch
Français
Nederlands
When you assume a language, you make an ass of you and of me. Don't be an ass. Be like Wikipedia.
A related issue that has me fuming is when, after arriving at a page of interest from a search engine, a modal popup forces me to select the country I'm from, and then promptly redirects me to the homepage of the regional website.
Some have a X button to close said dialog, but many don't which is really aggravating.
Google does this. I want to check out the new device they just released - "sorry, this product is not available in your country". I just wanted to read the specs, not buy.
Well, it's in an order, but I don't know about alphabetical. I clicked on today's English featured article and looked at the languages: "中文", "Italiano" are "suggested", then the remainder are grouped by geographic region, and aren't particularly alphabetical. They appear to be in groups which are still not alphabetical. Europe seems to have a Cyrillic group but "Қазақша" is shown after "Українська" which isn't accurate in Kazakh and probably also unexpected for anybody who isn't familiar with the letter Қ (Қ isn't a letter in Russian, this is probably why this happens). The Chinese languages don't seem to be in stroke order (no expert here), although Korean is below them (because of course, K for Korean alphabetizes after C for Chinese).
Anyways, no hate for Wikipedia; they do a great job of localizing. Just a bit of nuance/pedantry about how you can't "alphabetize" language names in their own language.
Not so, this sort order has been standardised as part of Unicode for at least 28 years. To see it in action, pipe the list of languages as a text file through a conforming tool like `ucsort`. When Қ is falsely sorted after Ч, then the wrong algorithm or no algorithm at all has been used.
> because of course, K for Korean alphabetizes after C for Chinese
It's a circular dependency: how do you sort and list the locales or languages for someone to pick one, when by definition you don't know their locale yet?
You have to either make some best-guess approximation (IP geo, browser headers, etc) or use a locale-invariant sort, both of which will be wrong in some cases.
We can find a sorting order with the minimal total distance between where we place a language entry and where this entry would be in that language. If there’s no pair of languages A and Ä such that A > Ä in one and Ä > A in the other, then (I guess???) this total distance will be zero.
That’s also part of the reason I’ve chosen it for a placeholder / variable name! The actual placing is not important as long as it’s where speakers of the Ä language expect to find it.
Or suppose there’s languages Ä₁ and Ä₂, where in Ä₁ the ‘ä’ is the umlauted ‘a’ and in Ä₂ it’s a distinct letter. The language list would be displayed as:
A Ä₁ B C Z Ä₂
The only problem / corner case would be such a language Ä₀ that would e.g. sort ‘ä’ before ‘a’. I would still put it after, since it’s where most other readers would expect to find it.
can't you just sort all as int? the codepages usually come roughly sorted, and while no one knows which of 檎 or 橙 comes first, I don't think it'll be particularly offensive whichever way a random app did, to most.
That would be one locale-invariant sort as I said. Sure, you can pick some way of doing it that's least-bad. The codepages are roughly sorted, but what we're debating is the cases where that fails some definition of correctness. The point is there can be no universally correct answer for sorting locales before the user picks one, because that can depend on already knowing the locale itself.
Yes, the DUCET is bound to disappoint everybody (especially users of the Latin script with diacritics, as none of them agree on the sort order and everyone’s preferences are tied to the specific subset of diacritics they need), but at least it disappoints everybody more or less equally.
(Do yourself a favor, though, and use the CLDR root collation instead of the raw DUCET—they are basically the same, except, and I’m quoting the standard here[1], “the DUCET is not entirely well-formed”.)
Yes, that’s confusing and probably hard to find a good balance. Someone speaking Greek or Czech may expect to find their language around E (Ελληνικά) or C (Čeština), but nope, on Wiki it’s all the way after Z.
The problem may be is that you need to set the locale in order to get certain alphabetization, but setting the locale won't happen until after the language is chosen.
A reasonable approach might be to sort the list of names by using, as the sort keys, the strings projected through a Unicode normalization function, followed by folding to upper case. Then Čestina gets mapped to CESTINA and at least appears among the C's.
Don’t special characters always go after the Latin alphabet? I think this is pretty common, and fairly expected behaviour. Of course nothing is perfect but I feel like the way Wikipedia handles it is consistent.
> If you want to see bizarre sort rules, look up how french sorts accent characters.
I tried to do this, but there do not appear to be any sources addressing this question.
I did find a French Stack Exchange question asking for this exact information, and complaining that there are no sources (other than an uncited wikipedia page) that address it. There is no answer posted, but there is a comment from a French guy suggesting that there are no official rules.
I notice that post suggests that Académie française specifies that accents should be sorted in reverse, and includes a link over the words "Académie française", and yet that link doesn't go to a supporting document.
A while ago I complained on this forum that Amazon's hyphenation for Kindle ebooks is abysmally bad. (Which is still true.) Someone responded to say that the hyphenation algorithm for English requires this. I pointed out that the hyphenation algorithm for English is a lookup table; each word has its hyphenation defined in the table, and when you need to hyphenate a word, you look up the hyphenation points.
Another response linked me to a paper describing how this table can be stored as a set of rules that provide hyphenation points in arbitrary letter sequences rather than dictionary words. That paper is very clear about its goals; it is an advance in data compression, proposing a method of storing a lookup table that takes less space than the table does. It carefully goes over how to produce the ruleset from the table.
But somewhere along the line, people confused the data compression algorithm (of storing the lookup table as a ruleset) for the hyphenation algorithm. They will now tell you with a straight face that a single ruleset that seems to have gone around represents the hyphenation algorithm for English, even if the word you want to hyphenate wasn't in the table that that ruleset was prepared from. And this is false.
It looks to me like something similar has happened in English speakers' understanding of French sorting order. It's very easy to explain why the example quadruplet has the sorting order it does:
cote
côte
coté
côté
(Note that the Stack Exchange question from 2024 and the blog post from 2004 use exactly the same example.)
These four words have two pronunciations, and the pronunciations are grouped with each other. After that, "cote" comes first by virtue of bearing no accents, and "o" comes before "ô" for the same reason.
What's happening here is that although French generally pretends that "e" and "é" are the same letter, they aren't, which forces -e (not pronounced) to come before -é (pronounced!). "o" and "ô" actually are the same letter, and can be ordered flexibly.
The rule "sort the accents in reverse" arises as a coincidence; it happens to be the case that this distinction is most significant at the end of French words. But French speakers would reject this ordering:
cetot
cétot
cetôt
cétôt
This doesn't come up because those words don't exist.
Well in my language "é" is absolutely not special, and should definitively be placed near "e" (to the point that uppercase é is often written E instead of É)
If I recall correctly, the default propose a first list that push items which are guessed most likely what the user expect, then a list more complete, and in any case let you filter by typing. I think it also can change the way it behave if you are connected and tweaked your preferences in the matter for your account.
Wikipedia uses UCA sort order in categories (depending on which lang wikipedia you are reading). Most other lists just sort using unicode codepoint order (in NFC). So it depends, but yes, for generated lists other than categories ascii characters usually come first.
That’s English hegemony. Languages have their own sorting that they expect. You can’t impose rules to other languages.
Of course at some point Unicode needs to be ordered, but you don’t get to impose technical details to people around the world because it matches with how English does it.
That’s where geo-ip guessing becomes relevant. Show a list with the most likely languages at the top.
Or use the Accept-Language. Since we already know the User understands that one, it's probably a reasonable choice for which sort order they expect too.
> The Chinese languages don't seem to be in stroke order (no expert here)
They are for me. In the Asia section, 中文 ["Chinese"] is listed first, followed by 吴语 ["Wu"] and then 粤语 ["Cantonese"]. Stroke order is first by stroke count and then by an obscure criterion that I don't know (and that, in my experience, Chinese people living in China also don't know), but stroke count is unambiguous and these are in order: 中 4, 吴 7, 粤 12.
Note that they aren't in alphabetical order: 中 Z, 吴 W, 粤 Y.
Japanese appears between Wu and Cantonese for unclear reasons.
It is sorted FIRST by radical and SECOND by stroke order. This is roughly equivalent to the Unicode codepoint sort if you stay in the basic multilingual plane. The order also puts literary chinese afer wu Chinese, which breaks with a pure stroke-count sort:
Dictionary lookup is done first by radical and second by stroke count. Collation is not. Stroke count is first.
For example, I have a book of 成语 stories that gives its table of contents in non-alphabetical order. (Since nobody understands the traditional ordering, I also have several such books that put their table of contents in alphabetical order.)
Note that 三's radical is 一, the first Kangxi radical, and that 一 is listed first. Your theory is wrong. 三 isn't even first among the 3-stroke characters, which start (among these) with 口.
Why did you make up a false answer to this question?
The Wikipedia sort for the languages is as I stated above, with Literary Chinese and Japanese between Wu Chinese and Yue Chinese. I explained why it was sorted that way, because radical is considered first. You could not explain why Japanese appeared between Wu and Yue because you insisted and continue to insist that radicals are not used.
I didn't say sorting is never done by stroke count alone. But I have seen radical+residual stroke count much more often than stroke count alone. Probably a result of the content I'm accessing. It's mostly Japanese and not intended for children.
The dictionary and non-dictionary sorting distinction that you make doesn't sound like a real thing. The audience, the country, and the number of items sorted are bigger factors. But you're not wrong in that stroke count is sometimes used alone.
> You could not explain why Japanese appeared between Wu and Yue because you insisted and continue to insist that radicals are not used.
I can't explain that because it's part of a different logical group, with its name written in a different script.† This puts it parallel to the Chinese options and to Korean.
> The Wikipedia sort for the languages is as I stated above
I took you to be describing the sort order for characters, not for wikipedia. Wikipedia doesn't obey that order either. You can check the page for Jiangsu, where all of the languages mentioned so far appear before the "Latin alphabet" style languages, but 閩南語 and 閩東語 appear after them.
† I also can't explain why wikipedia seems to have chosen 吴语 but 粵語, 客家語, and 贛語. Jiangsu is on the mainland... and so are Jiangxi and Guangdong.
> all of the languages mentioned so far appear before the "Latin alphabet"
> style languages, but 閩南語 and 閩東語 appear after them.
Could it have something to do with Minnan and Mindong Chinese articles being written in a Latin script, (despite the language name showing in both Chinese characters and Latin letters) ?
Oh nice! I never noticed that "Suggested languages" shows languages I previously selected.
But additionally, I like how it's not simply "pushing to the top", it does shows a previously selected language on the top, but it still keeps it duplicate in the list below, in case the user is going by muscle memory.
To me this is the best way.
Either make it VERY OBVIOUS that you're removing the item from the bottom of the list (which wouldn't be possible here), or don't remove it at all.
If I had a cent for each time a SaaS made my life harder by trying to "help me" I would be CEO of every SaaS I use.
I can give a perfect bad example: the Youtube app on my iPhone somehow determined to change to Amharic. This is the Google support article: https://support.google.com/youtube/answer/87604 telling me the buttons to press in English. Also, I don't know/speak Amharic, and so at the time had no idea what language it was, and the iPhone translate doesn't even recognize this Ethiopian language. Bit of a pickle that could have easily been mitigated by a universal multilanguage icon.
Yes, actually. Is there an article about it? After updating to iOS 18.4, Amharic was appended to the end of the list of preferred languages in the iPhone Settings app. However, what's interesting is it was ranked below English, and apps are supposed to use the languages list in order, but perhaps Youtube is alphabetically sorting the list?
Or the ChatGPT app which can be baffling. My phone language is English, I've set ChatGPT app specifically to English in the app settings, I ask my questions in English, and every now and then it still decides to answer in German.
There are two levels of this. If I get some other European language, I can generally figure out which is their word for English and it's not a big deal to switch. But if it gives me a script I don't know, like Bengali or something, it's a problem.
Perhaps every "choose language" menu should include English and Chinese in non-localized form, as an escape hatch, since almost every web users can recognize enough of them to navigate a menu to find their actual language.
Just include languages in their own script only. Why would a user need to select a language from that menu for which they DON’T know the target script? Showing Bengali in Bengali script is exactly what you want.
My favorite is the sites who do parse Accept-Languages, but then pick the last one in the list instead of the first. I have mine in rough order of my competence in them, which gets me my least-competent language on some sites even when my most-competent is supported.
I get a kick out of it when I see it, because you can understand how it happens. "Well, at least you tried."
It is wrong to blame the server here. For better results in content-negotiation, a user-agent should allow you to assign numeric weights instead of just a list (implying the same level of preference). Example:
Accept-Language: en;q=0.7,pt;q=0.3
If you already send something similar to this, and the server gets it wrong, then this is an outright bug, the software or its operator is out of compliance with HTTP.
This parameter, at first glance, appears to be used as numeric weights for automatic translations served by default, what turns browsing very uncomfortable (wrong translations, distorted texts).
Ie. Google, Youtube, Reddit.
Automatic translations should never be served by default, but only be loaded if the user requests it. The classical "do you want translate".
I don't need help, I'm criticising the use of the automatic translations served by default, and that are being used the weights of this parameter to do it.
The full picture? The weights seem to be more useful for fingerprinting and perhaps for server SEO than to help the users. Users who in the end will have to give the same weight to all the languages, or rewrite the outgoing headers, in order to be able to browse the Internet.
99% agree, but there is a problem on mobile, to switch from Spanish to English when I click the glass to search for alternatives. I have to type "Ing" (that are the first letters of "Inglés") while it shows "English" in the list of matches. It would be better if I can type both "Ing" or "Eng".
It's even more amusing when the displayed list looks like it's sorted randomly, but in reality it's sorted alphabetically in a different language to the display language..
e.g.
Nederlands
English
Français
Deutsch
Espagnol
(but if sorted in English: Dutch, English, French, German, Spanish)
I don't understand why you would want to select Inglés instead of English? You want to selecf English, or Español, or Nederlands, or Deutsch, or whatever language. If makes no sense for it to be translated.
Only in a very few weird corner cases. If it's an article about a city in Germany, I may like to see the article in German and use autotranslation to read t in English or Spanish.
Sometimes the article in the local language has more info. I had that problem in comments about places or events in Argentina. Sometimes the English article has less info than the Spanish article, so I made a link to the autotranslation.
Ironically, given TFA, it seems to be primarily using the user's IP:
> How does Universal Language Selector determine which languages I may understand
> ULS queries a service that determines your originating country based on your IP address. This is inaccurate in some cases. Based on the country code, most often spoken languages are suggested for you.
Indeed. Wikimedia wikis' language selection feature relies on Unicode CLDR language-territory data. This is very complex to maintain (and there are still many mistakes to fix), because reliable data is expensive to collect.
E.g. an English Wikipedia page will present me with the following language suggestions:
When you assume a language, you make an ass of you and of me. Don't be an ass. Be like Wikipedia.