Page 1 of 2
Umlauts and other diacritics are broken
Posted: Thu May 28, 2026 1:54 pm
by Garry
Try Games > Browse by language and select a language such as (say) Danish or German that uses diacritical marks. Now browse through the list of games and you'll see all sorts of strange characters where umlauts or other non-ASCII characters should appear in game titles and author's names.
Can someone please fix this?
Re: Umlauts and other diacritics are broken
Posted: Thu May 28, 2026 9:42 pm
by Alastair
Thanks Garry for bringing this to our attention. I noticed this for one entry back on the 16th, but I hadn't realised it was site wide. I hope there is an easy solution, the thought of correcting each error individually does not appeal.
Re: Umlauts and other diacritics are broken
Posted: Fri May 29, 2026 11:29 pm
by Gunness
I presume that it's a display error, as I don't know why these specific characters should have been corrupted.
As far as I can tell, all pages are displayed with UTF-8, which *is* able to display special characters such as these. I don't have any ideas straight away but I'll investigate.
Re: Umlauts and other diacritics are broken
Posted: Sat May 30, 2026 6:33 am
by Mr Creosote
My guess is on a database upgrade gone wrong. I can check this afternoon.
Re: Umlauts and other diacritics are broken
Posted: Sat May 30, 2026 11:08 am
by Gunness
Much appreciated!
Re: Umlauts and other diacritics are broken
Posted: Sat May 30, 2026 12:41 pm
by Mr Creosote
Can you help assembling a list of characters which will need replacing - i.e. how they look now and what they are supposed to be? I have so far:
Code: Select all
'ü' => 'ü',
'ä' => 'ä',
'ö' => 'ö',
'Ãœ' => 'Ü',
'é' => 'é',
'á' => 'á',
'ê' => 'ê'
Also, are other fields than the title also affected?
Re: Umlauts and other diacritics are broken
Posted: Sat May 30, 2026 9:03 pm
by Alastair
Add
Code: Select all
½ => ½
â => â
è => è
Ç => Ç
Å« => ū
also I am seeing î (see links below), I think it should be Î or î
I'm guessing that the ¿ in ¿La sentencia? -
https://solutionarchive.com/game/id%2C6 ... 2C+La.html - should be ¿ and the à in the author's name "José Luis DÃaz" should be í but then we have an instance where just one rather than two faulty characters represents the real one.
(Note that since ½ => ½ and ¿ may => ¿ it could be that all instances of Â{character} resolve to {character}.)
I also see
https://solutionarchive.com/game/id%2C9 ... A9lye.html what the title and the author's name are supposed to be I cannot guess.
Mr Creosote wrote: Sat May 30, 2026 12:41 pm
Also, are other fields than the title also affected?
Yes, see
https://solutionarchive.com/game/id%2C5 ... glub!.html for an example where "Related" (no surprise since it contains the game's title) and "Notes" are affected.
Re: Umlauts and other diacritics are broken
Posted: Sat May 30, 2026 11:42 pm
by Alastair
This is a rough equation in hex I've devised that may explain what is going on, where:
x is the value of the actual character
y is the value of the first rogue character
z is the value of the second rogue character
I'm getting the values from
https://en.wikipedia.org/wiki/List_of_U ... characters
x = 40(y - C2) + z
some examples which all correlate with what is known (for the third example see the note):
Code: Select all
à = 00C3
¤ = 00A4
x = 40(C3 - C2) + A4 = 40 + A4 = E4
00E4 = ä
Code: Select all
Å = 00C5
« = 00AB
x = 40(C5 - C2) + AB = C0 + AB = 16B
016B = ū
Code: Select all
à = 00C3
{Soft hyphen} = 00AD
x = 40(C3 - C2) + AD = ED
00ED = í
N.B. The soft hyphen would explain the "Díaz" issue.
Re: Umlauts and other diacritics are broken
Posted: Mon Jun 01, 2026 6:44 am
by Gunness
Mr Creosote wrote: Sat May 30, 2026 12:41 pm
Can you help assembling a list of characters which will need replacing - i.e. how they look now and what they are supposed to be? I have so far:
Code: Select all
'ü' => 'ü',
'ä' => 'ä',
'ö' => 'ö',
'Ãœ' => 'Ü',
'é' => 'é',
'á' => 'á',
'ê' => 'ê'
Also, are other fields than the title also affected?
I have these:
'Ã¥' => 'å'
'Ø' => 'ø'
'ø' => 'Ø'
'æ' => 'æ'
'Ä' => 'Ä'
This ought to cover the various Scandinavian languages.
A few French characters:
'Ç' => 'Ç'
'â' => 'â'
'è' => 'è'
Re: Umlauts and other diacritics are broken
Posted: Mon Jun 01, 2026 1:51 pm
by Gunness
Mr Creosote wrote: Sat May 30, 2026 12:41 pmAlso, are other fields than the title also affected?
Yes, the user comments - see:
https://solutionarchive.com/game/id%2C3 ... Karma.html:
"I’m a big fan of Avalon Hill Microcomputer Games and have nearly all of these titles in my game collection. Lords of Karma can’t be 1978 given one primary fact. Avalon Hill in context to computer games first presented offerings to the public for sale at the Origins Gaming Convention on June 27–29 1980"
What's worrying here is that the bug also seems to affect apostrophes and dashes?
Re: Umlauts and other diacritics are broken
Posted: Mon Jun 01, 2026 4:38 pm
by Mr Creosote
Well, I can run a script over a number of database fields to re-encode characters. Though honestly, it will never be fully complete. You don't happen to have a backup from before this happened, do you?
Re: Umlauts and other diacritics are broken
Posted: Mon Jun 01, 2026 5:16 pm
by Gunness
No, it would be pretty outdated, I'm afraid.
To avoid several passes, maybe we should ensure that the list of characters to be replaced is as complete as possible. I can take another look tomorrow, but don't have the time tonight.
Equally important, what can be done to avoid this in the future, other than restoring backups or running char replacement scripts?
Re: Umlauts and other diacritics are broken
Posted: Mon Jun 01, 2026 5:55 pm
by Alastair
Gunness wrote: Mon Jun 01, 2026 1:51 pm
Yes, the user comments - see:
https://solutionarchive.com/game/id%2C3 ... Karma.html:
"I’m a big fan of Avalon Hill Microcomputer Games and have nearly all of these titles in my game collection. Lords of Karma can’t be 1978 given one primary fact. Avalon Hill in context to computer games first presented offerings to the public for sale at the Origins Gaming Convention on June 27–29 1980"
What's worrying here is that the bug also seems to affect apostrophes and dashes?
Looking at that page in the Wayback Machine -
https://web.archive.org/web/20251015052 ... Karma.html - shows that the apostrophes and dash in 27–29 are not standard ASCII
' and
- they probably came from cutting and pasting from a word processor.
The Wayback Machine also shows that the problem occurred between 3rd April -
https://web.archive.org/web/20260403032 ... chive.com/ - and 8th May -
https://web.archive.org/web/20260508140 ... chive.com/ (search for "sur ma Cour" or "ologie en Aveugle" for a couple of examples).
Re: Umlauts and other diacritics are broken
Posted: Thu Jun 04, 2026 4:51 pm
by Mr Creosote
I think I have an algorithmic solution, not relying on a whitelist of specific characters (thanks, Alastair, your math approach nudged me in the right direction).
Here is an overview of what it would do if I put it live:
https://solutionarchive.com/__umlauts/ (temporary page, will be deleted again after committing the fix). The ones marked in red do contain question marks, which could happen if my algorithm fails. Though of course, a question mark may be correctly part of a game title or a sentence. I.e. it just means we should manually check whether this proposed result is correct. A bit of sanity checking would be appreciated.
I'm still looking into what other database fields could be affected. Is there any impact on the forums as well or only on the website?
Re: Umlauts and other diacritics are broken
Posted: Thu Jun 04, 2026 10:52 pm
by Alastair
Mr Creosote wrote: Thu Jun 04, 2026 4:51 pm
I'm still looking into what other database fields could be affected. Is there any impact on the forums as well or only on the website?
The umlauts in the thread
Jörg Walkowiak's "Gold Fever" -
https://solutionarchive.com/phpBB3/viewtopic.php?t=871 - are present and correct. So the forum is probably unaffected.