AW: Umlaut (Ü, Ä, Ö) in attribute resolution not UTF-8 encoded

Fri Feb 14 10:39:33 UTC 2025

Hi Timothy and everyone,

thank you for your help, I think I found the issue.
We don’t access a table directly, but a View that is being created with a SELECT statement.
The view itself is UTF-8, but the Column “surname” is converted to Binary for some reason:
                ,cast(`u`.`surname` AS CHAR charset BINARY) AS `surname`
This results in it not having a utf8 character set:
+-------------+--------------------+
| column_name | character_set_name |
+-------------+--------------------+
| surname     | NULL               |
+-------------+--------------------+

As we didn’t create the view/database our self, I’m gonna have to ask why that is done and everything else.

Thanks again, you all were a huge help!

Greetings
Philip

Von: Timothy Spear <n614cd at gmail.com>
Gesendet: Donnerstag, 13. Februar 2025 20:15
An: Grüter, Philip <philip.grueter at secunet.com>
Betreff: Re: Umlaut (Ü, Ä, Ö) in attribute resolution not UTF-8 encoded

Philip,

The fact that forcing a conversion to UTF-8 in the result heavily implies the character encoding is incorrect or lost somewhere further up the chain. I am a few years out of date on using international character encoding with MySQL, so take what I am going to cover in detail with a grain of salt. And as a developer, one of the things I have learned is to always question my own assumptions and verify.

MySQL has six areas of character encoding.
Database, Table, Column, Client, Connection, Result Set.

I can never recall the details of who wins except the generic advice of the most specific setting wins. e.g. If you set the character encoding of a column, it will override the character encoding of the connection and the table. But set the character encoding of the result set, it will win...

Therefore, I would approach this the following way. Connect to MySQL using the mysql command line interface. The reason is for using the CLI is because the CLI usually has the fewest assumptions and overrides applied.
I would start by verifying the character encoding is as you expect for the database, table and column. Then the JDBC connection related items.
(I have to web search these queries, I have not tested them)

For schema/database

SELECT default_character_set_name FROM information_schema.SCHEMATA
WHERE schema_name = "userdb";

For Tables:

SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
       information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
WHERE CCSA.collation_name = T.table_collation
  AND T.table_schema = "userdb"
  AND T.table_name = "moodle2_auth";

For Columns:

SELECT character_set_name FROM information_schema.`COLUMNS`
WHERE table_schema = "userdb"
  AND table_name = "moodle2_auth"
  AND column_name = "surname";

"status\G" should show the other three; note: these will be default values which is what you are looking for. This could provide a hint on a specific character set which is not being forced to your standard in the JDBC connection.

If all of these are correct. Then the next place would be to check the data being inserted. To me, it is a tossup if I would pursue JDBC connection logs or start from the other end and verify the data coming into Shibboleth.

Tim

On Thu, Feb 13, 2025 at 1:12 PM Grüter, Philip <philip.grueter at secunet.com<mailto:philip.grueter at secunet.com>> wrote:
Hi Tim,

my JDBC connection string is already enforcing the UTF-8 character set:
jdbc:mysql://REDACTED IP:6033/userdb?useUnicode=true&characterEncoding=UTF-8
And the database and table is also in UTF-8.
That’s why I’m confused about the wrong encoding.

Greetings
Philip

Von: users <users-bounces at shibboleth.net<mailto:users-bounces at shibboleth.net>> Im Auftrag von Timothy Spear via users
Gesendet: Donnerstag, 13. Februar 2025 19:09
An: Shib Users <users at shibboleth.net<mailto:users at shibboleth.net>>
Cc: Timothy Spear <n614cd at gmail.com<mailto:n614cd at gmail.com>>
Betreff: Re: Umlaut (Ü, Ä, Ö) in attribute resolution not UTF-8 encoded

Philip,

That suggests the JDBC connection is using the default latin1 character set.
Try forcing the character set assignment on the JDBC URL connection string.

Tim

On Thu, Feb 13, 2025 at 12:58 PM Grüter, Philip <philip.grueter at secunet.com<mailto:philip.grueter at secunet.com>> wrote:
Hi Scott and Gianluca,

I don't really think it has something to do with my terminal or even the output of the aacli.sh tool.
The IDP is connected via SAML2 to a moodle system, which is also getting the wrong umlauts.
Also, the log is also displaying the wrong encoding. When I use the same terminal to connect to the MySQL Database, it displays the Ü correctly.

Also, I got it working, but I don't consider my current solution a good one. I changed the SQL Query in my attribute-resolver config to:
SELECT email,forename,CONVERT(surname USING utf8) AS surname FROM moodle2_auth WHERE username='$resolutionContext.principal'
This outputs the correct result. So in my opinion, the wrong encoding has something to do with the gathering of the data.

- Philip

-----Ursprüngliche Nachricht-----
Von: users <users-bounces at shibboleth.net<mailto:users-bounces at shibboleth.net>> Im Auftrag von Cantor, Scott via users
Gesendet: Donnerstag, 13. Februar 2025 17:20
An: Shib Users <users at shibboleth.net<mailto:users at shibboleth.net>>
Cc: Cantor, Scott <cantor.2 at osu.edu<mailto:cantor.2 at osu.edu>>
Betreff: Re: Umlaut (Ü, Ä, Ö) in attribute resolution not UTF-8 encoded

> Are you sure it is not a problem related to your terminal ?

Without going that far, I will say it's incredibly hard to test and debug this sort of thing because of all the issues with log files, terminal display, etc. It's easy to mix up where the issue is.

But I wouldn't swear the output logic for the view in that flow isn't wrong somewhere. I would suggest as one thing to try the --saml2 switch and see if the XML seems to be right.

I'm in the middle of working out some of the Unicode issues with the SP redesign, and I truly believe aliens invading and mandating a new language would be simpler for the world than handling Unicode in C++.

-- Scott

--
For Consortium Member technical support, see https://shibboleth.atlassian.net/wiki/x/ZYEpPw
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net<mailto:users-unsubscribe at shibboleth.net>
--
For Consortium Member technical support, see https://shibboleth.atlassian.net/wiki/x/ZYEpPw
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net<mailto:users-unsubscribe at shibboleth.net>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20250214/a2fb2988/attachment.htm>