My Japanese is garbled when I connect to mysql from python

Asked 1 years ago, Updated 1 years ago, 408 views

When I tried to create python and mysql containers using docker and connect python to mysql and display the table in fetchall, the Japanese was garbled.
When you enter the mysql container and view it directly, the characters may not be garbled due to the character code settings.

for garbled characters:

+-----------------------------------------------------------------------------------
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/|
+--------------------------+----------------------------+

without garbled characters:

+-----------------------------------------------------------------------------------
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | latin1 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results|latin1|
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/|
+--------------------------+----------------------------+

Due to the data we handle, it seems to be displayed correctly if you read it on latin1.
So I changed the character code from the Python code and tried again while displaying the settings.

('character_set_client', 'latin1')
('character_set_connection', 'utf8mb4')
('character_set_database', 'utf8mb4')
('character_set_filesystem', 'binary')
('character_set_results', 'latin1' )
('character_set_server', 'utf8mb4')
('character_set_system', 'utf8')
('character_sets_dir', '/usr/share/mysql/charsets/')

The character code settings are the same as , but when I read from python, the characters got garbled.I have tried many things like charset and changing the character code (e.g., making them all utf8mb4 or utf8) but the characters get garbled.

#coding:latin1
import mysql.connector

# Connected from python to mysql
cnx = mysql.connector.connect(
host=',
port = '3306',
user='user',
password = 'pass',
database='test_database',
charset='latin1'
    )
cursor_=cnx.cursor()

cursor_.execute("use test_database;")

# Change character code to get configuration status
query1 = "SET CHARACTER SET latin1;"
cursor_.execute(query1)

cursor_.execute("show variables like 'chara%';")
rows=cursor_.fetchall()
For row in rows:
  print(row)

cursor_.execute("select * from test;")
rows=cursor_.fetchall()
For row in rows:
  print(row)

Also, the attempt to add coding to the print failed because it was a tuple type.I don't know how to resolve this either.

for row in rows:
  print(row.coding(utf8 or latin1))
'tuple' object has no attribute'encord'

Does anyone know how to display this data from python?Can't you display the data on latin1?

python mysql

2023-01-08 14:09

1 Answers

The question is not answered because it is not accurate at all and the cause is unknown.

Assume the question is an environment built with the following code:

MySQL image:mysql:8.0.31
Python: 3.10.9
mysql-connector-python:8.0.31
PyMySQL:1.0.2
Server and Client IP: Different

Code

空 Run in an empty directory
create_env.sh

WAIT_SETUP_MYSQL=10
cat>my.cnf<<EOF
[mysqld]
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
# [client]
# default-character-set=utf8mb4
EOF
cat>docker-compose.yml<EOF
version: '3.1'
services:
  db:
    image:mysql:8
    command:--default-authentication-plugin=mysql_native_password 
    volumes:
      - ./data: /var/lib/mysql
      - ./logs: /var/log/mysql
      - ./my.cnf: /etc/mysql/conf.d/my.cnf
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD—example
      TZ: 'Asia/Tokyo'
  python:
    image:python:3.10
    volumes:
      - ./python/:/opt/project/
EOF
docker-compose up-ddb
docker-compose exec-Tdbbash <<EOF
while sleep1; do if [-S/var/run/mysqld/mysqld.sock]; then break; fi; done
sleep$WAIT_SETUP_MYSQL
EOF
docker-compose exec-Tdbbash <<EOF
mysql --user=root --password=example
use mysql
create user\`user\`@\`%\`IDENTIFIED BY'python';
create database sampledb character set utf8mb4 collate utf8mb4_bin;
grant all on sampledb.* to user;
use sampleb
create table sample(
id varchar(20)primary key,
name varchar (1024)
);
insert into sample values ('1001', 'Japanese garbled when connected from python to mysql using docker');
insert into sample values ('1002', 'create python and mysql containers using docker');
insert into sample values ('1003', 'When python connects to mysql and tries to display the table in fetchall');
insert into sample values ('1004', '🤧😷🤒🤕🤑🤠😈👿👹👺🤡💩👻💀');
exit
EOF
docker-compose exec-Tdbbash <<EOF
mysql --user=user --password=python --database=sampledb
show variables like 'chara%';
select * from sample;
exit
EOF
docker-compose exec-Tdbbash <<EOF
mysql --user=user --password=python --database=sampledb --default-character-set=utf8mb4
show variables like 'chara%';
select * from sample;
exit
EOF
docker-compose run --rmpython bash <<EOF
cd/opt/project
python-mvenvenv
./env/bin/python-mpip install mysql-connector-python PyMySQL
cat>sample.py<<INNER_EOF
import mysql.connector
cnx = mysql.connector.connect(
    host='db',
    port = '3306',
    user='user',
    password = 'python',
    database='sampledb',
    charset='utf8mb4'
)
cursor_=cnx.cursor()
cursor_.execute("show variables like 'chara%';")
rows=cursor_.fetchall()
For row in rows:
    print(row)
cursor_.execute("select*from sample;")
rows=cursor_.fetchall()
For row in rows:
    print(row)
INNER_EOF
cat>sample2.py<<INNER_EOF
import pymysql.cursors
cnx = pymysql.connect(
    host='db',
    port = 3306,
    user='user',
    password = 'python',
    database='sampledb',
    charset='utf8mb4'
)
cursor_=cnx.cursor()
cursor_.execute("show variables like 'chara%';")
rows=cursor_.fetchall()
For row in rows:
    print(row)
cursor_.execute("select*from sample;")
rows=cursor_.fetchall()
For row in rows:
    print(row)
INNER_EOF
./env/bin/python sample.py
./env/bin/python sample2.py
EOF
docker-compose down

Output

$sh../create_env.sh 
Creating network "tmp_default" with the default driver
Creating tmp_db_1 ... done
mysql: [Warning] Using a password on the command line interface can be secured.
ERROR1396(HY000) at line 2: Operation CREATE USER failed for 'user'@'%'
mysql: [Warning] Using a password on the command line interface can be secured.
Variable_name Value
character_set_client latin1
character_set_connection latin1
character_set_database utf8mb4
character_set_filesystem binary
character_set_results latin1
character_set_server utf8mb4
character_set_system utf8mb3
character_sets_dir/usr/share/mysql-8.0/charsets/
id name
Japanese is garbled when connecting from python to mysql using 1001 docker
Create python and mysql containers using 1002 docker.
1003 python connects to mysql and tries to display the table in fetchall
1004    🤧 😷 🤒 🤕 🤑 🤠 😈 👿 👹 👺 🤡 💩 👻 💀
mysql: [Warning] Using a password on the command line interface can be secured.
Variable_name Value
character_set_client utf8mb4
character_set_connection utf8mb4
character_set_database utf8mb4
character_set_filesystem binary
character_set_results outf8mb4
character_set_server utf8mb4
character_set_system utf8mb3
character_sets_dir/usr/share/mysql-8.0/charsets/
id name
1001 dockerã‚' "'y'thon pythonã‹s‚ myãmysqlã«Ž™¥¶çšãq㨋ock™æ‚ —ãŒæž ————————————————————————— ''ã™ã‚‹ª
1002 dockerã‚' "'y'th'on'ã pythons mymysqlã®ã‚³ƒã³ƒã†ƒã' 'ãžã‚ŒŒã —ˆäœ½ã‚æ 100 dã2Š‚ãockl„ãqer¨¦ã¨ p 'ç y ã —  ˆthœ  Œ   —
1003 pythonã s f l ¥ Ž ¶ ç š ã et — fetchallch ã § ƒ ƒ ã ƒ ã ' ‚ on ã ¨ p 100 ™ ¡ 3 º è th « y ‹ † q ã ã ¤ ˆ ¨ ã ã ‹ ¨ ã all ‚ ã ‚ ¼ ç ã ƒ ã ã ‰ — — my ã « – ‚ † æ ã   
1004🠤 ð · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
Requirement already satified: mysql-connector-python in./env/lib/python 3.10/site-packages (8.0.31)
Requirement already satified: PyMySQL in./env/lib/python 3.10/site-packages (1.0.2)
Requirement already satified:protobuf<=3.20.1,>=3.11.0 in./env/lib/python 3.10/site-packages (from mysql-connector-python) (3.20.1)
('character_set_client', 'utf8mb4')
('character_set_connection', 'utf8mb4')
('character_set_database', 'utf8mb4')
('character_set_filesystem', 'binary')
('character_set_results', 'utf8mb4')
('character_set_server', 'utf8mb4')
('character_set_system', 'utf8mb3')
('character_sets_dir', '/usr/share/mysql-8.0/charsets/')
('1001', 'dockerã '' "" \x81„81\x81th python \\x81‹\x81s xq81mysql x\x81« x™¶\x81™‚ x81‹ã\x81¨— —œ x¬81èŒ\x81 'æã x \ —\x81æ – )y8181 xadã x x xžª 100on '81ãŒã –ã‹‚81 ' ( \ã ' \ \lãå \ã \å d‡ p '‚er¨ock¦ \¥ my‰š¥æ ,181çãã‚ã
('1002', 'dockerã '' "" \x81„ x\x81th python x\x81smysql x\x81®‚lƒãƒ xム'†ã81‚ã'\x81\x9dã x \Œ81\x81 ' xãä\x81 — x )\x81 \ã‚90y\x81—)
('1003', 'python \\x81‹s‚qmysql x\x81«Ž x f¶\x81—fetchallã\x81§ƒƒã¼ƒ – x81ãã x3 )‚8181 '81 100 \º x¤ —81 x x8181 x xth¨\x81—çl81 \ã\x81ã\x81‹ 'ã™et‚ch¨all \ˆ‚81 'ã \ ( '¨ p† \ã —ã 'ãã \ \ my¡ ,‰ãonãš«yèç¥ãã \†æƒã8181  ) 
('1004', '🤧 😷 🤒 🤕 🤑 ðŸ¤\xa0 😈 👿 👹 👺 🤡 💩 👻 💀')
('character_set_client', 'utf8mb4')
('character_set_connection', 'utf8mb4')
('character_set_database', 'utf8mb4')
('character_set_filesystem', 'binary')
('character_set_results', 'utf8mb4')
('character_set_server', 'utf8mb4')
('character_set_system', 'utf8mb3')
('character_sets_dir', '/usr/share/mysql-8.0/charsets/')
('1001', 'dockerを用ã\x81„ã\x81¦pythonã\x81‹ã‚‰mysqlã\x81«æŽ¥ç¶šã\x81™ã‚‹ã\x81¨æ—¥æœ¬èªžã\x81Œæ–‡å\xad—化ã\x81‘ã\x81™ã‚‹')
('1002', 'dockerを用ã\x81„ã\x81¦pythonã\x81¨mysqlã\x81®ã‚³ãƒ³ãƒ†ãƒŠã‚’ã\x81\x9dã‚Œã\x81žã‚Œä½œæˆ\x90ã\x81—')
('1003', 'pythonã\x81‹ã‚‰mysqlã\x81«æŽ¥ç¶šã\x81—fetchallã\x81§ãƒ†ãƒ¼ãƒ–ルを表示ã\x81—よã\x81†ã\x81¨ã\x81™ã‚‹ã\x81¨')
('1004', '🤧 😷 🤒 🤕 🤑 ðŸ¤\xa0 😈 👿 👹 👺 🤡 💩 👻 💀')
Stopping tmp_db_1 ... done
Removing tmp_db_1 ... done
Removing network tmp_default
$ 

Unknown.
character_set_system is different from client-side default_character_set or somehow it was handled by latin1.

Uncomment my.cnf

Output

$sh../create_env.sh 
Creating network "tmp_default" with the default driver
Creating tmp_db_1 ... done
mysql: [Warning] Using a password on the command line interface can be secured.
mysql: [Warning] Using a password on the command line interface can be secured.
Variable_name Value
character_set_client utf8mb4
character_set_connection utf8mb4
character_set_database utf8mb4
character_set_filesystem binary
character_set_results outf8mb4
character_set_server utf8mb4
character_set_system utf8mb3
character_sets_dir/usr/share/mysql-8.0/charsets/
id name
Japanese is garbled when connecting from python to mysql using 1001 docker
Create python and mysql containers using 1002 docker.
1003 python connects to mysql and tries to display the table in fetchall
1004    🤧 😷 🤒 🤕 🤑 🤠 😈 👿 👹 👺 🤡 💩 👻 💀
mysql: [Warning] Using a password on the command line interface can be secured.
Variable_name Value
character_set_client utf8mb4
character_set_connection utf8mb4
character_set_database utf8mb4
character_set_filesystem binary
character_set_results outf8mb4
character_set_server utf8mb4
character_set_system utf8mb3
character_sets_dir/usr/share/mysql-8.0/charsets/
id name
Japanese is garbled when connecting from python to mysql using 1001 docker
Create python and mysql containers using 1002 docker.
1003 python connects to mysql and tries to display the table in fetchall
1004    🤧 😷 🤒 🤕 🤑 🤠 😈 👿 👹 👺 🤡 💩 👻 💀
Collecting mysql-connector-python
  Downloading mysql_connector_python-8.0.31-cp310-cp310-manylinux1_x86_64.whl (23.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.5/23.5 MB 27.1 MB/seta 0:00:00
Collecting PyMySQL
  Downloading PyMySQL-1.0.2-py3-none-any.whl (43kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.8/43.8 kB 5.7 MB/seta 0:00:00
Collecting protobuf<=3.20.1,>=3.11.0
  Downloading protobuf-3.20.1-cp310-cp310-manylinux_2_12_x86_64.manylinux 2010_x86_64.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 12.6 MB/seta 0:00:00
Installing collected packages: PyMySQL, protobuf, mysql-connector-python
Successfully installed PyMySQL-1.0.2 mysql-connector-python-8.0.31 protobuf-3.20.1
('character_set_client', 'utf8mb4')
('character_set_connection', 'utf8mb4')
('character_set_database', 'utf8mb4')
('character_set_filesystem', 'binary')
('character_set_results', 'utf8mb4')
('character_set_server', 'utf8mb4')
('character_set_system', 'utf8mb3')
('character_sets_dir', '/usr/share/mysql-8.0/charsets/')
('1001', 'Japanese garbled when connecting from python to mysql using docker')
('1002', 'Create python and mysql containers using docker' respectively)
('1003', 'When trying to connect from python to mysql and display the table in fetchall')
('1004', '🤧 😷 🤒 🤕 🤑 🤠 😈 👿 👹 👺 🤡 💩 👻 💀')
('character_set_client', 'utf8mb4')
('character_set_connection', 'utf8mb4')
('character_set_database', 'utf8mb4')
('character_set_filesystem', 'binary')
('character_set_results', 'utf8mb4')
('character_set_server', 'utf8mb4')
('character_set_system', 'utf8mb3')
('character_sets_dir', '/usr/share/mysql-8.0/charsets/')
('1001', 'Japanese garbled when connecting from python to mysql using docker')
('1002', 'Create python and mysql containers using docker' respectively)
('1003', 'When trying to connect from python to mysql and display the table in fetchall')
('1004', '🤧 😷 🤒 🤕 🤑 🤠 😈 👿 👹 👺 🤡 💩 👻 💀')
Stopping tmp_db_1 ... done
Removing tmp_db_1 ... done
Removing network tmp_default
$ 


2023-01-09 01:40

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.