Attempt to insert crawled data into mysql table. My code is
def process_item(self, item, spider):
self.cursor.execute("SELECT * FROM dmmactress_jpconvert WHERE jpname = %s AND koname = %s AND enname = %s", (item['jpname'], item['koname'], item['enname']))
result = self.cursor.fetchone()
if result:
print("data already exist")
else:
try:
self.cursor.execute("INSERT INTO dmmactress_jpconvert(jpname, koname, enname) VALUES (%s, %s, %s)", (item['jpname'], item['koname'], item['enname']))
self.conn.commit()
except MySQLdb.Error as e:
print ("Error %d: %s" % (e.args[0], e.args[1]))
return item
This is.
The problem is that the id should go up to 8000 times in an empty table without data If you execute the above code, the data already exists will be printed and only about 20% will be stored in the data. I don't think there's a problem with the code. I wonder when I get into trouble like myself.
+--------+--------------+------+-----+---------+----------------+
| | Field | Type | Null | Key | Default | Extra |
+--------+--------------+------+-----+---------+----------------+
| | id | int(11) | NO | PRI | NULL | auto_increment |
| | jpname | varchar(100) | NO | | NULL | |
| | koname | varchar(100) | NO | | NULL | |
| | enname | varchar(100) | NO | | NULL | |
+--------+--------------+------+-----+---------+----------------+
Crawled data seems to have duplicate values.
self.cursor.execute("SELECT * FROM dmmactress_jpconvert WHERE jpname = %s AND koname = %s AND enname = %s", (item['jpname'], item['koname'], item['enname']))
result = self.cursor.fetchone()
If you look at , check through fetchone to see if DB has values of 'jpname', 'koname', and 'enname' If the value is already in the DB, it seems to show the phrase "data already exist" or put the data in the DB.
An empty table should contain up to 8,000 id's, but in fact, up to 2000 means that out of 8,000 data, there are about 6,000 data that overlap jpname, koname, enname.
© 2024 OneMinuteCode. All rights reserved.