I'm curious about Python curling using selenium.

I'd like to use Selenium to read comments from Naver News. Currently using Python 3.4 and Windows 8.1 64bit.

Currently, Naver News' comments can only be viewed by clicking on the "More" window, and before that, they only show 20 comments. I'd like to use selenium to run "more" five times and then retrieve the total comments.

from bs4 import BeautifulSoup from

selenium import webdriver

driver=webdriver.Chrome("C:\Users\kimty\Desktop\chromedriver.exe")

driver.get('http://news.naver.com/main/ranking/read.nhn?mid=etc&sid1=111&rankingType=popular_day&oid=214&aid=0000661050&date=20160820&type=2&rankingSectionId=101&rankingSeq=1&m_view=1')

html=driver.page_source

soup=BeautifulSoup(html,"html.parser")

typing=soup.find_all("span",{"class":"u_cbox_contents"})

print(typing)

driver.execute_script("u_cbox_page_more")

sleep(5)

Here's the code I made. There was an error, so I ran it one by one

driver.execute_script("u_cbox_page_more")

In this part,

Traceback (most recent call last):
File "", line 1, in driver.execute_script("u_cbox_page_more") File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 465, in execute_script 'args': converted_args})['value'] File "C:\Python34\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 236, in execute self.error_handler.check_response(response) File "C:\Python34\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 192, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: unknown error: u_cbox_page_more is not defined
(Session info: chrome=52.0.2743.116)
(Driver info: chromedriver=2.23.409699 (49b0fa931cda1caad0ae15b7d1b68004acd05129),platform=Windows NT 6.3.9600 x86_64)

I got this error

So I did the urea test again

How can I do what I want if it is not in the form of a Java script?

python selenium selenium-webdrive crawling crawler

2022-09-22 08:27

1 Answers

It's like a question about how to search for tags using HTML properties with JavaScript, where you can use JavaScript's DOM function to perform the function.

// A tag with that class can be obtained by the function below.
// Note, return an array to match the name "Elements".
var btns = document.getElementsByClassName('u_cbox_page_more');
// Assume that there are many HTML tags with the same class as the function name.
// Gets the first tag of them.
var btn = btns[0];
// Call the click event to that button.
btn.click();

You can run a script like this.

2022-09-22 08:27

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656