Scroll down question in Python Selenium Web Crawling.

Asked 2 years ago, Updated 2 years ago, 75 views

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys 
from selenium import webdriver
import time

driver = webdriver.Chrome()
url = 'https://pubmed.ncbi.nlm.nih.gov/' 
driver.get(url)  
driver.find_element_by_xpath('/html/body/div[2]/main/div[1]/div/form/div/div[1]/div/span/input').send_keys('human AND well-being AND University')
driver.find_element_by_class_name('search-btn').click()

last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    new_height = driver.execute_script("return document.body.scrollHeight")
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    driver.find_element_by_class_name('load-button.next-page').click()
    time.sleep(1)
    if new_height == last_height:
            try:
                driver.execute_script("window.scrollTo(document.body.scrollHeight, 0);")
            except:
                break
    last_height = new_height

I googled and turned it around, and when I type it in the search box, the scroll bar goes down to the next page, and the scroll bar goes down to the next page, and after about 40 pages, there's no more literature, so I want to put it up at the top.

However, if there are no more pages, then driver.find_element_by_class_name('load-button.next-page').An error code appears in click() and does not proceed to the next step.

Please answer me.

web-crawling python selenium

2022-09-20 15:42

1 Answers

The first problem is that an error has occurred and then the code is not executed.

If there is no more literature to import, it seems to be an error caused by the disappearance of certain buttons.

Try exception can be used to avoid errors as follows:

try:
    driver.find_element_by_class_name('load-button.next-page').click()
except:
    pass


2022-09-20 15:42

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.