I'm practicing crawl, and I'd like to collect reviews on several pages as above.
Page 1 barely succeeded in crawling, but from the next page
Even if I go to the next page, the url doesn't change, so I'm having a hard time curling
from bs4 import BeautifulSoup
from selenium import webdriver
from time import *
driver = webdriver.Chrome()
driver.get('https://sokoglam.com/collections/soko-glam-best-of-beauty-awards/products/hanskin-cleansing-oil-blackhead-pha')
for a in range(9):
html = driver.page_source
soup = BeautifulSoup(html,'html.parser')
r = soup.select('.yotpo-review-wrapper')
for i in r:
print(i.select_one('.content-review').text)
sleep(3)
The url does not change even if you go to the next page, so after crawling one page through the web drive, sleep (3) quickly presses the next page to crawl manually
How can I crawl several pages when url doesn't change even if I go to the next page?
python crawling
You only need to change the yoReviewsPage
value at the address below and request it.
© 2024 OneMinuteCode. All rights reserved.