How to crawl multiple pages of fixed url?

Asked 2 years ago, Updated 2 years ago, 91 views

I'm practicing crawl, and I'd like to collect reviews on several pages as above.

Page 1 barely succeeded in crawling, but from the next page

Even if I go to the next page, the url doesn't change, so I'm having a hard time curling

from bs4 import BeautifulSoup
from selenium import webdriver
from time import *

driver = webdriver.Chrome()
driver.get('https://sokoglam.com/collections/soko-glam-best-of-beauty-awards/products/hanskin-cleansing-oil-blackhead-pha')

for a in range(9):
    html = driver.page_source
    soup = BeautifulSoup(html,'html.parser')
    r = soup.select('.yotpo-review-wrapper')
    for i in r:
        print(i.select_one('.content-review').text)
    sleep(3)

The url does not change even if you go to the next page, so after crawling one page through the web drive, sleep (3) quickly presses the next page to crawl manually

How can I crawl several pages when url doesn't change even if I go to the next page?

python crawling

2022-09-22 18:41

1 Answers

You only need to change the yoReviewsPage value at the address below and request it.

https://sokoglam.com/collections/soko-glam-best-of-beauty-awards/products/hanskin-cleansing-oil-blackhead-pha?yoReviewsPage=1


2022-09-22 18:41

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.