When you scrap a text, you will get a message like no such element: Possible to location element

from bs4 import BeautifulSoup
import requests
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

session = requests.Session()

url = 'https://amzn.to/35lnOFf'
res = session.get(url, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'}).content
soup = BeautifulSoup(res, 'lxml')

chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome('./chromedriver', options=chrome_options)
driver.get(url)

asin = driver.find_element_by_id('detailBullets_feature_div').text
print(asin)

================================================================= This error occurs when executing the above code.You want to collect all text under this id. Please do something about it.Thank you. selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"detailBullets_feature_div"} (Session info: headless chrome=78.0.3904.108) (Driver info: chromedriver=2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT 6.1.7601 SP1 x86_64)

python

2022-09-22 18:10

1 Answers

from requests_html import HTMLSession


session = HTMLSession()
url = 'https://amzn.to/35lnOFf'

res = session.get(url)
res.html.render()  # Javascript Render

targets = [x.text for x in res.html.find("#detailBullets_feature_div li")]
for target in targets:
    print(target)

I think you used Selenium because you had to render Javascript.
If it's headless anyway, requests-html, rendering and Element Select are very convenient.

pip3 install requests-html

2022-09-22 18:10

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656