Yahoo Search Scraping Can't Get Only Ads

Asked 2 years ago, Updated 2 years ago, 182 views

I'm currently learning scraping at Python 3 and I know it's a gray zone to scrape Yahoo searches when I've tried various sites, but when I tried it (I don't use anything), I couldn't get an ad, so I wondered.
Originally, I used requests + beautifulSoup, but when I tried selenium, I couldn't.
So I thought I couldn't search Google, so I tried it, but it seems that it was done.

Does Yahoo intentionally make it impossible to do intentionally?
Then I was curious about many things such as why Google didn't adopt it.
Also, is there a way that Yahoo can do it?I would appreciate it if you could let me know as knowledge without using it.

Additional
Here is the code to the left using requests

import requests
from bs4 import BeautifulSoup
import json
import time

r=requests.get ("https://search.yahoo.co.jp/search?p=cheap rental car")
r.raise_for_status()
soup = BeautifulSoup(r.content, "html.parser")

print(soup)

Here is the code used in selenium.

 from selenium import webdriver
from selenium.webdriver.chrome.options import Options

driver=webdriver.Chrome()

# Generate options instance
options=Options()
# Set options
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')

driver.get ('https://search.yahoo.co.jp/search?p=cheap rental')

drive=driver.page_source

print(drive)

python3 web-scraping selenium beautifulsoup python-requests

2022-09-30 21:46

1 Answers

From metropolis
When I set up the User-Agent and run requests.get(), I was able to get the same page as when I accessed it with a web browser (Firefox in my environment).


2022-09-30 21:46

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.