Questions about how to crawl information to Python only when you refresh the website

Asked 1 years ago, Updated 1 years ago, 58 views

안녕하세요.

I'm trying to make a code to compare the price of Wine List with the overseas price. I'm using Wine Searsher as a site to check overseas prices, and I'm inquiring because there's a problem.

You can search for wine using the URL combination, but strangely, if you enter the URL for the first time, the average price does not come out, and if you re-enter the URL in the same state, the average price is displayed.

https://www.wine-searcher.com/find/p%C3%AAra+manca+tinto?Xcurrencycode=EUR&Xtax_mode=e&Xsavecurrency=Y

At first, there was no average price in the wine information as shown in the picture above,

If you re-enter a URL, such as Refresh, the average price is displayed as shown in the picture above.

The price I need is the average price, but I can't get that value at once.

Is there any solution to making the code with Requests?

I was thinking about using Selenium, but I think it will be hard because Captcha is displayed right away when I access Selenium from that site.

python requests selenium

2022-09-20 19:01

1 Answers

from msedge.selenium_tools import EdgeOptions
from msedge.selenium_tools import Edge
from time import sleep

options = EdgeOptions()
options.use_chromium = True
# # options.add_argument('headless')
# # options.add_argument('disable-gpu')
# # options.add_argument("--log-level=OFF")
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--ignore-certificate-errors')
driver = Edge(executable_path='C:/edgedriver_win64/msedgedriver.exe', options=options)

url = "https://www.wine-searcher.com/find/pêra+manca+tinto?Xcurrencycode=EUR&Xtax_mode=e&Xsavecurrency=Y"
driver.get(url)
driver.refresh()

sleep(10) driver.close()

It's an option that I put in because I kept getting a warning that ignore something is not a private connection, and if it doesn't come up, you don't have to put it in. The annotated part prevents the window from turning on and off when the web driver is turned on in the selenium and makes it turn completely in the background. I used an edge browser and copied and pasted it to fit the edge, and if you use Firefox or Chrome, you'll have to look for the code separately.


2022-09-20 19:01

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.