Scrapy redirect problem

Asked 1 years ago, Updated 1 years ago, 84 views

Hello, this is a crawling question about Scrappy or beautiful soup.

After the crawler is written, it is being tested while actually curling. Other parts work normally, but redirection is the problem.

For example, if you send a request to www.example.com/abc, the response is www.example.com/en/abc It will be redirected to the English version page.

It's not like that on a web browser, but it's just bot, so it seems like it's filtered from the server.

I set the user-agent differently, but it keeps redirecting to the English page.

What should I think of in this case?

Please give me a hint ㅠ<

python scrapy

2022-09-21 21:49

1 Answers

If you have a browser that opens the page of your choice, why don't you visit http://www.whoishostingthis.com/tools/user-agent/ and set it up?


2022-09-21 21:49

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.