Python request delay question!!

When crawling the web

res = requests.get(url, headers=custom_headers, params = paramDict,cookies=cookies)

If you send a request by getting like this,

When the website receives a request, it does not spray it immediately, but before it is sprayed

On an empty homepage

I'm going to put this on the web page, and I'm going to put it on the web page.

So in my res, document.cookie = "testck2=e1f8c5e995f64c61ad0e0891fba09fc0; path=/"; loading...

It only comes out like this.

I want to crawl the values after they are all sprayed

Is requests too much? Should I use selenium?

python requests crawling

2022-09-21 10:22

1 Answers

It usually appears as "Loading..." when the client renders it to javascript. This is difficult with a typical GET because you need to use the state of the DOM that is created after Javascript runs.

2022-09-21 10:22

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656