Enable Web Crawling Infinite (?)

Asked 2 years ago, Updated 2 years ago, 97 views

Hello, I created coding to crawl some stock-related financial information as below

It works well when you play only two sports with code_list_test that is annotated I turned the whole event code_list (2000 or more) and

It doesn't go on after a long time in progress. Masters, help me! What's the problem?

crawling beautifulsoup

2022-09-21 11:08

1 Answers

It's not a problem, it's a long run because it's going to run more than 2000 sequentially. * Indicates that the mark is running.

If you use async/waite to perform concurrent, you can reduce the io latency of http.


2022-09-21 11:08

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.