I want Python scraping on the cloud (preferably in parallel)

I summarized it briefly under the constraints of the cloud environment.

• Use cron to launch scripts periodically
  • cron is a daemon that runs jobs periodically, and you can specify when to run them in minutes.
  • cron-ArchWiki
• Periodically run the acquisition process while starting the application all the time
  • A library called schedule is useful for Python.However, this method requires action if the application terminates unexpectedly.
  • schedule—schedule 0.4.0 documentation

• cron is a daemon that runs jobs periodically, and you can specify when to run them in minutes.
• cron-ArchWiki

• A library called schedule is useful for Python.However, this method requires action if the application terminates unexpectedly.
• schedule—schedule 0.4.0 documentation

These methods are traditional and documentative, but they also incur the cost of constantly starting the server and monitoring and maintaining the server itself.

Therefore, in recent years, the following methods have become more mainstream:

• You can register functions in AWS Lambda or Google Cloud Functions and run them periodically like cron.In this case, I think that the function A that you want to parallelize and the function B that calls the function you want to parallelize are registered, the function B is executed periodically, and the function A is called separately.
  • AWS Lambda | AWS
  • Cloud Functions - Event-Driven Serverless Computing|Cloud Functions|Google Cloud

• AWS Lambda | AWS
• Cloud Functions - Event-Driven Serverless Computing|Cloud Functions|Google Cloud

However, this method cannot store data locally, so we need to prepare a separate destination for storing data.You also need to write code according to FaaS style (function name and execution time are limited).

Choose which one you want to choose according to your requirements.