Error while running web scraping

I have tried web scraping for the first time by referring to the site below.
http://blog.aidemy.net/entry/2017/12/17/214715

The following error occurred when I tried and obtained only API_KEY and CUSTOM_SEARCH_ENGINE and used the source code of the image collection as it was.

Traceback (most recent call last):
  File "gazou_shutoku.py", line 52, in<module>
    img_list = get_image_url (keywords[j], 100)
  File "gazou_shutoku.py", line 26, inget_image_url
    res=urllib.request.urlopen(query_img)
  File"C:\Users\Username\Anaconda3\envs\gazou_atsume\lib\urllib\request.
py", line 223, in urlopen
    return opener.open(url,data,timeout)
  File"C:\Users\Username\Anaconda3\envs\gazou_atsume\lib\urllib\request.
py", line 532, in open
    response=meth(req, response)
  File"C:\Users\Username\Anaconda3\envs\gazou_atsume\lib\urllib\request.
py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File"C:\Users\Username\Anaconda3\envs\gazou_atsume\lib\urllib\request.
py", line570, in error
    return self._call_chain(*args)
  File"C:\Users\Username\Anaconda3\envs\gazou_atsume\lib\urllib\request.
py", line 504, in_call_chain
    result=func(*args)
  File"C:\Users\Username\Anaconda3\envs\gazou_atsume\lib\urllib\request.
py", line 650, in http_error_default
    raise HTTPError(req.full_url,code,msg,hdrs,fp)
urllib.error.HTTPError:HTTPError403:Forbidden

What is the problem?Thank you for your cooperation.

python web-scraping

2022-09-30 11:23

1 Answers

From the error message, the error occurred in the following places:

res=urllib.request.urlopen(query_img)

Therefore, I think there is a problem with the query_image content.

query_img="https://www.googleapis.com/customsearch/v1?key="+API_KEY+"&cx="+CUSTOM_SEARCH_ENGINE+"&num="+str(10 if(total_num-i)>10 else(total_num-i))+"&start="+str(i+1)+quear="&quarech"+"+"

First, to see if the Google customsearch API is enabled, enter the following URL directly in the browser URL field:For API_KEY and CUSTOM_SEARCH_ENGINE, enter your number.

https://www.googleapis.com/customsearch/v1?key=API_KEY&cx=CUSTOM_SEARCH_ENGINE&q=Erika Ikuta & searchType=image

So, if you get an error, please check the API_KEY and CUSTOM_SEARCH_ENGINE settings again.The error is marked as Forbidden, so there is probably a problem with the configuration.Google API misconfiguration is common.

If you can see it correctly, print the query_img value, etc., and check if the query_img value is correct because Custom Search APIs Explorer is located below.
https://developers.google.com/apis-explorer/?hl=ja#p/customsearch/v1/search.cse.list

2022-09-30 11:23

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656