Python scraping doesn't work.

https://ntp.msn.com/edge/ntp?locale=ja

Microsoft Edge ↑ I took the news from this URL and tried to display it, but an error occurred to see if BeautifulSoup was working properly.

import requests
from bs4 import BeautifulSoup

load_url="https://ntp.msn.com/edge/ntp?locale=ja"
html=requests.get(load_url)
soup = BeautifulSoup(html.content, "html.parser")
print(soup)

topic=soup.find(class_="news-list")
for element intopic.find_all("a"):
    print(element.text)

This code

topic=soup.find(class_="news-list")

in the line of

An exception has occurred: AttributeError
'NoneType' object has no attribute 'find_all'
  File "Hidden Path", line 10, in <module>
    for element intopic.find_all("a"):

The error
occurs. If anyone knows how to solve it, please reply.

python beautifulsoup python-requests

2022-09-30 19:29

1 Answers

You probably don't understand where the error occurred.

The question states that an error occurred on the following line:

topic=soup.find(class_="news-list")

The error message indicates that an error occurred on the following line:

for element intopic.find_all("a"):

Since it is actually 'NoneType' object has no attribute 'find_all', and the object topic is NoneType (=None), the first problem is presented because it does not have the attribute find_all (in this case, method) _group>.

If there is still a gap in perception around here, it is more likely that the investigation will never progress and will not be resolved by making mistakes in what or what needs to be investigated and addressed.

Has it been confirmed that there is an element with the class name "news-list" in the web page contents that can be retrieved manually by using the MS-Edge browser instead of the Python program?

I think it would be better to work step by step so that you can gradually move from those prerequisites to the program.



		
		
			

				

					
				

				
					2022-09-30 19:29

			
			If you have any answers or tips



		

	
		Popular Tags
	
	python x 4647
android x 1593
java x 1494
javascript x 1427
c x 927
c++ x 878
ruby-on-rails x 696
php x 692
python3 x 685
html x 656
	


	
		Popular Questions
	
	
	609 GDB gets version error when attempting to debug with the Presense SDK (IDE)

	578 Understanding How to Configure Google API Key

	910 When building Fast API+Uvicorn environment with PyInstaller, console=False results in an error

	1022 In Java servlet, when SHA-256 sends WW-Authenticate header for digest authentication, the client does not return the result.

	571 rails db:create error: Could not find mysql2-0.5.4 in any of the sources