Python Beautiful Soup select result is strange.

Asked 2 years ago, Updated 2 years ago, 88 views

I'm practicing Python crawling, and I'm collecting sections from Naver's integrated search as follows.

If you search with the following code, there should be 9 words for section, but there are only 2 words. How can we solve this?

I want to resolve it using the body > div > div > div > section or #main_pack > section path.

If you enter #main_pack > section , you should get 6 results, but only 1 result.

import requests
import urllib.request
from bs4 import BeautifulSoup

url = 'https://search.naver.com/search.naver?where=nexearch&sm=top_hty&fbm=1&ie=utf8&query=%EC%82%BC%EC%84%B1%EC%A0%84%EC%9E%90'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')

sections = soup.select('body > div > div > div > div > section')
for section in sections:
       section = section.name
       print (section)

python crawling beautifulsoup

2022-09-20 19:26

1 Answers

import requests
import urllib.request
from bs4 import BeautifulSoup

url = 'https://search.naver.com/search.naver?where=nexearch&sm=top_hty&fbm=1&ie=utf8&query=%EC%82%BC%EC%84%B1%EC%A0%84%EC%9E%90'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
sections = soup.select('#main_pack section')
#sections = soup.select('body > div > div > div > div > section')
for section in sections:
       section = section.name
       print (section)

Once this is done, the six are normally output.


2022-09-20 19:26

If you have any answers or tips


© 2024 OneMinuteCode. All rights reserved.