Python Beautiful Soup select result is strange.

I'm practicing Python crawling, and I'm collecting sections from Naver's integrated search as follows.

If you search with the following code, there should be 9 words for section, but there are only 2 words. How can we solve this?

I want to resolve it using the body > div > div > div > section or #main_pack > section path.

If you enter #main_pack > section , you should get 6 results, but only 1 result.

import requests
import urllib.request
from bs4 import BeautifulSoup

url = 'https://search.naver.com/search.naver?where=nexearch&sm=top_hty&fbm=1&ie=utf8&query=%EC%82%BC%EC%84%B1%EC%A0%84%EC%9E%90'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')

sections = soup.select('body > div > div > div > div > section')
for section in sections:
       section = section.name
       print (section)

python crawling beautifulsoup

2022-09-20 19:26

1 Answers

import requests
import urllib.request
from bs4 import BeautifulSoup

url = 'https://search.naver.com/search.naver?where=nexearch&sm=top_hty&fbm=1&ie=utf8&query=%EC%82%BC%EC%84%B1%EC%A0%84%EC%9E%90'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
sections = soup.select('#main_pack section')
#sections = soup.select('body > div > div > div > div > section')
for section in sections:
       section = section.name
       print (section)

Once this is done, the six are normally output.

2022-09-20 19:26

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656