curling is not working well;

Crawling through search keyword.

These days, Daum and Naver have changed from crawling to not being able to do it I was going to do it with selenium If you run the code below, you will get an empty value.
What's wrong?

from selenium import webdriver
import requests
from bs4 import BeautifulSoup
import time

dict_keyword = {}

def daum():

    browser = webdriver.Chrome("C:/Users/eheee/Desktop/webdriver/chromedriver.exe")
    browser.get("https://www.naver.com")

    html = browser.page_source

    soup = BeautifulSoup(driver.page_source, "html.parser")


    search_word = soup.select("#sp_nws_all1 > dl > dt")
    for i in search_word:
        title = i.find("a").attrs['href']
        link = "https://www.daum.net" + i.find("a")["href"]
        dict_keyword[title.text] = title.get('href')

    #return dict_keyword
    print(dict_keyword)

python crawling selenium

2022-09-21 11:38

2 Answers

Did you call daum()?

2022-09-21 11:38

Selenium is only imported, and it seems to crawl with beautiful soup

2022-09-21 11:38

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656