51Testing软件测试论坛

标题: selenium自动化爬虫运行越来越慢，如何解决？ [打印本页]

---

作者: 测试积点老人    时间: 2021-6-17 13:45
标题: selenium自动化爬虫运行越来越慢，如何解决？

最近用selenium写了一个小爬虫，需要循环在搜索框内输入内容，然后模拟点击搜索，在前进的新的页面获取数据，之后循环之前步骤，知道搜索结束。在爬虫刚开始运行的时候速度还可以，大约一秒一个页面，随着运行，速度开始越来越慢。

这里贴出代码

1. browser = webdriver.Chrome(executable_path="D:\GeckoDriver\chromedriver")
   
2. browser.get("https://www.qcc.com/")
   
3. # #4.设置浏览器的大小
   
4. browser.maximize_window()
   
5. 
   
6. login = browser.find_element_by_xpath('/html/body/header/div/ul/li[10]/a')
   
7. login.click()
   
8. # sleep(30)
   
9. # print("30 seconds later")
   
10. x = input("登录后请按y")
    
11. cookies = browser.get_cookies()
    
12. browser.quit()
    
13. browser = webdriver.Chrome(executable_path="D:\GeckoDriver\chromedriver",options = chrome_options)
    
14. # , options = chrome_options
    
15. browser.get("https://www.qcc.com/")
    
16. for cookie in cookies:
    
17. browser.add_cookie(cookie)
    
18. browser.get("https://www.qcc.com/")
    
19. browser.maximize_window()
    
20. qccinput = browser.find_element_by_css_selector("#searchkey")
    
21. # qccinput.clear()
    
22. qccinput.send_keys(companyNames[random.randint(0, len(companyNames))])
    
23. qccbutton = browser.find_element_by_css_selector(".index-searchbtn")
    
24. sleep(0.5)
    
25. qccbutton.click()
    
26. qccbutton = browser.find_element_by_css_selector(".input-group-btn")
    
27. sleep(0.5)
    
28. qccbutton.click()
    
29. pbar = tqdm(range(len(companyNames)))
    
30. for companyName, i in zip(companyNames, pbar):
    
31. browser.forward()
    
32. # browser.delete_all_cookies()
    
33. # browser.refresh();
    
34. lem = WebDriverWait(browser, 15, 0.5).until(EC.presence_of_element_located((By.ID, "searchKey"))) # 节约时间，网页出现这个元素再操作
    
35. seach = browser.find_element_by_css_selector("#searchKey")
    
36. seach.clear()
    
37. seach.send_keys(companyName)
    
38. seachButton = browser.find_element_by_css_selector(".btn-primary")
    
39. seachButton.click()
    
40. response = browser.page_source
    
41. html = etree.HTML(response)
    
42. result = etree.tostring(html)
    
43. cookies = browser.get_cookies()
    
44. try:
    
45. companyName = html.xpath(
    
46. 'normalize-space(/html/body/div[1]/div[2]/div[2]/div[3]/div/div[2]/div/table/tr[1]/td[3]/div/a[1])') # 去掉tbody
    
47. urls = html.xpath(
    
48. '/html/body/div[1]/div[2]/div[2]/div[3]/div/div[2]/div/table/tr[1]/td[3]/div/a[1]/@href') # 去掉tbody
    
49. getCompanyNames.append(companyName)
    
50. getTaxpayerNumber(urls, cookies)
    
51. except Exception as r:
    
52. getCompanyNames.append("无法搜索到公司")
    
53. taxpayerNumber.append("无法查到税号")
    
54. # browser = webdriver.Chrome(profile)
    
55. # send_command = ('POST', '/session/$sessionId/chromium/send_command')
    
56. # browser.command_executor._commands['SEND_COMMAND'] = send_command
    
57. # browser.execute('SEND_COMMAND', dict(cmd='Network.clearBrowserCache', params={}))
    
58. browser.delete_all_cookies()
    
59. for cookie in cookies:
    
60. browser.add_cookie(cookie)

复制代码

---

欢迎光临 51Testing软件测试论坛 (http://bbs.51testing.com/)

Powered by Discuz! X3.2