TA的每日心情 | 擦汗 前天 09:07 |
---|
签到天数: 527 天 连续签到: 4 天 [LV.9]测试副司令
|
1测试积点
python的返回值news_detail4无法被get_equal_rate_1认定为字符串,请问如何能把news_detail4也一起进行比较?
我这里是先获取新闻网页内容,然后进行比较,前三个爬取返回值可以进行比较,第四个不行,请问该怎么办?
- <p>import difflib
- from xml.etree.ElementTree import tostring
- import requests
- from lxml import etree
- import time
- from gne import GeneralNewsExtractor
- from selenium.webdriver import Chrome
- from selenium.webdriver.chrome.options import Options
- def get_chinanew_data():
- cookies = {
- 'Hm_lvt_0da10fbf73cda14a786cd75b91f6beab': '1587367903',
- 'Hm_lpvt_0da10fbf73cda14a786cd75b91f6beab': '1587375545',
- }
- headers = {
- 'Connection': 'keep-alive',
- 'Cache-Control': 'max-age=0',
- 'Upgrade-Insecure-Requests': '1',
- 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36',
- 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
- 'Accept-Language': 'zh-CN,zh;q=0.9',
- }</p><p>response = requests.get('http://www.chinanews.com/gn/2020/04-20/9162019.shtml', headers=headers, cookies=cookies,
- verify=False)
- html = response.content.decode(errors='ignore')
- etree_html = etree.HTML(html)
- main = etree_html.xpath('//div[@id="cont_1_1_2"]')[0]
- title = main.xpath('./h1/text()')[0]
- pub_time = main.xpath(".//div[3]/div[@class='left-t']/text()")[0]
- author = main.xpath('./div[5]/div[2]/div/span/text()')[0][:-2].split(':')[1]
- pubtime = pub_time.split()[0] + ' ' + pub_time.split()[1]
- content = ''.join(main.xpath('./div[@class="left_zw"]/p/text()')).strip()
- site_url = 'http://www.chinanews.com/gn/2020/04-20/9162019.shtml'
- site_name = '中国新闻网'
- news_detail = {
- 'pub_time': pubtime.replace('年', '-').replace('月', '-').replace('日', ''),
- 'author': author,
- 'title': title,
- 'content': content.replace('\u3000', ''),
- 'site_url': site_url,
- 'site_name': site_name,
- }
- return news_detail
- def selenium_download_data():
- options = Options()
- options.add_argument('--headless')
- driver = Chrome(options=options,executable_path=r"C:\Users\常乐添\AppData\Local\Google\Chrome\Application\chromedriver.exe")
- url_list = [
- 'https://news.sina.com.cn/gov/xlxw/2020-04-20/doc-iircuyvh8766402.shtml',
- 'https://news.ifeng.com/c/7vovtvQ2gVc',
- 'https://baijiahao.baidu.com/s?id=1664460259411900230&wfr=spider&for=pc']</p>
复制代码
|
|