Home >  > Requests与Xpath搭配

Requests与Xpath搭配

0

今天发现使用Requests与Xpath还是挺搭的,而且配合chrome的xpath插件与scrapy shell,操作起来非常简单:

Updated:
其实可以使用浏览器的copy xpath功能。

附代码:

import requests
from lxml import etree

def getResult(i):
	url = "http://www.xuetu123.com/space-uid-{}.html".format(i)
	r = requests.get(url)
	selector = etree.HTML(r.text)
	userName = selector.xpath("//div[@class='nex_Home_tops_Main']/h2/text()")[0]
	userName = userName.replace('\r\n','')
	userName = userName.strip()
	#没有管理组的情况
	try:
		userType = selector.xpath("//div[@class='pbm mbm bbda cl']/ul//li/span/a/text()")[0]
	except:
	#有管理组:
		userType = selector.xpath("//div[@class='pbm mbm bbda cl']/ul/li[2]/span/a/font/text()")[0]
	return (userName,userType)

for i in range(1,2361):
	try:
		tempresult = getResult(i)
		print("*"*22)
		print(tempresult[0])
		print(tempresult[1])
		with open("result.txt","a") as f:
			f.write(tempresult[0]+tempresult[1]+'\n')
		print("写入第{}条记录成功".format(i))			
	except:
		print("写入第{}条记录错误".format(i))
		continue

本文暂无标签

发表评论

*

*