新人學(xué)scrapy，最近在爬金華信義居的房屋信息新房 - 列表，樓盤詳細(xì)信息都很順利的爬取成功了，但抓樓盤單元就報(bào)401的錯(cuò)，抓破腦袋不知道咋搞！

百度了說是401 是需要驗(yàn)證用戶信息，但具體應(yīng)該怎么操作呢？？？

貼上代碼

import datetime import json import pandas as pd import scrapy from jinhua.items import HouseItemclass ProjectSpider(scrapy.Spider):name = 'jhhouse'allowed_domains = ['https://www.jhtmsf.com']start_urls = ['https://www.jhtmsf.com/House/GetPageForRoom']spiderTime = datetime.datetime.now().strftime('%Y-%m-%d')def start_requests(self):df = pd.read_excel('../采集結(jié)果/py_jhtmsf_loupan.xlsx')for idx, row in df.iterrows():project_id = row['project_id']referer_url = 'https://www.jhtmsf.com/House/Room/' + str(project_id)yield scrapy.FormRequest(self.start_urls[0],headers={'Referer': referer_url},cookies={'__RequestVerificationToken': 'XpgK_gMXlG71JzgTKt27kPr9ZQE1Ptbm6DhRfN7Ol7OMuuS_p43T6XOKkwg48zNUlI5jSYlJA97oO_KoupElbXV5Zm-1ldmVCCjUkltPB8c1','ASP.NET_SessionId': 'vwtw2pdanxrkdnfbqimzol1u','Hm_lvt_88b265ab6b07373c61ffa7d36d6db2c3': '1634609773,1634711852,1634882161,1634883812','Hm_lpvt_88b265ab6b07373c61ffa7d36d6db2c3': '1634883832'},formdata={'eid': str(project_id), 'bulid': '', 'layer': '', 'status': '0', 'pageNumber': '1', 'pageSize': '15', 'sortName': 'StartDate', 'sortOrder': 'desc'},callback=self.parse,meta={'project_id': project_id, 'referer_url': referer_url})def parse(self, response):jsonBody = json.loads(response.body)page = jsonBody["TotalPage"]total = jsonBody["Total"]project_id = response.meta['project_id']referer_url = response.meta['referer_url']for pg in range(1, int(page) + 1):yield scrapy.FormRequest(self.start_urls[0], headers={'Referer': str(referer_url)},formdata={'eid': str(project_id), 'bulid': '', 'layer': '', 'status': "0", 'pageNumber': str(pg),'pageSize': "15", 'sortName': 'StartDate', 'sortOrder': 'desc'},callback=self.content_parse, meta={'project_id': project_id, 'total': total})def content_parse(self, response):jsonBody = json.loads(response.body)jrows = jsonBody["Rows"]if jrows:for row in jrows:item = HouseItem()item['project_id'] = response.meta['project_id']item['total'] = response.meta['total']item['area'] = row['Area']item['build_nb'] = row['Bulid']item['on_layer'] = row['Layer']item['price'] = row['Price']item['room_nb'] = row['RoomNO']item['start_time'] = row['StartDate']item['house_status'] = row['Status']item['spider_time'] = self.spiderTimeyield item

總結(jié)

以上是生活随笔為你收集整理的scrapy 报错401的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。