一、媒體管道

1.1、媒體管道的特性

媒體管道實現了以下特性:

• 避免重新下載最近下載的媒體

• 指定存儲位置（文件系統目錄，Amazon S3 bucket，谷歌云存儲bucket）

圖像管道具有一些額外的圖像處理功能:

• 將所有下載的圖片轉換為通用格式(JPG)和模式(RGB)
• 生成縮略圖
• 檢查圖像的寬度/高度，進行最小尺寸過濾

---

1.2、媒體管道的設置

ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 120} 啟用 FILES_STORE = '/path/to/valid/dir' 文件管道存放位置 IMAGES_STORE = '/path/to/valid/dir' 圖片管道存放位置 FILES_URLS_FIELD = 'field_name_for_your_files_urls' 自定義文件url字段 FILES_RESULT_FIELD = 'field_name_for_your_processed_files' 自定義結果字段 IMAGES_URLS_FIELD = 'field_name_for_your_images_urls' 自定義圖片url字段 IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images' 結果字段 FILES_EXPIRES = 90 文件過期時間 默認90天 IMAGES_EXPIRES = 90 圖片過期時間 默認90天 IMAGES_THUMBS = {'small': (50, 50), 'big':(270, 270)} 縮略圖尺寸 IMAGES_MIN_HEIGHT = 110 過濾最小高度 IMAGES_MIN_WIDTH = 110 過濾最小寬度 MEDIA_ALLOW_REDIRECTS = True 是否重定向

---

二、ImagesPipeline類簡介

#解析settings里的配置字段 def __init__(self, store_uri, download_func=None, settings=None) #圖片下載 def image_downloaded(self, response, request, info) #圖片獲取 圖片大小的過濾 #縮略圖的生成 def get_images(self, response, request, info) #轉化圖片格式 def convert_image(self, image, size=None) #生成媒體請求 可重寫 def get_media_requests(self, item, info)return [Request(x) for x in item.get(self.images_urls_field, [])] #得到圖片url 變成請求 發給引擎 #此方法獲取文件名 進行改寫 def item_completed(self, results, item, info) #文件路徑 def file_path(self, request, response=None, info=None) #縮略圖的存儲路徑 def thumb_path(self, request, thumb_id, response=None, info=None):

三、小案例：使用圖片管道爬取百度圖片

（當然不使用圖片管道的話也是可以爬取百度圖片的，但這還需要我們去分析網頁的代碼，還是有點麻煩，使用圖片管道就可以省去這個步驟了）

3.1、spider文件

注意：由于需要添加所有的請求頭，所以我們要重寫start_requests函數

import reimport scrapy from ..items import DbimgItemclass DbSpider(scrapy.Spider):name = 'db'# allowed_domains = ['xxx.com']start_urls = ['https://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=index&fr=&hs=0&xthttps=111110&sf=1&fmq=&pv=&ic=0&nc=1&z=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&word=%E7%8B%97&oq=%E7%8B%97&rsp=-1']def start_requests(self): #因為需要添加所有的請求頭，所以我們要重寫start_requests函數# url = 'https://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1&fm=index&fr=&hs=0&xthttps=111110&sf=1&fmq=&pv=&ic=0&nc=1&z=&se=1&showtab=0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&word=%E7%8B%97&oq=%E7%8B%97&rsp=-1'headers = {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","Accept-Encoding": "gzip, deflate, br","Accept-Language": "zh-CN,zh;q=0.9","Cache-Control": "max-age=0","Connection": "keep-alive","Cookie": "BIDUPSID=4B61D634D704A324E3C7E274BF11F280; PSTM=1624157516; BAIDUID=4B61D634D704A324C7EA5BA47BA5886E:FG=1; __yjs_duid=1_f7116f04cddf75093b9236654a2d70931624173362209; BAIDUID_BFESS=101022AEE931E08A9B9A3BA623709CFE:FG=1; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; BDRCVFR[dG2JNJb_ajR]=mk3SLVN4HKm; cleanHistoryStatus=0; H_PS_PSSID=34099_33969_34222_31660_34226_33848_34113_34073_33607_34107_34134_34118_26350_22159; delPer=0; PSINO=6; BA_HECTOR=24ak842ka421210koq1gdtj070r; BDRCVFR[X_XKQks0S63]=mk3SLVN4HKm; userFrom=www.baidu.com; firstShowTip=1; indexPageSugList=%5B%22%E7%8B%97%22%2C%22%E7%8C%AB%E5%92%AA%22%2C%22%E5%B0%8F%E9%80%8F%E6%98%8E%22%5D; ab_sr=1.0.1_OGYwMTZiMjg5ZTNiYmUxODIxOTgyYTllZGMyMzhjODE2ZWE5OGY4YmEyZWVjOGZhOWIxM2NlM2FhZTQxMmFjODY0OWZiNzQxMjVlMWIyODVlZWFiZjY2NTQyMTZhY2NjNTM5NDNmYTFmZjgxMTlkOGYxYTUzYTIzMzA0NDE3MGNmZDhkYTBkZmJiMmJhZmFkZDNmZTM1ZmI2MWZkNzYyYQ==","Host": "image.baidu.com","Referer": "https://image.baidu.com/","sec-ch-ua": '" Not;A Brand";v="99", "Google Chrome";v="91", "Chromium";v="91"',"sec-ch-ua-mobile": "?0","Sec-Fetch-Dest": "document","Sec-Fetch-Mode": "navigate","Sec-Fetch-Site": "same-origin","Sec-Fetch-User": "?1","Upgrade-Insecure-Requests": "1","User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Safari/537.36"}for url in self.start_urls:yield scrapy.Request(url,headers=headers,callback=self.parse,dont_filter=True)def parse(self, response):img_urls = re.findall('"thumbURL":"(.*?)"', response.text)# print(img_urls)item = DbimgItem()item['image_urls'] = img_urlsyield item

3.2、items文件

import scrapy class DbimgItem(scrapy.Item):# define the fields for your item here like:# name = scrapy.Field()image_urls = scrapy.Field()

3.3、settings文件

ROBOTSTXT_OBEY = False #打開我們寫的管道 ITEM_PIPELINES = {# 'dbimg.pipelines.DbimgPipeline': 300,'dbimg.pipelines.ImgPipe': 300, } #圖片存放位置 IMAGES_STORE = 'D:/python test/爬蟲/scrapy6/dbimg/imgs'

3.4、pipelines文件

import os from itemadapter import ItemAdapter from scrapy.pipelines.images import ImagesPipeline import settings""" def item_completed(self, results, item, info):with suppress(KeyError):ItemAdapter(item)[self.images_result_field] = [x for ok, x in results if ok]return item """ class ImgPipe(ImagesPipeline):num=0#重寫此函數修改獲取的圖片的名字 不然圖片名稱就是一串數字字母def item_completed(self, results, item, info):images_path = [x['path'] for ok, x in results if ok]#print('results: ',results) 先查看下results的數據格式，然后才能獲取到我們需要的值for image_path in images_path:os.rename(settings.IMAGES_STORE + "/" + image_path, settings.IMAGES_STORE + "/" + str(self.num) + ".jpg")self.num += 1

結果：

總結

以上是生活随笔為你收集整理的爬虫学习笔记（十一）—— Scrapy框架（六）：媒体管道的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。