如果不懂爬蟲，請看鏈接
爬蟲第一課：爬蟲的基本原理

什么是Urllib

Urllib是python 內置的http的請求庫。包含四個模塊

• urllib.request 請求模塊
• urllib.error 異常處理模塊
• urllib.parse url解析模塊
• urllib.robotparser robots.txt解析模塊（識別哪些網站可以爬）
  本文主要講解前面三個模塊

和python2中urlib模塊區別

Python2
import urllib2
response = urllib2.urlopen(‘http://www.baidu.com’)

Python3
import urllib.request
response = urllib.request.urlopen(‘http://www.baidu.com’)

用法講解

請求urlopen
urlopen

第一個參數URL請求網站，第二個參數data，post請求時需要，第三參數timeout限時設置，后面暫時用不到

#%% raw urllib.request.urlopen(url, data=None, [timeout, ]*, cafile=None, capath=None, cadefault=False, context=None)

get請求網頁打開

import urllib.requestresponse = urllib.request.urlopen('https://www.baidu.com/') print(response.read().decode('utf-8'))#解碼

post請求網頁打開

import urllib.request import urllib.parsedata = bytes(urllib.parse.urlencode({'word': 'hello'}), encoding='utf8')#編碼 response = urllib.request.urlopen('http://httpbin.org/post', data=data) print(response.read())

timeout
設置一個超時時間，如果在規定時間沒有完成請求，會報錯

import urllib.requestresponse = urllib.request.urlopen('http://httpbin.org/get', timeout=0.1) print(response.read()) import socket import urllib.request import urllib.errortry:response = urllib.request.urlopen('http://httpbin.org/get', timeout=0.1) except urllib.error.URLError as e:if isinstance(e.reason, socket.timeout):print('TIME OUT')

響應

響應類型

import urllib.requestresponse = urllib.request.urlopen('https://www.python.org') print(type(response))

狀態碼和響應頭
響應里面，包括兩個重要信息：狀態碼和響應頭

import urllib.requestresponse = urllib.request.urlopen('https://www.python.org') print(response.status)#狀態碼 print(response.getheaders())#響應頭 print(response.getheader('Server'))#查看響應頭具體某一項

讀取響應read

#讀取的響應為字節流型數據，需要解碼為utf-8

import urllib.requestresponse = urllib.request.urlopen('https://www.python.org') print(response.read().decode('utf-8'))

Request
通過前面urlopen 我們可以請求一個網頁并獲取響應，但urlopen沒法添加額外信息，如請求頭，所以需要更高級的請求，Request

讀取網頁

import urllib.requestrequest = urllib.request.Request('https://python.org')#聲明一個request對象 response = urllib.request.urlopen(request) print(response.read().decode('utf-8'))

添加請求頭，同時post方式讀取網頁

from urllib import request, parseurl = 'http://httpbin.org/post' headers = {'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)','Host': 'httpbin.org' } dict = {'name': 'Germey' } data = bytes(parse.urlencode(dict), encoding='utf8')#編碼 req = request.Request(url=url, data=data, headers=headers, method='POST') response = request.urlopen(req) print(response.read().decode('utf-8'))

前面添加請求頭是通過參數，還可以通過add_header方法

from urllib import request, parseurl = 'http://httpbin.org/post' dict = {'name': 'Germey' } data = bytes(parse.urlencode(dict), encoding='utf8') req = request.Request(url=url, data=data, method='POST') req.add_header('User-Agent', 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)') response = request.urlopen(req) print(response.read().decode('utf-8'))

Handler代理

import urllib.requestproxy_handler = urllib.request.ProxyHandler({'http': 'http://127.0.0.1:9743',#服務器網址'https': 'https://127.0.0.1:9743' }) opener = urllib.request.build_opener(proxy_handler) response = opener.open('http://httpbin.org/get') print(response.read())

Cookie
Cookie是在客戶端保存的，用來記錄用戶身份的文本文件

如我們打開淘寶網頁，左上角顯示著用戶名信息

審查元素，找到淘寶cookie，右鍵clear

刷新網頁 用戶登錄信息消失

cookie如果有，可以維持用戶登錄信息，在做爬蟲時，可以爬取一些登錄后的信息。
1.打印cookie

import http.cookiejar, urllib.requestcookie = http.cookiejar.CookieJar()#聲明cookie對象 handler = urllib.request.HTTPCookieProcessor(cookie) opener = urllib.request.build_opener(handler) response = opener.open('http://www.baidu.com/')#百度，我登錄了百度賬號 for item in cookie:print(item.name+"="+item.value) 2.保存cookie為文本文件

方式1

import http.cookiejar, urllib.request filename = "cookie.txt" cookie = http.cookiejar.MozillaCookieJar(filename) handler = urllib.request.HTTPCookieProcessor(cookie) opener = urllib.request.build_opener(handler) response = opener.open('http://www.baidu.com/') cookie.save(ignore_discard=True, ignore_expires=True)

方式2

import http.cookiejar, urllib.request filename = 'cookie.txt' cookie = http.cookiejar.LWPCookieJar(filename) handler = urllib.request.HTTPCookieProcessor(cookie) opener = urllib.request.build_opener(handler) response = opener.open('http://www.baidu.com/') cookie.save(ignore_discard=True, ignore_expires=True)

3.帶cookie讀取網頁

import http.cookiejar, urllib.request cookie = http.cookiejar.LWPCookieJar()#要和前面保存方式一樣 cookie.load('cookie.txt', ignore_discard=True, ignore_expires=True) handler = urllib.request.HTTPCookieProcessor(cookie) opener = urllib.request.build_opener(handler) response = opener.open('http://www.baidu.com/') print(response.read().decode('utf-8'))

異常處理

from urllib import request, error try:response = request.urlopen('http://cuiqingcai.com/index.htm') except error.URLError as e:print(e.reason) from urllib import request, errortry:response = request.urlopen('http://cuiqingcai.com/index.htm') except error.HTTPError as e:print(e.reason, e.code, e.headers, sep='\n') except error.URLError as e:print(e.reason) else:print('Request Successfully') import socket import urllib.request import urllib.errortry:response = urllib.request.urlopen('https://www.baidu.com', timeout=0.01) except urllib.error.URLError as e:print(type(e.reason))if isinstance(e.reason, socket.timeout):print('TIME OUT')

URL解析
urlparse
傳入一個url，然后把一個url分割成幾個部分。協議，域名，參數等

urllib.parse.urlparse(urlstring, scheme=’’, allow_fragments=True)

from urllib.parse import urlparseresult = urlparse('http://www.baidu.com/index.html;user?id=5#comment') print(type(result), result)

指定協議類型，傳入的ur無協議

from urllib.parse import urlparseresult = urlparse('www.baidu.com/index.html;user?id=5#comment', scheme='https') print(result)

指定默認協議為ihttps，如果傳入的url無協議就會被指定為https

from urllib.parse import urlparseresult = urlparse('http://www.baidu.com/index.html;user?id=5#comment', scheme='https') print(result)

錨點拼接allow_fragments

from urllib.parse import urlparseresult = urlparse('http://www.baidu.com/index.html;user?id=5#comment', allow_fragments=False)#allow_fragments錨點 print(result)

urlunparse：拼接URL

from urllib.parse import urlunparsedata = ['http', 'www.baidu.com', 'index.html', 'user', 'a=6', 'comment'] print(urlunparse(data))

urljoin

from urllib.parse import urljoinprint(urljoin('http://www.baidu.com', 'FAQ.html')) print(urljoin('http://www.baidu.com', 'https://cuiqingcai.com/FAQ.html')) print(urljoin('http://www.baidu.com/about.html', 'https://cuiqingcai.com/FAQ.html')) print(urljoin('http://www.baidu.com/about.html', 'https://cuiqingcai.com/FAQ.html?question=2')) print(urljoin('http://www.baidu.com?wd=abc', 'https://cuiqingcai.com/index.php')) print(urljoin('http://www.baidu.com', '?category=2#comment')) print(urljoin('www.baidu.com', '?category=2#comment')) print(urljoin('www.baidu.com#comment', '?category=2'))

urlencode
把字典對象轉換為get請求參數

from urllib.parse import urlencodeparams = {'name': 'germey','age': 22 } base_url = 'http://www.baidu.com?' url = base_url + urlencode(params) print(url)

作者：電氣-余登武。
創作不易，大佬請留步… 來個贊再走唄 (???￩?)

總結

以上是生活随笔為你收集整理的爬虫笔记：Urllib库详解的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。