#下載網(wǎng)頁 #具有功能：捕獲異常，重試下載并設(shè)置用戶代理 import urllib.request import urllib.error #下載網(wǎng)頁 #wscp:默認用戶代理 web scraping with python 縮寫 def download(url, user_agent='wscp',num_retries=2):print('Downloading:',url) #打印下載路徑headers={'USer-Agent':user_agent}request=urllib.request.Request(url,headers=headers)try:html=urllib.request.urlopen(request).read()except urllib.error.URLError as e:print('download error:',e.reason)html=Noneif num_retries>0:#下載遇到錯誤時嘗試下載if hasattr(e,'code') and 500 <=e.code <600:#404 notfound 這種錯誤，說明網(wǎng)頁不存在，故不需要重新下載print(user_agent)return download(url,user_agent,num_retries-1)return html download('http://example.webscraping.com/') download('http://httpstat.us/500')#測試錯誤500 # print(dir(urllib)) Downloading: http://example.webscraping.com/ Downloading: http://httpstat.us/500 download error: Internal Server Error wscp Downloading: http://httpstat.us/500 download error: Internal Server Error wscp Downloading: http://httpstat.us/500 download error: Internal Server Error

?

轉(zhuǎn)載于:https://www.cnblogs.com/liangliangzz/p/10160482.html

總結(jié)

以上是生活随笔為你收集整理的python 多功能下载网页的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。