Scrapy Django项目
生活随笔
收集整理的這篇文章主要介紹了
Scrapy Django项目
小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.
需求: 編寫爬蟲項(xiàng)目與Django項(xiàng)目詳解和, 將爬取到的數(shù)據(jù)展示到前端頁面上
# spider編寫: import scrapy from dl.items import DlItem class PSpider(scrapy.Spider):name = 'p'# allowed_domains = ['www.baidu.com']start_urls = ['https://www.kuaidaili.com/free/']def parse(self, response):# print(response)tr_list = response.xpath('//*[@id="list"]/table/tbody/tr')# print(tr_list)for tr in tr_list:ip = tr.xpath('./td[1]/text()').extract_first()port = tr.xpath('./td[2]/text()').extract_first()typ = tr.xpath('./td[3]/text()').extract_first()protocal = tr.xpath('./td[4]/text()').extract_first()position = tr.xpath('./td[5]/text()').extract_first()# print(ip, port, protocal, position)item = DlItem()item['ip'] = ipitem['port'] = portitem['typ'] = typitem['protocal'] = protocalitem['position'] = positionprint(item)yield item # items編碼 import scrapy class DlItem(scrapy.Item):ip = scrapy.Field()port = scrapy.Field()typ = scrapy.Field()protocal = scrapy.Field()position = scrapy.Field() # Django項(xiàng)目創(chuàng)建與所有配置: 1.models創(chuàng)建: from django.db import models# Create your models here.class Proxy(models.Model):ip = models.CharField(max_length=50)port = models.CharField(max_length=50)typ = models.CharField(max_length=50)protocal = models.CharField(max_length=50)position = models.CharField(max_length=50)2.在scrapy框架項(xiàng)目中嵌入django import os import sys sys.path.append(os.path.dirname(os.path.abspath('.'))) os.environ['DJANGO_SETTINGS_MODULE'] = 'proxyscan.settings' # 手動初始化Django： import django django.setup()3.修改爬蟲item: import scrapy from scrapy_djangoitem import DjangoItem from proxy import models class DlItem(DjangoItem):django_model = models.Proxy4.pipeline編碼: class DlPipeline(object):def process_item(self, item, spider):print('開啟數(shù)據(jù)庫, 進(jìn)行數(shù)據(jù)存儲')item.save()print('關(guān)閉數(shù)據(jù)庫')return item5.Django項(xiàng)目遷移數(shù)據(jù)庫與admin后臺配置 Python manage.py makemigrations python manage.py migratefrom proxy.models import Proxy admin.site.register(Proxy)# 創(chuàng)建超級用戶: Python manage.py createsuperuser # 路由: from django.conf.urls import url from django.contrib import admin from proxy.views import indexurlpatterns = [url(r'^admin/', admin.site.urls),url(r'^index/', index), ]# 視圖函數(shù): from django.shortcuts import render from proxy.models import Proxy def index(requests):p = Proxy.objects.all()return render(requests, 'index.html', {"p":p}) # 前端代碼: <!DOCTYPE html> <html lang="en"> <head><meta charset="UTF-8"><title>Title</title><script src="https://cdn.bootcss.com/jquery/3.4.1/jquery.min.js"></script><link href="https://cdn.bootcss.com/twitter-bootstrap/4.3.1/css/bootstrap.min.css" rel="stylesheet"> </head> <body> <div class="container"><div class="row" ><div class="col-md-10 col-md-offset-2" style="margin:0 auto"><div class="panel panel-primary"><div class="panel-heading" style="margin-top:50px"><h3 class="panel-title">代理IP一覽表</h3></div><div class="panel-body"><table class="table table-striped"><thead><tr><th>IP</th><th>Port</th><th>Type</th><th>Protocal</th><th>Positon</th></tr></thead><tbody>{% for i in p %}<tr><th>{{ i.ip }}</th><td>{{ i.port }}</td><td>{{ i.typ }}</td><td>{{ i.protocal }}</td><td>{{ i.position }}</td></tr>{% endfor %}</tbody></table></div></div></div></div> </div></body> </html>總結(jié)
以上是生活随笔為你收集整理的Scrapy Django项目的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 增量式爬虫与分布式爬虫
- 下一篇： 图片爬取数据解析数据持久化