緊跟著上一節說的文章，雖然已經放出了所寫的全代碼，但還是再解釋一下另外一個頁面的請求和分析過程吧。

PS：又可以愉快的水一章了，咕嘿嘿。

---

頁面分析

上回說到下載按鈕的href屬性是javascript:;偽協議，導致了新打開的頁面鏈接攜帶#符號，但是我們通過了phantomjs已經解決了第一次跳轉的問題。

下載頁面

事實證明，這里更加狠，連個偽協議都沒有。不過沒關系，我們還是沿用上回的那個方法，使用phantomjs來渲染頁面并且將跳轉的頁面鏈接以響應返回給我們的客戶端請求。

實現

采用上一節所說的讓Phantomjs作為服務端，然后去請求它，讓它把要爬取的結果反饋給.net。注意，這里的返回給客戶端的響應結果可以是網頁頁面，也可以是Phantomjs進行HTML解析完的真實數據。

.Net Core代碼

public async Task<string> GetDownloadPageAsync(string url){string result = string.Empty;//請求phantomjs 獲取下載頁面string dom = "Tappable-inactive animated fadeIn";KeyValuePair<string, string> url2dom = new KeyValuePair<string, string>(url, dom);var postData = JsonConvert.SerializeObject(url2dom);CookieContainer cc = new CookieContainer(); ?HttpHelpers helper = new HttpHelpers(); ?HttpItems items = new HttpItems();HttpResults hr = new HttpResults();items.Url = this.PostUrl1;items.Method = "POST";items.Container = cc;items.Postdata = postData;items.Timeout = 100000;hr = await helper.GetHtmlAsync(items);var downloadPageUrl = hr.Html;Console.WriteLine($"first => { downloadPageUrl }");if(downloadPageUrl.Contains("http")){//獲取百度云下載地址和分享密碼//string code1 = "1";dom = "Tappable-inactive btn btn-success btn-block"; // 下載鏈接url2dom = new KeyValuePair<string, string>(downloadPageUrl, dom);postData = JsonConvert.SerializeObject(url2dom);items = new HttpItems{Url = this.PostUrl2};items.Method = "POST";items.Container = cc;items.Postdata = postData;items.Timeout = 1000000;hr = await helper.GetHtmlAsync(items);result = hr.Html; //返回json數據Console.WriteLine($"second => { result }");}else{result = downloadPageUrl; //輸出錯誤信息}return result;}

這里包含了第一次在詳情頁獲取下載頁的那個請求，以及下載頁面獲取百度云鏈接和分享密碼的請求。

JavaScript代碼

"use strict"; var port = 8089; var server = require('webserver').create();server.listen(8089, function (request, response) {//傳入的參數有待更改，目前為//{"Key":"https://acg12.com/download/#60e21d8417ab60fbfJfcqnT1BC8Qd20PehAIKv3J4ZO%2FJCo0htE9hP5IFZU", //"Value":"Tappable-inactive btn btn-success btn-block"}的json字符竄//第一個參數為經過第一次請求所返回的下載頁面，第二個為下載按鈕的Domvar data = JSON.parse(request.postRaw);var url = data.Key.toString();console.log(url);var dom = data.Value.toString();console.log(dom);var code = 200;var pwdArray = new Array();var result = new Array();var page = require('webpage').create();page.onInitialized = function() {page.customHeaders = {};};page.settings.loadImages = false;page.customHeaders = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36","Referer": url};response.headers = {'Cache': 'no-cache','Content-Type': 'text/plain','Connection': 'Keep-Alive','Keep-Alive': 'timeout=40, max=100'};//根據Phantomjs的官網，這個回調在打開新標簽頁會觸發page.onPageCreated = function(newPage) {//console.log('A new child page was created! Its requested URL is not yet available, though.');page.onInitialized = function() {newPage.customHeaders = {};};newPage.settings.loadImages = false;newPage.customHeaders = {"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"};//newPage.viewportSize = { width: 1920, height: 1080 };//當百度云頁面打開并渲染完成時觸發newPage.onLoadFinished = function(status) {//console.log('A child page is Loaded: ' + newPage.url);//newPage.render('newPage.png', {format: 'png', quality: '100'});//console.log(pwdArray.length);if(pwdArray.length > 0){//console.log("enter");//從數組中pop出密碼，當無密碼時pop的數據為null字符竄var temp = {"url": newPage.url.toString(), "password": pwdArray.pop().toString()};console.log(JSON.stringify(temp));result.push(temp); // 將json數據push進返回結果}};};page.open(url, function (status) {console.log("----" + status);if (status !== 'success') {code = 400;response.write('4XX');response.statusCode = code;response.close();} else {code = 200;window.setTimeout(function (){//var dom = dom;pwdArray = page.evaluate(function(dom) {console.log(dom);var pwdArray = new Array();var btnList = document.getElementsByClassName(dom); // 百度云鏈接for(var i = 0; i < btnList.length;i ++ ){//猜測所有下載節點都有密碼var temp = document.getElementById("downloadPwd-" + i);if(temp != undefined){//console.log("****" + temp.value);pwdArray.push(temp.value); // 有密碼push進數組}else{//console.log("****null");pwdArray.push("null"); // 無密碼則push進null字符竄，這樣做到和url的一一對應}}for(var i = 0; i < btnList.length;i ++ ){//console.log("click");btnList[i].click(); // 點擊下載，打開新標簽頁}return pwdArray;}, dom);}, 6000);}});//設置等待20秒后才發送客戶端的響應結果，這樣保證上述方法都能成功運行結束window.setTimeout(function(){var rs = JSON.stringify(result)console.log(rs);response.write(rs);response.statusCode = code;response.close();}, 20000);page.onConsoleMessage = function(msg, lineNum, sourceId) {console.log("$$$$$" + msg);};page.onError = function(msg, trace) {var msgStack = ['PHANTOM ERROR: ' + msg];if (trace && trace.length) {msgStack.push('TRACE:');trace.forEach(function(t) {msgStack.push(' -> ' + (t.file || t.sourceURL) + ': ' + t.line + (t.function ? ' (in function ' + t.function +')' : ''));});}console.log(msgStack.join('\n'));phantom.exit(1);}; }); phantom.onError = function(msg, trace) {var msgStack = ['PHANTOM ERROR: ' + msg];if (trace && trace.length) {msgStack.push('TRACE:');trace.forEach(function(t) {msgStack.push(' -> ' + (t.file || t.sourceURL) + ': ' + t.line + (t.function ? ' (in function ' + t.function +')' : ''));});}console.log(msgStack.join('\n'));phantom.exit(1);};

完整的源代碼已經放在Github上了，里面有寫好的bat文件，直接運行run.bat就行。當然前提，第一節的那些環境都配置完成了。大家下周見，下周可能可以嘗試用用DotnetSpider，這是借鑒了WebMagic寫的.net core地爬蟲框架，有興趣的可以先去嘗試一下玩玩。

原文地址：http://www.jianshu.com/p/27bf3bb9ca60

---

.NET社區新聞，深度好文，微信中搜索dotNET跨平臺或掃描二維碼關注

總結

以上是生活随笔為你收集整理的.net core之ACG小站爬虫（二）的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。