背景

?Selenium是一個(gè)用于Web應(yīng)用程序測(cè)試的工具。Selenium測(cè)試直接運(yùn)行在瀏覽器中，就像真正的用戶在操作一樣。而對(duì)于爬蟲來說，使用Selenium操控瀏覽器來爬取網(wǎng)上的數(shù)據(jù)那么肯定是爬蟲中的殺手武器。這里，我將介紹selenium + 谷歌瀏覽器的一般使用。

需求

在平常的爬蟲開發(fā)中，有時(shí)候網(wǎng)頁是一堆js堆起來的代碼，涉及很多異步計(jì)算，如果是普通的http 控制臺(tái)請(qǐng)求，那么得到的源文件是一堆js ，需要自己在去組裝數(shù)據(jù)，很費(fèi)力；但是采用Selenium+ChromeDriver可以達(dá)到所見即所得的完美效果。

實(shí)現(xiàn)方式

項(xiàng)目結(jié)構(gòu)：為了方便使用，用的winform程序，附nuget包

?以下是form1.cs的代碼，這里就只放關(guān)鍵方法代碼了。需要安裝最新的chrome瀏覽器+代碼中使用的chromedriver是 v2.9.248315

?

private void crawlingWebFunc(){SetText("\r\n開始嘗試...");List<testfold> surls = new List<testfold>();string path = System.Environment.CurrentDirectory + "\\圖片url\\";DirectoryInfo root = new DirectoryInfo(path);DirectoryInfo[] dics = root.GetDirectories();foreach (var itemdic in dics){string txt = "";StreamReader sr = new StreamReader(itemdic.FullName + "\\data.txt");while (!sr.EndOfStream){string str = sr.ReadLine();txt += str;// + "\n";}sr.Close();surls.Add(new testfold() { key = itemdic.FullName, picurl = txt });}ChromeDriverService service = ChromeDriverService.CreateDefaultService(System.Environment.CurrentDirectory);// service.HideCommandPromptWindow = true;ChromeOptions options = new ChromeOptions();options.AddArguments("--test-type", "--ignore-certificate-errors");options.AddArgument("enable-automation");// options.AddArgument("headless");// options.AddArguments("--proxy-server=http://user:password@yourProxyServer.com:8080");using (IWebDriver driver = new OpenQA.Selenium.Chrome.ChromeDriver(service, options, TimeSpan.FromSeconds(120))){driver.Url = "https://www.1688.com/";Thread.Sleep(200);try{int a = 1;foreach (var itemsurls in surls){SetText("\r\n第" + a.ToString() + "個(gè)");driver.Navigate().GoToUrl(itemsurls.picurl);//登錄if (driver.Url.Contains("login.1688.com")){SetText("\r\n需要登錄，開始嘗試...");trylogin(driver); //嘗試登錄完成//再試試driver.Navigate().GoToUrl("https://s.1688.com/youyuan/index.htm?tab=imageSearch&imageType=oss&imageAddress=cbuimgsearch/eWXC7XHHPN1607529600000&spm=");if (driver.Url.Contains("login.1688.com")){//沒辦法退出SetText("\r\n退出，換ip重試...");return;}}//鼠標(biāo)放上去的內(nèi)容因?yàn)轫撁孀詭е荒茱@示一個(gè)的原因 沒辦法做到全部顯示 然后在下載 只能是其他方式下載// var elements = document.getElementsByClassName('hover-container');// Array.prototype.forEach.call(elements, function(element) {// element.style.display = "block";// console.log(element);// });// IJavaScriptExecutor js = (IJavaScriptExecutor)driver;// var sss = js.ExecuteScript(" var elements = document.getElementsByClassName('hover-container'); Array.prototype.forEach.call(elements, function(element) { console.log(element); element.setAttribute(\"class\", \"測(cè)試title\"); element.style.display = \"block\"; console.log(element); });");Thread.Sleep(500);var responseModel = Write(itemsurls.key, driver.PageSource, Pagetypeenum.列表);Thread.Sleep(500);int i = 1;foreach (var offer in responseModel?.data?.offerList ?? new List<OfferItemModel>()){driver.Navigate().GoToUrl(offer.information.detailUrl);string responseDatadetail = driver.PageSource;Write(itemsurls.key, driver.PageSource, Pagetypeenum.詳情);SetText("\r\n第" + a.ToString() + "-" + i.ToString() + "個(gè)");Thread.Sleep(500);i++;}}}catch (Exception ex){CloseChromeDriver(driver);throw;}}}

?　

#region 異常 退出chromedriver[DllImport("user32.dll", EntryPoint = "FindWindow")]private extern static IntPtr FindWindow(string lpClassName, string lpWindowName);[DllImport("user32.dll", EntryPoint = "SendMessage")]public static extern int SendMessage(IntPtr hWnd, int Msg, int wParam, int lParam);public const int SW_HIDE = 0;public const int SW_SHOW = 5;[DllImport("user32.dll", EntryPoint = "ShowWindow")]public static extern int ShowWindow(IntPtr hwnd, int nCmdShow);///?<summary>///?獲取窗口句柄///?</summary>///?<returns></returns>public IntPtr GetWindowHandle(){string name = (Environment.CurrentDirectory + "\\chromedriver.exe");IntPtr hwd = FindWindow(null, name);return hwd;}///?<summary>///?關(guān)閉chromedriver窗口///?</summary>public void CloseWindow(){try{IntPtr hwd = GetWindowHandle();SendMessage(hwd, 0x10, 0, 0);}catch { }}///?<summary>///?退出chromedriver///?</summary>///?<param?name="driver"></param>public void CloseChromeDriver(IWebDriver driver){try{driver.Quit();driver.Dispose();}catch { }CloseWindow();}#endregion 異常 退出chromedriver

　效果

總結(jié)

說一下思路:

1.跳轉(zhuǎn)到指定的網(wǎng)頁driver.Navigate().GoToUrl

2.確定數(shù)據(jù)源，從driver.PageSource讀取數(shù)據(jù)

3.對(duì)html數(shù)據(jù)進(jìn)行解析

總結(jié)

以上是生活随笔為你收集整理的用C#+Selenium+ChromeDriver 爬取网页，完美模拟真实的用户浏览行为的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。