学术前沿趋势分析Task01
生活随笔
收集整理的這篇文章主要介紹了
学术前沿趋势分析Task01
小編覺得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.
一、任務(wù)說明
- 任務(wù)主題：論文數(shù)量統(tǒng)計(jì)，即統(tǒng)計(jì)2019年全年計(jì)算機(jī)各個(gè)方向論文數(shù)量；
- 任務(wù)內(nèi)容：賽題的理解、使用 Pandas 讀取數(shù)據(jù)并進(jìn)行統(tǒng)計(jì)；
- 任務(wù)內(nèi)容：任務(wù)成果：學(xué)習(xí) Pandas 的基礎(chǔ)操作；
二、數(shù)據(jù)集介紹
數(shù)據(jù)集的格式如下：
- id：arXiv ID，可用于訪問論文；
- submitter：論文提交者；
- authors：論文作者；
- title：論文標(biāo)題；
- comments：論文頁數(shù)和圖表等其他信息；
- journal-ref：論文發(fā)表的期刊的信息；
- doi：數(shù)字對(duì)象標(biāo)識(shí)符，https://www.doi.org；
- report-no：報(bào)告編號(hào)；
- categories：論文在 arXiv 系統(tǒng)的所屬類別或標(biāo)簽；
- license：文章的許可證；
- abstract：論文摘要；
- versions：論文版本；
- authors_parsed：作者的信息。
三、arxiv論文類別介紹
‘a(chǎn)stro-ph’: ‘Astrophysics’,
‘a(chǎn)stro-ph.CO’: ‘Cosmology and Nongalactic Astrophysics’,
‘a(chǎn)stro-ph.EP’: ‘Earth and Planetary Astrophysics’,
‘a(chǎn)stro-ph.GA’: ‘Astrophysics of Galaxies’,
‘cs.AI’: ‘Artificial Intelligence’,
‘cs.AR’: ‘Hardware Architecture’,
‘cs.CC’: ‘Computational Complexity’,
‘cs.CE’: ‘Computational Engineering, Finance, and Science’,
‘cs.CV’: ‘Computer Vision and Pattern Recognition’,
‘cs.CY’: ‘Computers and Society’,
‘cs.DB’: ‘Databases’,
‘cs.DC’: ‘Distributed, Parallel, and Cluster Computing’,
‘cs.DL’: ‘Digital Libraries’,
‘cs.NA’: ‘Numerical Analysis’,
‘cs.NE’: ‘Neural and Evolutionary Computing’,
‘cs.NI’: ‘Networking and Internet Architecture’,
‘cs.OH’: ‘Other Computer Science’,
‘cs.OS’: ‘Operating Systems’,
四、具體代碼實(shí)現(xiàn)以及講解
4.1 導(dǎo)入package并讀取原始數(shù)據(jù)
# 導(dǎo)入所需的package import seaborn as sns #用于畫圖 from bs4 import BeautifulSoup #用于爬取arxiv的數(shù)據(jù) import re #用于正則表達(dá)式，匹配字符串的模式 import requests #用于網(wǎng)絡(luò)連接，發(fā)送網(wǎng)絡(luò)請(qǐng)求，使用域名獲取對(duì)應(yīng)信息 import json #讀取數(shù)據(jù)，我們的數(shù)據(jù)為json格式的 import pandas as pd #數(shù)據(jù)處理，數(shù)據(jù)分析 import matplotlib.pyplot as plt #畫圖工具 # 讀入數(shù)據(jù) data = [] #初始化 #使用with語句優(yōu)勢(shì)：1.自動(dòng)關(guān)閉文件句柄；2.自動(dòng)顯示（處理）文件讀取數(shù)據(jù)異常 with open("arxiv-metadata-oai-snapshot.json", 'r') as f: for line in f: data.append(json.loads(line))data = pd.DataFrame(data) #將list變?yōu)閐ataframe格式，方便使用pandas進(jìn)行分析 data.shape #顯示數(shù)據(jù)大小 >>> (1796911, 14)(1796911, 14) 其中的1796911表示數(shù)據(jù)總量，14表示特征數(shù)，對(duì)應(yīng)我們1.2節(jié)說明的論文的14種信息。
data.head()| 0704.0001 | Pavel Nadolsky | C. Bal\'azs, E. L. Berger, P. M. Nadolsky, C.-... | Calculation of prompt diphoton production cros... | 37 pages, 15 figures; published version | Phys.Rev.D76:013009,2007 | 10.1103/PhysRevD.76.013009 | ANL-HEP-PR-07-12 | hep-ph | None | A fully differential calculation in perturba... | [{'version': 'v1', 'created': 'Mon, 2 Apr 2007... | 2008-11-26 | [[Balázs, C., ], [Berger, E. L., ], [Nadolsky,... |
| 0704.0002 | Louis Theran | Ileana Streinu and Louis Theran | Sparsity-certifying Graph Decompositions | To appear in Graphs and Combinatorics | None | None | None | math.CO cs.CG | http://arxiv.org/licenses/nonexclusive-distrib... | We describe a new algorithm, the $(k,\ell)$-... | [{'version': 'v1', 'created': 'Sat, 31 Mar 200... | 2008-12-13 | [[Streinu, Ileana, ], [Theran, Louis, ]] |
| 0704.0003 | Hongjun Pan | Hongjun Pan | The evolution of the Earth-Moon system based o... | 23 pages, 3 figures | None | None | None | physics.gen-ph | None | The evolution of Earth-Moon system is descri... | [{'version': 'v1', 'created': 'Sun, 1 Apr 2007... | 2008-01-13 | [[Pan, Hongjun, ]] |
| 0704.0004 | David Callan | David Callan | A determinant of Stirling cycle numbers counts... | 11 pages | None | None | None | math.CO | None | We show that a determinant of Stirling cycle... | [{'version': 'v1', 'created': 'Sat, 31 Mar 200... | 2007-05-23 | [[Callan, David, ]] |
| 0704.0005 | Alberto Torchinsky | Wael Abu-Shammala and Alberto Torchinsky | From dyadic $\Lambda_{\alpha}$ to $\Lambda_{\a... | None | Illinois J. Math. 52 (2008) no.2, 681-689 | None | None | math.CA math.FA | None | In this paper we show how to compute the $\L... | [{'version': 'v1', 'created': 'Mon, 2 Apr 2007... | 2013-10-15 | [[Abu-Shammala, Wael, ], [Torchinsky, Alberto, ]] |
4.2 數(shù)據(jù)預(yù)處理
- count：一列數(shù)據(jù)的元素個(gè)數(shù)；
- unique：一列數(shù)據(jù)中元素的種類；
- top：一列數(shù)據(jù)中出現(xiàn)頻率最高的元素；
- freq：一列數(shù)據(jù)中出現(xiàn)頻率最高的元素的個(gè)數(shù)；
以上的結(jié)果表明：共有1,796,911個(gè)數(shù)據(jù)，有62,055個(gè)子類，其中最多的種類是astro-ph，即Astrophysics（天體物理學(xué)），共出現(xiàn)了86914次。
由于部分論文的類別不止一種，所以下面我們判斷在本數(shù)據(jù)集中共出現(xiàn)了多少種獨(dú)立的數(shù)據(jù)集。
# 所有的種類(獨(dú)立的) unique_categories = set([i for l in [x.split(' ') for x in data['categories']] for i in l]) len(unique_categories) >>> 176 unique_categories{'acc-phys','adap-org','alg-geom',...'stat.OT','stat.TH','supr-con'}這里使用了 split 函數(shù)將多類別使用 “ ”（空格）分開，組成list，并使用 for 循環(huán)將獨(dú)立出現(xiàn)的類別找出來，并使用 set 類別，將重復(fù)項(xiàng)去除得到最終所有的獨(dú)立paper種類。
從以上結(jié)果發(fā)現(xiàn)，共有176種論文種類，而官網(wǎng)上是153種論文種類，這說明存在一些官網(wǎng)上沒有的類別，不過對(duì)于我們的計(jì)算機(jī)方向的論文沒有影響。
我們的任務(wù)要求對(duì)于2019年以后的paper進(jìn)行分析，所以首先對(duì)于時(shí)間特征進(jìn)行預(yù)處理，從而得到2019年以后的所有種類的論文：
data['update_date'].astype # object類型<bound method NDFrame.astype of 0 2008-11-26 1 2008-12-13 2 2008-01-13 3 2007-05-23 4 2013-10-15... 1796906 2009-10-30 1796907 2016-11-18 1796908 2009-10-30 1796909 2009-10-30 1796910 2009-10-30 Name: update_date, Length: 1796911, dtype: object> #將update_date從例如2019-02-20的str變?yōu)閐atetime格式，并提取處year data['year'] = pd.to_datetime(data['update_date']).dt.year #找出 year 中2019年以后的數(shù)據(jù) data = data[data['year'] >= 2019] data.shape # 還剩395123條數(shù)據(jù) >>> (395123, 15) data.reset_index(drop=True, inplace=True) #重新編號(hào) data.head() #查看結(jié)果| 0704.0297 | Sung-Chul Yoon | Sung-Chul Yoon, Philipp Podsiadlowski and Step... | Remnant evolution after a carbon-oxygen white ... | 15 pages, 15 figures, 3 tables, submitted to M... | None | 10.1111/j.1365-2966.2007.12161.x | None | astro-ph | None | We systematically explore the evolution of t... | [{'version': 'v1', 'created': 'Tue, 3 Apr 2007... | 2019-08-19 | [[Yoon, Sung-Chul, ], [Podsiadlowski, Philipp,... | 2019 |
| 0704.0342 | Patrice Ntumba Pungu | B. Dugmore and PP. Ntumba | Cofibrations in the Category of Frolicher Spac... | 27 pages | None | None | None | math.AT | None | Cofibrations are defined in the category of ... | [{'version': 'v1', 'created': 'Tue, 3 Apr 2007... | 2019-08-19 | [[Dugmore, B., ], [Ntumba, PP., ]] | 2019 |
| 0704.0360 | Zaqarashvili | T.V. Zaqarashvili and K Murawski | Torsional oscillations of longitudinally inhom... | 6 pages, 3 figures, accepted in A&A | None | 10.1051/0004-6361:20077246 | None | astro-ph | None | We explore the effect of an inhomogeneous ma... | [{'version': 'v1', 'created': 'Tue, 3 Apr 2007... | 2019-08-19 | [[Zaqarashvili, T. V., ], [Murawski, K, ]] | 2019 |
| 0704.0525 | Sezgin Ayg\"un | Sezgin Aygun, Ismail Tarhan, Husnu Baysal | On the Energy-Momentum Problem in Static Einst... | This submission has been withdrawn by arXiv ad... | Chin.Phys.Lett.24:355-358,2007 | 10.1088/0256-307X/24/2/015 | None | gr-qc | None | This paper has been removed by arXiv adminis... | [{'version': 'v1', 'created': 'Wed, 4 Apr 2007... | 2019-10-21 | [[Aygun, Sezgin, ], [Tarhan, Ismail, ], [Baysa... | 2019 |
| 0704.0535 | Antonio Pipino | Antonio Pipino (1,3), Thomas H. Puzia (2,4), a... | The Formation of Globular Cluster Systems in M... | 32 pages (referee format), 9 figures, ApJ acce... | Astrophys.J.665:295-305,2007 | 10.1086/519546 | None | astro-ph | None | The most massive elliptical galaxies show a ... | [{'version': 'v1', 'created': 'Wed, 4 Apr 2007... | 2019-08-19 | [[Pipino, Antonio, ], [Puzia, Thomas H., ], [M... | 2019 |
這里我們就已經(jīng)得到了所有2019年以后的論文，下面我們挑選出計(jì)算機(jī)領(lǐng)域內(nèi)的所有文章：
#爬取所有的類別 website_url = requests.get('https://arxiv.org/category_taxonomy').text #獲取網(wǎng)頁的文本數(shù)據(jù) soup = BeautifulSoup(website_url,'lxml') #爬取數(shù)據(jù)，這里使用lxml的解析器，加速 root = soup.find('div',{'id':'category_taxonomy_list'}) #找出 BeautifulSoup 對(duì)應(yīng)的標(biāo)簽入口 tags = root.find_all(["h2","h3","h4","p"], recursive=True) #讀取 tagshttps://www.jianshu.com/p/fb6ee6cc5c1c
import requests
requests.get(url) #get請(qǐng)求核心代碼是requests.get(url)
requests.post(url) #post請(qǐng)求核心代碼是requests.post(url,data={請(qǐng)求體的字典})
requests.put(url)
requests.delete(url)
requests.head(url)
requests.options(url)
https://cuiqingcai.com/1319.html
Beautiful Soup 將復(fù)雜 HTML 文檔轉(zhuǎn)換成一個(gè)復(fù)雜的樹形結(jié)構(gòu)，每個(gè)節(jié)點(diǎn)都是 Python 對(duì)象，所有對(duì)象可以歸納為 4 種:
- Tag：就是 HTML 中的一個(gè)個(gè)標(biāo)簽，可以利用 soup 加標(biāo)簽名輕松地獲取這些標(biāo)簽的內(nèi)容
- NavigableString：獲取標(biāo)簽內(nèi)部的文字用 .string 即可，它的類型是一個(gè) NavigableString
- BeautifulSoup：BeautifulSoup 對(duì)象表示的是一個(gè)文檔的全部內(nèi)容。大部分時(shí)候，可以把它當(dāng)作 Tag 對(duì)象，是一個(gè)特殊的 Tag，我們可以分別獲取它的類型，名稱，以及屬性
- Comment：一個(gè)特殊類型的 NavigableString 對(duì)象，輸出的內(nèi)容不包括注釋符號(hào)
| Computer Science | Computer Science | Computer Science | Artificial Intelligence | cs.AI | Covers all areas of AI except Vision, Robotics... |
| Computer Science | Computer Science | Computer Science | Hardware Architecture | cs.AR | Covers systems organization and hardware archi... |
| Computer Science | Computer Science | Computer Science | Computational Complexity | cs.CC | Covers models of computation, complexity class... |
| Computer Science | Computer Science | Computer Science | Computational Engineering, Finance, and Science | cs.CE | Covers applications of computer science to the... |
| Computer Science | Computer Science | Computer Science | Computational Geometry | cs.CG | Roughly includes material in ACM Subject Class... |
| Computer Science | Computer Science | Computer Science | Artificial Intelligence | cs.AI | Covers all areas of AI except Vision, Robotics... |
| Computer Science | Computer Science | Computer Science | Hardware Architecture | cs.AR | Covers systems organization and hardware archi... |
| Computer Science | Computer Science | Computer Science | Computational Complexity | cs.CC | Covers models of computation, complexity class... |
| Computer Science | Computer Science | Computer Science | Computational Engineering, Finance, and Science | cs.CE | Covers applications of computer science to the... |
| Computer Science | Computer Science | Computer Science | Computational Geometry | cs.CG | Roughly includes material in ACM Subject Class... |
| ... | ... | ... | ... | ... | ... |
| Statistics | Statistics | Statistics | Computation | stat.CO | Algorithms, Simulation, Visualization |
| Statistics | Statistics | Statistics | Methodology | stat.ME | Design, Surveys, Model Selection, Multiple Tes... |
| Statistics | Statistics | Statistics | Machine Learning | stat.ML | Covers machine learning papers (supervised, un... |
| Statistics | Statistics | Statistics | Other Statistics | stat.OT | Work in statistics that does not fit into the ... |
| Statistics | Statistics | Statistics | Statistics Theory | stat.TH | stat.TH is an alias for math.ST. Asymptotics, ... |
155 rows × 6 columns
說明一下上面代碼中的正則操作，這里我們使用re.sub來用于替換字符串中的匹配項(xiàng)
raw = 'Astrophysics(astro-ph)' re.sub(r"(.*)\((.*)\)",r"\2",raw) >>> 'astro-ph'對(duì)應(yīng)的參數(shù)：
- 正則中的模式字符串 pattern 的格式為 “任意字符” + “(” + “任意字符” + “)”。
- 替換的字符串 repl 為第2個(gè)分組的內(nèi)容。
- 要被查找替換的原始字符串 string 為原始的爬取的數(shù)據(jù)。
這里推薦大家一個(gè)在線正則表達(dá)式測試的網(wǎng)站：https://tool.oschina.net/regex/
4.3 數(shù)據(jù)分析及可視化
接下來我們首先看一下所有大類的paper數(shù)量分布：
# 使用merge函數(shù)，以兩個(gè)dataframe共同的屬性 “categories” 進(jìn)行合并，并以 “group_name” 作為類別進(jìn)行統(tǒng)計(jì)，統(tǒng)計(jì)結(jié)果放入 “id” 列中并排序。 _df = data.merge(df_taxonomy, on="categories", how="left").drop_duplicates(["id","group_name"]).groupby("group_name").agg({"id":"count"}).sort_values(by="id",ascending=False).reset_index()_df| Physics | 79985 |
| Mathematics | 51567 |
| Computer Science | 40067 |
| Statistics | 4054 |
| Electrical Engineering and Systems Science | 3297 |
| Quantitative Biology | 1994 |
| Quantitative Finance | 826 |
| Economics | 576 |
下面我們使用餅圖進(jìn)行上圖結(jié)果的可視化：
fig = plt.figure(figsize=(15,12)) explode = (0, 0, 0, 0.2, 0.3, 0.3, 0.2, 0.1) plt.pie(_df["id"], labels=_df["group_name"], autopct='%1.2f%%', startangle=160, explode=explode) plt.tight_layout() # tight_layout會(huì)自動(dòng)調(diào)整子圖參數(shù)，使之填充整個(gè)圖像區(qū)域。 plt.show()https://www.cnblogs.com/biyoulin/p/9565350.html
def pie(x, explode=None, labels=None, colors=None, autopct=None,
pctdistance=0.6, shadow=False, labeldistance=1.1, startangle=None,
radius=None, counterclock=True, wedgeprops=None, textprops=None,
center=(0, 0), frame=False, rotatelabels=False, hold=None, data=None)
下面統(tǒng)計(jì)在計(jì)算機(jī)各個(gè)子領(lǐng)域2019年后的paper數(shù)量：
# 我們同樣使用 merge 函數(shù)，對(duì)于兩個(gè)dataframe 共同的特征 categories 進(jìn)行合并并且進(jìn)行查詢。然后我們?cè)賹?duì)于數(shù)據(jù)進(jìn)行統(tǒng)計(jì)和排序： group_name="Computer Science" cats = data.merge(df_taxonomy, on="categories").query("group_name == @group_name") cats.groupby(["year","category_name"]).count().reset_index().pivot(index="category_name", columns="year",values="id")| 558 | 757 |
| 2153 | 2906 |
| 131 | 188 |
| 108 | 205 |
| 199 | 216 |
| 281 | 323 |
| 5559 | 6517 |
| 346 | 564 |
| 1067 | 1238 |
| 711 | 902 |
| 282 | 342 |
| 125 | 157 |
| 84 | 81 |
| 715 | 774 |
| 101 | 84 |
| 152 | 137 |
| 5 | 5 |
| 116 | 151 |
| 95 | 159 |
| 420 | 580 |
| 245 | 331 |
| 470 | 504 |
| 177 | 538 |
| 27 | 45 |
| 85 | 90 |
| 76 | 66 |
| 864 | 783 |
| 235 | 279 |
| 40 | 11 |
| 36 | 33 |
| 67 | 69 |
| 45 | 51 |
| 268 | 294 |
| 917 | 1298 |
| 202 | 325 |
| 659 | 804 |
| 7 | 4 |
| 44 | 36 |
| 415 | 133 |
我們可以從結(jié)果看出，Computer Vision and Pattern Recognition（計(jì)算機(jī)視覺與模式識(shí)別）類是CS中paper數(shù)量最多的子類，遙遙領(lǐng)先于其他的CS子類，并且paper的數(shù)量還在逐年增加；另外，Computation and Language（計(jì)算與語言）、Cryptography and Security（密碼學(xué)與安全）以及 Robotics（機(jī)器人學(xué)）的2019年paper數(shù)量均超過1000或接近1000，這與我們的認(rèn)知是一致的。
總結(jié)
以上是生活随笔為你收集整理的学术前沿趋势分析Task01的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： CC1310切换速率的办法
- 下一篇： fat16和fat32文件系统学习