keepalived+haproxy（双主）+nginx(静态)+lamp（动态）部署phpBB
簡(jiǎn)介：
haproxy為高性能的反向代理服務(wù)器，在向后端server調(diào)度方面支持很多的調(diào)度算法：roundrobin，source，uri，url_param，hdr()，leastconn等，且支持動(dòng)態(tài)權(quán)重調(diào)整的一致性hash，后端為cache server時(shí)很好的提高cache命中率；相比nginx反向代理，haproxy調(diào)度功能更為強(qiáng)大，另外haproxy提供自帶的gui接口，方便直接在web頁(yè)面管理前、后端服務(wù)器，只需要在配置文件中開(kāi)啟即可。
nginx自身在處理web靜態(tài)頁(yè)面方面非常強(qiáng)大，其采用模塊化設(shè)計(jì)，有較好的擴(kuò)展性，但不支持模塊動(dòng)態(tài)裝卸載（Tengine支持模塊動(dòng)態(tài)裝卸載），nginx具有高可靠性，支持熱部署，低內(nèi)存消耗等優(yōu)點(diǎn)，其在設(shè)計(jì)之初就吸納了最新的技術(shù)，如支持事件驅(qū)動(dòng)、異步IO等，并發(fā)處理能力強(qiáng)大。
apache結(jié)合php處理動(dòng)態(tài)頁(yè)面功能強(qiáng)大，php可直接以模塊的形式加載到httpd，而nginx需要通過(guò)fastcgi協(xié)議與后端php通信。所以結(jié)合各組件優(yōu)點(diǎn)，我們這里采用haproxy(支持https)+nginx(靜態(tài))+lamp(動(dòng)態(tài))架構(gòu)，最后通過(guò)keepalived實(shí)現(xiàn)對(duì)haproxy高可用。
下圖為本次實(shí)驗(yàn)架構(gòu)圖：
??
?????????????????? 圖1
???
????? ? ? ? ? ? ? ? ?? 圖2 ?? ????
架構(gòu)圖說(shuō)明
這里有2幅架構(gòu)圖，圖1中，后端各主機(jī)均在同一網(wǎng)段，這也是我們這次實(shí)驗(yàn)所采用的架構(gòu)；圖2中后端static server group和dynamic server group不在同一網(wǎng)段，二者之間需要路由器相連，Router的地址：eth0 --> 192.168.100.80eth1 --> 192.168.200.80非但如此，采用圖2架構(gòu)還需在haproxy和各后端server上添加靜態(tài)路由：
haproxy1：? route add -net 192.168.200.0/24 gw 192.168.100.80 dev eth1
haproxy2：? route add -net 192.168.100.0/24 gw 192.168.200.80 dev eth1
Nginx1：?? routeadd -net 192.168.200.0/24 gw 192.168.100.80 dev eth0
Nginx2：?? routeadd -net 192.168.200.0/24 gw 192.168.100.80 dev eth0
LAMP1：??? routeadd -net 192.168.100.0/24 gw 192.168.200.80 dev eth0
LAMP2：??? routeadd -net 192.168.100.0/24 gw 192.168.200.80 dev eth0
為簡(jiǎn)便，這里我們采用架構(gòu)圖1。這里我們用virtualbox準(zhǔn)備6臺(tái)linux主機(jī)，各主機(jī)IP地址、Gateway及所需安裝軟件等如下表所配置，這里我們將192.168.0.0/24網(wǎng)絡(luò)模擬為公網(wǎng)，192.168.100.0/24為服務(wù)器內(nèi)網(wǎng)網(wǎng)段。前端為2臺(tái)haproxy組成的反向代理服務(wù)器，并實(shí)現(xiàn)雙主高可用：所謂雙主，即haproxy1和haproxy2都正常工作，均同時(shí)為后端server提供反向代理和調(diào)度功能，用戶(hù)可通過(guò)DNS解析后的192.168.0.50和192.168.0.51兩個(gè)地址訪問(wèn)我們的網(wǎng)站。客戶(hù)端請(qǐng)求的所有phpBB靜態(tài)頁(yè)面由前端的haproxy調(diào)度至后端的2臺(tái)nginx組成的服務(wù)器組響應(yīng)；用戶(hù)請(qǐng)求的所有phpBB動(dòng)態(tài)頁(yè)面由前端的haproxy調(diào)度至后端的2臺(tái)lamp組成的服務(wù)器組響應(yīng)。
6臺(tái)linux主機(jī)上所需資源如下：
IP Address | Hosts Name | OS | Software Required | comments |
eth0 -->192.168.0.16; GW -->192.168.0.1 | HAProxy1(active) | centos6.8 | keepalived、haproxy、ntpd、雙網(wǎng)卡 | HAProxy（active） |
eth0:0 -->192.168.0.50(由keepalived配置文件自動(dòng)生成) | ||||
eth1 -->192.168.100.16 | ||||
eth0 -->192.168.0.66; GW -->192.168.0.1 | HAProxy2(active) | centos6.8 | keepalived、haproxy、雙網(wǎng)卡 | HAProxy（active） |
eth0:1 -->192.168.0.51(由keepalived配置文件自動(dòng)生成) | ||||
eth1 -->192.168.100.66 | ||||
eth0 -->192.168.100.64; GW -->192.168.100.16 | Nginx1 | centos6.8 | nginx、phpBB、單網(wǎng)卡 | Static Server Group |
eth0 -->192.168.100.65; GW ?-->192.168.100.66 | Nginx2 | centos6.8 | nginx、phpBB、單網(wǎng)卡 | |
enp0s3 -->192.168.100.70; GW ?-->192.168.100.16 | LAMP1 | centos7.3 | httpd、mariadb/mysql、php、單網(wǎng)卡、phpBB | Dynamic Server Group |
enp0s3 -->192.168.100.71; GW ?-->192.168.100.66 | LAMP2 | centos7.3 | httpd、mariadb/mysql、php、單網(wǎng)卡、phpBB |
?
軟件安裝
按照上面的清單，分別在各主機(jī)上部署所需軟件，對(duì)于清單中HAProxy需要安裝ntpd服務(wù)器的原因在于keepalived的運(yùn)行需要后端server與前端服務(wù)器時(shí)鐘保持同步，否則對(duì)后端server的健康狀態(tài)檢測(cè)會(huì)出現(xiàn)異常。
①、haproxy
haproxy在centos6.8上可以直接使用yum方式：
[root@centos68 ~]# yum install -y haproxy
目前系統(tǒng)提供的默認(rèn)版本是1.5.18，且支持openssl：
[root@centos68 ~]# rpm -q haproxy ; ldd $(which haproxy)| grep ssl
haproxy-1.5.18-1.el6.x86_64
libssl.so.10 => /usr/lib64/libssl.so.10(0x00007f05e085c000)
如果是手動(dòng)編譯安裝，則需要先在官網(wǎng)下載程序包，并添加對(duì)openssl的支持，下面是手動(dòng)編譯步驟：
[root@centos68 ~]# yum install -y openssl openssl-devel readline-devel pcre-devellibssl-dev libpcre3
[root@centos68 ~]# wget http://www.haproxy.org/download/1.6/src/haproxy-1.6.13.tar.gz
[root@centos68 ~]# tar xf haproxy-1.6.13.tar.gz ; cd haproxy-1.6.13
[root@centos68 haproxy-1.6.13]# make TARGET=linux2628 USE_PCRE=1USE_OPENSSL=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LIBCRYPT=1
[root@centos68 haproxy-1.6.13]# ldd $(which haproxy) | grep ssl
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f05e085c000)
[root@centos68 haproxy-1.6.13]# make install PREFIX=/usr/local/haproxy
[root@centos68 haproxy-1.6.13]# vi /etc/profile.d/haproxy.sh
#!/bin/bash
export PATH=$PATH:/usr/local/haproxy/sbin/
[root@centos68 haproxy-1.6.13]# source /etc/profile.d/haproxy.sh
創(chuàng)建配置文件目錄及拷貝并修改啟動(dòng)腳本：
[root@centos68 haproxy-1.6.13]# mkdir -p /etc/haproxy
[root@centos68 haproxy-1.6.13]# cp examples/haproxy.init/etc/rc.d/init.d/haproxy
[root@centos68 haproxy-1.6.13]# vi /etc/rc.d/init.d/haproxy
BIN=/usr/local/haproxy/sbin/$BASENAME
CFG=/etc/$BASENAME/$BASENAME.cfg
PIDFILE=/var/run/$BASENAME.pid
LOCKFILE=/var/lock/subsys/$BASENAME
?
至此haproxy已經(jīng)安裝完畢，這里我們根據(jù)架構(gòu)圖提供haproxy的配置文件：
?
[root@centos68 ~]# vi /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
??? log???????? 127.0.0.1 local2
??? chroot????? /var/lib/haproxy
??? pidfile???? /var/run/haproxy.pid
??? maxconn???? 40000
??? user??????? haproxy
??? group???? ??haproxy
??? daemon
??? tune.ssl.default-dh-param 2048
??? # turn on stats unix socket
??? stats socket/var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
??? mode??????????????????? http
??? log???????????????????? global
??? option????????????????? httplog
??? option????????????????? dontlognull
??? option ????? ????????????http-server-close
??? option forwardfor?????? except 127.0.0.0/8
??? option????????????????? redispatch
??? retries???????????????? 3
??? timeout http-request??? 10s
??? timeout queue?????????? 1m
??? timeout connect???????? 10s
??? timeout client????????? 1m
??? timeout server????????? 1m
??? timeout http-keep-alive 10s
??? timeout check???????? ??10s
??? maxconn???????????????? 3000
?
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend? http_frontend
??? bind?? *:80
??? bind *:443 ssl crt /etc/haproxy/haproxy.pem? # 網(wǎng)站證書(shū)文件
??? mode http
??? log global
??? option httpclose
??? option logasap
??? option dontlognull
??? option forwardfor except127.0.0.1 header X-Client if-none
??? capture request header Host len20
??? capture request header Refererlen 60
??? reqadd X-Forwarded-Proto:\ https
??? #rspadd Via:\ haproxy.example.com
??? acl url_static path_beg -i/static /p_w_picpaths /javascript /stylesheets
??? acl url_static path_end -i .jpg.jpeg .png .bmp .gif .css .js
?? ?redirect scheme https if !{ ssl_fc } ??????# 全站采用https協(xié)議
??? use_backend static_servers ifurl_static
??? default_backend dynamic_servers
???
???
#---------------------------------------------------------------------
# static backend for serving up p_w_picpaths, stylesheets and such
#---------------------------------------------------------------------
backend static_servers
??? mode http
??? balance??? roundrobin
??? #option???? httpchk?HEAD? /healthchk.html
??? server???? static1 192.168.100.64:80 check inter 1000rise 2 fall 4 weight 1 maxconn 6000
??? server???? static2 192.168.100.65:80 check inter 1000rise 2 fall 4 weight 1 maxconn 6000
??? http-request set-headerX-Forwarded-Port %[dst_port]
??? http-request add-headerX-Forwarded-Proto https if { ssl_fc }
backend dynamic_servers
??? mode http
??? balance??? roundrobin
??? #cookie???? dynamic_cookie insert nocache indirect
??? server???? https_dynamic1 192.168.200.70:80 checkinter 2000 rise 2 fall 4 weight 1 maxconn 1000
??? server???? https_dynamic2 192.168.200.71:80 checkinter 2000 rise 2 fall 4 weight 1 maxconn 1000
??? http-request set-headerX-Forwarded-Port %[dst_port]
??? http-request add-headerX-Forwarded-Proto https if { ssl_fc }
?
listen stats
??? bind *:9090
??? stats enable
??? stats uri /haproxy?stats
??? stats hide-version
??? stats auth admin:yourpassword
??? stats admin if TRUE
?
②、haproxy.cfg配置說(shuō)明
haproxy的配置文件位于：/etc/haproxy/haproxy.cfg，該配置文件中的一個(gè)重點(diǎn)是添加對(duì)https協(xié)議的支持，為了實(shí)現(xiàn)該目的，需要做如下操作：
我們知道，haproxy代理ssl有三種方式：第一種，haproxy 自身提供ssl 證書(shū)，僅是客戶(hù)端與haproxy之間傳輸https協(xié)議，而haproxy與后端的web server仍然傳輸http；第二種為SSL透?jìng)?#xff0c;haproxy 本身工作于tcp傳輸層，https由后端server直接處理，但haproxy不對(duì)https報(bào)文做任何修改，這就意味著后端server失去了對(duì)客戶(hù)端IP、端口及使用協(xié)議的記錄；第三種，SSL連接在負(fù)載均衡器處終止，按需求調(diào)整，然后作為新的SSL連接代理到后端服務(wù)器。這可能會(huì)提供最大的安全性和發(fā)送客戶(hù)端信息的能力。這樣做的代價(jià)是更多的CPU能耗和稍復(fù)雜的配置。實(shí)驗(yàn)中我們采用第一種，所以我們需要提供haproxy的ssl證書(shū)文件：
因?yàn)槭菍?shí)驗(yàn)，所以我們需要生成自簽署證書(shū)cacert.pem、應(yīng)用程序haproxy證書(shū)haproxy.crt,應(yīng)用程序haproxy私鑰haproxy.key等文件，具體私鑰及證書(shū)生成步驟這里不再詳述。最后將haproxy.crt和haproxy.key合并為一個(gè)文件：
[root@centos68 ~]# cd /etc/haproxy
[root@centos68 haproxy]# cat haproxy.crt haproxy.key | tee haproxy.pem
而上面生成的haproxy.pem正是haproxy.cfg所需要的：
bind *:443 ssl crt /etc/haproxy/haproxy.pem
③、keepalived安裝與配置
我們需要在前端部署有haproxy的兩臺(tái)server上安裝keepalived，其安裝過(guò)程較為簡(jiǎn)單，直接使用yum安裝即可：
[root@centos68 ~]# yum install -y keepalived
[root@centos68 ~]# rpm -q keepalived
keepalived-1.2.13-5.el6_6.x86_64
?
haproxy1上的keepalived配置：
?
! Configuration File for keepalived
?
global_defs {
?? notification_email {
???? root@localhost
?? }
?? notification_email_fromAlexandre.Cassen@firewall.loc
?? smtp_server 127.0.0.1
?? smtp_connect_timeout 30
?? router_id www.example.com
?
}
?
vrrp_script chk_haproxy {
?? script "killall -0 haproxy&> /dev/null"
?? interval 1
?? weight -20
}
vrrp_instance VI_1 {
??? state MASTER
??? interface eth0
??? virtual_router_id 51
??? priority 100
??? advert_int 1
??? authentication {
??????? auth_type PASS
??????? auth_pass 256f6df6
??? }
??? virtual_ipaddress {
?????? 192.168.0.50/24 dev eth0label eth0:0
??? }
??? track_script {?
??????? chk_haproxy
??? }
??? notify_master"/etc/keepalived/notify.sh master"
??? notify_backup"/etc/keepalived/notify.sh backup"
??? notify_fault"/etc/keepalived/notify.sh fault"
}
?
vrrp_instance VI_2 {
??? state BACKUP
??? interface eth0
??? virtual_router_id 52
??? priority 99
??? advert_int 1
??? authentication {
??????? auth_type PASS
??????? auth_pass 19afa2c5
??? }
??? virtual_ipaddress {
?????? 192.168.0.51/24 dev eth0label eth0:1
??? }
??? track_script {?
??????? chk_haproxy
??? }
??? notify_master"/etc/keepalived/notify.sh master"
?? ?notify_backup "/etc/keepalived/notify.shbackup"
??? notify_fault"/etc/keepalived/notify.sh fault"
}
?
haproxy2上的keepalived配置：
?
?! Configuration File for keepalived
?
global_defs {
?? notification_email {
???? root@localhost
?? }
?? notification_email_fromAlexandre.Cassen@firewall.loc
?? smtp_server 127.0.0.1
?? smtp_connect_timeout 30
?? router_id centos66.example.com
}
vrrp_script chk_haproxy {
?? script "killall -0 haproxy&> /dev/null"
?? interval 1
?? weight -20
?? }
?
vrrp_instance VI_1 {
??? state BACKUP
??? interface eth0
??? virtual_router_id 51
??? priority 99
??? advert_int 1
??? authentication {
??????? auth_type PASS
??????? auth_pass 256f6df6
??? }
??? virtual_ipaddress {
??????? 192.168.0.50/24 dev eth0label eth0:0
??? }
??? track_script {?
??????? chk_haproxy
??? }
??? notify_master"/etc/keepalived/notify.sh master"
??? notify_backup"/etc/keepalived/notify.sh backup"
??? notify_fault"/etc/keepalived/notify.sh fault"
}
vrrp_instance VI_2 {
??? state MASTER
??? interface eth0
??? virtual_router_id 52
??? priority 100
??? advert_int 1
??? authentication {
??????? auth_type PASS
??????? auth_pass 19afa2c5
??? }
??? virtual_ipaddress {
??????? 192.168.0.51/24 dev eth0label eth0:1
??? }
??? track_script {?
??????? chk_haproxy
??? }
??? notify_master "/etc/keepalived/notify.shmaster"
??? notify_backup"/etc/keepalived/notify.sh backup"
??? notify_fault"/etc/keepalived/notify.sh fault"
}
?
消息通知腳本notify.sh：
#!/bin/bash
#Date:2017-10-07
vip=192.168.0.50
contact="root@localhost"
?
notify() {
?? mailsubject="`hostname` tobe $1: $vip floating"
?? mailbody="`date +'%F%H:%M:%S'`: vrrp transition, `hostname` changed to be $1"
?? echo "$mailbody" | mail-s "$mailsubject" $contact
}
case "$1" in
master)
??? notify master
??? exit 0
??? ;;
backup)
??? notify backup
??? exit 0
??? ;;
fault)
??? notify fault
??? exit 0
??? ;;
*)
??? echo "Usage: `basename $0`{master|backup|fault}"
??? exit 1
??? ;;
esac
?
④、ntpd的安裝與配置
由于前端keepalived與后端各server通信需要保證時(shí)鐘同步，所以我們需要在前端的其中一臺(tái)主機(jī)上安裝ntpd服務(wù)器，注意本次實(shí)驗(yàn)采用的是架構(gòu)圖1，如果是架構(gòu)圖2，那么前端的2臺(tái)haproxy服務(wù)器上均要部署ntpd。
[root@centos66 ~]# yum install -y ntpd
[root@centos66 ~]# vi /etc/ntpd.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5),ntp_mon(5).
?
driftfile /var/lib/ntp/drift
?
# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
?
restrict ntp1.aliyun.com
restrict ntp2.aliyun.com
restrict ntp3.aliyun.com
restrict ntp4.aliyun.com
restrict ntp5.aliyun.com
# Permit all access over the loopback interface.? This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
restrict 192.168.100.0 mask 255.255.255.0 nomodify
server ntp1.aliyun.com prefer
server ntp2.aliyun.com
server ntp3.aliyun.com
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
?
安裝完成后啟動(dòng)即可：service ntpd start ;chkconfig ntpd on
后端各server添加crontab任務(wù)計(jì)劃：
[root@www ~]# crontab -e
MAILTO=""
*/10 * * * * /usr/sbin/ntpdate 192.168.100.16 &> /dev/null
這樣設(shè)置完成后，后端的各server每10分鐘就會(huì)與前端的ntpd進(jìn)行一次時(shí)鐘同步。
⑤、Static Server Group
靜態(tài)服務(wù)器組的組成是由各linux主機(jī)上部署nginx來(lái)完成，在centos6.8上也可以直接使用yum來(lái)安裝：
[root@centos66 ~]# yum install -y nginx
然后對(duì)后端各主機(jī)上的nginx配置文件進(jìn)行配置：
[root@centos66 ~]# vi /etc/nginx/nginx.conf
# nginx.conf
user? nginx;
worker_processes? 1;
worker_rlimit_nofile 51200;
pid /var/run/nginx.pid;
events {
??? worker_connections? 51200;
}
?
http {
??? include?????? mime.types;
??? default_type? application/octet-stream;
?
??? log_format? main?'$remote_addr - $remote_user [$time_local] "$request" '
????????????????????? '$status$body_bytes_sent "$http_referer" '
?????????????????????'"$http_user_agent" "$http_x_forwarded_for"';
?
??? access_log? /var/log/nginx/access.log? main;
??? error_log/var/log/nginx/error.log? notice;
?
??? sendfile??????? on;
??? keepalive_timeout? 65;
??? server {
??????? listen?????? 80;
??????? server_name? localhost;
?
??????? location / {
??????????? root??/var/www/html/phpBB;
??????????? index? index.html index.htm;
??????? }
??????? error_page?? 500 502 503 504? /50x.html;
??????? location = /50x.html {
??????????? root?? html;
??????? }
?
????? }
?
? }
?
⑥、Dynamic Server Group
在后端各server上需要部署httpd+mariadb+php，在centos7上安裝過(guò)程比較簡(jiǎn)單，直接使用yum安裝即可，這里不再具體給出各軟件安裝步驟，我們需要在/etc/httpd/conf.d中添加虛擬主機(jī)：
[root@www ~]# cd /etc/httpd/conf.d/
[root@www conf.d]# vi vhosts.conf
<VirtualHost *:80>
?? DocumentRoot"/var/www/html"
?? ServerName? ftp.example.com
</VirtualHost>
?
<Directory "/var/www/html/phpBB">
?? Options Includes ExecCGIFollowSymLinks
?? AllowOverride None
?? Require all granted
</Directory>
?
<VirtualHost *:80>
?? DocumentRoot "/var/www/html/phpBB"
?? ServerName? www.example.com:80
</VirtualHost>
另外需要將配置文件：/etc/httpd/conf/httpd.conf中的"DocumentRoot"這一行注釋掉并修改日志格式：
#DocumentRoot "/var/www/html"
LogFormat "%{X-Client}i%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b\"%{Referer}i\" \"%{User-Agent}i\"" combined
#SetEnvIf Request_URI "^/test.html(/.*)?$" dontlog
#CustomLog "logs/access_log" combined env=!dontlog
?
⑦、部署phpBB
從phpBB官網(wǎng)下載安裝包，將安裝包分別拷貝至后端各server：Nginx1,Ngine2,LAMP1,LAMP2上的/var/www/html目錄下，解壓縮并修改所屬用戶(hù)和組，這里以后端的LAMP1來(lái)演示，其它各server上的操作步驟與下面類(lèi)似：
[root@www ~]# cp phpBB-3.2.0.tar.bz2 /var/www/html
[root@www ~]# cd /var/www/html
[root@www html]# tar xf phpBB-3.2.0.tar.bz2
[root@www html]# ls
phpBB3
[root@www html]# chown -R apache:apache phpBB3
[root@www html]# ln -s phpBB3 phpBB
[root@www html]# chown -R apache:apache phpBB
?
最后我們?cè)?/span>LAMP1和LAMP2上創(chuàng)建數(shù)據(jù)庫(kù)
MariaDb[(none)] > create database phpBB
MariaDb[(none)] > grant all privileges on phpBB.* to phpBB@'127.0.0.1'identified by 'yourpassword';
MariaDb[(none)] > flush privileges;
上面創(chuàng)建了phpBB數(shù)據(jù)庫(kù)名為：phpBB，數(shù)據(jù)庫(kù)用戶(hù)名：phpBB，密碼：yourpassword，這在后續(xù)的phpBB頁(yè)面安裝過(guò)程中需要。
?
至此各server配置已經(jīng)完成，最后就需要我們?cè)跒g覽器安裝phpBB了，在瀏覽器中輸入www.example.com進(jìn)入安裝。
轉(zhuǎn)載于:https://blog.51cto.com/11829889/1977173
總結(jié)
以上是生活随笔為你收集整理的keepalived+haproxy（双主）+nginx(静态)+lamp（动态）部署phpBB的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。
- 上一篇： 转：java中static、final、
- 下一篇： 20155301《信息安全系统设计基础》