Varnish purges 缓存清除
生活随笔
收集整理的這篇文章主要介紹了
Varnish purges 缓存清除
小編覺得挺不錯的,現在分享給大家,幫大家做個參考.
?
Varnish的緩存清除非常復雜。無論是Varnish的清除方式還是清除時候使用的語法規則等，都是比較復雜。為了理解他，我花費了不少時間，現在我很高興我知道怎么來解釋給大家聽了。
1、Varnish有兩種方式來清除緩存，其中一種方式是通過命中對象的單一變體，所以在他命中一個沒有壓縮的對象的時候他不能清除一個已經壓縮的對象。這個方式也就是強制過期(forced expiry)，他是通過設置你想清除的對象的TTL為0去強制它過期。VCL設置如下：
acl purge {"localhost";"192.0.2.14";
}sub vcl_recv {if (req.request == "PURGE") {if (!client.ip ~ purge) {error 405 "Not allowed.";}lookup;}
}sub vcl_hit {if (req.request == "PURGE") {set obj.ttl = 0s;error 200 "Purged.";}
}sub vcl_miss {if (req.request == "PURGE") {error 404 "Not in cache.";}
} 2、另外一種方式是使用purge_url，VCL設置如下：
acl purge {"localhost";"192.0.2.14";
}sub vcl_recv {if (req.request == "PURGE") {if (!client.ip ~ purge) {error 405 "Not allowed.";}purge("req.url == " req.url);}
通過以上在VCL文件的設置，我們通過HTTP來執行PURGE。這意味著你現在發送了一個：
PURGE / HTTP/1.0 Host: www.example.com
通過80端口給了Varnish。但是，這種執行PURGE的方式不支持正則。如果你想支持，可以按照這樣來設置VCL：
acl purge {"localhost";"192.0.2.14";
}sub vcl_recv {if (req.request == "PURGE") {if (!client.ip ~ purge) {error 405 "Not allowed.";}purge("req.url ~ " req.url);} 3、對于purge的方式，除了像上邊第2點那樣設置VCL來允許PURGE外，其實我們還可以通過Varnish的管理端口發送靈活的PURGE命令來清除緩存。
3.1 首先讓我們來看看管理端口的help(Varnish版本2.1)
[root@varnish4 varnish]# telnet 192.168.1.185 3500 Trying 192.168.1.185... Connected to 192.168.1.185 (192.168.1.185). Escape character is '^]'. 200 154 ----------------------------- Varnish HTTP accelerator CLI. ----------------------------- Type 'help' for command list. Type 'quit' to close CLI session.help 200 377 help [command] ping [timestamp] auth response quit banner status start stop stats vcl.load <configname> <filename> vcl.inline <configname> <quoted_VCLstring> vcl.use <configname> vcl.discard <configname> vcl.list vcl.show <configname> param.show [-l] [<param>] param.set <param> <value> purge.url <regexp> purge <field> <operator> <arg> [&& <field> <oper> <arg>]... purge.list
3.2 help中和purge有關的命令有三個，其中purge.list是查看purge的列表，能執行purge的是purge.url和purge兩個命令。
3.2.1 purge.url命令它只支持url的purge,如清除http://blog.izhoufeng.com/test.html。
[root@varnish2 varnish]# telnet 192.168.1.185 3500 Trying 192.168.1.185... Connected to varnish1 (192.168.1.185). Escape character is '^]'. 200 154 ----------------------------- Varnish HTTP accelerator CLI. ----------------------------- Type 'help' for command list. Type 'quit' to close CLI session.purge.url test.html 200 0
除用CLI接口外也可以用：
/usr/local/varnish-2.1/bin/varnishadm -T 192.168.1.185:3500 purge.url ^test.html$
3.2.2 purge命令則很靈活,請看列子：
清除http://izhoufeng.com/somedirectory/和目錄下的所有頁面。
purge req.http.host == izhoufeng.com && req.url ~ ^/somedirectory/.*$ or purge req.url ~ ^/somedirectory/ && req.http.host == izhoufeng.com
清除所有帶“Cache-Control: max-age=3600”的對象。
purge obj.http.Cache-Control ~ max-age=3600 or purge obj.http.Cache-Control ~ max-age ?= ?3600[^0-9]
4、對于大量清除，需要程序接口來做。
4.1 通過HTTP的PURGE的接口。
<?php
//刷新varnish緩存的函數，$ip為varnish服務器IP地址， $host為要刷新的網站域名，$url為要刷新的不含域名的URL地址
function varnish_purge($ip, $host, $url)
{ $errstr = ''; $errno = '';$fp = fsockopen ($ip, 80, $errno, $errstr, 10);if (!$fp) { return false; } else { $out = "PURGE {$url} HTTP/1.1\r\n"; $out .= "Host:{$host}\r\n"; $out .= "Connection: close\r\n\r\n"; fputs ($fp, $out); $out = fgets($fp , 4096);fclose ($fp); return true; }
} //用法：假設 192.168.1.185（varnish1）和192.168.1.186（varnish2）是兩臺varnish緩存服務器的內網IP地址，http://blog.izhoufeng.com/housing1d/08041110_2372147.htm為要刷新的地址
varnish_purge("varnish1", "blog.izhoufeng.com", "/housing1d/08041110_2372147.htm");
varnish_purge("varnish2", "blog.izhoufeng.com", "/housing1d/08041110_2372147.htm");
?>4.2 通過Varnish的管理端口的接口。
對下邊接口，先建立表把需要PURGE的URL放入表內。
mysql> show create table dirty_url;
+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| dirty_url | CREATE TABLE `dirty_url` (`id` int(11) NOT NULL AUTO_INCREMENT,`url` varchar(600) NOT NULL,`is_done` tinyint(2) NOT NULL DEFAULT '0',`ip` varchar(15) NOT NULL,`time` datetime NOT NULL,PRIMARY KEY (`id`),KEY `is_done` (`is_done`)
) ENGINE=InnoDB AUTO_INCREMENT=20627685 DEFAULT CHARSET=utf8 |
+-----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)mysql> select * from dirty_url limit 10;
+----------+---------------------------------------------------------------+---------+-----------------+---------------------+
| id | url | is_done | ip | time |
+----------+---------------------------------------------------------------+---------+-----------------+---------------------+
| 20544199 | http://blog.izhoufeng.com/xianzhilipin/10040810_3231622.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544200 | http://blog.izhoufeng.com/yingyouyunfu/10040810_3231623.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544201 | http://blog.izhoufeng.com/ershoubijibendiannao/10040810_3231624.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544202 | http://blog.izhoufeng.com/zhaoshangjiameng/10032610_2222213.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544203 | http://blog.izhoufeng.com/fushixiaobaxuemao/10040810_3231625.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544204 | http://blog.izhoufeng.com/wupinjiaohuan/10040810_3231626.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544205 | http://blog.izhoufeng.com/fushixiaobaxuemao/10040810_3231627.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544206 | http://blog.izhoufeng.com/wupinjiaohuan/10040810_3231628.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544207 | http://blog.izhoufeng.com/qitawupin/10040810_3231629.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
| 20544208 | http://blog.izhoufeng.com/shoujitongxun/10040810_3231630.htm | 0 | 192.168.1.186 | 2010-04-08 10:43:11 |
+----------+---------------------------------------------------------------+---------+-----------------+---------------------+<?php
//error_reporting(E_ALL);
//ini_set("log_errors", "1");
//ini_set("display_errors", "0");
set_time_limit(0);$adminHost[0] = "192.168.1.185"; // IP address to connect to
$adminHost[1] = "192.168.1.186"; // IP address to connect to
$adminHost[2] = "192.168.1.187"; // IP address to connect to
$adminPort = "3500"; // Port to connect tofunction pollServer($command,$adminHost,$adminPort) {$socket = socket_create(AF_INET, SOCK_STREAM, getprotobyname("tcp"));if ((!socket_set_option($socket, SOL_SOCKET, SO_RCVTIMEO, Array("sec" => "5", "usec" => "0"))) OR (!socket_set_option($socket, SOL_SOCKET, SO_SNDTIMEO, Array("sec" => "5", "usec" => "0")))) {die("Unable to set socket timeout");}if (@socket_connect($socket, $adminHost, $adminPort)) {$data = "";if (!$socket) {die("Unable to open socket to " . $server . ":" . $port . "\n");}socket_write($socket, $command . "\n");socket_recv($socket, $buffer, 65536, 0);$data .= $buffer;socket_close($socket);return $data;}else {return "Unable to connect: " . socket_strerror(socket_last_error()) . "\n";}
}
//this function is to add the web into varnish
function socketHttpGet($url,$hostIp)
{$urlInfo = parse_url($url);$urlInfo["path"] = ($urlInfo["path"] == "" ? "/" : $urlInfo["path"]);$urlInfo["port"] = (!isset($urlInfo["port"]) ? 80 : $urlInfo["port"]);$urlInfo["request"] = $urlInfo["path"] .(empty($urlInfo["query"]) ? "" : "?" . $urlInfo["query"]) .(empty($urlInfo["fragment"]) ? "" : "#" . $urlInfo["fragment"]);$fsock = fsockopen($hostIp, $urlInfo["port"], $errno, $errstr, 10);if (false == $fsock) {return false;}/* begin get */$in = "GET " . $urlInfo["request"] . " HTTP/1.0\r\n";$in .= "Host: " . $urlInfo["host"] . "\r\n";$in .= "Accept: */*\r\n";$in .= "Accept-Language: zh-CN\r\n";$in .= "Accept-Encoding: gzip, deflate\r\n";$in .= "Accept-Language: zh-CN\r\n";$in .= "User-Agent: Mozilla/4.0 (Auto add the Web to Varnish)\r\n";$in .= "Cache-Control: no-cache\r\n";$in .= "Connection: close\r\n\r\n";//stream_set_timeout($fsock, 10);if (!fwrite($fsock, $in, strlen($in))) {fclose($fsock);return false;}unset($in);//fclose($fsock);return;//process response$out = "";while ($buff = fgets($fsock, 2048)) {$out .= $buff;}//finish socketfclose($fsock);$pos = strpos($out, "\r\n\r\n");$head = substr($out, 0, $pos);return $head;
}
//now to purge from the mysql
while(true)
{
$conn=mysql_connect("192.168.1.186","username","password");
mysql_query("set names utf8");
mysql_select_db("queue",$conn);
$query="select url,id from dirty_url where is_done =0 order by id asc limit 200";
$results=mysql_query($query);
while($arr=mysql_fetch_array($results))
{$url=parse_url($arr['url']);$purge_url="purge.url ".$url['path'];$stats=0;foreach($adminHost as $value){$result=pollServer($purge_url,$value,$adminPort);$status = explode(" ", $result);if($status[0]=="200"){//socketHttpGet($arr['url'],$value);$stats++;}}if($stats==3){$ups="update dirty_url set is_done =1 where id=$arr[id]";mysql_query($ups);}
}
sleep(1);
}
?>我想接口固然是接口，但是如果到大型網站，每天刷新量有10多萬，就需要用到多線程來做了，下圖就是我開發的刷新程序，希望能給大家些啟發。
5、對于PURGE的幾種方式，原理都一樣，我們可以從purge.list的輸出結果就可以看出。
5.1 用2大點中用VCL來控制刷新，然后通過HTTP發送PURGE命令的日志如下：
384 req.url == /fang5/10040721_93041.htm0 req.url == /jiajiao/10040721_25874.htm0 req.url == /fang5/10040721_530212.htm0 req.url == /jiaoyou9/10040721_3814.htm0 req.url == /fang1/10040721_152079.htm0 req.url == /fang5/10040721_284698.htm0 req.url == /fang1/10040721_625739.htm0 req.url == /fang5/10040721_388442.htm0 req.url == /fang1/10040721_450056.htm0 req.url == /fang5/10040721_704267.htm0 req.url == /fang5/10040721_704266.htm0 req.url == /fang1/10040721_625738.htm0 req.url == /fang5/10040721_704265.htm0 req.url == /fang1/10040721_71558.htm0 req.url == /fang5/10040721_226345.htm0 req.url == /fang5/10040721_121378.htm0 req.url == /fang1/10040721_818489.htm
5.2 用purge.url命令的日志是：
0x2aaaaec44640 1270695397.844543 0 req.url ~ ^test.html$
5.3 用purge命令的日志是：
0x2aaaaecc9fa0 1270698757.617076 0 req.http.host ~ sh.izhoufeng.com && req.url ~ ^/jzjiuba/10040810_1173704.htm$ 0x2aaaaecc9f40 1270698757.616768 0 req.http.host ~ bj.izhoufeng.com && req.url ~ ^/fang5/10040711_6414363.htm$ 0x2aaaaecc9e80 1270698757.547097 0 req.http.host ~ sh.izhoufeng.com && req.url ~ ^/zpshichangyingxiao/09112600_1315775.htm$ 0x2aaaaecc9e20 1270698755.967497 0 req.http.host ~ sh.izhoufeng.com && req.url ~ ^/jzjiuba/10040810_1173702.htm$ 0x2aaaaecc9dc0 1270698755.665087 0 req.http.host ~ bj.izhoufeng.com && req.url ~ ^/zpsiji/10040811_3258734.htm$ 0x2aaaaecc9d60 1270698755.591958 0 req.http.host ~ bj.izhoufeng.com && req.url ~ ^/zpbaomu/10040811_3258735.htm$ 0x2aaaaecc9ca0 1270698755.356326 0 req.http.host ~ bj.izhoufeng.com && req.url ~ ^/zpyinyuebiaoyanzhuchi/10040811_3258736.htm$ 0x2aaaaecc9c40 1270698755.291577 0 req.http.host ~ bj.izhoufeng.com && req.url ~ ^/zpwenan/10040811_3258737.htm$ 0x2aaaaecc9be0 1270698755.290569 0 req.http.host ~ sh.izhoufeng.com && req.url ~ ^/lvshi/10040809_2410859.htm$
6、Varnish刷新原理是通過ban列表來操作的。
當我們執行如下：
purge req.url ~ \.png
這個操作就添加到活躍的bans的前邊。當有同樣的purge操作時，他就會一直添加，Varnish不可能遍歷它緩存的幾億個緩存對象以確定誰受影響。代替Varnish從緩存中查找對象是它通過比較purge list的bans。
如果匹配的ban被找到，這個ban和緩存中的對像進行比較。當有一個匹配的時候，對象被標記為不可用，除非另外個合適的對象能被找到，緩存hit將被一緩存miss替代，促使對象從后端獲取。
新創建的對象不被老bans約束，當我們插入一個對象到緩存，他被標記為checked以針對所有的在列表中的現有bans。
針對一個ban當所有的對象都已經被檢查過，這個ban就被從purge list中移除，同時內存重新可用。
7、我參考的文檔如下：
http://varnish-cache.org/wiki/Purging
http://kristian.blog.linpro.no/2010/02/02/varnish-purges/
http://varnish-cache.org/wiki/VCLExamplePurging
總結
以上是生活随笔為你收集整理的Varnish purges 缓存清除的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： “名为公器无多取”下一句是什么
- 下一篇： 暴风电视机开不了机如何回事？