從一個表里group by 之后 計算累加值、去重值:

為了效率設置并行：set hive.exec.parallel=true（可選：set hive.exec.parallel.thread.number=16）、set hive.groupby.skewindata=true、set hive.map.aggr=true

select plat, pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat, pagetype union all select plat, 'all' pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat union all select 'all' plat, pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by pagetype union all select 'all' plat, 'all' pagetype, count(*) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19'

壞就壞在：set hive.map.aggr=true，map端聚合的設置；

出來的pv數跟真實值對不上；

改成下邊代碼運行正確；

select plat, pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat, pagetype union all select plat, 'all' pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by plat union all select 'all' plat, pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19' group by pagetype union all select 'all' plat, 'all' pagetype, sum(1) pv, count(distinct userkey) uv from client_pv_form where dt = '2015-08-19'

?

轉載于:https://www.cnblogs.com/sudz/p/4745985.html

與50位技術專家面對面20年技術見證，附贈技術全景圖

總結

以上是生活随笔為你收集整理的set hive.map.aggr=true 时统计PV数据错误的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。