背景:python
梳理統計框架時不可避免會涉及到get的數據和插入數據庫的字段,梳理完的初版統計框架仍是偏醜陋的(邊寫邊改,沒好好總體的規劃)。有大俠耐心看完的話能夠提一些指導性建議,先在此感謝。mysql
假設有2個不可合併統計需求,一個統計結果(表結構)是:pv, uv;另外一個是 device_type, dau, new_device. 2個統計都是直接經過SQL(Mysql或者Hive)實現, mysql_excete(sql) 和 hive_execute(sql) 分別是數據獲取接口,返回[(),()...]。sql
若是單獨寫2個統計腳本:數據庫
1. 腳本 #數據獲取 result = mysql_excete(sql) for res in result: pv, uv = res #數據存儲 '''insert into t1(pv, uv) values(%d, %d)''' %(pv, uv) 2.腳本 #數據獲取 result = hive_execute(sql) for res in result: device_type, dau, new_device = res #數據存儲 '''insert into t1(device_type, dau, new_device) values(%s, %d, %d)''' %(device_type, dau, new_device)
要合併成公共函數,必須考慮的就是動態的變量,以及賦值,緣由和實現可參考代碼。另外因爲中文的問題,暫時每條記錄保存一次,而且將結果保存到dict中。框架
只開放了部分代碼,參考思路便可:
函數
簡單父類思路: class father(): def __init__(self): self.get_sql = '' self.column = '' self.get_result = [] self.save_table = '' self.save_map = {} self.save_sql = '' def get_from_mysql(self): result = cursor.fetchall() for res in result: ex_cmd = "%s=%s" %(self.get_column, res) print ex_cmd exec(ex_cmd) for key in self.save_map: if key not in ("save_table", "save_column"): self.save_map[key] = eval(key) self.save_data() cursor.close() def save_data(self): cursor = xxx try: cursor.execute(self.save_sql % self.save_map) except Exception, e: print e print self.save_sql % self.save_map finally: cursor.close() xxx.commit() def run(self): pass class daily_stat(father): def __init__(self, dt): fbase.__init__(self, dt) def stat_pv_uv(self): self.get_sql = '''select pv, uv .... ''' self.column = '(pv, uv)' self.save_map = {'pv':0, 'uv_name':0} self.save_map['save_table'] = self.save_table self.save_map['save_column'] = self.column self.save_table = 't1' self.save_sql = "insert into %(save_table)s %(save_column)s values ( %(pv)d, %(uv) )" self.get_from_mysql_many() def stat_device(self): self.get_sql = '''select pv, uv .... ''' self.column = '(device_type, dau, new_device)' self.save_map = {'device_type':'', 'dau':0, :0} self.save_map['save_table'] = self.save_table self.save_map['save_column'] = self.column self.save_table = 't2' self.save_sql = "insert into %(save_table)s %(save_column)s values ( '%(device_type)s', %(dau)d, %(new_device)d )" self.get_from_mysql_many() def run(self): self.stat_pv_uv() self.stat_device()
核心問題點: fetch
1. tuple的動態賦值:spa
s = (k1, k2, k3) s = res #(res = (1,2,3) ) 結果 k1 = 1, k2 = 2, k3 = 3
試了幾個方案,最終實現是用exec:code
s = '(k1, k2, k3)' ex_cmd = "%s=%s" %(s, res) #(res = (1,2,3) ) exec(ex_cmd) 結果實現
2. map中key的字段名 和 變量名的切換:接口
(k1, k2, k3) = (1, 2, 3) m = {'k1':0, 'k2':0, 'k3':0} for key in m: m[key] = eval(key)