使用gitlab-ci、sonarqube、sonar-scanner 實現以下功能css
1.一旦提交代碼就進行代碼質量檢測html
2. 發送檢測報告郵件給提交者java
先來看下最終結果,郵件中有檢測報告,具體bug等詳細狀況可點擊郵件中的 url 跳轉到檢測結果進行查看python
sonarqube中的概況mysql
Sonarqube中代碼bug等具體信息linux
Gitlab-ci 結果git
若是這也是你想實現的功能的話,那麼請往下看,不然就不須要浪費時間了github
Jenkins結合sonarqube可參考 http://www.javashuo.com/article/p-tvkbvkni-bu.htmlweb
centso: 7.4 gitlab: 12.2.3sql
jdk: 11.0.3 Scanner: 4.0.0.1744
python: 3.6.8
centso: 7.4 docker: 19.03.13
jdk: 11.0.3 sonarqube: 7.9.4
postgres:13
轉載請在文章開頭附上原文連接地址:http://www.javashuo.com/article/p-fzctmsru-mv.html
gitlab、gitlab-runner、jdk 安裝與配置請自行解決
由於不支持mysql了,oracle和SqlServer又不想用。
docker pull postgres
啓動並設置用戶名和密碼 均爲 sonarqube
docker run --name=postgresql -p 5432:5432 -e POSTGRES_DB=sonarqube \
-e POSTGRES_USER=sonarqube -e POSTGRES_PASSWORD=sonarqube -d postgres
(1)編輯 /etc/security/limits.conf,新增以下兩項。sonarqube爲用戶名,待會會新增這個用戶
sonarqube soft nofile 65536
sonarqube hard nofile 65536
(2) 設置 max_map_count
sysctl -w vm.max_map_count=262144 sysctl -p
新建sonarqube用戶
useradd sonarqube
切換至sonarqube
su - sonarqube
sonarqube官網下載:
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.4.zip -d /opt/
sonarqube配置:
sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube?currentSchema=my_schema sonar.jdbc.username=sonarqube sonar.jdbc.password=sonarqube sonar.jdbc.url=jdbc:postgresql://127.0.0.1/sonarqube sonar.web.port=9000
注意: 若是不是yum安裝jdk的話,還須要改 wrapper.conf 中的 wrapper.java.command配置
下載插件,達到全部分支都可掃描
wget https://github.com/mc1arke/sonarqube-community-branch-plugin/releases/download/1.3.2/sonarqube-community-branch-plugin-1.3.2.jar
cp sonarqube-community-branch-plugin-1.3.2.jar /opt/sonarqube-7.9.4/extensions/plugins/
cp sonarqube-community-branch-plugin-1.3.2.jar /opt/sonarqube-7.9.4/lib/common/
啓動sonarqube,不能以root用戶啓動,我這裏使用的是sonarqube用戶
/opt/sonarqube-7.9.4/bin/linux-x86-64/sonar.sh start
官網下載:
wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.0.0.1744-linux.zip
解壓並配置
unzip sonar-scanner-cli-4.0.0.1744-linux.zip -d /opt/
編輯 /opt/sonar-scanner-4.0.0.1744-linux/conf/sonar-scanner.properties
sonar.host.url=https://your-sonarqube.com # sonarqube 的url sonar.login=admin # sonarqube 的用戶名和密碼 sonar.password=admin sonar.sourceEncoding=UTF-8 sonar.language=java sonar.sources=. sonar.java.binaries=.
配置環境變量
新增文件 /etc/profile.d/sonar-scanner.sh,內容以下
export PATH=$PATH:/opt/sonar-scanner-4.0.0.1744-linux/bin/sonar-scanner
source /etc/profile.d/sonar-scanner.sh
檢查是否安裝成功
sonar-scanner -v
INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties INFO: Project root configuration file: NONE INFO: SonarQube Scanner 4.0.0.1744 INFO: Java 11.0.3 AdoptOpenJDK (64-bit) INFO: Linux 3.10.0-693.el7.x86_64 amd64
目的就是一旦用戶提交代碼,觸發代碼掃描併發送郵件給代碼提交者。
在項目中新增 .gitlab-ci.yml 文件
stages:
- sonarqube_scan
- sendmail
sonarqube_scan_job:
stage: sonarqube_scan
script:
- sonar-scanner -Dsonar.projectName=$CI_PROJECT_NAME -Dsonar.projectKey=$CI_PROJECT_NAME -Dsonar.branch.name=${CI_COMMIT_REF_NAME} -Dsonar.language=java -Dsonar.host.url=https://your-sonarqube.com -Dsonar.login=admin -Dsonar.password=admin
tags:
- sonar-scanner
when: always
sendmail_job:
stage: sendmail
script:
- echo $GITLAB_USER_EMAIL
- echo $CI_PROJECT_NAME
- echo $CI_COMMIT_REF_NAME
- python3 /opt/sonarqube_api.py $CI_PROJECT_NAME $CI_COMMIT_REF_NAME $GITLAB_USER_EMAIL
tags:
- sonar-scanner
參數說明:
tag: gitlab-runner中的tag, 我配置的tag是sonar-scanner
$CI_PROJECT_NAME: gitlab內置參數,爲項目名稱
$GITLAB_USER_EMAIL: gitlab內置參數,提交者的郵箱,傳遞給Python,爲了後邊發郵件用
$CI_COMMIT_REF_NAME: gitlab內置參數,本次提交的分支
-Dsonar.projectName=$CI_PROJECT_NAME 爲在sonarqube中項目的名稱
-Dsonar.projectKey=$CI_PROJECT_NAME 爲在sonarqube中項目的惟一標識
-Dsonar.host.url=https://your-sonarqube.com sonarqube的url
-Dsonar.login=admin -Dsonar.password=admin sonarqube的用戶名和密碼配置
代碼以下
import sys import smtplib from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from sonarqube import SonarQubeClient def sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password): mail_msg = MIMEMultipart() mail_msg['Subject'] = subject mail_msg['From'] = fromaddr mail_msg['To'] = ','.join(toaddrs) mail_msg.attach(MIMEText(msg, 'html', 'utf-8')) try: s = smtplib.SMTP_SSL(smtpserver) s.connect(smtpserver, 465) # 鏈接smtp服務器 s.login(fromaddr, password) # 登陸郵箱 s.sendmail(fromaddr, toaddrs, mail_msg.as_string()) # 發送郵件 s.quit() print("send successful!") except Exception as e: print(e) print("Failed to send ") def getSonarqubeInfo(branch="master", component=None, url=None, username=None, password=None): sonar = SonarQubeClient(sonarqube_url=url) sonar.auth.authenticate_user(login=username, username=username, password=password) component_data = sonar.measures.get_component_with_specified_measures( component=component, branch=branch, fields="metrics,periods", metricKeys=""" code_smells,bugs,coverage,duplicated_lines_density,ncloc, security_rating,reliability_rating,vulnerabilities,comment_lines_density, ncloc_language_distribution,alert_status,sqale_rating """ ) result_dict = {} for info_dict in component_data["component"]["measures"]: result_dict[info_dict["metric"]] = info_dict["value"] # print(result_dict) return result_dict def main(): url = "https://your-sonarqube.com" username = "admin" password = "admin" branch = sys.argv[2] project = sys.argv[1] project_url = "{}/dashboard?id={}&branch={}".format(url, project, branch)
user_email = sys.argv[3]
sonarqube_data = getSonarqubeInfo(branch=branch, component=project, url=url, username=username, password=password) html_text = """ <!DOCTYPE html> <html lang="en"> <head> <title></title> <meta charset="utf-8"> </head> <body> <div class="page" style="margin-left: 30px"> <h3>{user_mail}, 你好</h3> <h3> 本次提交代碼檢查結果以下</h3> <h3> 項目名稱:{project} </h3> <h3> 分支:{branch} </h3> <h4>1、整體狀況</h4> <ul> <li style="font-weight:bold;"> 本次掃描代碼行數: <span style="color:blue">{lines} </span>, bugs: <span style="color:red">{bugs}</span>, Vulnerabilities: <span style="color:red">{vulnerabilities}</span>, Code Smells: <span style="color:red">{code_smells}</span> </li> <li style="font-weight:bold;margin-top: 10px;"> URL地址: <a style="font-weight:bold;" href={project_url}>{project_url} </a> </li> </ul> <h4>2、信息詳情</h4> <ul> <li style="font-weight:bold;"> 綜合等級: {sqale_rating} </li> <li style="font-weight:bold;"> 各語言掃描行數: {ncloc_language_distribution} </li> <li style="font-weight:bold;"> 代碼重複率: {duplicated_lines_density} </li> <li style="font-weight:bold;"> 安全等級: {security_rating} </li> <li style="font-weight:bold;"> 可靠等級: {reliability_rating} </li> <li style="font-weight:bold;"> 註釋行密度: {comment_lines_density} </li> </ul> </div> </body> </html> """.format(project_url=project_url, user_mail=user_email, project=project, branch=branch, lines=sonarqube_data["ncloc"], bugs=sonarqube_data["bugs"], vulnerabilities=sonarqube_data["vulnerabilities"], code_smells=sonarqube_data["code_smells"], ncloc_language_distribution=sonarqube_data["ncloc_language_distribution"], duplicated_lines_density=sonarqube_data["duplicated_lines_density"], reliability_rating=sonarqube_data["reliability_rating"], security_rating=sonarqube_data["security_rating"], comment_lines_density=sonarqube_data["comment_lines_density"], sqale_rating=sonarqube_data["sqale_rating"] ) fromaddr = "gitlab@xxx.com" smtpserver = "smtpdm-ap-southeast-1.aliyun.com" toaddrs = [user_email, ] subject = "Gitlab代碼質量檢測" password = "xxxx" msg = html_text # print(msg) sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password) if __name__ == '__main__': main()
建議用方法一,方法二太挫了
用sonar-scanner掃描代碼以後,去查sonarqube的數據庫而後在把數據拼湊成郵件進行發送,
因爲7.9已經不支持mysql,我這裏用的postgresql,其餘數據庫改下Python所用模塊和鏈接就行,sql語句應該不須要改動就可以使用
編輯 sonarqube.py
import os import sys import smtplib import psycopg2 from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from jinja2 import FileSystemLoader, Environment def select_project_uuid(project_name): db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() select_p_uuid = "SELECT project_uuid,kee FROM projects WHERE name= '%s'" % (project_name) cursor.execute(select_p_uuid) result = cursor.fetchone() p_uuid = result[0] projectKey = result[1] db.close() return (p_uuid, projectKey) def select_total_info(p_uuid): total_info = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() select_p_links = "SELECT text_value FROM project_measures WHERE text_value LIKE 'java=%' and component_uuid='{}'".format( str(p_uuid)) cursor.execute(select_p_links) p_links = str(cursor.fetchone()[0].split("=")[1]).split(";")[0] sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =%s" for leak in [2, 3, 1]: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) total_info.append(cursor.fetchone()[0]) db.close() return p_links, total_info def select_bugs(p_uuid): bugs = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =2 AND severity ='%s'" for leak in ['BLOCKER', 'CRITICAL', "MAJOR", 'MINOR', 'INFO']: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) bugs.append(cursor.fetchone()[0]) db.close() return bugs def select_leaks(p_uuid): leaks = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =3 AND severity ='%s'" for leak in ['BLOCKER', 'CRITICAL', "MAJOR", 'MINOR', 'INFO']: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) leaks.append(cursor.fetchone()[0]) db.close() return leaks def select_bad_tastes(p_uuid): tastes = [] db = psycopg2.connect(database=database, user=user, password=password, host=host, port=port) cursor = db.cursor() sql_info = "SELECT count(*) FROM issues WHERE project_uuid='%s' and issue_type =1 AND severity ='%s'" for leak in ['BLOCKER', 'CRITICAL', "MAJOR", 'MINOR', 'INFO']: search_data = sql_info % (p_uuid, leak) cursor.execute(search_data) tastes.append(cursor.fetchone()[0]) db.close() return tastes def generate_errmsg_table(s_lines="", total_data=[], bugs=[], leaks=[], tastes=[], report_url="", project_name=sys.argv[1], user_email=sys.argv[2]): env = Environment(loader=FileSystemLoader(curpath, 'utf-8')) # 建立一個包加載器對象 template = env.get_template(table_tem_name) html_content = ( template.render( lins=s_lines, total_data=total_data, bugs=bugs, leaks=leaks, tastes=tastes, report_url=report_url, project_name=project_name, user_email=user_email, )) fh = open(report_html_path, 'w') fh.write(html_content) fh.close() def sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password): """ :param subject: 郵件主題 :param msg: 郵件內容 :param toaddrs: 收信人的郵箱地址 :param fromaddr: 發信人的郵箱地址 :param smtpserver: smtp服務地址 :param password: 發信人的郵箱密碼 :return: """ mail_msg = MIMEMultipart() mail_msg['Subject'] = subject mail_msg['From'] = fromaddr mail_msg['To'] = ','.join(toaddrs) mail_msg.attach(MIMEText(msg, 'html', 'utf-8')) try: s = smtplib.SMTP_SSL(smtpserver) s.connect(smtpserver, 465) # 鏈接smtp服務器 s.login(fromaddr, password) # 登陸郵箱 s.sendmail(fromaddr, toaddrs, mail_msg.as_string()) # 發送郵件 s.quit() print("send successful!") except Exception as e: print(e) print("Failed to send ") def main(): p_uuid, projectKey = select_project_uuid(project_name) s_lines, total_data = select_total_info(p_uuid) bugs = select_bugs(p_uuid) leaks = select_leaks(p_uuid) tastes = select_bad_tastes(p_uuid) report_url = "https://your-sonarqube.com/dashboard?id=%s" % (projectKey) generate_errmsg_table(s_lines, total_data, bugs, leaks, tastes, report_url, project_name, user_email) with open("report_%s.html" % (project_name)) as f: message = str(f.read()) f.close() #配置郵件服務器,我這裏用的是阿里的郵件服務器 fromaddr = "xxxx@xxx.com" smtpserver = "xxx.com" toaddrs = [user_email, ] subject = "Gitlab代碼質量檢測" password = "yourpassword" msg = message sendmail(subject, msg, toaddrs, fromaddr, smtpserver, password) curpath = os.getcwd() table_tem_name = "table.html" project_name = sys.argv[1] user_email = sys.argv[2] report_html_path = "report_" + project_name + ".html" # sonarqube數據庫帳號密碼等 database = "sonarqube" user = "sonarqube" password = "sonarqube" host = "xx.xx.xx.xx" port = "5432" if __name__ == '__main__': main()
table.html , table.html須要和sonarqube.py放在同一個目錄下
<!DOCTYPE html> <html lang="en"> <head> <title></title> <meta charset="utf-8"> <style type="text/css"> table { text-align: center; border-style: solid solid solid solid; border-collapse: collapse; width:550px ; height:120px; } tr:hover { background-color: #F7F9FF; cursor: pointer; } .page { margin-left: 30px; } </style> </head> <body> <div class="page"> <h3>{{ user_email }}, 你好</h3> <h3> 本次提交代碼檢查結果以下</h3> <h3> 項目名稱:{{ project_name }} </h3> <h4>1、整體狀況</h4> <ul> <li style="font-weight:bold;"> 總體運行狀況:掃描代碼行數: <span style="color:blue">{{lins}} </span>, bugs: <span style="color:red">{{total_data[0]}}</span>, Vulnerabilities: <span style="color:red">{{total_data[1]}}</span>, Code Smells: <span style="color:red">{{total_data[2]}}</span> </li> <li style="font-weight:bold;margin-top: 10px;"> URL地址: <a style="font-weight:bold;" href={{report_url}}>{{report_url}}</a> </li> </ul> <h4>2、信息詳情</h4> <table cellspacing="0" border="1px" > <thead> <tr style="background-color:#F7F9FF"> <th>{{ project_name }}</th> <th>阻斷</th> <th>嚴重</th> <th>主要</th> <th>次要</th> <th>提示</th> <th>總數</th> </tr> </thead> <tbody> <tr> <td>bugs</td> <td>{{bugs[0]}}</td> <td>{{bugs[1]}}</td> <td>{{bugs[2]}}</td> <td>{{bugs[3]}}</td> <td>{{bugs[4]}}</td> <td style="color:red">{{total_data[0]}}</td> </tr> <tr> <td>Vulnerabilities</td> <td>{{leaks[0]}}</td> <td>{{leaks[1]}}</td> <td>{{leaks[2]}}</td> <td>{{leaks[3]}}</td> <td>{{leaks[4]}}</td> <td style="color:red">{{total_data[1]}}</td> </tr> <tr> <td>Code Smells</td> <td>{{tastes[0]}}</td> <td>{{tastes[1]}}</td> <td>{{tastes[2]}}</td> <td>{{tastes[3]}}</td> <td>{{tastes[4]}}</td> <td style="color:red">{{total_data[2]}}</td> </tr> </tbody> </table> </div> </body> </html>
方法二郵件以下圖
感謝閱讀,有問題歡迎找我交流
參考文章:
https://docs.gitlab.com/ee/ci/variables/
https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37115
https://docs.sonarqube.org/7.9/setup/install-server/
https://pypi.org/project/python-sonarqube-api/