.NetCore實踐篇：分佈式監控Zipkin持久化之殤
前言
本系列已寫了四篇文章，讀本篇以前，能夠先讀前面幾篇。
思考大綱：.Net架構篇:思考如何設計一款實用的分佈式監控系統？
實踐篇一：.NetCore實踐篇：分佈式監控客戶端ZipkinTracer從入門到放棄之路
實踐篇二：.NetCore實踐篇：分佈式監控系統zipkin踩坑之路（二）
實踐篇三：.NetCore實踐篇：成功解決分佈式監控ZipKin聚合依賴問題（三）javascript
簡要回顧
zipkin
Zipkin是一種分佈式跟蹤系統。它有助於收集解決微服務架構中的延遲問題所需的時序數據
zipkin官網html
zipkin4Net
zipkin4net是.NET客戶端庫。
zipkin4netjava
zipkin-dependencies
這是一個Spark做業，它將從您的數據存儲區收集跨度，分析服務之間的連接，並存儲它們以供之後在Web UI中呈現。node
zipkin-dependencies
使用方法
若是內存不足時，java後跟上-Xmx1024m -Xms1024m參數，JAVA_OPTS的一些參數可參考Oracle官方說明配置默認JVM和Java參數mysql
# ex to run the job to process yesterday's traces on OS/X
$ STORAGE_TYPE=cassandra3 java -jar zipkin-dependencies.jar `date -uv-1d +%F`
# or on Linux
$ STORAGE_TYPE=cassandra3 java -jar zipkin-dependencies.jar `date -u -d '1 day ago' +%F`
MySQL 存儲
* `MYSQL_DB`: 使用的數據庫，默認是 "zipkin".
* `MYSQL_USER` and `MYSQL_PASS`: MySQL受權, 默認是空.
* `MYSQL_HOST`: 默認主機（域名/ip）是localhost
* `MYSQL_TCP_PORT`: 默認端口是 3306
* `MYSQL_USE_SSL`: 驗證 `javax.net.ssl.trustStore` 和 `javax.net.ssl.trustStorePassword`,默認不驗證。
示例
$ STORAGE_TYPE=mysql MYSQL_USER=root java -jar zipkin-dependencies.jar
詳情參考：
zipkin-dependencieslinux
實踐
建立使用數據庫
mysql> create database mytestdb;
Query OK, 1 row affected (0.01 sec)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| mytestdb |
| performance_schema |
| sys |
| ttt |
+--------------------+
7 rows in set (0.00 sec)
mysql> use mytestdb
Database changed
使用sql語句建立zipkin表
CREATE TABLE IF NOT EXISTS zipkin_spans (
`trace_id_high` BIGINT NOT NULL DEFAULT 0COMMENT 'If non zero, this means the trace uses 128 bit traceIds instead of 64bit',
`trace_id` BIGINT NOT NULL,
`id` BIGINT NOT NULL,
`name` VARCHAR(255) NOT NULL,
`parent_id` BIGINT,
`debug` BIT(1),
`start_ts` BIGINT COMMENT 'Span.timestamp():epoch micros used for endTs query and to implement TTL',
`duration` BIGINT COMMENT 'Span.duration():micros used for minDuration and maxDuration query'
)ENGINE=InnoDB ROW_FORMAT=COMPRESSED CHARACTER SET=utf8 COLLATE utf8_general_ci;
ALTER TABLE zipkin_spans ADD UNIQUE KEY(`trace_id_high`, `trace_id`, `id`) COMMENT'ignore insert on duplicate';
ALTER TABLE zipkin_spans ADD INDEX(`trace_id_high`, `trace_id`, `id`) COMMENT 'forjoining with zipkin_annotations';
ALTER TABLE zipkin_spans ADD INDEX(`trace_id_high`, `trace_id`) COMMENT 'forgetTracesByIds';
ALTER TABLE zipkin_spans ADD INDEX(`name`) COMMENT 'for getTraces and getSpanNames';
ALTER TABLE zipkin_spans ADD INDEX(`start_ts`) COMMENT 'for getTraces ordering andrange';
CREATE TABLE IF NOT EXISTS zipkin_annotations (
`trace_id_high` BIGINT NOT NULL DEFAULT 0COMMENT 'If non zero, this means the trace uses 128 bit traceIds instead of 64bit',
`trace_id` BIGINT NOT NULL COMMENT 'coincideswith zipkin_spans.trace_id',
`span_id` BIGINT NOT NULL COMMENT 'coincideswith zipkin_spans.id',
`a_key` VARCHAR(255) NOT NULL COMMENT'BinaryAnnotation.key or Annotation.value if type == -1',
`a_value` BLOB COMMENT'BinaryAnnotation.value(), which must be smaller than 64KB',
`a_type` INT NOT NULL COMMENT'BinaryAnnotation.type() or -1 if Annotation',
`a_timestamp` BIGINT COMMENT 'Used toimplement TTL; Annotation.timestamp or zipkin_spans.timestamp',
`endpoint_ipv4` INT COMMENT 'Null whenBinary/Annotation.endpoint is null',
`endpoint_ipv6` BINARY(16) COMMENT 'Null whenBinary/Annotation.endpoint is null, or no IPv6 address',
`endpoint_port` SMALLINT COMMENT 'Null whenBinary/Annotation.endpoint is null',
`endpoint_service_name` VARCHAR(255) COMMENT'Null when Binary/Annotation.endpoint is null'
)ENGINE=InnoDB ROW_FORMAT=COMPRESSED CHARACTER SET=utf8 COLLATE utf8_general_ci;
ALTER TABLE zipkin_annotations ADD UNIQUE KEY(`trace_id_high`, `trace_id`, `span_id`,`a_key`, `a_timestamp`) COMMENT 'Ignore insert on duplicate';
ALTER TABLE zipkin_annotations ADD INDEX(`trace_id_high`, `trace_id`, `span_id`)COMMENT 'for joining with zipkin_spans';
ALTER TABLE zipkin_annotations ADD INDEX(`trace_id_high`, `trace_id`) COMMENT 'forgetTraces/ByIds';
ALTER TABLE zipkin_annotations ADD INDEX(`endpoint_service_name`) COMMENT 'forgetTraces and getServiceNames';
ALTER TABLE zipkin_annotations ADD INDEX(`a_type`) COMMENT 'for getTraces';
ALTER TABLE zipkin_annotations ADD INDEX(`a_key`) COMMENT 'for getTraces';
ALTER TABLE zipkin_annotations ADD INDEX(`trace_id`, `span_id`, `a_key`) COMMENT 'fordependencies job';
CREATE TABLE IF NOT EXISTS zipkin_dependencies (
`day` DATE NOT NULL,
`parent` VARCHAR(255) NOT NULL,
`child` VARCHAR(255) NOT NULL,
`call_count` BIGINT
)ENGINE=InnoDB ROW_FORMAT=COMPRESSED CHARACTER SET=utf8 COLLATE utf8_general_ci;
ALTER TABLE zipkin_dependencies ADD UNIQUE KEY(`day`, `parent`, `child`);
建立成功後，查詢結果。git
mysql> show tables;
+---------------------+
| Tables_in_mytestdb |
+---------------------+
| zipkin_annotations |
| zipkin_dependencies |
| zipkin_spans |
+---------------------+
3 rows in set (0.00 sec)
啓動zipkin-dependencies
最開始個人密碼是【四個字母一個感嘆號一個數字】，再執行啓動命令時，密碼那塊給我報錯自動換成【四個字母rm -f】，我修改爲【四個字母一個#號一個數字】就能執行了github
執行成功後，依然提示Access denied for user 'root'@'localhost' (using password: NO),但我在linux的命令中直接用mysql -u root -p相同密碼是能夠登陸成功的。因此問題出如今哪呢？spring
[root@izwz9fwifc2eniq3lbdzmgz cusD]# STORAGE_TYPE=mysql MYSQL_HOST=localhost MYSQL_TCP_PORT=3306 MYSQL_DB=mytestdb MYSQL_USER=XXXXX MYSQL_PASS=XXXXX java -Xmx1024m -Xms1024m -jar zipkin-dependencies.jar
Exception in thread "main" java.lang.RuntimeException: java.sql.SQLInvalidAuthorizationSpecException: Access denied for user 'root'@'localhost' (using password: NO)
at zipkin2.dependencies.mysql.MySQLDependenciesJob.hasTraceIdHigh(MySQLDependenciesJob.java:233)
at zipkin2.dependencies.mysql.MySQLDependenciesJob.run(MySQLDependenciesJob.java:184)
at zipkin2.dependencies.ZipkinDependenciesJob.main(ZipkinDependenciesJob.java:65)
Caused by: java.sql.SQLInvalidAuthorizationSpecException: Access denied for user 'root'@'localhost' (using password: NO)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:173)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1115)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:502)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:154)
at org.mariadb.jdbc.Driver.connect(Driver.java:86)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at zipkin2.dependencies.mysql.MySQLDependenciesJob.hasTraceIdHigh(MySQLDependenciesJob.java:229)
... 2 more
Caused by: java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: NO)
Current charset is UTF-8. If password has been set using other charset, consider using option 'passwordCharacterEncoding'
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authentication(AbstractConnectProtocol.java:862)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:785)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:456)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1111)
... 8 more
追溯源碼
逼不得已，走上查看源碼之路，Idea打開zipkin-dependencies/mysql源碼，查看相關部分代碼。sql
public static final class Builder {
Map<String, String> sparkProperties = ImmutableMap.of(
"spark.ui.enabled", "false"
);
String db = getEnv("MYSQL_DB", "zipkin");
String host = getEnv("MYSQL_HOST", "localhost");
int port = Integer.parseInt(getEnv("MYSQL_TCP_PORT", "3306"));
String user = getEnv("MYSQL_USER", "");
String password = getEnv("MYSQL_PASS", "");
int maxConnections = Integer.parseInt(getEnv("MYSQL_MAX_CONNECTIONS", "10"));
boolean useSsl = Boolean.parseBoolean(getEnv("MYSQL_USE_SSL", "false"));
// local[*] master lets us run & test the job locally without setting a Spark cluster
String sparkMaster = getEnv("SPARK_MASTER", "local[*]");
// By default the job only works on traces whose first timestamp is today
long day = midnightUTC(System.currentTimeMillis());
/** *爲減小篇幅，中間設置屬性部分代碼省略。 */
public MySQLDependenciesJob build() {
return new MySQLDependenciesJob(this);
}
}
/** *爲減小篇幅，中間部分代碼省略。 */
MySQLDependenciesJob(Builder builder) {
this.db = builder.db;
this.day = builder.day;
SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");
df.setTimeZone(TimeZone.getTimeZone("UTC"));
this.dateStamp = df.format(new Date(builder.day));
this.url = new StringBuilder("jdbc:mysql://")
.append(builder.host).append(":").append(builder.port)
.append("/").append(builder.db)
.append("?autoReconnect=true")
.append("&useSSL=").append(builder.useSsl).toString();
this.user = builder.user;
this.password = builder.password;
this.conf = new SparkConf(true)
.setMaster(builder.sparkMaster)
.setAppName(getClass().getName());
if (builder.jars != null) conf.setJars(builder.jars);
for (Map.Entry<String, String> entry : builder.sparkProperties.entrySet()) {
conf.set(entry.getKey(), entry.getValue());
}
this.logInitializer = builder.logInitializer;
}
void saveToMySQL(List<DependencyLink> links) {
try (Connection con = DriverManager.getConnection(url, user, password)) {
PreparedStatement replace = con.prepareStatement(
"REPLACE INTO zipkin_dependencies (day, parent, child, call_count, error_count) VALUES (?,?,?,?,?)");
for (DependencyLink link : links) {
replace.setDate(1, new java.sql.Date(day));
replace.setString(2, link.parent());
replace.setString(3, link.child());
replace.setLong(4, link.callCount());
replace.setLong(5, link.errorCount());
replace.executeUpdate();
}
} catch (SQLException e) {
throw new RuntimeException("Could not save links " + links, e);
}
}
然並卵，看完以後，沒看出明顯問題。難道仍是我本身的mysql配置問題？仍是啓動部分的參數問題？代碼部分也是有些疑惑,password和root爲何沒放進url裏，難道是爲了安全考慮麼？
this.url = new StringBuilder("jdbc:mysql://")
.append(builder.host).append(":").append(builder.port)
.append("/").append(builder.db)
.append("?autoReconnect=true")
.append("&useSSL=").append(builder.useSsl).toString();
this.user = builder.user;
this.password = builder.password;
文中還提到 Current charset is UTF-8. If password has been set using other charset, consider using option 'passwordCharacterEncoding'，編碼格式是否有不一樣呢？
查看mysql數據庫及表編碼格式
mysql> show variables like 'character_set_database';
+------------------------+---------+
| Variable_name | Value | +------------------------+---------+ | character_set_database | utf8mb4 |
+------------------------+---------+
1 row in set (0.01 sec)
參考連接
https://jira.mariadb.org/browse/CONJ-480
https://samebug.io/exceptions/2980875/java.sql.SQLInvalidAuthorizationSpecException/could-not-connect-access-denied-for-user
將請求參數做爲UTF-8編碼的字符串傳遞[重複]
zipkin
zipkin集成到node，C#
微服務之分佈式跟蹤系統（springboot+zipkin+mysql）
第二十九章 springboot + zipkin + mysql
Linux下的Mysql用命令執行sql文件
修改MySQL管理員密碼
zipkin-server
總結
因爲啓動zipkin-dependencies連接mysql報Access denied for user 'root'@'localhost' (using password: NO)錯誤，本次持久化之路最終失敗。但因爲我直接使用【mysql -u 用戶 -p】是能登陸成功的，因此我猜想了如下緣由：
- 客戶端本身的bug,和我服務器mysql版本不兼容？
- 編碼問題，編碼二者不符？
- 用戶名和密碼沒有共享全局，只對一個數據庫有效？
別人的博文是面向教學成功編程，個人是面向失敗編程，也別有一番趣味。留下疑問，待往後解決調。雖然失敗了，但我又收集了一堆連接，增添了mysql一些故障解決的認識。
今天換了一款我的很喜歡的皮膚，會根據h1,h2自動生成目錄，以前的博文我也都檢查了下，有很大失位的我都調整了過來，不標準的暫時不改了，我之後的博文都按照要求的格式寫，排版美觀度提高了不少，感謝做者bndong，若是有打算使用這個皮膚的，必定要開啓控件顯示公告。
相關文章
- 1. .NetCore實踐篇：分佈式監控系統zipkin踩坑之路（二）
- 2. .NetCore實踐篇：成功解決分佈式監控ZipKin聚合依賴問題（三）
- 3. 分佈式監控系統zipkin介紹
- 4. 分佈式緩存redis之持久化
- 5. redis持久化和分佈式實現
- 6. redis持久化和監控
- 7. .NetCore 使用Zipkin 分佈式服務追蹤監控服務性能
- 8. NetCore實踐篇：分佈式監控客戶端ZipkinTracer從入門到放棄之路
- 9. Zabbix監控之分佈式監控zabbix_proxy
- 10. 打造立體化監控體系的最佳實踐——分佈式調用跟蹤和監控實踐
- 更多相關文章...
- • Redis的兩種備份（持久化）方式：RDB和AOF - Redis教程
- • Hibernate持久化對象的狀態轉換實例 - Hibernate教程
- • 再有人問你分佈式事務，把這篇扔給他
- • 常用的分佈式事務解決方案
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章