本文做爲第2部分，主要根據原始論文介紹幾篇基礎的工做，主要包括GNN，GCN及變體，DCNN，Tree-LSTM,包括模型的詳解和模型訓練，以及模型評價。html
這篇論文是第一個提出Graph Neural Network模型的論文，它將神經網絡使用在圖結構數據上，並細述了神經網絡模型告終構組成、計算方法、優化算法、流程實現等等。論文後面還對模型的複雜度進行了評估，以及在現實任務上進行了實驗和比較(比較算法爲NL、L、FNN)。該報告暫時主要關注模型設計部分和實驗結果部分，忽略複雜性評估部分 。node
對於圖領域問題，假設函數
τ
\tau
τ
G
G
G
n
n
n
τ
(
G
,
n
)
∈
R
m
\tau(G,n)\in{R^m}
τ ( G , n ) ∈ R m python
圖領域的應用主要能夠分爲兩種類型：專一於圖的應用(graph-focused)和專一於節點的應用(node-focused) 。對於graph-focused的應用，函數
τ
\tau
τ
τ
(
G
)
\tau(G)
τ ( G )
τ
\tau
τ
n
n
n
τ
(
G
,
n
)
\tau(G,n)
τ ( G , n ) web
圖 (a) 是一個化學分子結構，可以使用圖
G
G
G
τ
(
G
)
\tau(G)
τ ( G )
圖 (b) 是一張城堡的圖片，圖片中的每一種結構都由節點表示，函數
τ
(
G
,
n
)
\tau(G,n)
τ ( G , n )
GNN模型基於信息傳播機制，每個節點經過相互交換信息來更新本身的節點狀態，直到達到某一個穩定值 ，GNN的輸出就是在每一個節點處，根據當前節點狀態分別計算輸出。算法
有以下定義：json
一個圖
G
G
G
(
N
,
E
)
(\boldsymbol{N}, \boldsymbol{E})
( N , E )
N
\boldsymbol{N}
N
E
\boldsymbol{E}
E 網絡
n
e
[
n
]
ne[n]
n e [ n ]
n
n
n app
c
o
[
n
]
co[n]
c o [ n ]
n
n
n 框架
l
n
∈
R
l
N
\boldsymbol{l}_{n} \in \mathbb{R}^{l_{N}}
l n ∈ R l N
n
n
n ide
l
(
n
1
,
n
2
)
∈
R
l
E
\boldsymbol{l}_{\left(n_{1}, n_{2}\right)} \in \mathbb{R}^{l_{E}}
l ( n 1 , n 2 ) ∈ R l E
(
n
1
,
n
2
)
(n_1,n_2)
( n 1 , n 2 )
l
\boldsymbol{l}
l
注 ：原論文裏面
l
\boldsymbol{l}
l features of objects related to nodes and features of the relationships between the objects，也就是相關特徵，因此這裏一概使用特徵向量翻譯。
論文將圖分爲positional graph和nonpositional graph，對於positional graph，對於每個節點
n
n
n
u
u
u
ν
n
(
u
)
\nu_{n}(u)
ν n ( u )
ν
n
:
n
e
[
n
]
→
{
1
,
…
∣
N
∣
}
\nu_{n} : \mathrm{ne}[n] \rightarrow\{1, \ldots|\mathbf{N}|\}
ν n : n e [ n ] → { 1 , … ∣ N ∣ }
假設存在一個圖-節點對的集合
D
=
G
×
N
\mathcal{D}=\mathcal{G} \times \mathcal{N}
D = G × N
G
\mathcal{G}
G
N
\mathcal{N}
N 圖領域問題能夠表示成一個有以下數據集的監督學習框架
L
=
{
(
G
i
,
n
i
,
j
,
t
i
,
j
)
∣
G
i
=
(
N
i
,
E
i
)
∈
G
;
n
i
,
j
∈
N
i
;
t
i
,
j
∈
R
m
,
1
≤
i
≤
p
,
1
≤
j
≤
q
i
}
\mathcal{L}=\left\{\left(\boldsymbol{G}_{i}, n_{i, j}, \boldsymbol{t}_{i, j}\right)| \boldsymbol{G}_{i}=\left(\boldsymbol{N}_{i}, \boldsymbol{E}_{i}\right) \in \mathcal{G}\right.;n_{i, j} \in \boldsymbol{N}_{i} ; \boldsymbol{t}_{i, j} \in \mathbb{R}^{m}, 1 \leq i \leq p, 1 \leq j \leq q_{i} \}
L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i }
n
i
,
j
∈
N
i
n_{i, j} \in \boldsymbol{N}_{i}
n i , j ∈ N i
N
i
∈
N
\boldsymbol{N}_{i} \in \mathcal{N}
N i ∈ N
j
j
j
t
i
,
j
\boldsymbol{t}_{i, j}
t i , j
n
i
j
n_{ij}
n i j
節點
n
n
n
x
n
∈
R
s
\boldsymbol{x}_{n} \in \mathbb{R}^{s}
x n ∈ R s
o
n
\boldsymbol{o}_{\boldsymbol{n}}
o n
f
w
f_{\boldsymbol{w}}
f w local transition function ，
g
w
g_{\boldsymbol{w}}
g w local output function ，那麼
x
n
\boldsymbol{x}_{n}
x n
o
n
\boldsymbol{o}_{\boldsymbol{n}}
o n
x
n
=
f
w
(
l
n
,
l
c
o
[
n
]
,
x
n
e
[
n
]
,
l
n
e
[
n
]
)
o
n
=
g
w
(
x
n
,
l
n
)
\begin{array}{l}{\boldsymbol{x}_{n}=f_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{\mathrm{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}, \boldsymbol{l}_{\mathrm{ne}\left[n\right]}\right)} \\ {\boldsymbol{o}_{n}=g_{\boldsymbol{w}}\left(\boldsymbol{x}_{n}, \boldsymbol{l}_{n}\right)}\end{array}
x n = f w ( l n , l c o [ n ] , x n e [ n ] , l n e [ n ] ) o n = g w ( x n , l n )
l
n
,
l
co
[
n
]
,
x
n
e
[
n
]
,
l
n
e
[
n
]
\boldsymbol{l}_{n}, \boldsymbol{l}_{\operatorname{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}, \boldsymbol{l}_{\mathrm{ne}[n]}
l n , l c o [ n ] , x n e [ n ] , l n e [ n ]
n
n
n
n
n
n
n
n
n
n
n
n
假設
x
,
o
,
l
,
l
N
\boldsymbol{x}, \boldsymbol{o}, \boldsymbol{l}, \boldsymbol{l}_{N}
x , o , l , l N
x
=
F
w
(
x
,
l
)
o
=
G
w
(
x
,
l
N
)
\begin{array}{l}{\boldsymbol{x}=F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})} \\ {\boldsymbol{o}=\boldsymbol{G}_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)}\end{array}
x = F w ( x , l ) o = G w ( x , l N )
F
w
F_{\boldsymbol{w}}
F w global transition function ，
G
w
G_{\boldsymbol{w}}
G w global output function ，分別是
f
w
f_{\boldsymbol{w}}
f w
g
w
g_{\boldsymbol{w}}
g w
根據Banach的不動點理論，假設
F
w
F_{\boldsymbol{w}}
F w
x
(
t
+
1
)
=
F
w
(
x
(
t
)
,
l
)
\boldsymbol{x}(t+1)=F_{\boldsymbol{w}}(\boldsymbol{x}(t), \boldsymbol{l})
x ( t + 1 ) = F w ( x ( t ) , l )
x
(
t
)
\boldsymbol{x}(t)
x ( t )
x
\boldsymbol{x}
x
t
t
t
x
n
(
t
+
1
)
=
f
w
(
l
n
,
l
c
o
[
n
]
,
x
n
e
[
n
]
(
t
)
,
l
n
e
[
n
]
)
o
n
(
t
)
=
g
w
(
x
n
(
t
)
,
l
n
)
,
n
∈
N
\begin{aligned} \boldsymbol{x}_{n}(t+1) &=f_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{\mathrm{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}(t), \boldsymbol{l}_{\mathrm{ne}[n]}\right) \\ \boldsymbol{o}_{n}(t) &=g_{\boldsymbol{w}}\left(\boldsymbol{x}_{n}(t), \boldsymbol{l}_{n}\right), \quad n \in \boldsymbol{N} \end{aligned}
x n ( t + 1 ) o n ( t ) = f w ( l n , l c o [ n ] , x n e [ n ] ( t ) , l n e [ n ] ) = g w ( x n ( t ) , l n ) , n ∈ N
T
T
T
GNN的學習就是估計參數
w
\boldsymbol{w}
w
φ
w
\varphi_{\boldsymbol{w}}
φ w
L
=
{
(
G
i
,
n
i
,
j
,
t
i
,
j
)
∣
G
i
=
(
N
i
,
E
i
)
∈
G
;
n
i
,
j
∈
N
i
;
t
i
,
j
∈
R
m
,
1
≤
i
≤
p
,
1
≤
j
≤
q
i
}
\mathcal{L}=\left\{\left(\boldsymbol{G}_{i}, n_{i, j}, \boldsymbol{t}_{i, j}\right)| \boldsymbol{G}_{i}=\left(\boldsymbol{N}_{i}, \boldsymbol{E}_{i}\right) \in \mathcal{G}\right.;n_{i, j} \in \boldsymbol{N}_{i} ; \boldsymbol{t}_{i, j} \in \mathbb{R}^{m}, 1 \leq i \leq p, 1 \leq j \leq q_{i} \}
L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i }
q
i
q_i
q i
G
i
G_{i}
G i
e
w
=
∑
i
=
1
p
∑
j
=
1
q
i
(
t
i
,
j
−
φ
w
(
G
i
,
n
i
,
j
)
)
2
e_{\boldsymbol{w}}=\sum_{i=1}^{p} \sum_{j=1}^{q_{i}}\left(\boldsymbol{t}_{i, j}-\varphi_{\boldsymbol{w}}\left(\boldsymbol{G}_{i}, n_{i, j}\right)\right)^{2}
e w = i = 1 ∑ p j = 1 ∑ q i ( t i , j − φ w ( G i , n i , j ) ) 2
按照迭代方程迭代
T
T
T
x
n
(
t
)
x_{n}(t)
x n ( t )
x
(
T
)
≈
x
\boldsymbol{x}(T) \approx \boldsymbol{x}
x ( T ) ≈ x
計算參數權重的梯度
∂
e
w
(
T
)
/
∂
w
\partial e_{\boldsymbol{w}}(T) / \partial \boldsymbol{w}
∂ e w ( T ) / ∂ w
使用該梯度來更新權重
w
\boldsymbol{w}
w
這裏假設函數
F
w
F_{\boldsymbol{w}}
F w backpropagation-through-time algorithm 。
爲了代表前面的方法是可行的，論文接着證實了兩個結論
理論1(可微性)：令
F
w
F_{\boldsymbol{w}}
F w
G
w
G_{\boldsymbol{w}}
G w
F
w
(
x
,
l
)
F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
F w ( x , l )
G
w
(
x
,
l
N
)
G_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)
G w ( x , l N )
x
\boldsymbol{x}
x
w
\boldsymbol{w}
w
φ
w
\varphi_{\boldsymbol{w}}
φ w
w
\boldsymbol{w}
w
理論2(反向傳播)：令
F
w
F_{\boldsymbol{w}}
F w
G
w
G_{\boldsymbol{w}}
G w
F
w
(
x
,
l
)
F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
F w ( x , l )
G
w
(
x
,
l
N
)
G_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)
G w ( x , l N )
x
\boldsymbol{x}
x
w
\boldsymbol{w}
w
z
(
t
)
\boldsymbol{z}(t)
z ( t )
z
(
t
)
=
z
(
t
+
1
)
⋅
∂
F
w
∂
x
(
x
,
l
)
+
∂
e
w
∂
o
⋅
∂
G
w
∂
x
(
x
,
l
N
)
z(t)=z(t+1) \cdot \frac{\partial F_{w}}{\partial x}(x, l)+\frac{\partial e_{w}}{\partial o} \cdot \frac{\partial G_{w}}{\partial x}\left(x, l_{N}\right)
z ( t ) = z ( t + 1 ) ⋅ ∂ x ∂ F w ( x , l ) + ∂ o ∂ e w ⋅ ∂ x ∂ G w ( x , l N )
z
(
T
)
,
z
(
T
−
1
)
,
…
\boldsymbol{z}(T), \boldsymbol{z}(T-1), \ldots
z ( T ) , z ( T − 1 ) , …
z
=
lim
t
→
−
∞
z
(
t
)
z=\lim _{t \rightarrow-\infty} z(t)
z = lim t → − ∞ z ( t )
z
(
T
)
\boldsymbol{z}(T)
z ( T )
∂
e
w
∂
w
=
∂
e
w
∂
o
⋅
∂
G
w
∂
w
(
x
,
l
N
)
+
z
⋅
∂
F
w
∂
w
(
x
,
l
)
\frac{\partial e_{w}}{\partial \boldsymbol{w}}=\frac{\partial e_{\boldsymbol{w}}}{\partial \boldsymbol{o}} \cdot \frac{\partial G_{\boldsymbol{w}}}{\partial \boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{N}\right)+\boldsymbol{z} \cdot \frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
∂ w ∂ e w = ∂ o ∂ e w ⋅ ∂ w ∂ G w ( x , l N ) + z ⋅ ∂ w ∂ F w ( x , l )
x
\boldsymbol{x}
x
算法流程以下FORWARD 用於迭代計算出收斂點，BACKWARD 用於計算梯度。
在GNN中，函數
g
w
g_{\boldsymbol{w}}
g w
f
w
f_{\boldsymbol{w}}
f w
f
w
f_{\boldsymbol{w}}
f w
Linear(nonpositional) GNN ：
對於節點
n
n
n
f
w
f_{\boldsymbol{w}}
f w
x
n
=
∑
u
∈
ne
∣
n
]
h
w
(
l
n
,
l
(
n
,
u
)
,
x
u
,
l
u
)
,
n
∈
N
\boldsymbol{x}_{n}=\sum_{u \in \text { ne } | n ]} h_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, u)}, \boldsymbol{x}_{u}, \boldsymbol{l}_{u}\right), \quad n \in \boldsymbol{N}
x n = u ∈ ne ∣ n ] ∑ h w ( l n , l ( n , u ) , x u , l u ) , n ∈ N
n
n
n
h
w
h_{\boldsymbol{w}}
h w
n
n
n
由此，對上式中的函數
h
w
h_{\boldsymbol{w}}
h w
h
w
(
l
n
,
l
(
n
,
a
)
,
x
u
,
l
u
)
=
A
n
,
u
x
u
+
b
n
h_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, \mathfrak{a})}, \boldsymbol{x}_{u}, \boldsymbol{l}_{u}\right) = \boldsymbol{A}_{n, u} \boldsymbol{x}_{u}+\boldsymbol{b}_{n}
h w ( l n , l ( n , a ) , x u , l u ) = A n , u x u + b n
b
n
∈
R
s
\boldsymbol{b}_{n} \in \mathbb{R}^{s}
b n ∈ R s
A
n
,
u
∈
R
s
×
s
\boldsymbol{A}_{n, u} \in \mathbb{R}^{s \times s}
A n , u ∈ R s × s
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u transition network ，產生向量
b
n
\boldsymbol{b}_{n}
b n forcing network
transition network表示爲
ϕ
w
\phi_{\boldsymbol{w}}
ϕ w
ϕ
w
:
R
2
l
N
+
l
E
→
R
s
2
\phi_{\boldsymbol{w}} : \mathbb{R}^{2 l_{N}+l_{E}} \rightarrow \mathbb{R}^{s^{2}}
ϕ w : R 2 l N + l E → R s 2
ρ
w
\rho_{\boldsymbol{w}}
ρ w
ρ
w
:
R
l
N
→
R
s
\rho_{\boldsymbol{w}} : \mathbb{R}^{l_{N}} \rightarrow \mathbb{R}^{s}
ρ w : R l N → R s
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u
b
n
\boldsymbol{b}_{n}
b n
A
n
,
u
=
μ
s
∣
ne
[
u
]
∣
⋅
Ξ
b
w
=
ρ
w
(
l
n
)
\begin{aligned} \boldsymbol{A}_{\boldsymbol{n}, \boldsymbol{u}} &=\frac{\mu}{s|\operatorname{ne}[u]|} \cdot \boldsymbol{\Xi} \\ \boldsymbol{b}_{\boldsymbol{w}} &=\rho_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}\right) \end{aligned}
A n , u b w = s ∣ n e [ u ] ∣ μ ⋅ Ξ = ρ w ( l n )
μ
∈
(
0
,
1
)
\mu \in(0,1)
μ ∈ ( 0 , 1 )
Ξ
=
resize
(
ϕ
w
(
l
n
,
l
(
n
,
u
)
,
l
u
)
)
\Xi=\operatorname{resize}\left(\phi_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, u)}, \boldsymbol{l}_{u}\right)\right)
Ξ = r e s i z e ( ϕ w ( l n , l ( n , u ) , l u ) )
resize
(
⋅
)
\text{resize}(\cdot)
resize ( ⋅ )
s
2
s^2
s 2
s
×
s
s\times{s}
s × s
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u
b
n
\boldsymbol{b}_{n}
b n
在這裏，假定
∥
ϕ
w
(
l
n
,
l
(
n
,
u
)
,
l
u
)
∥
1
≤
s
\left\|\phi_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(\boldsymbol{n}, \boldsymbol{u})}, \boldsymbol{l}_{u}\right)\right\|_{1} \leq \boldsymbol{s}
∥ ∥ ϕ w ( l n , l ( n , u ) , l u ) ∥ ∥ 1 ≤ s
t
a
n
h
(
)
tanh()
t a n h ( )
F
w
(
x
,
l
)
=
A
x
+
b
F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})=\boldsymbol{A} \boldsymbol{x}+\boldsymbol{b}
F w ( x , l ) = A x + b
A
\boldsymbol{A}
A
b
\boldsymbol{b}
b
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u
b
n
\boldsymbol{b}_{n}
b n
∥
∂
F
w
∂
x
∥
1
=
∥
A
∥
1
≤
max
u
∈
N
(
∑
n
∈
ne
[
u
]
∥
A
n
,
u
∥
1
)
≤
max
u
∈
N
(
μ
s
∣
ne
[
u
]
∣
⋅
∑
n
∈
n
e
[
u
]
∥
Ξ
∥
1
)
≤
μ
\begin{aligned}\left\|\frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{x}}\right\|_{1} &=\|\boldsymbol{A}\|_{1} \leq \max _{u \in \boldsymbol{N}}\left(\sum_{n \in \operatorname{ne}[u]}\left\|\boldsymbol{A}_{n, u}\right\|_{1}\right) \\ & \leq \max _{u \in N}\left(\frac{\mu}{s|\operatorname{ne}[u]|} \cdot \sum_{n \in \mathrm{ne}[u]}\|\mathbf{\Xi}\|_{1}\right) \leq \mu \end{aligned}
∥ ∥ ∥ ∥ ∂ x ∂ F w ∥ ∥ ∥ ∥ 1 = ∥ A ∥ 1 ≤ u ∈ N max ⎝ ⎛ n ∈ n e [ u ] ∑ ∥ A n , u ∥ 1 ⎠ ⎞ ≤ u ∈ N max ⎝ ⎛ s ∣ n e [ u ] ∣ μ ⋅ n ∈ n e [ u ] ∑ ∥ Ξ ∥ 1 ⎠ ⎞ ≤ μ
F
w
F_{\boldsymbol{w}}
F w
w
\boldsymbol{w}
w
矩陣
M
M
M
∥
M
∥
1
=
max
j
∑
i
∣
m
i
,
j
∣
\|M\|_{1}=\max _{j} \sum_{i}\left|m_{i, j}\right|
∥ M ∥ 1 = j max i ∑ ∣ m i , j ∣
Nonelinear(nonpositional) GNN ：在這個結構中，
h
w
h_{\boldsymbol{w}}
h w
w
\boldsymbol{w}
w
F
w
F_{\boldsymbol{w}}
F w
e
w
=
∑
i
=
1
p
∑
j
=
1
q
i
(
t
i
,
j
−
φ
w
(
G
i
,
n
i
,
j
)
)
2
+
β
L
(
∥
∂
F
w
∂
x
∥
)
e_{\boldsymbol{w}}=\sum_{i=1}^{p} \sum_{j=1}^{q_{i}}\left(\boldsymbol{t}_{i, j}-\varphi_{\boldsymbol{w}}\left(\boldsymbol{G}_{i}, n_{i, j}\right)\right)^{2}+\beta L\left(\left\|\frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{x}}\right\|\right)
e w = i = 1 ∑ p j = 1 ∑ q i ( t i , j − φ w ( G i , n i , j ) ) 2 + β L ( ∥ ∥ ∥ ∥ ∂ x ∂ F w ∥ ∥ ∥ ∥ )
L
(
y
)
L(y)
L ( y )
y
>
μ
y>\mu
y > μ
(
y
−
μ
)
2
(y-\mu)^2
( y − μ ) 2
y
≤
μ
y\le{\mu}
y ≤ μ
μ
∈
(
0
,
1
)
\mu\in(0,1)
μ ∈ ( 0 , 1 )
F
w
F_{\boldsymbol{w}}
F w
論文將GNN模型在三個任務上進行了實驗：子圖匹配(subgraph matching)任務，誘變(mutagenesis)任務和網頁排序(web page ranking)任務。在這些任務上使用linear和nonlinear的模型測試，其中nonlinear模型中的激活函數使用sigmoid函數。
子圖匹配任務爲在一個大圖
G
G
G
S
S
S
τ
\tau
τ
n
i
,
j
n_{i,j}
n i , j
G
G
G
τ
(
G
i
,
n
i
,
j
)
=
1
\tau(G_i,n_{i,j})=1
τ ( G i , n i , j ) = 1
τ
(
G
i
,
n
i
,
j
)
=
−
1
\tau(G_i,n_{i,j})=-1
τ ( G i , n i , j ) = − 1
誘變問題任務是對化學分子進行分類，識別出誘變化合物，採用二分類方法。實驗結果是nonlinear效果較好，但不是最好。
網頁排序任務是學會網頁排序。實驗代表雖然訓練集只包含50個網頁，可是仍然沒有產生過擬合的現象。
在模擬的節點分類任務上實現該論文的GNN模型。
任務要求 爲輸入一個graph，該graph的全部節點都有標籤，而後對部分節點進行訓練，在驗證階段使用另一部分節點進行驗證。輸入的數據圖以下圖：
{
n
1
,
n
2
,
.
.
.
,
n
18
}
\{n1,n2,...,n18\}
{ n 1 , n 2 , . . . , n 1 8 }
{
0
,
1
,
2
}
\{0,1,2\}
{ 0 , 1 , 2 }
{
n
1
,
n
2
,
n
3
,
n
7
,
n
8
,
n
9
,
n
13
,
n
14
,
n
15
}
\{n1,n2,n3,n7,n8,n9,n13,n14,n15\}
{ n 1 , n 2 , n 3 , n 7 , n 8 , n 9 , n 1 3 , n 1 4 , n 1 5 }
輸入的數據爲(node,label)列表和(node1, node2)邊列表，表示以下
N = [ ( "n{}" . format ( i) , 0 ) for i in range ( 1 , 7 ) ] + \
[ ( "n{}" . format ( i) , 1 ) for i in range ( 7 , 13 ) ] + \
[ ( "n{}" . format ( i) , 2 ) for i in range ( 13 , 19 ) ]
E = [ ( "n1" , "n2" ) , ( "n1" , "n3" ) , ( "n1" , "n5" ) ,
( "n2" , "n4" ) ,
( "n3" , "n6" ) , ( "n3" , "n9" ) ,
( "n4" , "n5" ) , ( "n4" , "n6" ) , ( "n4" , "n8" ) ,
( "n5" , "n14" ) ,
( "n7" , "n8" ) , ( "n7" , "n9" ) , ( "n7" , "n11" ) ,
( "n8" , "n10" ) , ( "n8" , "n11" ) , ( "n8" , "n12" ) ,
( "n9" , "n10" ) , ( "n9" , "n14" ) ,
( "n10" , "n12" ) ,
( "n11" , "n18" ) ,
( "n13" , "n15" ) , ( "n13" , "n16" ) , ( "n13" , "n18" ) ,
( "n14" , "n16" ) , ( "n14" , "n18" ) ,
( "n15" , "n16" ) , ( "n15" , "n18" ) ,
( "n17" , "n18" ) ]
N
N
N
E
E
E
模型部分使用論文的linear函數來設計
f
w
f_w
f w
g
w
g_w
g w
Ξ
\Xi
Ξ
ρ
\rho
ρ
class Xi ( nn. Module) :
def __init__ ( self, ln, s) :
. . .
def forward ( self, X) :
. . .
class Rou ( nn. Module) :
def __init__ ( self, ln, s) :
. . .
def forward ( self, X) :
. . .
class Hw ( nn. Module) :
def __init__ ( self, ln, s, mu= 0.9 ) :
. . .
def forward ( self, X, H, dg_list) :
. . .
class AggrSum ( nn. Module) :
def __init__ ( self, node_num) :
. . .
def forward ( self, H, X_node) :
. . .
class OriLinearGNN ( nn. Module) :
def __init__ ( self, node_num, feat_dim, stat_dim, T) :
. . .
def forward ( self, X_Node, X_Neis, dg_list) :
. . .
for t in range ( self. T) :
H = torch. index_select( self. node_states, 0 , X_Node)
H = self. Hw( X, H, dg_list)
self. node_states = self. Aggr( H, X_Node)
. . . 能夠看出，在模型訓練階段，每次forward，都會直接循環計算T次
f
w
f_w
f w
模型訓練部分按照常規的分類模型進行訓練，採用Adam優化器，學習率保持爲0.01，權重衰減爲0.01，使用交叉熵做爲損失函數，模型訓練部分代碼以下
def CalAccuracy ( output, label) :
. . .
def train ( node_list, edge_list, label_list, T, ndict_path= "./node_dict.json" ) :
. . .
. . .
. . .
. . .
train_node_list = [ 0 , 1 , 2 , 6 , 7 , 8 , 12 , 13 , 14 ]
train_node_label = [ 0 , 0 , 0 , 1 , 1 , 1 , 2 , 2 , 2 ]
test_node_list = [ 3 , 4 , 5 , 9 , 10 , 11 , 15 , 16 , 17 ]
test_node_label = [ 0 , 0 , 0 , 1 , 1 , 1 , 2 , 2 , 2 ]
model = OriLinearGNN( node_num= len ( node_list) ,
feat_dim= 2 ,
stat_dim= 2 ,
T= T)
optimizer = torch. optim. Adam( model. parameters( ) , lr= 0.01 , weight_decay= 0.01 )
criterion = nn. CrossEntropyLoss( size_average= True )
min_loss = float ( 'inf' )
node_inds_tensor = Variable( torch. Tensor( node_inds) . long ( ) )
node_neis_tensor = Variable( torch. Tensor( node_neis) . long ( ) )
train_label = Variable( torch. Tensor( train_node_label) . long ( ) )
for ep in range ( 500 ) :
res = model( node_inds_tensor, node_neis_tensor, dg_list)
train_res = torch. index_select( res, 0 , torch. Tensor( train_node_list) . long ( ) )
test_res = torch. index_select( res, 0 , torch. Tensor( test_node_list) . long ( ) )
loss = criterion( input = train_res,
target= train_label)
loss_val = loss. item( )
train_acc = CalAccuracy( train_res. cpu( ) . detach( ) . numpy( ) , np. array( train_node_label) )
test_acc = CalAccuracy( test_res. cpu( ) . detach( ) . numpy( ) , np. array( test_node_label) )
optimizer. zero_grad( )
loss. backward( retain_graph= True )
optimizer. step( )
if loss_val < min_loss:
min_loss = loss_val
print ( "==> [Epoch {}] : loss {:.4f}, min_loss {:.4f}, train_acc {:.3f}, test_acc {:.3f}" . format ( ep, loss_val, min_loss, train_acc, test_acc) )
模型的訓練和評估結果以下圖:
2
3
\frac{2}{3}
3 2
按照圖傅里葉變換的性質，能夠獲得以下圖卷積的定義
(
f
∗
h
)
G
=
Φ
diag
[
h
^
(
λ
1
)
,
…
,
h
^
(
λ
n
)
]
Φ
T
f
(\boldsymbol{f} * \boldsymbol{h})_{\mathcal{G}}=\boldsymbol{\Phi} \operatorname{diag}\left[\hat{h}\left(\lambda_{1}\right), \ldots, \hat{h}\left(\lambda_{n}\right)\right] \mathbf{\Phi}^{T} \boldsymbol{f}
( f ∗ h ) G = Φ d i a g [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ] Φ T f
對於圖$ \boldsymbol{f}
的
傅
裏
葉
變
換
爲
的傅里葉變換爲
的 傅 裏 葉 變 換 爲
對於卷積核的圖傅里葉變換：
h
^
=
(
h
^
1
,
…
,
h
^
n
)
\hat{h}=\left(\hat{h}_{1}, \ldots, \hat{h}_{n}\right)
h ^ = ( h ^ 1 , … , h ^ n )
h
^
k
=
⟨
h
,
ϕ
k
⟩
,
k
=
1
,
2
…
,
n
\hat{h}_{k}=\left\langle h, \phi_{k}\right\rangle, k=1,2 \ldots, n
h ^ k = ⟨ h , ϕ k ⟩ , k = 1 , 2 … , n
h
^
=
Φ
T
h
\hat{\boldsymbol{h}}=\mathbf{\Phi}^{T} \boldsymbol{h}
h ^ = Φ T h
對二者的傅里葉變換向量
f
^
∈
R
N
×
1
\hat{f} \in \mathbb{R}^{N \times 1}
f ^ ∈ R N × 1
h
^
∈
R
N
×
1
\hat{h} \in \mathbb{R}^{N \times 1}
h ^ ∈ R N × 1 等價於將
h
\boldsymbol{h}
h ，即
diag
[
h
^
(
λ
k
)
]
∈
R
N
×
N
\operatorname{diag}\left[\hat{h}\left(\lambda_{k}\right)\right] \in \mathbb{R}^{N \times N}
d i a g [ h ^ ( λ k ) ] ∈ R N × N
diag
[
h
^
(
λ
k
)
]
\operatorname{diag}\left[\hat{h}\left(\lambda_{k}\right)\right]
d i a g [ h ^ ( λ k ) ]
f
\boldsymbol{f}
f
求上述結果的傅里葉逆變換，即左乘
Φ
\mathbf{\Phi}
Φ
深度學習中的卷積就是要設計trainable的卷積核 ，從上式能夠看出，就是要設計
diag
[
h
^
(
λ
1
)
,
…
,
h
^
(
λ
n
)
]
\operatorname{diag}\left[\hat{h}\left(\lambda_{1}\right), \ldots, \hat{h}\left(\lambda_{n}\right)\right]
d i a g [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ]
diag
[
θ
1
,
…
,
θ
n
]
\operatorname{diag}\left[\theta_{1}, \ldots, \theta_{n}\right]
d i a g [ θ 1 , … , θ n ] 不須要再將卷積核進行傅里葉變換 ，由此，至關於直接將變換後的參量進行學習。
第一代GCN爲
y
output
=
σ
(
Φ
g
θ
Φ
T
x
)
=
σ
(
Φ
diag
[
θ
1
,
…
,
θ
n
]
Φ
T
x
)
\boldsymbol{y}_{\text {output}}=\sigma\left(\mathbf{\Phi} \boldsymbol{g}_{\theta} \mathbf{\Phi}^{T} \boldsymbol{x}\right)=\sigma\left(\boldsymbol{\Phi} \operatorname{diag}\left[\theta_{1}, \ldots, \theta_{n}\right] \mathbf{\Phi}^{T} \boldsymbol{x}\right)
y output = σ ( Φ g θ Φ T x ) = σ ( Φ d i a g [ θ 1 , … , θ n ] Φ T x )
x
\boldsymbol{x}
x
x
=
(
x
1
,
x
2
,
…
,
x
n
)
x=\left(x_{1}, x_{2}, \ldots, x_{n}\right)
x = ( x 1 , x 2 , … , x n )
y
output
\boldsymbol{y}_{\text {output}}
y output
一些缺點：
須要對拉普拉斯矩陣進行譜分解來求
Φ
\mathbf{\Phi}
Φ
O
(
n
2
)
O(n^2)
O ( n 2 )
卷積核參數爲
n
n
n
n
n
n
卷積核的spatial localization很差。
圖傅里葉變換是關於特徵值(至關於普通傅里葉變換的頻率)的函數，也就是
F
(
λ
1
)
,
…
,
F
(
λ
n
)
F\left(\lambda_{1}\right), \ldots, F\left(\lambda_{n}\right)
F ( λ 1 ) , … , F ( λ n )
F
(
Λ
)
F(\mathbf{\Lambda})
F ( Λ )
g
θ
\boldsymbol{g}_{\theta}
g θ
g
θ
(
Λ
)
\boldsymbol{g}_{\theta}(\Lambda)
g θ ( Λ )
g
θ
(
Λ
)
\boldsymbol{g}_{\theta}(\Lambda)
g θ ( Λ ) k階多項式
g
θ
′
(
Λ
)
≈
∑
k
=
0
K
θ
k
′
Λ
k
g_{\theta^{\prime}}(\mathbf{\Lambda}) \approx \sum_{k=0}^{K} \theta_{k}^{\prime} \mathbf{\Lambda}^{k}
g θ ′ ( Λ ) ≈ k = 0 ∑ K θ k ′ Λ k
g
θ
′
∗
x
≈
Φ
∑
k
=
0
K
θ
k
′
Λ
k
Φ
T
x
=
∑
k
=
0
K
θ
k
′
(
Φ
Λ
k
Φ
T
)
x
=
∑
k
=
0
K
θ
k
′
(
Φ
Λ
Φ
T
)
k
x
=
∑
k
=
0
K
θ
k
′
L
k
x
\begin{aligned} g_{\theta^{\prime}} * x & \approx \Phi \sum_{k=0}^{K} \theta_{k}^{\prime} \mathbf{\Lambda}^{k} \mathbf{\Phi}^{T} \boldsymbol{x} \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} \mathbf{\Lambda}^{k} \mathbf{\Phi}^{T}\right) x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} \mathbf{\Lambda} \mathbf{\Phi}^{T}\right)^{k} x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime} \boldsymbol{L}^{k} x \end{aligned}
g θ ′ ∗ x ≈ Φ k = 0 ∑ K θ k ′ Λ k Φ T x = k = 0 ∑ K θ k ′ ( Φ Λ k Φ T ) x = k = 0 ∑ K θ k ′ ( Φ Λ Φ T ) k x = k = 0 ∑ K θ k ′ L k x
L
k
\boldsymbol{L}^{k}
L k
O
(
K
n
2
)
O(Kn^2)
O ( K n 2 )
對於每一次Laplacian矩陣
L
\boldsymbol{L}
L
x
\mathbf{x}
x
n
n
n
n
e
[
n
]
ne[n]
n e [ n ]
n
n
n
k
k
k
n
n
n 所以，實際上只利用了節點的K-Localized信息 。
另外，可使用切比雪夫展開式來近似
L
k
\boldsymbol{L}^{k}
L k 任何k次多項式均可以使用切比雪夫展開式來近似 ，由此，引入切比雪夫多項式的
K
K
K
L
k
\boldsymbol{L}^{k}
L k
g
θ
(
Λ
)
g_{\theta}(\mathbf{\Lambda})
g θ ( Λ )
g
θ
′
(
Λ
)
≈
∑
k
=
0
K
θ
k
′
T
k
(
Λ
~
)
g_{\theta^{\prime}}(\mathbf{\Lambda}) \approx \sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\mathbf{\Lambda}})
g θ ′ ( Λ ) ≈ k = 0 ∑ K θ k ′ T k ( Λ ~ )
Λ
~
=
2
λ
max
Λ
−
I
n
\tilde{\mathbf{\Lambda}}=\frac{2}{\lambda_{\max }} \mathbf{\Lambda}-\boldsymbol{I}_{n}
Λ ~ = λ max 2 Λ − I n
θ
′
∈
R
K
\boldsymbol{\theta}^{\prime} \in \mathbb{R}^{K}
θ ′ ∈ R K
θ
k
′
\theta_{k}^{\prime}
θ k ′
k
k
k
T
k
(
x
)
T_{k}(x)
T k ( x )
T
k
(
x
)
=
2
x
T
k
−
1
(
x
)
−
T
k
−
2
(
x
)
T_{k}(x)=2 x T_{k-1}(x)-T_{k-2}(x)
T k ( x ) = 2 x T k − 1 ( x ) − T k − 2 ( x )
T
0
(
x
)
=
1
,
T
1
(
x
)
=
x
T_{0}(x)=1, T_{1}(x)=x
T 0 ( x ) = 1 , T 1 ( x ) = x
此時，帶入到卷積公式
g
θ
′
∗
x
≈
Φ
∑
k
=
0
K
θ
k
′
T
k
(
Λ
~
)
Φ
T
x
≈
∑
k
=
0
K
θ
k
′
(
Φ
T
k
(
Λ
~
)
Φ
T
)
x
=
∑
k
=
0
K
θ
k
′
T
k
(
L
~
)
x
\begin{aligned} \boldsymbol{g}_{\boldsymbol{\theta}^{\prime}} * \boldsymbol{x} & \approx \mathbf{\Phi} \sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{\Lambda}}) \mathbf{\Phi}^{T} \boldsymbol{x} \\ &\approx \sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} T_{k}(\tilde{\mathbf{\Lambda}}) \mathbf{\Phi}^{T}\right) x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{L}}) \boldsymbol{x} \end{aligned}
g θ ′ ∗ x ≈ Φ k = 0 ∑ K θ k ′ T k ( Λ ~ ) Φ T x ≈ k = 0 ∑ K θ k ′ ( Φ T k ( Λ ~ ) Φ T ) x = k = 0 ∑ K θ k ′ T k ( L ~ ) x
L
~
=
2
λ
max
L
−
I
n
\tilde{\boldsymbol{L}}=\frac{2}{\lambda_{\max }} \boldsymbol{L}-\boldsymbol{I}_{n}
L ~ = λ max 2 L − I n
所以，能夠獲得輸出爲
y
output
=
σ
(
∑
k
=
0
K
θ
k
′
T
k
(
L
~
)
x
)
\boldsymbol{y}_{\text {output}}=\sigma\left(\sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{L}}) \boldsymbol{x}\right)
y output = σ ( k = 0 ∑ K θ k ′ T k ( L ~ ) x )
這一代GCN直接取切比雪夫多項式中
K
=
1
K=1
K = 1
將
K
=
1
K=1
K = 1
λ
max
=
2
\lambda_{\max }=2
λ max = 2
g
θ
′
∗
x
≈
θ
0
′
x
+
θ
1
′
(
L
−
I
n
)
x
=
θ
0
′
x
+
θ
1
′
(
L
−
I
n
)
x
=
θ
0
′
x
−
θ
1
′
(
D
−
1
/
2
W
D
−
1
/
2
)
x
\begin{aligned} \boldsymbol{g}_{\boldsymbol{\theta}^{\prime}} * \boldsymbol{x} & \approx \boldsymbol{\theta}_{0}^{\prime} \boldsymbol{x}+\theta_{1}^{\prime}\left(\boldsymbol{L}-\boldsymbol{I}_{n}\right) \boldsymbol{x} \\ &=\boldsymbol{\theta}_{0}^{\prime} \boldsymbol{x}+\theta_{1}^{\prime}\left(\boldsymbol{L}-\boldsymbol{I}_{n}\right) \boldsymbol{x} \\ &=\theta_{0}^{\prime} \boldsymbol{x}-\theta_{1}^{\prime}\left(\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}\right) \boldsymbol{x} \end{aligned}
g θ ′ ∗ x ≈ θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x − θ 1 ′ ( D − 1 / 2 W D − 1 / 2 ) x
L
=
D
−
1
/
2
(
D
−
W
)
D
−
1
/
2
=
I
n
−
D
−
1
/
2
W
D
−
1
/
2
\boldsymbol{L}=\boldsymbol{D}^{-1 / 2}(\boldsymbol{D}-\boldsymbol{W}) \boldsymbol{D}^{-1 / 2}=\boldsymbol{I}_{n}-\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}
L = D − 1 / 2 ( D − W ) D − 1 / 2 = I n − D − 1 / 2 W D − 1 / 2 爲了進一步簡化，令
θ
0
′
=
−
θ
1
′
\theta_{0}^{\prime}=-\theta_{1}^{\prime}
θ 0 ′ = − θ 1 ′
θ
\theta
θ
g
θ
′
∗
x
=
θ
(
I
n
+
D
−
1
/
2
W
D
−
1
/
2
)
x
g_{\theta^{\prime}} * x=\theta\left(I_{n}+D^{-1 / 2} W D^{-1 / 2}\right) x
g θ ′ ∗ x = θ ( I n + D − 1 / 2 W D − 1 / 2 ) x
I
n
+
D
−
1
/
2
W
D
−
1
/
2
\boldsymbol{I}_{n}+\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}
I n + D − 1 / 2 W D − 1 / 2 譜半徑
[
0
,
2
]
[0,2]
[ 0 , 2 ]
I
n
+
D
−
1
/
2
W
D
−
1
/
2
→
D
~
−
1
/
2
W
~
D
~
−
1
/
2
\boldsymbol{I}_{n}+\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2} \rightarrow \tilde{\boldsymbol{D}}^{-1 / 2} \tilde{\boldsymbol{W}} \tilde{\boldsymbol{D}}^{-1 / 2}
I n + D − 1 / 2 W D − 1 / 2 → D ~ − 1 / 2 W ~ D ~ − 1 / 2
W
~
=
W
+
I
n
\tilde{\boldsymbol{W}}=\boldsymbol{W}+\boldsymbol{I}_{n}
W ~ = W + I n
D
~
i
j
=
Σ
j
W
~
i
j
\tilde{D}_{i j}=\Sigma_{j} \tilde{W}_{i j}
D ~ i j = Σ j W ~ i j
由此，帶入卷積公式
g
θ
′
∗
x
⏟
R
n
×
1
=
θ
(
D
~
−
1
/
2
W
~
D
~
−
1
/
2
⏟
R
n
×
n
)
x
⏟
R
n
×
1
\underbrace{g_{\theta^{\prime}} * x}_{\mathbb{R}^{n \times 1}}=\theta\left(\underbrace{\tilde{D}^{-1 / 2} \tilde{W} \tilde{D}^{-1 / 2}}_{\mathbb{R}^{n \times n}}\right) \underbrace{x}_{\mathbb{R}^{n \times 1}}
R n × 1
g θ ′ ∗ x = θ ( R n × n
D ~ − 1 / 2 W ~ D ~ − 1 / 2 ) R n × 1
x 推廣到多通道 ，至關於每個節點的信息是向量
x
∈
R
N
×
1
→
X
∈
R
N
×
C
x \in \mathbb{R}^{N \times 1} \rightarrow X \in \mathbb{R}^{N \times C}
x ∈ R N × 1 → X ∈ R N × C
N
N
N 數量 ，
C
C
C 維度數 。
X
\mathbf{X}
X 特徵矩陣 。
相應的卷積核參數變化
θ
∈
R
→
Θ
∈
R
C
×
F
\theta \in \mathbb{R} \rightarrow \Theta \in \mathbb{R}^{C \times F}
θ ∈ R → Θ ∈ R C × F
F
F
F
那麼卷積結果寫成矩陣形式爲
Z
⏟
R
N
×
F
=
D
~
−
1
/
2
W
~
D
~
−
1
/
2
⏟
R
N
×
N
X
⏟
R
N
×
C
Θ
⏟
R
C
×
F
\underbrace{Z}_{\mathbb{R}^{N \times F}}=\underbrace{\tilde{D}^{-1 / 2} \tilde{W} \tilde{D}^{-1 / 2}}_{\mathbb{R}^{N \times N}} \underbrace{X}_{\mathbb{R}^{N \times C}} \underbrace{\mathbf{\Theta}}_{\mathbb{R}^{C \times F}}
R N × F
Z = R N × N
D ~ − 1 / 2 W ~ D ~ − 1 / 2 R N × C
X R C × F
Θ 能夠疊加多層，對上述輸出激活一下，就能夠做爲下一層節點的特徵矩陣 。
這一代GCN特色：
取
K
=
1
K=1
K = 1
3
×
3
3\times{3}
3 × 3
因爲卷積核寬度減少，能夠經過增長卷積層數來擴大感覺野，從而加強網絡的表達能力。
增長了參數約束，好比
λ
max
≈
2
\lambda_{\max } \approx 2
λ max ≈ 2
論文采用兩層的GCN，用來在graph上進行半監督的節點分類任務，鄰接矩陣爲
A
A
A
A
^
=
D
~
−
1
2
A
~
D
~
−
1
2
\hat{A}=\tilde{D}^{-\frac{1}{2}} \tilde{A} \tilde{D}^{-\frac{1}{2}}
A ^ = D ~ − 2 1 A ~ D ~ − 2 1
Z
=
f
(
X
,
A
)
=
softmax
(
A
^
ReLU
(
A
^
X
W
(
0
)
)
W
(
1
)
)
Z=f(X, A)=\operatorname{softmax}\left(\hat{A} \operatorname{ReLU}\left(\hat{A} X W^{(0)}\right) W^{(1)}\right)
Z = f ( X , A ) = s o f t m a x ( A ^ R e L U ( A ^ X W ( 0 ) ) W ( 1 ) )
W
(
0
)
∈
R
C
×
H
W^{(0)} \in \mathbb{R}^{C \times H}
W ( 0 ) ∈ R C × H
H
H
H
W
(
1
)
∈
R
H
×
F
W^{(1)} \in \mathbb{R}^{H \times F}
W ( 1 ) ∈ R H × F
softmax
(
x
i
)
=
1
Z
exp
(
x
i
)
\operatorname{softmax}\left(x_{i}\right)=\frac{1}{\mathcal{Z}} \exp \left(x_{i}\right)
s o f t m a x ( x i ) = Z 1 exp ( x i )
Z
=
∑
i
exp
(
x
i
)
\mathcal{Z}=\sum_{i} \exp \left(x_{i}\right)
Z = ∑ i exp ( x i ) 交叉熵損失函數 爲
L
=
−
∑
l
∈
Y
L
∑
f
=
1
F
Y
l
f
ln
Z
l
f
\mathcal{L}=-\sum_{l \in \mathcal{Y}_{L}} \sum_{f=1}^{F} Y_{l f} \ln Z_{l f}
L = − l ∈ Y L ∑ f = 1 ∑ F Y l f ln Z l f
Y
L
\mathcal{Y}_{L}
Y L
C
C
C
F
F
F
Y
1
Y_1
Y 1
Y
2
Y_2
Y 2
論文在以下幾個任務中進行實驗
在citation network中進行半監督的document classification。
在從knowledge graph中提取的bipartite graph中進行半監督的entity classification
實驗數聽說明以下
結果以下：
能夠看出，在比較的幾種算法中，論文GCN的在準確率和時間上都最好。
該模型對每個節點(或邊、或圖)採用H個hop的矩陣進行表示，每個hop都表示該鄰近範圍的鄰近信息，由此，對於局部信息的獲取效果比較好，獲得的節點的representation的表示能力很強。
假設有以下定義
一個graph數據集
G
=
{
G
t
∣
t
∈
1
…
T
}
\mathcal{G}=\left\{G_{t} | t \in 1 \ldots T\right\}
G = { G t ∣ t ∈ 1 … T }
graph定義爲
G
t
=
(
V
t
,
E
t
)
G_{t}=\left(V_{t}, E_{t}\right)
G t = ( V t , E t )
V
t
V_t
V t
E
t
E_t
E t
全部節點的特徵矩陣定義爲
X
t
X_t
X t
N
t
×
F
N_t\times{F}
N t × F
N
t
N_t
N t
G
t
G_t
G t
F
F
F
邊信息
E
t
E_t
E t
N
t
×
N
t
N_t\times{}N_t
N t × N t
A
t
A_t
A t 節點度(degree)歸一化 的轉移機率矩陣
P
t
P_t
P t
i
i
i
j
j
j
對於graph來講沒有任何限制，graph能夠是帶權重的或不帶權重的，有向的或無向的。
模型的目標爲預測
Y
Y
Y 在每一種狀況中，模型輸入部分帶有標籤的數據集合，而後預測剩下的數據的標籤 。
DCNN模型輸入圖
G
\mathcal{G}
G
Y
Y
Y
P
(
Y
∣
X
)
\mathbb{P}(Y|X)
P ( Y ∣ X )
H
×
F
H\times{}F
H × F
H
H
H
t
t
t
N
t
×
H
×
F
N_t\times{H}\times{F}
N t × H × F
Z
t
Z_t
Z t
Z
t
Z_t
Z t
H
×
F
H\times{F}
H × F
Z
t
Z_t
Z t
M
t
×
H
×
F
M_t\times{H}\times{F}
M t × H × F
節點分類任務 ，假設
P
t
∗
P_t^*
P t ∗
P
t
P_t
P t
N
t
×
H
×
N
t
N_t\times{H}\times{N_t}
N t × H × N t
t
t
t
i
i
i
j
j
j
k
k
k
Z
t
i
j
k
Z_{tijk}
Z t i j k
Z
t
i
j
k
=
f
(
W
j
k
c
⋅
∑
l
=
1
N
t
P
t
i
j
l
∗
X
t
l
k
)
Z_{t i j k}=f\left(W_{j k}^{c} \cdot \sum_{l=1}^{N_{t}} P_{t i j l}^{*} X_{t l k}\right)
Z t i j k = f ( W j k c ⋅ l = 1 ∑ N t P t i j l ∗ X t l k )
Z
t
=
f
(
W
c
⊙
P
t
∗
X
t
)
Z_{t}=f\left(W^{c} \odot P_{t}^{*} X_{t}\right)
Z t = f ( W c ⊙ P t ∗ X t )
⊙
\odot
⊙
H
H
H
O
(
H
×
F
)
O(H\times{F})
O ( H × F ) 使得diffusion-convolutional representation不受輸入大小的限制 。
在計算出
Z
Z
Z
Y
Y
Y
Y
^
\hat{Y}
Y ^
P
(
Y
∣
X
)
\mathbb{P}(Y|X)
P ( Y ∣ X )
Y
^
=
arg
max
(
f
(
W
d
⊙
Z
)
)
\hat{Y}=\arg \max \left(f\left(W^{d} \odot Z\right)\right)
Y ^ = arg max ( f ( W d ⊙ Z ) )
P
(
Y
∣
X
)
=
softmax
(
f
(
W
d
⊙
Z
)
)
\mathbb{P}(Y | X)=\operatorname{softmax}\left(f\left(W^{d} \odot Z\right)\right)
P ( Y ∣ X ) = s o f t m a x ( f ( W d ⊙ Z ) )
對於圖分類任務 ，直接採用全部節點表示的均值做爲graph的representation
Z
t
=
f
(
W
c
⊙
1
N
t
T
P
t
∗
X
t
/
N
t
)
Z_{t}=f\left(W^{c} \odot 1_{N_{t}}^{T} P_{t}^{*} X_{t} / N_{t}\right)
Z t = f ( W c ⊙ 1 N t T P t ∗ X t / N t )
1
N
t
1_{N_t}
1 N t
N
t
×
1
N_t\times{1}
N t × 1
對於邊分類任務 ，經過將每一條邊轉化爲一個節點來進行訓練和預測 ，這個節點與原來的邊對應的首尾節點相連，轉化後的圖的鄰接矩陣
A
t
′
A_t'
A t ′
A
t
A_t
A t incidence matrix 獲得
A
t
′
=
(
A
t
B
t
T
B
t
0
)
A_{t}^{\prime}=\left(\begin{array}{cc}{A_{t}} & {B_{t}^{T}} \\ {B_{t}} & {0}\end{array}\right)
A t ′ = ( A t B t B t T 0 )
A
t
′
A_t'
A t ′
P
t
′
P_t'
P t ′
P
t
P_t
P t
對於模型訓練 ，使用梯度降低法，並採用early-stop方式獲得最終模型。
節點分類任務的實驗數據集使用Cora和Pubmed數據集，包含scientific papers(至關於node)、citations(至關於edge)和subjects(至關於label)。實驗評估標準使用分類準確率以及F1值。
節點分類的實驗結果 以下
能夠看出使用各類評估標準，DCNN效果都是最好的。
圖分類任務 的實驗結果以下
優勢 ：
缺點 ：
內存佔用大 ：DCNN創建在密集的張量計算上，須要存儲大量的張量，須要
O
(
N
t
2
H
)
O(N_t^2H)
O ( N t 2 H ) 長距離信息傳播不足 ：模型對於局部的信息獲取較好，可是遠距離的信息傳播不足。
將序列型的LSTM模型擴展到樹型的LSTM模型 ，簡稱Tree-LSTM，並根據孩子節點是否有序，論文提出了兩個模型變體，Child-Sum Tree-LSTM模型和N-ary Tree-LSTM模型 。和序列型的LSTM模型的主要不一樣點在於，序列型的LSTM從前一時刻獲取隱藏狀態
h
t
h_t
h t
Tree-LSTM模型對於每個孩子節點都會產生一個「遺忘門」
f
j
k
f_{jk}
f j k 從全部的孩子節點選擇性地獲取信息和結合信息 。
該模型的更新方程以下
h
~
j
=
∑
k
∈
C
(
j
)
h
k
i
j
=
σ
(
W
(
i
)
x
j
+
U
(
i
)
h
~
j
+
b
(
i
)
)
f
j
k
=
σ
(
W
(
f
)
x
j
+
U
(
f
)
h
k
+
b
(
f
)
)
o
j
=
σ
(
W
(
o
)
x
j
+
U
(
o
)
h
~
j
+
b
(
o
)
)
u
j
=
tanh
(
W
(
u
)
x
j
+
U
(
u
)
h
~
j
+
b
(
u
)
)
c
j
=
i
j
⊙
u
j
+
∑
k
∈
C
(
j
)
f
j
k
⊙
c
k
h
j
=
o
j
⊙
tanh
(
c
j
)
\begin{aligned} \tilde{h}_{j} &=\sum_{k \in C(j)} h_{k} \\ i_{j} &=\sigma\left(W^{(i)} x_{j}+U^{(i)} \tilde{h}_{j}+b^{(i)}\right) \\ f_{j k} &=\sigma\left(W^{(f)} x_{j}+U^{(f)} h_{k}+b^{(f)}\right) \\ o_{j} &=\sigma\left(W^{(o)} x_{j}+U^{(o)} \tilde{h}_{j}+b^{(o)}\right) \\ u_{j} &=\tanh \left(W^{(u)} x_{j}+U^{(u)} \tilde{h}_{j}+b^{(u)}\right) \\ c_{j} &=i_{j} \odot u_{j}+\sum_{k \in C(j)} f_{j k} \odot c_{k} \\ h_{j} &=o_{j} \odot \tanh \left(c_{j}\right) \end{aligned}
h ~ j i j f j k o j u j c j h j = k ∈ C ( j ) ∑ h k = σ ( W ( i ) x j + U ( i ) h ~ j + b ( i ) ) = σ ( W ( f ) x j + U ( f ) h k + b ( f ) ) = σ ( W ( o ) x j + U ( o ) h ~ j + b ( o ) ) = tanh ( W ( u ) x j + U ( u ) h ~ j + b ( u ) ) = i j ⊙ u j + k ∈ C ( j ) ∑ f j k ⊙ c k = o j ⊙ tanh ( c j )
C
(
j
)
C(j)
C ( j )
j
j
j
h
k
h_k
h k
k
k
k
i
j
i_j
i j
j
j
j
f
j
k
f_{jk}
f j k
j
j
j
k
k
k
o
j
o_j
o j
j
j
j
這裏的關鍵點在於第三個公式的
f
j
k
f_{jk}
f j k
j
j
j
k
k
k
c
j
c_j
c j
因爲該模型是對全部的孩子節點求和，因此這個模型對於節點順序不敏感的 ，適合於孩子節點無序的狀況。
假如一個樹的最大分支數爲
N
N
N
N
N
N
j
j
j
k
k
k
h
j
k
h_{jk}
h j k
c
j
k
c_{jk}
c j k
i
j
=
σ
(
W
(
i
)
x
j
+
∑
ℓ
=
1
N
U
ℓ
(
i
)
h
j
ℓ
+
b
(
i
)
)
f
j
k
=
σ
(
W
(
f
)
x
j
+
∑
ℓ
=
1
N
U
k
ℓ
(
f
)
h
j
ℓ
+
b
(
f
)
)
o
j
=
σ
(
W
(
o
)
x
j
+
∑
ℓ
=
1
N
U
ℓ
(
o
)
h
j
ℓ
+
b
(
o
)
)
u
j
=
tanh
(
W
(
u
)
x
j
+
∑
ℓ
=
1
N
U
ℓ
(
a
)
h
j
ℓ
+
b
(
a
)
)
c
j
=
i
j
⊙
u
j
+
∑
ℓ
=
1
N
f
j
ℓ
⊙
c
j
ℓ
h
j
=
o
j
⊙
tanh
(
c
j
)
\begin{aligned} i_{j} &=\sigma\left(W^{(i)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(i)} h_{j \ell}+b^{(i)}\right) \\ f_{j k} &=\sigma\left(W^{(f)} x_{j}+\sum_{\ell=1}^{N} U_{k \ell}^{(f)} h_{j \ell}+b^{(f)}\right) \\ o_{j} &=\sigma\left(W^{(o)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(o)} h_{j \ell}+b^{(o)}\right) \\ u_{j} &=\tanh \left(W^{(u)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(a)} h_{j \ell}+b^{(a)}\right) \\ c_{j} &=i_{j} \odot u_{j}+\sum_{\ell=1}^{N} f_{j \ell} \odot c_{j \ell} \\ h_{j} &=o_{j} \odot \tanh \left(c_{j}\right) \end{aligned}
i j f j k o j u j c j h j = σ ( W ( i ) x j + ℓ = 1 ∑ N U ℓ ( i ) h j ℓ + b ( i ) ) = σ ( W ( f ) x j + ℓ = 1 ∑ N U k ℓ ( f ) h j ℓ + b ( f ) ) = σ ( W ( o ) x j + ℓ = 1 ∑ N U ℓ ( o ) h j ℓ + b ( o ) ) = tanh ( W ( u ) x j + ℓ = 1 ∑ N U ℓ ( a ) h j ℓ + b ( a ) ) = i j ⊙ u j + ℓ = 1 ∑ N f j ℓ ⊙ c j ℓ = o j ⊙ tanh ( c j ) 該模型爲每一個孩子節點都單獨地設置了參數
U
l
U_{l}
U l 。
分類任務定義爲在類別集
Y
\mathcal{Y}
Y
y
^
\hat{y}
y ^
j
j
j
y
^
j
\hat{y}_j
y ^ j
h
j
h_j
h j
p
^
θ
(
y
∣
{
x
}
j
)
=
softmax
(
W
(
s
)
h
j
+
b
(
s
)
)
y
^
j
=
arg
max
y
p
^
θ
(
y
∣
{
x
}
j
)
\begin{aligned} \hat{p}_{\theta}\left(y |\{x\}_{j}\right) &=\operatorname{softmax}\left(W^{(s)} h_{j}+b^{(s)}\right) \\ \hat{y}_{j} &=\arg \max _{y} \hat{p}_{\theta}\left(y |\{x\}_{j}\right) \end{aligned}
p ^ θ ( y ∣ { x } j ) y ^ j = s o f t m a x ( W ( s ) h j + b ( s ) ) = arg y max p ^ θ ( y ∣ { x } j )
J
(
θ
)
=
−
1
m
∑
k
=
1
m
log
p
^
θ
(
y
(
k
)
∣
{
x
}
(
k
)
)
+
λ
2
∥
θ
∥
2
2
J(\theta)=-\frac{1}{m} \sum_{k=1}^{m} \log \hat{p}_{\theta}\left(y^{(k)} |\{x\}^{(k)}\right)+\frac{\lambda}{2}\|\theta\|_{2}^{2}
J ( θ ) = − m 1 k = 1 ∑ m log p ^ θ ( y ( k ) ∣ { x } ( k ) ) + 2 λ ∥ θ ∥ 2 2
m
m
m
λ
\lambda
λ
L
2
L2
L 2
該任務給定一個句子對(sentence pair)，模型須要預測出一個範圍在
[
1
,
K
]
[1,K]
[ 1 , K ]
論文首先對每個句子產生一個representation，兩個句子的表示分別用
h
L
h_L
h L
h
R
h_R
h R
(
h
L
,
h
R
)
(h_L,h_R)
( h L , h R )
h
×
=
h
L
⊙
h
R
h
+
=
∣
h
L
−
h
R
∣
h
s
=
σ
(
W
(
×
)
h
×
+
W
(
+
)
h
+
+
b
(
h
)
)
p
^
θ
=
softmax
(
W
(
p
)
h
s
+
b
(
p
)
)
y
^
=
r
T
p
^
θ
\begin{aligned} h_{ \times} &=h_{L} \odot h_{R} \\ h_{+} &=\left|h_{L}-h_{R}\right| \\ h_{s} &=\sigma\left(W^{( \times)} h_{ \times}+W^{(+)} h_{+}+b^{(h)}\right) \\ \hat{p}_{\theta} &=\operatorname{softmax}\left(W^{(p)} h_{s}+b^{(p)}\right) \\ \hat{y} &=r^{T} \hat{p}_{\theta} \end{aligned}
h × h + h s p ^ θ y ^ = h L ⊙ h R = ∣ h L − h R ∣ = σ ( W ( × ) h × + W
web