shell腳本處理JSON數據工具jq

shell腳本如何方便地處理JSON格式的數據呢，這裏介紹一個工具：jq。

官方教程簡單翻譯以下。

一、獲取JSON數據

咱們以github上jq項目最新5條評論的JSON數據爲例。獲取數據以下:

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5'

結果以下：

[ { "sha": "d25341478381063d1c76e81b3a52e0592a7c997f", "commit": { "author": { "name": "Stephen Dolan", "email": "mu@netsoc.tcd.ie", "date": "2013-06-22T16:30:59Z" }, "committer": { "name": "Stephen Dolan", "email": "mu@netsoc.tcd.ie", "date": "2013-06-22T16:30:59Z" }, "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161", "tree": { "sha": "6ab697a8dfb5a96e124666bf6d6213822599fb40", "url": "https://api.github.com/repos/stedolan/jq/git/trees/6ab697a8dfb5a96e124666bf6d6213822599fb40" }, "url": "https://api.github.com/repos/stedolan/jq/git/commits/d25341478381063d1c76e81b3a52e0592a7c997f", "comment_count": 0 }, "url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f", "html_url": "https://github.com/stedolan/jq/commit/d25341478381063d1c76e81b3a52e0592a7c997f", "comments_url": "https://api.github.com/repos/stedolan/jq/commits/d25341478381063d1c76e81b3a52e0592a7c997f/comments", "author": { "login": "stedolan", ...

二、高亮並按屬性排序顯示

用 jq '.'便可：

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'

結果：

[ { "parents": [ { "html_url": "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7", "url": "https://api.github.com/repos/stedolan/jq/commits/54b9c9bdb225af5d886466d72f47eafc51acb4f7", "sha": "54b9c9bdb225af5d886466d72f47eafc51acb4f7" }, { "html_url": "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a", "url": "https://api.github.com/repos/stedolan/jq/commits/8b1b503609c161fea4b003a7179b3fbb2dd4345a", "sha": "8b1b503609c161fea4b003a7179b3fbb2dd4345a" } ], "committer": { "type": "User", "received_events_url": "https://api.github.com/users/stedolan/received_events", "events_url": "https://api.github.com/users/stedolan/events{/privacy}", "repos_url": "https://api.github.com/users/stedolan/repos", "organizations_url": "https://api.github.com/users/stedolan/orgs", ...

這裏的評論內容比較多，咱們如今想拿第一個評論。

三、獲取數組某一項

curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[0]'

結果：

  View Code

能夠看到，咱們已經拿到了一條完整的評論內容。但咱們真正關心的只是評論內容和用戶名，下面來獲取這兩項內容。

四、自定義格式輸出某一項

jq '.[0] | {message: .commit.message, name: .commit.committer.name}'

結果：

{ "name": "Stephen Dolan", "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161" }

咱們能夠看到，已經拿到了想要的內容，而且已經按咱們本身定義的格式顯示了。

這裏 | 後面的內容是之前面的內容爲輸入的，.commit 中的 . 就是指 .[0] 中的內容。

五、自定義格式輸出多項

jq '.[] | {message: .commit.message, name: .commit.committer.name}'

結果：

{ "name": "Stephen Dolan", "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161" } { "name": "Stephen Dolan", "message": "Reject all overlong UTF8 sequences." } { "name": "Stephen Dolan", "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences." } { "name": "Stephen Dolan", "message": "Fix example in manual for `floor`. See #155." } { "name": "Nicolas Williams", "message": "Document floor" }

這裏 .[] 獲取的是數組中的全部項。

咱們看到，結果是一個個獨立的JSON對象，如何把結果組合成一個數組呢？

六、以數組形式自定義輸出多項

jq '[.[] | {message: .commit.message, name: .commit.committer.name}]'

結果：

[ { "name": "Stephen Dolan", "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161" }, { "name": "Stephen Dolan", "message": "Reject all overlong UTF8 sequences." }, { "name": "Stephen Dolan", "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences." }, { "name": "Stephen Dolan", "message": "Fix example in manual for `floor`. See #155." }, { "name": "Nicolas Williams", "message": "Document floor" } ]

咱們能夠看到，只要在上一步的命令中內容的兩端加個中括號便可。

最後，咱們若是想獲取每一個評論的引用評論的url（在parents節點中，有一個或多個）呢？

七、獲取其餘內容

jq '[.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]}]'

結果：

[ { "parents": [ "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7", "https://github.com/stedolan/jq/commit/8b1b503609c161fea4b003a7179b3fbb2dd4345a" ], "name": "Stephen Dolan", "message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161" }, { "parents": [ "https://github.com/stedolan/jq/commit/ff48bd6ec538b01d1057be8e93b94eef6914e9ef" ], "name": "Stephen Dolan", "message": "Reject all overlong UTF8 sequences." }, { "parents": [ "https://github.com/stedolan/jq/commit/54b9c9bdb225af5d886466d72f47eafc51acb4f7" ], "name": "Stephen Dolan", "message": "Fix various UTF8 parsing bugs.\n\nIn particular, parse bad UTF8 by replacing the broken bits with U+FFFD\nand resychronise correctly after broken sequences." }, { "parents": [ "https://github.com/stedolan/jq/commit/3dcdc582ea993afea3f5503a78a77675967ecdfa" ], "name": "Stephen Dolan", "message": "Fix example in manual for `floor`. See #155." }, { "parents": [ "https://github.com/stedolan/jq/commit/7c4171d414f647ab08bcd20c76a4d8ed68d9c602" ], "name": "Nicolas Williams", "message": "Document floor" } ]

這裏用 .parents[].html_url 獲取當前項的 parents 節點中的全部項的 html_url 屬性的內容，而後兩邊加個中括號組裝成數組輸出。

怎麼樣，通過這個例子能夠看出，用jq處理JSON數據仍是很方便強大的吧。

八、注意點

jq能處理的須要是嚴格的JSON格式數據，JSON對象和JSON字符串是不行的，以下面的兩種格式數據jq是不能處理的：

json對象：

{ a: 1, b: { c: "abc" } }

json字符串：

'{"a":1,"b":{"c":"abc"}}'

正確的JSON格式：

{ "a": 1, "b": { "c": "abc" } }

關於什麼是JSON格式數據，請參考：http://www.json.org/