Shell命令行處理JSON

基本介紹

常常用python或java處理json數據，其實在Shell命令行，咱們能夠用開源JQ工具來處理Json數據。會更加簡潔和高效。

CentOS 安裝JQ

wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64

chmod +x ./jq

cp jq /usr/bin

JQ經常使用舉例

一、原始JSON數據

 cat jsonData.json

{"productId":"2723","click":60,"view":300,"deal":2,"day":"20130919"}

{"productId":"2728","click":130,"view":800,"deal":10,"day":"20130919"}

{"productId":"3609","click":50,"view":400,"deal":3,"day":"20130919"}

{"productId":"3783","click":375,"view":1200,"deal":50,"day":"20130919"}

{"productId":"3522","click":87,"view":573,"deal":15,"day":"20130919"}

二、格式化輸出

 jq -r . jsonData.json

{

  "productId": "2723",

  "click": 60,

  "view": 300,

  "deal": 2,

  "day": "20130919"

}

{

  "productId": "2728",

  "click": 130,

  "view": 800,

  "deal": 10,

  "day": "20130919"

}

...

三、提取json指定字段轉csv

這裏咱們須要獲得productId_day,click,view,deal這種格式的csv文件。

jq -r '[.productId+"_"+.day,(.click|tostring),(.view|tostring),(.deal|tostring)]|join(",")' jsonData.json

結果：

2723_20130919,60,300,2

2728_20130919,130,800,10

3609_20130919,50,400,3

3783_20130919,375,1200,50

3522_20130919,87,573,15

注意：

  .productId:取productId字段的值

　.click|tostring:取click字段的值並轉化爲string類型

　[]:數組

　join(","):數組中的元素經過","鏈接。

四、複雜json數據查詢

 complexJson='{"uid":"2378","nameInfo":[{"firstName":"nova","lastName":"jos"}]}'

echo ${complexJson} |jq

{

  "uid": "2378",

  "nameInfo": [

    {

      "firstName": "nova",

      "lastName": "jos"

    }

  ]

}

echo ${complexJson} |jq -r '.nameInfo[].lastName'

結果:jos

注意:.nameInfo[].lastName是獲取.nameInfo[] json array中每一個json object的lastName字段。

五、提取指定字段造成新的json

 jq -c '{productId:.productId,click:.click,dayYmd:(.day|tonumber)}' jsonData.json >>newjson.json

結果:

cat newjson.json

{"productId":"2723","click":60,"dayYmd":20130919}

{"productId":"2728","click":130,"dayYmd":20130919}

{"productId":"3609","click":50,"dayYmd":20130919}

{"productId":"3783","click":375,"dayYmd":20130919}

{"productId":"3522","click":87,"dayYmd":20130919}

注意:

 -c/--compact-output:jq 默認是pretty-prints JSON 輸出，加上這個參數，一行一個json對象。

六、csv文件轉json文件

 cat csvData.csv

1478,john,38

1529,lucy,25

1673,iris,22

jq -R -c 'split(",")|{"uid":.[0],"name":.[1],"age":.[2]|tonumber}' csvData.csv > csv2json.json

cat csv2json.json

{"uid":"1478","name":"john","age":38}

{"uid":"1529","name":"lucy","age":25}

{"uid":"1673","name":"iris","age":22}

主要參考文檔

一、https://programminghistorian.org/lessons/json-and-jq 

二、JQ使用手冊:https://stedolan.github.io/jq/manual/#Basicfilters 

三、JQ Cookbook:https://github.com/stedolan/jq/wiki/Cookbook

--------------------- 

做者：wangpei1949 

來源：CSDN 

原文：https://blog.csdn.net/wangpei1949/article/details/78702272