一個小遊戲：網易雲音樂資源爬取（登陸+評論）

如何爬取網易雲音樂的評論呢？

推薦：微信小遊戲大全

1.文本以華晨宇的《個人滑板鞋2016》爲例

http://music.163.com/#/song?i...

2.使用瀏覽器的工具，查找獲取評論的url

http://music.163.com/weapi/v1...

3.不難發現，此API是經過POST請求得到咱們想要的評論信息的，須要POST的參數有params和encSecKey，網易爲了反爬蟲，加密了這2個參數，不過沒關係，下面有加密過程

4.POST請求成功後，會獲得一個JSON字符串，裏面就包含了各類評論信息

5.下面就用JAVA來實現評論的獲取

AES加密工具類

import java.util.Base64;

import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;

public class EncryptTools {
    //AES加密
    public static String encrypt(String text, String secKey) throws Exception {
        byte[] raw = secKey.getBytes();
        SecretKeySpec skeySpec = new SecretKeySpec(raw, "AES");
        // "算法/模式/補碼方式"
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding"); 
        // 使用CBC模式，須要一個向量iv，可增長加密算法的強度
        IvParameterSpec iv = new IvParameterSpec("0102030405060708".getBytes());
        cipher.init(Cipher.ENCRYPT_MODE, skeySpec, iv);
        byte[] encrypted = cipher.doFinal(text.getBytes());
        return Base64.getEncoder().encodeToString(encrypted);
    }
  //字符填充
  public static String zfill(String result, int n) {
        if (result.length() >= n) {
            result = result.substring(result.length() - n, result.length());
        } else {
            StringBuilder stringBuilder = new StringBuilder();
            for (int i = n; i > result.length(); i--) {
                stringBuilder.append("0");
            }
            stringBuilder.append(result);
            result = stringBuilder.toString();
        }
        return result;
    }
}

推薦：微信小遊戲大全

評論獲取

public void commentAPI() throws Exception {
        //私鑰，隨機16位字符串（本身可改）
        String secKey = "cd859f54539b24b7";
        String text = "{\"username\": \"\", \"rememberLogin\": \"true\", \"password\": \"\"}";
        String modulus = "00e0b509f6259df8642dbc35662901477df22677ec152b5ff68ace615bb7b725152b3ab17a876aea8a5aa76d2e417629ec4ee341f56135fccf695280104e0312ecbda92557c93870114af6c9d05c4f7f0c3685b7a46bee255932575cce10b424d813cfe4875d3e82047b97ddef52741d546b8e289dc6935b3ece0462db0a22b8e7";
        String nonce = "0CoJUm6Qyw8W8jud";
        String pubKey = "010001";
        //2次AES加密，獲得params
        String params = EncryptTools.encrypt(EncryptTools.encrypt(text, nonce), secKey);
        StringBuffer stringBuffer = new StringBuffer(secKey);
        //逆置私鑰
        secKey = stringBuffer.reverse().toString();
        String hex = Hex.encodeHexString(secKey.getBytes());
        BigInteger bigInteger1 = new BigInteger(hex, 16);
        BigInteger bigInteger2 = new BigInteger(pubKey, 16);
        BigInteger bigInteger3 = new BigInteger(modulus, 16);
        //RSA加密計算
        BigInteger bigInteger4 = bigInteger1.pow(bigInteger2.intValue()).remainder(bigInteger3);
        String encSecKey= Hex.encodeHexString(bigInteger4.toByteArray());
        //字符填充
        encSecKey= EncryptTools.zfill(encSecKey, 256);
        //評論獲取
        Document document = Jsoup.connect("http://music.163.com/weapi/v1/resource/comments/R_SO_4_437859519/").cookie("appver", "1.5.0.75771")
                .header("Referer", "http://music.163.com/").data("params", params).data("encSecKey", encSecKey)
                .ignoreContentType(true).post();
        System.out.println("評論：" + document.text());
    }

那麼如何登陸網易雲音樂呢？

1.查找登陸的API地址(在瀏覽器登陸一次，在network中查看url)

2.查看登陸時post須要提交的參數

3.是否是很熟悉？和剛纔獲取評論的時候同樣，該如何作呢？看看下面的示例代碼

在上面AES加密工具類中添加MD5加密方法

public static String md5(String pwd) {
        try {
            MessageDigest digest = MessageDigest.getInstance("md5");
            byte[] bs = digest.digest(pwd.getBytes());
            String hexString = "";
            for (byte b : bs) {
                int temp = b & 255;
                if (temp < 16 && temp >= 0) {
                    hexString = hexString + "0" + Integer.toHexString(temp);
                } else {
                    hexString = hexString + Integer.toHexString(temp);
                }
            }
            return hexString;
        } catch (NoSuchAlgorithmException e) {
            e.printStackTrace();
        }
        return "";
    }

登陸請求

public void loginAPI(String username, String password) throws Exception {
        password = EncryptTools.md5(password);
        // 私鑰，隨機16位字符串（本身可改）
        String secKey = "cd859f54539b24b7";
        String text = "{\"username\": \"" + username + "\", \"rememberLogin\": \"true\", \"password\": \"" + password
                + "\"}";
        String modulus = "00e0b509f6259df8642dbc35662901477df22677ec152b5ff68ace615bb7b725152b3ab17a876aea8a5aa76d2e417629ec4ee341f56135fccf695280104e0312ecbda92557c93870114af6c9d05c4f7f0c3685b7a46bee255932575cce10b424d813cfe4875d3e82047b97ddef52741d546b8e289dc6935b3ece0462db0a22b8e7";
        String nonce = "0CoJUm6Qyw8W8jud";
        String pubKey = "010001";
        // 2次AES加密，獲得params
        String params = EncryptTools.encrypt(EncryptTools.encrypt(text, nonce), secKey);
        StringBuffer stringBuffer = new StringBuffer(secKey);
        // 逆置私鑰
        secKey = stringBuffer.reverse().toString();
        String hex = Hex.encodeHexString(secKey.getBytes());
        BigInteger bigInteger1 = new BigInteger(hex, 16);
        BigInteger bigInteger2 = new BigInteger(pubKey, 16);
        BigInteger bigInteger3 = new BigInteger(modulus, 16);
        // RSA加密計算
        BigInteger bigInteger4 = bigInteger1.pow(bigInteger2.intValue()).remainder(bigInteger3);
        String encSecKey = Hex.encodeHexString(bigInteger4.toByteArray());
        // 字符填充
        encSecKey = EncryptTools.zfill(encSecKey, 256);
        // 登陸請求
        Document document = Jsoup.connect("http://music.163.com/weapi/login/").cookie("appver", "1.5.0.75771")
                .header("Referer", "http://music.163.com/").data("params", params).data("encSecKey", encSecKey)
                .ignoreContentType(true).post();
        System.out.println("登陸結果：" + document.text());
    }

參考資料

網易雲音樂登陸信息加密算法詳解
推薦：微信小遊戲大全

本文由博客羣發一文多發等運營工具平臺 OpenWrite 發佈