Matrix 源碼分析 ———— Apk Canary

時間 2019-11-07

原文原文鏈接

概述

年前，微信開源了Matrix項目，提供了Android、ios的APM實現方案。對於Android端實現，主要包括APK Checker、Resource Canary、Trace Canary、SQLite Lint、IO Canary五部分。本文主要介紹APK Checker的源碼實現，其餘部分的源碼分析將在後續推出。java

代碼框架分析

總體代碼結構比較清晰，主要包括三部分：ApkJob、Task、 Result。ApkJob是表示總體這個apk檢測任務，Task表示每一步細分的檢測任務、Result表示檢測任務的結果。整體流程以下：ApkJob讀取配置信息，實例化相關的Task任務；相關Task任務執行以後輸出Result到文件（默認爲MMTaskJsonResult）。android

Task任務實現分析

UnZipTask

目的：解壓Apk，解析Class混淆規則、Res混淆規則，並輸出apk中每一個entry原始大小、zip包中壓縮後的大小。主要存儲了一些原始數據，爲後續的Task作準備。ios

@Override
    public TaskResult call() throws TaskExecuteException {

        try {
            //apk文件
            ZipFile zipFile = new ZipFile(inputFile);
            ...
            //Result輸出對象
            TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
            ...
            //apk總大小
            ((TaskJsonResult) taskResult).add("total-size", inputFile.length());
            //讀取Class的mapping規則，並存儲到config對象中
            readMappingTxtFile();
            config.setProguardClassMap(proguardClassMap);
            //讀取Res的mapping規則，並存儲到config對象中
            readResMappingTxtFile();
            config.setResguardMap(resguardMap);

            Enumeration entries = zipFile.entries();
            JsonArray jsonArray = new JsonArray();
            String outEntryName = "";
            while (entries.hasMoreElements()) {
                ZipEntry entry = (ZipEntry) entries.nextElement();
                outEntryName = writeEntry(zipFile, entry);
                if (!Util.isNullOrNil(outEntryName)) {
                    JsonObject fileItem = new JsonObject();
                    //輸出Apk中每一個item的名字、壓縮後的大小
                    fileItem.addProperty("entry-name", outEntryName);
                    fileItem.addProperty("entry-size", entry.getCompressedSize());
                    jsonArray.add(fileItem);
                    //Map：解壓後文件（相對路徑）-> (未壓縮Size，壓縮後Size)
                    entrySizeMap.put(outEntryName, Pair.of(entry.getSize(), entry.getCompressedSize()));
                    //Map：Apk中文件名 -> ：解壓後文件（相對路徑）
                    entryNameMap.put(entry.getName(), outEntryName);
                }
            }
            //存儲到config對象
            config.setEntrySizeMap(entrySizeMap);
            config.setEntryNameMap(entryNameMap);
            //輸出到Result
            ((TaskJsonResult) taskResult).add("entries", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

重點講解一下Task任務中mapping文件的解析規則：git

Class Mapping

class mapping文件截取片斷：github

...
android.arch.core.executor.ArchTaskExecutor$1 -> android.arch.a.a.a$1:
    42:42:void <init>() -> <init>
    45:46:void execute(java.lang.Runnable) -> execute
android.arch.core.executor.ArchTaskExecutor$2 -> android.arch.a.a.a$2:
    50:50:void <init>() -> <init>
    53:54:void execute(java.lang.Runnable) -> execute
android.arch.core.executor.DefaultTaskExecutor -> android.arch.a.a.b:
    java.lang.Object mLock -> a
    java.util.concurrent.ExecutorService mDiskIO -> b
    android.os.Handler mMainHandler -> c
    31:33:void <init>() -> <init>
    40:41:void executeOnDiskIO(java.lang.Runnable) -> a
    45:54:void postToMainThread(java.lang.Runnable) -> b
    58:58:boolean isMainThread() -> b
    ...
複製代碼

* 原始類名 -> 混淆後類名 （頂格）
* 原始字段名 -> 混淆後字段名   （行首預留一個Tab）
* 原始函數名 -> 混淆後函數名   （行首預留一個Tab）
複製代碼

Res Mapping

res mapping文件截取片斷：json

res path mapping:
    res/layout-v22 -> r/a
    res/drawable -> r/b
    res/color-night-v8 -> r/c
    res/xml -> r/d
    res/layout -> r/e
  ...
  
res id mapping:
    com.example.app.R.attr.avatar_border_color -> com.example.app.R.attr.a
    com.example.app.R.attr.actualImageScaleType -> com.example.app.R.attr.b
    com.example.app.R.attr.backgroundImage -> com.example.app.R.attr.c
    com.example.app.R.attr.fadeDuration -> com.example.app.R.attr.d
    com.example.app.R.attr.failureImage -> com.example.app.R.attr.e
複製代碼

* 原始資源目錄 -> 混淆後資源目錄
* 原始資源名 -> 混淆後資源名
複製代碼

ManifestAnalyzeTask

目的：解析Manifest文件、arsc文件api

public TaskResult call() throws TaskExecuteException {
        try {
            ManifestParser manifestParser = null;
            //建立Manifest解析對象
            if (!FileUtil.isLegalFile(arscFile)) {
                manifestParser = new ManifestParser(inputFile);
            } else {
                manifestParser = new ManifestParser(inputFile, arscFile);
            }
            TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
            if (taskResult == null) {
                return null;
            }
            long startTime = System.currentTimeMillis();
            JsonObject jsonObject = manifestParser.parse();
            //輸出Manifest解析結果
            ((TaskJsonResult) taskResult).add("manifest", jsonObject);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

此處講解下arsc文件。arsc文件以二進制形式存在，存儲了資源的索引信息，基本文件格式以下圖（圖片來源網絡）：數組

用二進制工具查看arsc文件的內容：sass

arsc的詳細文件格式暫時不展開，可參考文章，此處僅簡單分析一下二進制工具中可視化展現的一些信息。bash

ResTable_header部分

ResStringPool部分

ResTablePackage

關於arsc文件解析的相關內容，詳見文章

ShowFileSizeTask

目的：統計超過閾值的文件。

public TaskResult call() throws TaskExecuteException {
        ...
            long startTime = System.currentTimeMillis();
            //獲取UnZipTask中記錄的 文件名->（文件壓縮後大小，文件壓縮前大小） map
            Map<String, Pair<Long, Long>> entrySizeMap = config.getEntrySizeMap();
            if (!entrySizeMap.isEmpty()) {                                        
                for (Map.Entry<String, Pair<Long, Long>> entry : entrySizeMap.entrySet()) {
                    final String suffix = getSuffix(entry.getKey());
                    Pair<Long, Long> size = entry.getValue();
                    // 記錄超出閾值的文件
                    if (size.getFirst() >= downLimit * ApkConstants.K1024) {
                        if (filterSuffix.isEmpty() || filterSuffix.contains(suffix)) {
                            entryList.add(Pair.of(entry.getKey(), size.getFirst()));
                        } 
                    } 
                }
            }

           ...
           //排序

            JsonArray jsonArray = new JsonArray();
            for (Pair<String, Long> sortFile : entryList) {
                JsonObject fileItem = new JsonObject();
                fileItem.addProperty("entry-name", sortFile.getFirst());
                fileItem.addProperty("entry-size", sortFile.getSecond());
                jsonArray.add(fileItem);
            }
            //輸出到結果
            ((TaskJsonResult) taskResult).add("files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

MethodCountTask

目的：統計在本dex文件內定義的方法數、未在本dex文件內定義的方法數。

public TaskResult call() throws TaskExecuteException {
        try {
            ...
            long startTime = System.currentTimeMillis();
            JsonArray jsonArray = new JsonArray();
            for (int i = 0; i < dexFileList.size(); i++) {
                RandomAccessFile dexFile = dexFileList.get(i);
                //計算dex中的方法信息
                countDex(dexFile);
                //dex內能找到定義的方法
                int totalInternalMethods = sumOfValue(classInternalMethod);
                //跨dex的方法
                int totalExternalMethods = sumOfValue(classExternalMethod);
                JsonObject jsonObject = new JsonObject();
                jsonObject.addProperty("dex-file", dexFileNameList.get(i));
                //按Class維度聚合
                if (JobConstants.GROUP_CLASS.equals(group)) {
                    List<String> sortList = sortKeyByValue(classInternalMethod);
                    JsonArray classes = new JsonArray();
                    for (String className : sortList) {
                        JsonObject classObj = new JsonObject();
                        classObj.addProperty("name", className);
                        classObj.addProperty("methods", classInternalMethod.get(className));
                        classes.add(classObj);
                    }
                    jsonObject.add("internal-classes", classes);
                    //按package維度聚合
                } else if (JobConstants.GROUP_PACKAGE.equals(group)) {
                    String packageName;
                    for (Map.Entry<String, Integer> entry : classInternalMethod.entrySet()) {
                        packageName = ApkUtil.getPackageName(entry.getKey());
                        if (!Util.isNullOrNil(packageName)) {
                            if (!pkgInternalRefMethod.containsKey(packageName)) {
                                pkgInternalRefMethod.put(packageName, entry.getValue());
                            } else {
                                pkgInternalRefMethod.put(packageName, pkgInternalRefMethod.get(packageName) + entry.getValue());
                            }
                        }
                    }
                    List<String> sortList = sortKeyByValue(pkgInternalRefMethod);
                    JsonArray packages = new JsonArray();
                    for (String pkgName : sortList) {
                        JsonObject pkgObj = new JsonObject();
                        pkgObj.addProperty("name", pkgName);
                        pkgObj.addProperty("methods", pkgInternalRefMethod.get(pkgName));
                        packages.add(pkgObj);
                    }
                    jsonObject.add("internal-packages", packages);
                }
                jsonObject.addProperty("total-internal-classes", classInternalMethod.size());
                jsonObject.addProperty("total-internal-methods", totalInternalMethods);

                if (JobConstants.GROUP_CLASS.equals(group)) {
                    List<String> sortList = sortKeyByValue(classExternalMethod);
                    JsonArray classes = new JsonArray();
                    for (String className : sortList) {
                        JsonObject classObj = new JsonObject();
                        classObj.addProperty("name", className);
                        classObj.addProperty("methods", classExternalMethod.get(className));
                        classes.add(classObj);
                    }
                    jsonObject.add("external-classes", classes);

                } else if (JobConstants.GROUP_PACKAGE.equals(group)) {
                    String packageName = "";
                    for (Map.Entry<String, Integer> entry : classExternalMethod.entrySet()) {
                        packageName = ApkUtil.getPackageName(entry.getKey());
                        if (!Util.isNullOrNil(packageName)) {
                            if (!pkgExternalMethod.containsKey(packageName)) {
                                pkgExternalMethod.put(packageName, entry.getValue());
                            } else {
                                pkgExternalMethod.put(packageName, pkgExternalMethod.get(packageName) + entry.getValue());
                            }
                        }
                    }
                    List<String> sortList = sortKeyByValue(pkgExternalMethod);
                    JsonArray packages = new JsonArray();
                    for (String pkgName : sortList) {
                        JsonObject pkgObj = new JsonObject();
                        pkgObj.addProperty("name", pkgName);
                        pkgObj.addProperty("methods", pkgExternalMethod.get(pkgName));
                        packages.add(pkgObj);
                    }
                    jsonObject.add("external-packages", packages);

                }
                jsonObject.addProperty("total-external-classes", classExternalMethod.size());
                jsonObject.addProperty("total-external-methods", totalExternalMethods);
                jsonArray.add(jsonObject);
            }
            ((TaskJsonResult) taskResult).add("dex-files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

這段代碼的重點是如何對dex文件進行靜態分析的

private void countDex(RandomAccessFile dexFile) throws IOException {
        classInternalMethod.clear();
        classExternalMethod.clear();
        pkgInternalRefMethod.clear();
        pkgExternalMethod.clear();
        DexData dexData = new DexData(dexFile);
        //加載dex數據
        dexData.load();
        MethodRef[] methodRefs = dexData.getMethodRefs();
        ClassRef[] externalClassRefs = dexData.getExternalReferences();
        //獲取混淆的Class maping規則
        Map<String, String> proguardClassMap = config.getProguardClassMap();
        String className = null;
        for (ClassRef classRef : externalClassRefs) {
            className = ApkUtil.getNormalClassName(classRef.getName());
            if (proguardClassMap.containsKey(className)) {
                //混淆前的原始className
                className = proguardClassMap.get(className);
            }
            if (className.indexOf('.') == -1) {
                continue;
            }
            classExternalMethod.put(className, 0);
        }
        for (MethodRef methodRef : methodRefs) {
            className = ApkUtil.getNormalClassName(methodRef.getDeclClassName());
            if (proguardClassMap.containsKey(className)) {
                className = proguardClassMap.get(className);
            }
            if (!Util.isNullOrNil(className)) {
                if (className.indexOf('.') == -1) {
                    continue;
                }
                if (classExternalMethod.containsKey(className)) {
                    classExternalMethod.put(className, classExternalMethod.get(className) + 1);
                } else if (classInternalMethod.containsKey(className)) {
                    classInternalMethod.put(className, classInternalMethod.get(className) + 1);
                } else {
                    classInternalMethod.put(className, 1);
                }
            }
        }

        //remove 0-method referenced class
        Iterator<String> iterator = classExternalMethod.keySet().iterator();
        while (iterator.hasNext()) {
            if (classExternalMethod.get(iterator.next()) == 0) {
                iterator.remove();
            }
        }
    }
複製代碼

理解上述代碼以前，先介紹下dex文件格式。 dex文件可分爲Header部分、String索引表、類型索引表、方法原型索引表、字段索引表、方法索引表、類定義、Data數據區。

Header部分

String索引表

類型索引表

方法原型索引表

字段索引表

方法索引表

類定義

經過二進制工具，大概講解了dex的文件格式。再回過頭看代碼，代碼中有一個classInternalMethod和classExternalMethod的區別；首先在解析TypeId的時候會有一個internal字段表示這個類型是否認義在這個dex文件內；

/** * Holds the contents of a type_id_item. * * This is chiefly a list of indices into the string table. We need * some additional bits of data, such as whether or not the type ID * represents a class defined in this DEX, so we use an object for * each instead of a simple integer. (Could use a parallel array, but * since this is a desktop app it's not essential.) */
    static class TypeIdItem {
        public int descriptorIdx;       // index into string_ids

        public boolean internal;        // defined within this DEX file?
    }
複製代碼

internal字段的賦值操做以下：

/**
     * Sets the "internal" flag on type IDs which are defined in the
     * DEX file or within the VM (e.g. primitive classes and arrays).
     */
    void markInternalClasses() {
        for (int i = mClassDefs.length - 1; i >= 0; i--) {
            mTypeIds[mClassDefs[i].classIdx].internal = true;
        }

        for (int i = 0; i < mTypeIds.length; i++) {
            String className = mStrings[mTypeIds[i].descriptorIdx];

            if (className.length() == 1) {
                // primitive class
                mTypeIds[i].internal = true;
            } else if (className.charAt(0) == '[') {
                mTypeIds[i].internal = true;
            }

            //System.out.println(i + " " +
            //    (mTypeIds[i].internal ? "INTERNAL" : "external") + " - " +
            //    mStrings[mTypeIds[i].descriptorIdx]);
        }
    }
複製代碼

在ClassDef中定義的類型都屬於internal，同時轉換後的className長度爲1的類型(基礎數據類型)也屬於interal，最後數組類型的也屬於internal。

classInternalMethod和classExternalMethod的具體劃分規則以下：

private void countDex(RandomAccessFile dexFile) throws IOException {
       ...
       ...
        for (ClassRef classRef : externalClassRefs) {
            className = ApkUtil.getNormalClassName(classRef.getName());
            if (proguardClassMap.containsKey(className)) {
                className = proguardClassMap.get(className);
            }
            if (className.indexOf('.') == -1) {
                continue;
            }
            //將類定義不在本dex文件中的類名加入map
            classExternalMethod.put(className, 0);
        }
        for (MethodRef methodRef : methodRefs) {
            className = ApkUtil.getNormalClassName(methodRef.getDeclClassName());
            if (proguardClassMap.containsKey(className)) {
                className = proguardClassMap.get(className);
            }
            if (!Util.isNullOrNil(className)) {
                if (className.indexOf('.') == -1) {
                    continue;
                }
                //根據類名加入不一樣的分類
                if (classExternalMethod.containsKey(className)) {
                    classExternalMethod.put(className, classExternalMethod.get(className) + 1);
                } else if (classInternalMethod.containsKey(className)) {
                    classInternalMethod.put(className, classInternalMethod.get(className) + 1);
                } else {
                    classInternalMethod.put(className, 1);
                }
            }
        }

        //remove 0-method referenced class
        Iterator<String> iterator = classExternalMethod.keySet().iterator();
        while (iterator.hasNext()) {
            if (classExternalMethod.get(iterator.next()) == 0) {
                iterator.remove();
            }
        }
    }
複製代碼

ResProguardCheckTask

目的：判斷apk是否執行了資源混淆。

@Override
    public TaskResult call() throws TaskExecuteException {
        File resDir = new File(inputFile, ApkConstants.RESOURCE_DIR_PROGUARD_NAME);
            ...
            if (resDir.exists() && resDir.isDirectory()) {
                Log.d(TAG, "find resource directory " + resDir.getAbsolutePath());
                //有名爲r的文件夾，執行了支援混淆
                ((TaskJsonResult) taskResult).add("hasResProguard", true);
            } else {
                resDir = new File(inputFile, ApkConstants.RESOURCE_DIR_NAME);
                if (resDir.exists() && resDir.isDirectory()) {
                    File[] dirs = resDir.listFiles();
                    boolean hasProguard = true;
                    for (File dir : dirs) {
                        //任意文件夾不符合資源混淆的命名規則，則未執行資源混淆
                        if (dir.isDirectory() && !fileNamePattern.matcher(dir.getName()).matches()) {
                            hasProguard = false;
                            Log.i(TAG, "directory " + dir.getName() + " has a non-proguard name!");
                            break;
                        }
                    }
                    ((TaskJsonResult) taskResult).add("hasResProguard", hasProguard);
             ...
    }
複製代碼

FindNonAlphaPngTask

目的：檢測出沒有透明度的png文件（應該使用jpg替換，佔用空間會更小）

private void findNonAlphaPng(File file) throws IOException {
        if (file != null) {
            if (file.isDirectory()) {
                File[] files = file.listFiles();
                for (File tempFile : files) {
                    findNonAlphaPng(tempFile);
                }
            } else if (file.isFile() && file.getName().endsWith(ApkConstants.PNG_FILE_SUFFIX) && !file.getName().endsWith(ApkConstants.NINE_PNG)) {
                BufferedImage bufferedImage = ImageIO.read(file);
                //沒有alpha信息
                if (!bufferedImage.getColorModel().hasAlpha()) {
                    String filename = file.getAbsolutePath().substring(inputFile.getAbsolutePath().length() + 1);
                    if (entryNameMap.containsKey(filename)) {
                        filename = entryNameMap.get(filename);
                    }
                    long size = file.length();
                    if (entrySizeMap.containsKey(filename)) {
                        size = entrySizeMap.get(filename).getFirst();
                    }
                    if (size >= downLimitSize * ApkConstants.K1024) {
                        nonAlphaPngList.add(Pair.of(filename, file.length()));
                    }
                }
            }
        }
    }
複製代碼

MultiLibCheckTask

目的：檢測lib文件夾中是否有多文件夾存在。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(getType(), TASK_RESULT_TYPE_JSON, config);
            if (taskResult == null) {
                return null;
            }
            long startTime = System.currentTimeMillis();
            JsonArray jsonArray = new JsonArray();
            if (libDir.exists() && libDir.isDirectory()) {
                File[] dirs = libDir.listFiles();
                for (File dir : dirs) {
                    if (dir.isDirectory()) {
                        jsonArray.add(dir.getName());
                    }
                }
            }
            ((TaskJsonResult) taskResult).add("lib-dirs", jsonArray);
            if (jsonArray.size() > 1) {
                ((TaskJsonResult) taskResult).add("multi-lib", true);
            } else {
                ((TaskJsonResult) taskResult).add("multi-lib", false);
            }
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

UncompressedFileTask

目的：比對apk壓縮包裏每個entry的壓縮後大小、壓縮前大小；若大小同樣，則表示文件未壓縮。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            ...
            if (!entrySizeMap.isEmpty()) {                                                          //take advantage of the result of UnzipTask.
                for (Map.Entry<String, Pair<Long, Long>> entry : entrySizeMap.entrySet()) {
                    final String suffix = getSuffix(entry.getKey());
                    Pair<Long, Long> size = entry.getValue();
                    if (filterSuffix.isEmpty() || filterSuffix.contains(suffix)) {
                        if (!uncompressSizeMap.containsKey(suffix)) {
                            uncompressSizeMap.put(suffix, size.getFirst());
                        } else {
                            uncompressSizeMap.put(suffix, uncompressSizeMap.get(suffix) + size.getFirst());
                        }
                        if (!compressSizeMap.containsKey(suffix)) {
                            compressSizeMap.put(suffix, size.getSecond());
                        } else {
                            compressSizeMap.put(suffix, compressSizeMap.get(suffix) + size.getSecond());
                        }
                    } else {
// Log.d(TAG, "file: %s, filter by suffix.", entry.getKey());
                    }
                }
            }

            for (String suffix : uncompressSizeMap.keySet()) {
            //大小比對
                if (uncompressSizeMap.get(suffix).equals(compressSizeMap.get(suffix))) {
                    JsonObject fileItem = new JsonObject();
                    fileItem.addProperty("suffix", suffix);
                    fileItem.addProperty("total-size", uncompressSizeMap.get(suffix));
                    jsonArray.add(fileItem);
                }
            }
            ((TaskJsonResult) taskResult).add("files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

CountRTask

目的：統計R文件數量。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            Map<String, String> classProguardMap = config.getProguardClassMap();
            for (RandomAccessFile dexFile : dexFileList) {
                DexData dexData = new DexData(dexFile);
                dexData.load();
                ClassRef[] defClassRefs = dexData.getInternalReferences();
                for (ClassRef classRef : defClassRefs) {
                    String className = ApkUtil.getNormalClassName(classRef.getName());
                    if (classProguardMap.containsKey(className)) {
                        className = classProguardMap.get(className);
                    }
                    //去掉內部類
                    String pureClassName = getOuterClassName(className);
                    //識別R文件
                    if (pureClassName.endsWith(".R") || "R".equals(pureClassName)) {
                        if (!classesMap.containsKey(pureClassName)) {
                            classesMap.put(pureClassName, classRef.getFieldArray().length);
                        } else {
                            classesMap.put(pureClassName, classesMap.get(pureClassName) + classRef.getFieldArray().length);
                        }
                    }
                }
            }

            JsonArray jsonArray = new JsonArray();
            long totalSize = 0;
            Map<String, String> proguardClassMap = config.getProguardClassMap();
            for (Map.Entry<String, Integer> entry : classesMap.entrySet()) {
                JsonObject jsonObject = new JsonObject();
                if (proguardClassMap.containsKey(entry.getKey())) {
                    jsonObject.addProperty("name", proguardClassMap.get(entry.getKey()));
                } else {
                    jsonObject.addProperty("name", entry.getKey());
                }
                jsonObject.addProperty("field-count", entry.getValue());
                totalSize += entry.getValue();
                jsonArray.add(jsonObject);
            }
            ((TaskJsonResult) taskResult).add("R-count", jsonArray.size());
            ((TaskJsonResult) taskResult).add("Field-counts", totalSize);

            ((TaskJsonResult) taskResult).add("R-classes", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

DuplicateFileTask

目的：經過計算md5，判斷apk中是否存在徹底同樣的文件。

private void computeMD5(File file) throws NoSuchAlgorithmException, IOException {
        if (file != null) {
            if (file.isDirectory()) {
                File[] files = file.listFiles();
                for (File resFile : files) {
                    computeMD5(resFile);
                }
            } else {
                MessageDigest msgDigest = MessageDigest.getInstance("MD5");
                BufferedInputStream inputStream = new BufferedInputStream(new FileInputStream(file));
                byte[] buffer = new byte[512];
                int readSize = 0;
                long totalRead = 0;
                while ((readSize = inputStream.read(buffer)) > 0) {
                    msgDigest.update(buffer, 0, readSize);
                    totalRead += readSize;
                }
                inputStream.close();
                if (totalRead > 0) {
                    final String md5 = Util.byteArrayToHex(msgDigest.digest());
                    String filename = file.getAbsolutePath().substring(inputFile.getAbsolutePath().length() + 1);
                    if (entryNameMap.containsKey(filename)) {
                        filename = entryNameMap.get(filename);
                    }
                    if (!md5Map.containsKey(md5)) {
                        md5Map.put(md5, new ArrayList<String>());
                        if (entrySizeMap.containsKey(filename)) {
                            fileSizeList.add(Pair.of(md5, entrySizeMap.get(filename).getFirst()));
                        } else {
                            fileSizeList.add(Pair.of(md5, totalRead));
                        }
                    }
                    //md5相同的文件列表
                    md5Map.get(md5).add(filename);
                }
            }
        }
    }
複製代碼

@Override
    public TaskResult call() throws TaskExecuteException {
            ...
            ...
            for (Pair<String, Long> entry : fileSizeList) {
                //md5相同的文件
                if (md5Map.get(entry.getFirst()).size() > 1) {
                    JsonObject jsonObject = new JsonObject();
                    jsonObject.addProperty("md5", entry.getFirst());
                    jsonObject.addProperty("size", entry.getSecond());
                    JsonArray jsonFiles = new JsonArray();
                    for (String filename : md5Map.get(entry.getFirst())) {
                        jsonFiles.add(filename);
                    }
                    jsonObject.add("files", jsonFiles);
                    jsonArray.add(jsonObject);
                }
            }
            ((TaskJsonResult) taskResult).add("files", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
        return taskResult;
    }
複製代碼

MultiSTLCheckTask

目的：判斷so是否帶有多份stl標準庫。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            ...
            for (File libFile : libFiles) {
                if (isStlLinked(libFile)) {
                    Log.d(TAG, "lib: %s has stl link", libFile.getName());

                    jsonArray.add(libFile.getName());
                }
            }
            ((TaskJsonResult) taskResult).add("stl-lib", jsonArray);
            if (jsonArray.size() > 1) {
                ((TaskJsonResult) taskResult).add("multi-stl", true);
            } else {
                ((TaskJsonResult) taskResult).add("multi-stl", false);
            }
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

private boolean isStlLinked(File libFile) throws IOException, InterruptedException {
        ProcessBuilder processBuilder = new ProcessBuilder(toolnmPath, "-D", "-C", libFile.getAbsolutePath());
        Process process = processBuilder.start();
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
        String line = reader.readLine();
        while (line != null) {
            String[] columns = line.split(" ");
// Log.d(TAG, "%s", line);
            if (columns.length >= 3 && columns[1].equals("T") && columns[2].startsWith("std::")) {
                return true;
            }
            line = reader.readLine();
        }
        reader.close();
        process.waitFor();
        return false;
    }
複製代碼

UnusedResourcesTask

目的：檢測出在代碼、資源文件中未被引用的資源。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            readMappingTxtFile();
            readResourceTxtFile();
            //添加全部聲明的資源
            unusedResSet.addAll(resourceDefMap.values());
            Log.d(TAG, "find resource declarations %d items.", unusedResSet.size());
            //找到全部代碼中使用的資源
            decodeCode();
            Log.d(TAG, "find resource references in classes: %d items.", resourceRefSet.size());
            //找到全部資源中引用的資源
            decodeResources();
            Log.d(TAG, "find resource references %d items.", resourceRefSet.size());
            //去掉被引用的資源
            unusedResSet.removeAll(resourceRefSet);
            Log.d(TAG, "find unused references %d items", unusedResSet.size());

            JsonArray jsonArray = new JsonArray();
            for (String name : unusedResSet) {
                jsonArray.add(name);
            }
            ((TaskJsonResult) taskResult).add("unused-resources", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

private void readMappingTxtFile() throws IOException {
        // com.tencent.mm.R$string -> com.tencent.mm.R$l:
        // int fade_in_property_anim -> aRW

        if (mappingTxt != null) {
            BufferedReader bufferedReader = new BufferedReader(new FileReader(mappingTxt));
            String line = bufferedReader.readLine();
            boolean readRField = false;
            String beforeClass = "", afterClass = "";
            try {
                while (line != null) {
                    if (!line.startsWith(" ")) {
                        String[] pair = line.split("->");
                        if (pair.length == 2) {
                            beforeClass = pair[0].trim();
                            afterClass = pair[1].trim();
                            afterClass = afterClass.substring(0, afterClass.length() - 1);
                            if (!Util.isNullOrNil(beforeClass) && !Util.isNullOrNil(afterClass) && ApkUtil.isRClassName(ApkUtil.getPureClassName(beforeClass))) {
// Log.d(TAG, "before:%s,after:%s", beforeClass, afterClass);
                                readRField = true;
                            } else {
                                readRField = false;
                            }
                        } else {
                            readRField = false;
                        }
                    } else {
                        if (readRField) {
                            String[] entry = line.split("->");
                            if (entry.length == 2) {
                                String key = entry[0].trim();
                                String value = entry[1].trim();
                                if (!Util.isNullOrNil(key) && !Util.isNullOrNil(value)) {
                                    String[] field = key.split(" ");
                                    if (field.length == 2) {
// Log.d(TAG, "%s -> %s", afterClass.replace('$', '.') + "." + value, getPureClassName(beforeClass).replace('$', '.') + "." + field[1]);
                                        //添加 R.java中混淆後的全路徑field -> R.java混淆前的全路徑field
                                        rclassProguardMap.put(afterClass.replace('$', '.') + "." + value, ApkUtil.getPureClassName(beforeClass).replace('$', '.') + "." + field[1]);
                                    }
                                }
                            }
                        }
                    }
                    line = bufferedReader.readLine();
                }
            } finally {
                bufferedReader.close();
            }
        }
    }
複製代碼

private void readResourceTxtFile() throws IOException {
        //讀取R.txt
        BufferedReader bufferedReader = new BufferedReader(new FileReader(resourceTxt));
        String line = bufferedReader.readLine();
        try {
            while (line != null) {
                String[] columns = line.split(" ");
                if (columns.length >= 4) {
                    final String resourceName = "R." + columns[1] + "." + columns[2];
                    if (!columns[0].endsWith("[]") && columns[3].startsWith("0x")) {
                        //int styleable ActionBar_title 27
                        if (columns[3].startsWith("0x01")) {
                            Log.d(TAG, "ignore system resource %s", resourceName);
                        } else {
                            final String resId = parseResourceId(columns[3]);
                            if (!Util.isNullOrNil(resId)) {
                                //資源id 資源名稱 映射
                                resourceDefMap.put(resId, resourceName);
                            }
                        }
                    } else {
                        //int[] styleable ActionMode { 0x7f030034, 0x7f030036, 0x7f030056, 0x7f0300ad, 0x7f030168, 0x7f03019e }
                        Log.d(TAG, "ignore resource %s", resourceName);
                        if (columns[0].endsWith("[]") && columns.length > 5) {
                            Set<String> attrReferences = new HashSet<String>();
                            for (int i = 4; i < columns.length; i++) {
                                if (columns[i].endsWith(",")) {
                                    attrReferences.add(columns[i].substring(0, columns[i].length() - 1));
                                } else {
                                    attrReferences.add(columns[i]);
                                }
                            }
                            //style映射
                            styleableMap.put(resourceName, attrReferences);
                        }
                    }
                }
                line = bufferedReader.readLine();
            }
        } finally {
            bufferedReader.close();
        }
    }
複製代碼

解析dex文件中的smali代碼：

private void decodeCode() throws IOException {
        for (String dexFileName : dexFileNameList) {
            DexBackedDexFile dexFile = DexFileFactory.loadDexFile(new File(inputFile, dexFileName), Opcodes.forApi(15));

            BaksmaliOptions options = new BaksmaliOptions();
            List<? extends ClassDef> classDefs = Ordering.natural().sortedCopy(dexFile.getClasses());

            for (ClassDef classDef : classDefs) {
                String[] lines = ApkUtil.disassembleClass(classDef, options);
                if (lines != null) {
                    readSmaliLines(lines);
                }
            }

        }
    }
複製代碼

private void readSmaliLines(String[] lines) {
        if (lines == null) {
            return;
        }
        for (String line : lines) {
            line = line.trim();
            if (!Util.isNullOrNil(line)) {
                if (line.startsWith("const")) {
                    String[] columns = line.split(",");
                    if (columns.length == 2) {
                        final String resId = parseResourceId(columns[1].trim());
                        //從id獲取資源名
                        if (!Util.isNullOrNil(resId) && resourceDefMap.containsKey(resId)) {
                            resourceRefSet.add(resourceDefMap.get(resId));
                        }
                    }
                } else if (line.startsWith("sget")) {
                    String[] columns = line.split(" ");
                    if (columns.length == 3) {
                        //獲取資源名稱
                        final String resourceRef = parseResourceNameFromProguard(columns[2]);
                        if (!Util.isNullOrNil(resourceRef)) {
                            //Log.d(TAG, "find resource reference %s", resourceRef);
                            if (styleableMap.containsKey(resourceRef)) {
                                //reference of R.styleable.XXX
                                for (String attr : styleableMap.get(resourceRef)) {
                                    resourceRefSet.add(resourceDefMap.get(attr));
                                }
                            } else {
                                resourceRefSet.add(resourceRef);
                            }
                        }
                    }
                }
            }
        }
    }
複製代碼

private String parseResourceNameFromProguard(String entry) {
        if (!Util.isNullOrNil(entry)) {
            // sget v6, Lcom/tencent/mm/R$string;->chatting_long_click_menu_revoke_msg:I
            // sget v1, Lcom/tencent/mm/libmmui/R$id;->property_anim:I
            // sput-object v0, Lcom/tencent/mm/plugin_welab_api/R$styleable;->ActionBar:[I
            // const v6, 0x7f0c0061
            String[] columns = entry.split("->");
            if (columns.length == 2) {
                int index = columns[1].indexOf(':');
                if (index >= 0) {
                    final String className = ApkUtil.getNormalClassName(columns[0]);
                    final String fieldName = columns[1].substring(0, index);
                    if (!rclassProguardMap.isEmpty()) {
                        String resource = className.replace('$', '.') + "." + fieldName;
                        if (rclassProguardMap.containsKey(resource)) {
                            return rclassProguardMap.get(resource);
                        } else {
                            return "";
                        }
                    } else {
                        if (ApkUtil.isRClassName(ApkUtil.getPureClassName(className))) {
                            return (ApkUtil.getPureClassName(className) + "." + fieldName).replace('$', '.');
                        }
                    }
                }
            }
        }
        return "";
    }
複製代碼

UnusedAssetsTask

目的：檢測出apk中未被使用的asset資源（代碼實現僅覆蓋了字符串常量的狀況，會有遺留）。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            File assetDir = new File(inputFile, ApkConstants.ASSETS_DIR_NAME);
            //找到全部asset文件
            findAssetsFile(assetDir);
            generateAssetsSet(assetDir.getAbsolutePath());
            Log.d(TAG, "find all assets count: %d", assetsPathSet.size());
            //解析代碼中的asset引用
            decodeCode();
            Log.d(TAG, "find reference assets count: %d", assetRefSet.size());
            //移除被引用的資源
            assetsPathSet.removeAll(assetRefSet);
            JsonArray jsonArray = new JsonArray();
            for (String name : assetsPathSet) {
                jsonArray.add(name);
            }
            ((TaskJsonResult) taskResult).add("unused-assets", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

private void generateAssetsSet(String rootPath) {
        HashSet<String> relativeAssetsSet = new HashSet<String>();
        for (String path : assetsPathSet) {
            int index = path.indexOf(rootPath);
            if (index >= 0) {
                String relativePath = path.substring(index + rootPath.length() + 1);
                //Log.d(TAG, "assets %s", relativePath);
                relativeAssetsSet.add(relativePath);
                if (ignoreAsset(relativePath)) {
                    Log.d(TAG, "ignore assets %s", relativePath);
                    //獲取asset使用時的相對路徑
                    assetRefSet.add(relativePath);
                }
            }
        }
        assetsPathSet.clear();
        assetsPathSet.addAll(relativeAssetsSet);
    }
複製代碼

private void readSmaliLines(String[] lines) {
        if (lines == null) {
            return;
        }
        for (String line : lines) {
            line = line.trim();
            // invoke-virtual {p0}, Lcom/ss/android/alog/App;->getAssets()Landroid/content/res/AssetManager;

            //move-result-object v1

            //const-string v2, "video"

            //invoke-virtual {v1, v2}, Landroid/content/res/AssetManager;->open(Ljava/lang/String;)Ljava/io/InputStream;
            //:try_end_13
            //.catch Ljava/io/IOException; {:try_start_a .. :try_end_13} :catch_1a
            //這個const-string判斷不是很完善，只能判斷寫死的值
            if (!Util.isNullOrNil(line) && line.startsWith("const-string")) {
                String[] columns = line.split(",");
                if (columns.length == 2) {
                    String assetFileName = columns[1].trim();
                    assetFileName = assetFileName.substring(1, assetFileName.length() - 1);
                    if (!Util.isNullOrNil(assetFileName)) {
                        //再判斷這個常量是否在asset文件名集合中
                        for (String path : assetsPathSet) {
                            if (path.endsWith(assetFileName)) {
                                assetRefSet.add(path);
                            }
                        }
                    }
                }
            }
        }
    }
複製代碼

UnStrippedSoCheckTask

目的：檢測出apk中未裁剪的so。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            ...
            if (libDir.exists() && libDir.isDirectory()) {
                File[] dirs = libDir.listFiles();
                for (File dir : dirs) {
                    if (dir.isDirectory()) {
                        File[] libs = dir.listFiles();
                        for (File libFile : libs) {
                            if (libFile.isFile() && libFile.getName().endsWith(ApkConstants.DYNAMIC_LIB_FILE_SUFFIX)) {
                                libFiles.add(libFile);
                            }
                        }
                    }
                }
            }
            for (File libFile : libFiles) {
                //判斷是否裁剪
                if (!isSoStripped(libFile)) {
                    Log.d(TAG, "lib: %s is not stripped", libFile.getName());

                    jsonArray.add(libFile.getName());
                }
            }
            ((TaskJsonResult) taskResult).add("unstripped-lib", jsonArray);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼

經過命令行判斷so是否被裁剪

private boolean isSoStripped(File libFile) throws IOException, InterruptedException {
        ProcessBuilder processBuilder = new ProcessBuilder(toolnmPath, libFile.getAbsolutePath());
        Process process = processBuilder.start();
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
        String line = reader.readLine();
        if (!Util.isNullOrNil(line)) {
            //Log.d(TAG, "%s", line);
            String[] columns = line.split(":");
            if (columns.length == 3 && columns[2].trim().equalsIgnoreCase("no symbols")) {
                return true;
            }
        }
        reader.close();
        process.waitFor();
        return false;
    }
複製代碼

CountClassTask

目的：統計類的數量。

@Override
    public TaskResult call() throws TaskExecuteException {
        try {
            TaskResult taskResult = TaskResultFactory.factory(type, TaskResultFactory.TASK_RESULT_TYPE_JSON, config);
            long startTime = System.currentTimeMillis();
            Map<String, String> classProguardMap = config.getProguardClassMap();
            JsonArray dexFiles = new JsonArray();

            for (int i = 0; i < dexFileList.size(); i++) {
                RandomAccessFile dexFile = dexFileList.get(i);
                DexData dexData = new DexData(dexFile);
                dexData.load();
                ClassRef[] defClassRefs = dexData.getInternalReferences();
                Set<String> classNameSet = new HashSet<>();
                for (ClassRef classRef : defClassRefs) {
                    String className = ApkUtil.getNormalClassName(classRef.getName());
                    if (classProguardMap.containsKey(className)) {
                        className = classProguardMap.get(className);
                    }
                    if (className.indexOf('.') == -1) {
                        continue;
                    }
                    classNameSet.add(className);
                }
                JsonObject jsonObject = new JsonObject();
                jsonObject.addProperty("dex-file", dexFileNameList.get(i));
                //Log.d(TAG, "dex %s, classes %s", dexFileNameList.get(i), classNameSet.toString());

                Map<String, Set<String>> packageClass = new HashMap<>();
                if (JobConstants.GROUP_PACKAGE.equals(group)) {
                    String packageName = "";
                    for (String clazzName : classNameSet) {
                        packageName = ApkUtil.getPackageName(clazzName);
                        if (!Util.isNullOrNil(packageName)) {
                            if (!packageClass.containsKey(packageName)) {
                                packageClass.put(packageName, new HashSet<String>());
                            }
                            //按package聚合
                            packageClass.get(packageName).add(clazzName);
                        }
                    }
                    JsonArray packages = new JsonArray();
                    for (Map.Entry<String, Set<String>> pkg : packageClass.entrySet()) {
                        JsonObject pkgObj = new JsonObject();
                        pkgObj.addProperty("package", pkg.getKey());
                        JsonArray classArray = new JsonArray();
                        for (String clazz : pkg.getValue()) {
                            classArray.add(clazz);
                        }
                        //單個package下的全部class
                        pkgObj.add("classes", classArray);
                        packages.add(pkgObj);
                    }
                    jsonObject.add("packages", packages);
                }
                dexFiles.add(jsonObject);
            }

            ((TaskJsonResult) taskResult).add("dex-files", dexFiles);
            taskResult.setStartTime(startTime);
            taskResult.setEndTime(System.currentTimeMillis());
            return taskResult;
        } catch (Exception e) {
            throw new TaskExecuteException(e.getMessage(), e);
        }
    }
複製代碼