Android Pie Feature: Restrictions on use of non-SDK interfaces

今年的6月12號 David Brazdil 和 Nicolas Geoffray 在 Android Developers Blog 上 post 了一篇題名 An Update on non-SDK restrictions in Android P 的文章。大體內容是他們發現不少開發者使用 non-SDK interfaces，這致使用戶的崩潰上漲以及開發者不斷採起應急措施，Google但願從這個版本開始着手解決此問題。2月份推送的 Developer Preview 和 Beta 1 版本上會經過 Toast 或 logcat 指示使用 non-SDK interfaces，另提供 StrictMode 檢測自家應用 restrictions 的行爲。

囉嗦一下，David Brazdil 是一名 Android ART compiler developer，hiddenapi這個Feature大部分commiter是他，review 這些能夠幫助快速理解hiddenapi具體作了哪些。若是你感興趣，能夠從下圖藍色星星的提交開始閱讀向上擼。說明下，non-SDK interfaces是口頭語，hiddenapi纔是代碼層面的術語。

android-review.googlesource.com/q/owner:dbr…

鎮定一下，正式分析開始。

首先，虛擬機加載的是dex文件（oat牽扯東西太多，因而沒有它的戲份），app developer經常使用的Api好比Activity.java / View.java是通過aosp源代碼編譯生成classes.dex進而打包到framework.jar中，還好比okHttp.jar。

簡單說明下dex format，它是一種8位字節的二進制流文件，各個數據按順序緊密的排列，無間隙且整個應用中全部的Java源文件都放在一個dex中

上圖中的文件頭部分，記錄了dex文件的信息，全部字段一個分部；索引區部分，主要包含字符串、類型、方法原型、域、方法的索引；索引區最終又被存儲在數據區，其中連接數據區，主要存儲動態連接庫，so庫的信息。

下圖以./leakcanary-sample/build/intermediates/transforms/dexBuilder/debug/11/com/example/leakcanary/MainActivity.dex爲例。恰好此MainActivity.java包含了private / protected / public三個modifier的Filed / Method，public這裏指的是<init>，它表明實例的初始化方法，還有一個<cinit>表明在jvm第一次加載class文件時調用，包括靜態變量初始化語句和靜態塊的執行。

dex文件中的register字段: Dalvik 最初目標是運行在以ARM 作CPU 的機器上的，ARM 芯片的一個主要特色是寄存器多。寄存器多的話有好處，就是能夠把操做數放在寄存器裏，而不是像傳統VM 同樣放在棧中。天然，操做寄存器是比操做內存（棧嘛，其實就是一塊內存區域）快。registers 變量表示該方法運行過程當中會使用多少個寄存器。

注意到access字段了嗎？ 它表明訪問標誌，定義在如下兩個結構體中，hiddenapi思路即是從access_flag下手。出於的考慮是不增長dex size，也沒有代碼侵入dex的gen過程，透過./host/linux-x86/bin/hiddenapi從新encode一次dex文件，修改access_flag的動做 David Brazdil 稱爲encode。

// A decoded version of the field of a class_data_item
  struct ClassDataField {
    uint32_t field_idx_delta_;  // delta of index into the field_ids array for FieldId
    uint32_t access_flags_;  // access flags for the field
    ClassDataField() :  field_idx_delta_(0), access_flags_(0) {}

   private:
    DISALLOW_COPY_AND_ASSIGN(ClassDataField);
  };
  ClassDataField field_;

  // Read and decode a field from a class_data_item stream into field
  void ReadClassDataField();

  // A decoded version of the method of a class_data_item
  struct ClassDataMethod {
    uint32_t method_idx_delta_;  // delta of index into the method_ids array for MethodId
    uint32_t access_flags_;
    uint32_t code_off_;
    ClassDataMethod() : method_idx_delta_(0), access_flags_(0), code_off_(0) {}

   private:
    DISALLOW_COPY_AND_ASSIGN(ClassDataMethod);
  };
複製代碼

下面看一下hiddenapi如何encode過程， 舉例aosp編譯okhttp.jar,

aosp自7.x開始逐步遷移到Go語言構建build system，爲了兼容mk增長一層映射，看到 BUILD_JAVA_LIBRARY 很熟悉了吧。

編譯的末尾環節調用hiddenapi加工dex，執行hiddenapi --dex=xx/classes1.dex --light-greylist=? --dark-greylist=? --blacklist=?

# ./build/core/java.mk
ifneq ($(filter $(LOCAL_MODULE),$(PRODUCT_BOOT_JARS)),) # is_boot_jar
  $(eval $(call hiddenapi-copy-dex-files,$(built_dex_intermediate),$(built_dex_hiddenapi)))
  built_dex_copy_from := $(built_dex_hiddenapi)
else # !is_boot_jar
  built_dex_copy_from := $(built_dex_intermediate)
endif # is_boot_jar

# ./build/core/definitions.mk
define hiddenapi-copy-dex-files
$(2): $(1) $(HIDDENAPI) $(INTERNAL_PLATFORM_HIDDENAPI_LIGHT_GREYLIST) \
      $(INTERNAL_PLATFORM_HIDDENAPI_DARK_GREYLIST) $(INTERNAL_PLATFORM_HIDDENAPI_BLACKLIST)
	@rm -rf $(dir $(2))
	@mkdir -p $(dir $(2))
	find $(dir $(1)) -maxdepth 1 -name "classes*.dex" | sort | \
		xargs -I{} cp -f {} $(dir $(2))
	find $(dir $(2)) -name "classes*.dex" | sort | sed 's/^/--dex=/' | \
		xargs $(HIDDENAPI) --light-greylist=$(INTERNAL_PLATFORM_HIDDENAPI_LIGHT_GREYLIST) \
		                   --dark-greylist=$(INTERNAL_PLATFORM_HIDDENAPI_DARK_GREYLIST) \
		                   --blacklist=$(INTERNAL_PLATFORM_HIDDENAPI_BLACKLIST)
endef
複製代碼

light-greylist / dark-greylist / blacklist 三個文件由external/doclava解析生成（包括檢查@hiden修飾，public / private / proteced / 包級別修飾等），這三個文件有默認，在以下路徑。咱們在最開始logcat中看到的Notification#isGroupChild，它在private list中，格式嘛：Methods are encoded as: class_descriptor->method_name(parameter_types)return_type，Fields are encoded as: class_descriptor->field_name:field_type

回到encode的邏輯，它是一個binary文件，代碼在./art/tools/hiddenapi/hiddenapi.cc，處理過程很是簡單，解析參數、修改access_flag。

OpenApiFile函數作的事兒:讀取上面三個文件，OpenDexFiles函數經過ArtDexFileLoader加載dex文件並建立dexFile結構體，它對應dex format描述。

//./art/tools/hiddenapi/hiddenapi.cc
  // Paths to text files which contain the lists of API members.
  std::string light_greylist_path_;
  std::string dark_greylist_path_;
  std::string blacklist_path_;
複製代碼

重要邏輯在CategorizeAllClasses方法中，而後盯着SetHidden，修改每個dexclass最後更新Checksums。閱讀代碼時注意下access_flag在規範中指定LEB128編碼，它是一中可變長度編碼，表示任意有符號或無符號整數的。在 .dex 文件中，LEB128 僅用於對 32 位數字進行編碼。實在不懂百度一下LEB編解碼過程，很基礎的東西。

至於爲何要編碼，請閱讀《信息論》通訊專業必修課。

void CategorizeAllClasses(const DexFile& dex_file) {
    for (uint32_t class_idx = 0; class_idx < dex_file.NumClassDefs(); ++class_idx) {
      DexClass klass(dex_file, class_idx);
      const uint8_t* klass_data = klass.GetData();
      if (klass_data == nullptr) {
        continue;
      }

      for (ClassDataItemIterator it(klass.GetDexFile(), klass_data); it.HasNext(); it.Next()) {
        DexMember member(klass, it);

        // Catagorize member and overwrite its access flags.
        // Note that if a member appears on multiple API lists, it will be categorized
        // as the strictest.
        bool is_hidden = true;
        if (member.IsOnApiList(blacklist_)) {
          member.SetHidden(HiddenApiAccessFlags::kBlacklist);
        } else if (member.IsOnApiList(dark_greylist_)) {
          member.SetHidden(HiddenApiAccessFlags::kDarkGreylist);
        } else if (member.IsOnApiList(light_greylist_)) {
          member.SetHidden(HiddenApiAccessFlags::kLightGreylist);
        } else {
          member.SetHidden(HiddenApiAccessFlags::kWhitelist);
          is_hidden = false;
        }

        if (print_hidden_api_ && is_hidden) {
          std::cout << member.GetApiEntry() << std::endl;
        }
      }
    }
  }
  
  // Sets hidden bits in access flags and writes them back into the DEX in memory.
  // Note that this will not update the cached data of ClassDataItemIterator
  // until it iterates over this item again and therefore will fail a CHECK if
  // it is called multiple times on the same DexMember.
  void SetHidden(HiddenApiAccessFlags::ApiList value) {
    const uint32_t old_flags = it_.GetRawMemberAccessFlags();
    const uint32_t new_flags = HiddenApiAccessFlags::EncodeForDex(old_flags, value);
    CHECK_EQ(UnsignedLeb128Size(new_flags), UnsignedLeb128Size(old_flags));

    // Locate the LEB128-encoded access flags in class data.
    // `ptr` initially points to the next ClassData item. We iterate backwards
    // until we hit the terminating byte of the previous Leb128 value.
    const uint8_t* ptr = it_.DataPointer();
    if (it_.IsAtMethod()) {
      ptr = ReverseSearchUnsignedLeb128(ptr);
      DCHECK_EQ(DecodeUnsignedLeb128WithoutMovingCursor(ptr), it_.GetMethodCodeItemOffset());
    }
    ptr = ReverseSearchUnsignedLeb128(ptr);
    DCHECK_EQ(DecodeUnsignedLeb128WithoutMovingCursor(ptr), old_flags);

    // Overwrite the access flags.
    UpdateUnsignedLeb128(const_cast<uint8_t*>(ptr), new_flags);
  }
複製代碼

CategorizeAllClasses完事兒後，dex中每個class的成員（field / method）根據三個名單打上kBlacklist / kDarkGreylist / kLightGreylist / kWhitelist 標籤。OK，接下來是runtime。

Runtime和Art虛擬機實例指得一回事兒，天然在./art/runtime/jni_internal.cc中。找方法和找變量見FindMethodID與FindFieldID，關鍵路徑上關注ShouldBlockAccessToMember方法。

static jmethodID FindMethodID(ScopedObjectAccess& soa, jclass jni_class, const char* name, const char* sig, bool is_static) REQUIRES_SHARED(Locks::mutator_lock_) {
  ObjPtr<mirror::Class> c = EnsureInitialized(soa.Self(), soa.Decode<mirror::Class>(jni_class));
  if (c == nullptr) {
    return nullptr;
  }
  ArtMethod* method = nullptr;
  auto pointer_size = Runtime::Current()->GetClassLinker()->GetImagePointerSize();
  if (c->IsInterface()) {
    method = c->FindInterfaceMethod(name, sig, pointer_size);
  } else {
    method = c->FindClassMethod(name, sig, pointer_size);
  }
  if (method != nullptr && ShouldBlockAccessToMember(method, soa.Self())) {
    method = nullptr;
  }
  if (method == nullptr || method->IsStatic() != is_static) {
    ThrowNoSuchMethodError(soa, c, name, sig, is_static ? "static" : "non-static");
    return nullptr;
  }
  return jni::EncodeArtMethod(method);
}

static jfieldID FindFieldID(const ScopedObjectAccess& soa, jclass jni_class, const char* name, const char* sig, bool is_static) REQUIRES_SHARED(Locks::mutator_lock_) {
  StackHandleScope<2> hs(soa.Self());
  Handle<mirror::Class> c(
      hs.NewHandle(EnsureInitialized(soa.Self(), soa.Decode<mirror::Class>(jni_class))));
  if (c == nullptr) {
    return nullptr;
  }
  ArtField* field = nullptr;
  mirror::Class* field_type;
  ClassLinker* class_linker = Runtime::Current()->GetClassLinker();
  if (sig[1] != '\0') {
    Handle<mirror::ClassLoader> class_loader(hs.NewHandle(c->GetClassLoader()));
    field_type = class_linker->FindClass(soa.Self(), sig, class_loader);
  } else {
    field_type = class_linker->FindPrimitiveClass(*sig);
  }
  if (field_type == nullptr) {
    // Failed to find type from the signature of the field.
    DCHECK(soa.Self()->IsExceptionPending());
    StackHandleScope<1> hs2(soa.Self());
    Handle<mirror::Throwable> cause(hs2.NewHandle(soa.Self()->GetException()));
    soa.Self()->ClearException();
    std::string temp;
    soa.Self()->ThrowNewExceptionF("Ljava/lang/NoSuchFieldError;",
                                   "no type \"%s\" found and so no field \"%s\" "
                                   "could be found in class \"%s\" or its superclasses", sig, name,
                                   c->GetDescriptor(&temp));
    soa.Self()->GetException()->SetCause(cause.Get());
    return nullptr;
  }
  std::string temp;
  if (is_static) {
    field = mirror::Class::FindStaticField(
        soa.Self(), c.Get(), name, field_type->GetDescriptor(&temp));
  } else {
    field = c->FindInstanceField(name, field_type->GetDescriptor(&temp));
  }
  if (field != nullptr && ShouldBlockAccessToMember(field, soa.Self())) {
    field = nullptr;
  }
  if (field == nullptr) {
    soa.Self()->ThrowNewExceptionF("Ljava/lang/NoSuchFieldError;",
                                   "no \"%s\" field \"%s\" in class \"%s\" or its superclasses",
                                   sig, name, c->GetDescriptor(&temp));
    return nullptr;
  }
  return jni::EncodeArtField(field);
}
複製代碼

ShouldBlockAccessToMember這方法用來check來自Reflection / JNI / Link的restricting invoke，link還記得發生在什麼階段嗎？類的加載過程。 oat的方法調用，虛擬機加載類文件後經過蹦牀（trampoline）函數找到方法入口地址。

enum AccessMethod {
  kNone,  // internal test that does not correspond to an actual access by app
  kReflection,
  kJNI,
  kLinking,
};

template<typename T>
ALWAYS_INLINE static bool ShouldBlockAccessToMember(T* member, Thread* self)
    REQUIRES_SHARED(Locks::mutator_lock_) {
  hiddenapi::Action action = hiddenapi::GetMemberAction(
      member, self, IsCallerTrusted, hiddenapi::kJNI);
  if (action != hiddenapi::kAllow) {
    hiddenapi::NotifyHiddenApiListener(member);
  }

  return action == hiddenapi::kDeny;
}

template<typename T>
inline Action GetMemberAction(T* member,
                              Thread* self,
                              std::function<bool(Thread*)> fn_caller_is_trusted,
                              AccessMethod access_method)
    REQUIRES_SHARED(Locks::mutator_lock_) {
  DCHECK(member != nullptr);

  // Decode hidden API access flags.
  // NB Multiple threads might try to access (and overwrite) these simultaneously,
  // causing a race. We only do that if access has not been denied, so the race
  // cannot change Java semantics. We should, however, decode the access flags
  // once and use it throughout this function, otherwise we may get inconsistent
  // results, e.g. print whitelist warnings (b/78327881).
  HiddenApiAccessFlags::ApiList api_list = member->GetHiddenApiAccessFlags();

  Action action = GetActionFromAccessFlags(member->GetHiddenApiAccessFlags());
  if (action == kAllow) {
    // Nothing to do.
    return action;
  }

  // Member is hidden. Invoke `fn_caller_in_platform` and find the origin of the access.
  // This can be *very* expensive. Save it for last.
  if (fn_caller_is_trusted(self)) {
    // Caller is trusted. Exit.
    return kAllow;
  }

  // Member is hidden and caller is not in the platform.
  return detail::GetMemberActionImpl(member, api_list, action, access_method);
}
複製代碼

GetHiddenApiAccessFlags就是前面取出encode的access flag，只是此處叫HiddenApiAccessFlags。

到這裏，基本Feature的workflow講解完畢，仍然存在不少細節沒有描述，好比：

1. doclava如何分類名單；
2. 虛擬機相關都沒有覆蓋，虛擬機解決架構上兩大需求，core-java加載，GC；
3. non-SDK interfaces對system_server、system app、other app如何區別對待，ApplicationInfo / Zygote有作一些改動。

昨天從Japan返來，調整下，明天開工啦