android 兼容性測試 CTS 測試過程（實踐測試驗證經過）

時間 2020-02-18

標籤 android 兼容性測試 cts 過程實踐驗證經過欄目 Android 简体版

原文原文鏈接

寫這個博客的時候是爲了記憶，建議你們仍是看官方的說明，官方說的很清楚，不想把官方大段大段的拷貝到這裏，官方的的確說的很清楚：html

http://source.android.com/compatibility/overview.html 左邊的相關的幾個連接
java

中文說明：具體的也能夠見http://source.android.com/compatibility/overview.html中有關Program goalslinux

1、Android的CTS測試，英文爲Compatibility Test Suite，意爲兼容性測試。只有經過CTS測試的設備纔有可能得到android的商標和享受Android Market的權限；Android的CTS目的與意義：用戶在android系統中有更好的用戶體驗，而且展現android應用的優越性，使得android開發者更容易編寫高質量的andorid程序。android

2、CTS是兼容性測試,Google爲了防止廠商對Android的改動影響其SDKAPI的兼容性,即第三方應用程序安裝到該廠商的機器上都能正常運行；這個差很少是自動跑的，會測試硬git

件（好比GPS,WIFI），還有其餘一系列的東西，跑完以後會出一份詳細的報告，告訴你哪些地方沒經過。shell

3、咱們實際使用CTS的過程當中，極可能須要根據特定的要求，來定製本身的TestPlan。這時就須要本身編譯CTSubuntu

官方說明api

1、有關CTS的目的和意圖說明：http://source.android.com/faqs.html#compatibility安全

Compatibility Test Suite

What is the purpose of the CTS?

The Compatibility Test Suite is a tool used by device manufacturers to help ensure their devices are compatible, and to report test results for validations. The CTS is intended to be run frequently by OEMs throughout the engineering process to catch compatibility issues early.session

What kinds of things does the CTS test?

The CTS currently tests that all of the supported Android strong-typed APIs are present and behave correctly. It also tests other non-API system behaviors such as application lifecycle and performance. We plan to add support in future CTS versions to test "soft" APIs such as Intents as well.

2、官方說明：http://source.android.com/compatibility/overview.html

CTS是一個免費的，商用級的測試套件，可供下載，CTS運行於臺式機的直接鏈接的設備或仿真器，並執行測試用例。 CTS是一套設計集成到平常工做流程的建築設備工程師（如經過持續構建系統）中的單元測試。它的目的是揭示不兼容的早期，並確保該軟件仍然是整個開發過程當中兼容。

Compatibility Test Suite (CTS)

The CTS is a free, commercial-grade test suite, available for download. The CTS represents the "mechanism" of compatibility.

The CTS runs on a desktop machine and executes test cases directly on attached devices or an emulator. The CTS is a set of unit tests designed to be integrated into the daily workflow (such as via a continuous build system) of the engineers building a device. Its intent is to reveal incompatibilities early on, and ensure that the software remains compatible throughout the development process.

一、查看ubuntu版本

pateo@pateo-B86N53X:/$ uname -a
Linux pateo-B86N53X 2.6.35-22-generic #33-Ubuntu SMP Sun Sep 19 20:34:50 UTC 2010 i686 GNU/Linux
備註說明： uname -m 若是出現i386 i686 i586則是32位若是出現amd64 則是64位系統；android官方給出說明froyo/2.2及之後版本默認只支持64-bit
解決方法：針對你的機器是32位的操做系統
步驟一
./external/clearsilver/cgi/Android.mk
./external/clearsilver/java-jni/Android.mk
./external/clearsilver/util/Android.mk
./external/clearsilver/cs/Android.mk
四個文件中的
LOCAL_CFLAGS += -m64
LOCAL_LDFLAGS += -m64
註釋掉，或者將「64」換成「32」
LOCAL_CFLAGS += -m32
LOCAL_LDFLAGS += -m32
步驟二
./build/core/main.mk 中的
ifneq (64,$(findstring 64,$(build_arch)))
改成：

ifneq (i686,$(findstring i686,$(build_arch)))

二、安裝jdk 官方：http://source.android.com/source/initializing.html

Installing the JDK

The Sun JDK is no longer in Ubuntu's main package repository. In order to download it, you need to add the appropriate repository and indicate to the system which JDK should be used.

Java 6: for Gingerbread and newer

$ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
$ sudo apt-get update
$ sudo apt-get install sun-java6-jdk

三、切換jdk

root@pateo-B86N53X:~# sudo update-alternatives --config java
要維持當前值[*]請按回車鍵，或者鍵入選擇的編號：2
root@pateo-B86N53X:~# sudo update-alternatives --config javac
There are 2 choices for the alternative javac (providing /usr/bin/javac).
選擇路徑優先級狀態
------------------------------------------------------------
0 /usr/lib/jvm/java-6-sun/bin/javac 63 自動模式
* 1 /usr/lib/jvm/java-1.5.0-sun/bin/javac 53 手動模式
2 /usr/lib/jvm/java-6-sun/bin/javac 63 手動模式
要維持當前值[*]請按回車鍵，或者鍵入選擇的編號：2

update-alternatives: 使用 /usr/lib/jvm/Java-6-sun/bin/javac 來提供 /usr/bin/javac (javac)，於手動模式中。

root@pateo-B86N53X:~# java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Server VM (build 20.1-b02, mixed mode)

四、編譯Code
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# . build/envsetup.sh
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# lunch 15
pateo@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0$ make

五、Build CTS
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# make cts
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0/out/host/linux-x86/cts# ls

all_cts_core_files_stamp all_cts_files_stamp android-cts android-cts.zip temp

root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0/out/host/linux-x86/cts/android-cts/repository/plans# ls
Android.xml AppSecurity.xml CTS.xml Java.xml Performance.xml RefApp.xml Signature.xml VM.xml

備註說明

（1）能夠從http://source.android.com/compatibility/downloads.html下載最新版本的Compatibility Test Suit；

（2）經過編譯Android源代碼的方式得到。在android源代碼目錄下輸入make cts命令來編譯CTS，以後會在out/host/Linux-x86/cts/下生成android-cts文件夾。

這個文件夾就是 Compatibility Test Suit。

（3）cts的測試主要須要符合Android的兼容性定義文件（CDD），CDD列舉兼容的Android設備的軟件和硬件要求。

有關CDD的官方說明：

Compatibility Definition Document (CDD)

For each release of the Android platform, a detailed Compatibility Definition Document (CDD) will be provided. The CDD represents the "policy" aspect of Android compatibility.

Android 2.3

Android 2.3 is the release of the development milestone code-named Gingerbread. Source code for Android 2.3 is found in the 'gingerbread' branch in the open-source tree.

csdn提供了Android 2.3 Compatibility Definition Document (CDD)相應的翻譯可供下載的文檔

http://download.csdn.net/detail/jianguo_liao19840726/4039034

六、Running CTS
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0/out/host/linux-x86/bin# ./cts
Listening for transport dt_socket at address: 1337
Android CTS version 2.3_r1
cts_host > ls --plan
List of plans (8 in total):
AppSecurity 沒有官方說明，猜想是針對Application安全性的測試集合
Performance 包含全部針對性能的測試，隨本版的跟新，本測試計劃也將更新
VM包含針對虛擬機的全部測試
RefApp 包含針對參考應用程序的全部測試，隨本版的跟新，本測試計劃也將更新
Signature 包含全部針對公有APIs的署名測試
Java包含全部針對Java核心library的測試
CTS 包含21000個測試，這些測試是檢驗兼容性所必須的，性能測試不包含在本計劃中，隨本版的跟新，本測試計劃也將更新

Android 包含針對androidAPIs的有所測試

備註說明：在「cts_host >」提示符下輸入命令,如下爲幾個經常使用的命令
如：start --plan CTS，start --plan plan_name --package package_name 運行一個特定的測試包，如： start --plan CTS --package android.bluetooth

官方說明：http://source.android.com/compatibility/cts-development.html cts start --plan CTS -p android.os.cts.BuildVersionTest

官方命令詳解：http://source.android.com/compatibility/downloads.html，點擊Compatibility Test Suite (CTS) User Manual會有android-cts-manual-r4.pdf文檔，此文檔裏面具體說明了命令的意思

Compatibility Test Suite Manual

The CTS user manual is applicable to any CTS version, but CTS 2.1 R2 and beyond require additional steps to run the accessibility tests.

Compatibility Test Suite (CTS) User Manual

中文說明：

CTS命令的意義，咱們必須明確Plan，Package，Testcase，Test，result_type,session的含義：

CTS將Test組合爲Testcase，Testcase再組合爲Package，最後由Package組合爲Plan。
CTS能夠執行一個Plan，一個Plan的某個Package，一個Plan的某個Test。
CTS執行的結果以Session_ID來標識一個測試結果。
CTS的結果類型result_type包含Pass，Fail，Timeout，NoExecuted四種

CTS的命令說明中文版：http://blog.csdn.net/jianguo_liao19840726/article/details/7225570

七、啓動模擬器

root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# make sdk

root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# . build/envsetup.sh
including device/fsl/imx5x/vendorsetup.sh
including device/pateo/yeagle/vendorsetup.sh
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# lunch 15
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=2.3.3
TARGET_PRODUCT=yeagle
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=GRI40
============================================
root@pateo-B86N53X:/work/project/Yeagle_turquoise_3.0# emulator
emulator: WARNING: system partition size adjusted to match p_w_picpath file (128 MB > 66 MB)

另外一種啓動：

模擬器

archermind@archermind-Lenovo-IdeaPad-Y470:/work/froyo/out/host/linux-x86/sdk/android-sdk_eng.archermind_linux-x86/tools$ emulator

備註說明：若是提示-avd 或 -data file 則能夠另外一種方式先 . build/envsetup.sh，而後lunch 1

八、啓動cts

archermind@archermind-Lenovo-IdeaPad-Y470:/work/froyo/out/host/linux-x86/cts/android-cts/tools$ gedit startcts

註釋#SDK_ROOT=NOT_CONFIGURED

增長SDK_ROOT=/work/froyo/out/host/linux-x86/sdk/android-sdk_eng.archermind_linux-x86

archermind@archermind-Lenovo-IdeaPad-Y470:/work/froyo/out/host/linux-x86/cts/android-cts/tools$ ./startcts

備註說明：若是運行成功會出現Android CTS version 2.3_r1的字樣（個人android的版本是2.3的）。若是有鏈接設備到PC上還會出現Device(設備ID)connected的字樣。這裏設備能夠是鏈接PC的android的機器，也能夠是模擬器，若是鏈接了多個設備的話需加上-d參數，後面跟上設備id來告訴CTS須要測試的設備，經過下面方法查看設備

Android CTS version 2.3_r1
cts_host >

若是你要連接真機須要以下的設置：

pateo@pateo-B86N53X:/etc/udev/rules.d$ ls
10-vboxdrv.rules 51-android.rules 51-android.rules. 70-persistent-cd.rules 70-persistent-net.rules README
pateo@pateo-B86N53X:/etc/udev/rules.d$ cat 51-android.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0666",
GROUP="plugdev"
pateo@pateo-B86N53X:/etc/udev/rules.d$ adb devices
* daemon not running. starting it now on port 5037 *
* daemon started successfully *
List of devices attached
emulator-5554 offline

Note:各品牌手機的USBVendor IDs分別見官方：http://developer.android.com/guide/developing/device.html

相應的生成的測試報告：out/host/linux-x86/cts/android-cts/repository/results/下面的testResult.xml

九、效果

你會看到不須要按鍵它也能自動測試

說明下面兩步驟是在以前需run cts以前作的，針對fail的咱們須要定位，這個時候咱們最好能看到cts的源碼，下面爲cts的源碼獲取方式：sudo Git clone https://android.googlesource.com/platform/cts/

拿到源碼後，咱們針對fail的log來進行定位,例如：

fail log： junit.framework.AssertionFailedError: expected:<1> but was:<0> at android.provider.cts.MediaStore_Images_ThumbnailsTest.testQueryExternalMiniThumbnails(MediaStore_Images_ThumbnailsTest.java:163)

找到相應的多少行進行定位

十、安裝CtsVerifier.apk

Verifier是cts的一個補充，可到download下載，其CTS Verifier主要測試api和functions，好比很差測的無需人工輸入的（如音頻質量，加速度等）

Compatibility Test Suite Verifier (CTS Verifier)

The Compatibility Test Suite Verifier (CTS Verifier) is a supplement to the Compatibility Test Suite (CTS), available for download. CTS Verifier provides tests for APIs and functions that cannot be tested on a stationary device without manual input (e.g. audio quality, accelerometer, etc).

十一、安裝CtsDelegatingAccessibilityService.apk，官方說明：http://source.android.com/compatibility/cts-intro.html

這個apk的來源

Workflow

Download the CTS.
Attach at least one device (or emulator) to your machine.
For CTS 2.1 R2 and beyond, setup your device (or emulator) to run the accessibility tests:
1. adb install -r android-cts/repository/testcases/CtsDelegatingAccessibilityService.apk
2. On the device, enable Settings > Accessibility > Accessibility > Delegating Accessibility Service
For CTS 2.3 R4 and beyond, setup your device to run the device administration tests:
1. adb install -r android-cts/repository/testcases/CtsDeviceAdmin.apk
2. On the device, enable all the android.deviceadmin.cts.* device administrators under Settings > Location & security > Select device administrators
Launch the CTS. The CTS test harness loads the test plan onto the attached devices. For each test in the test harness:

The test harness pushes a .apk file to each device, executes the test through instrumentation, and records test results.
The test harness removes the .apk file from each device.

Once all the tests are executed, you can view the test results in your browser and use the results to adjust your design. You can continue to run the CTS throughout your development process.

When you are ready, you can submit the report generated by the CTS to cts@android.com. The report is a .zip archived file that contains XML results and supplemental information such as screen captures.

十二、有關本身編寫測試用例的官方說明：http://source.android.com/compatibility/cts-development.html，

重點學習怎麼寫本身的測試用例：http://developer.android.com/guide/topics/testing/index.html

什麼都是從HelloWord開始的，測試單元測試用例也是同樣：http://developer.android.com/resources/tutorials/testing/helloandroid_test.html

其具體的api見官方http://developer.android.com/reference/android/test/InstrumentationTestRunner.html

Writing CTS Tests

CTS tests use JUnit and the Android testing APIs. Review the Testing and Instrumentation tutorial while perusing the existing tests under the cts/tests/tests directory. You will see that CTS tests mostly follow the same conventions used in other Android tests.

Since CTS runs across many production devices, the tests must follow these rules:

Must take into account varying screen sizes, orientations, and keyboard layouts.
Only use public API methods. In other words, avoid all classes, methods, and fields that are annotated with the "hide" annotation.
Avoid relying upon particular view layouts or depend on the dimensions of assets that may not be on some device.
Don't rely upon root privileges.

Test Naming and Location

Most CTS test cases target a specific class in the Android API. These tests have java package names with a cts suffix and class names with the Test suffix. Each test case consists of multiple tests, where each test usually exercises a particular API method of the API class being tested. These tests are arranged in a directory structure where tests are grouped into different categories like "widgets" and "views."

For example, the CTS test for android.widget.TextView is android.widget.cts.TextViewTest found under the cts/tests/tests/widget/src/android/widget/cts directory with its Java package name asandroid.widget.cts and its class name as TextViewTest. The TextViewTest class has a test called testSetText that exercises the "setText" method and a test named "testSetSingleLine" that calls the setSingleLinemethod. Each of those tests have @TestTargetNew annotations indicating what they cover.

Some CTS tests do not directly correspond to an API class but are placed in the most related package possible. For instance, the CTS test, android.NET.cts.ListeningPortsTest, is in the android.net.cts, because it is network related even though there is no android.Net.ListeningPorts class. You can also create a new test package if necessary. For example, there is an "android.security" test package for tests related to security. Thus, use your best judgement when adding new tests and refer to other tests as examples.

Finally, a lot of tests are annotated with @TestTargets and @TestTargetNew. These are no longer necessary so do not annotate new tests with these.

New Test Packages

When adding new tests, there may not be an existing directory to place your test. In that case, refer to the example under cts/tests/tests/example and create a new directory. Furthermore, make sure to add your new package's module name from its Android.mk to CTS_COVERAGE_TEST_CASE_LIST in cts/CtsTestCaseList.mk. This Makefile is used by build/core/tasks/cts.mk to glue all the tests together to create the final CTS package.

Test Stubs and Utilities

Some tests use additional infrastructure like separate activities and various utilities to perform tests. These are located under the cts/tests/src directory. These stubs aren't separated into separate test APKs like the tests, so the cts/tests/src directory does not have additional top level directories like "widget" or "view." Follow the same principle of putting new classes into a package with a name that correlates to the purpose of your new class. For instance, a stub activity used for testing OpenGL like GLSurfaceViewStubActivity belongs in the android.opengl.cts package under the cts/tests/src/android/opengl directory.

Other Tasks

Besides adding new tests there are other ways to contribute to CTS:

Fix or remove tests annotated with BrokenTest and KnownFailure.

Submitting Your Changes

Follow the Android Contributors' Workflow to contribute changes to CTS. A reviewer will be assigned to your change, and your change should be reviewed shortly!

1三、有關Monkey的用法網上資源不少，具體見官方：http://developer.android.com/guide/developing/tools/monkey.html

　 1)monkey是一個android自帶的命令行工具。它向系統發送僞隨機的用戶事件流，實現對正在開發的應用程序進行壓力測試。

　 2)方法
　　在設備端打開setting界面
　　 $ adb shell
# monkey -p com.android.settings -v 500
　　此時能夠看到界面不斷被切換

重點是掌握相關的Monkey的一些命令便可，在實踐中把這些命令都試試經常使用便可。

Monkey 的×××地址：http://download.csdn.net/detail/jianguo_liao19840726/4039754

有關monkeyrunner的用法：http://developer.android.com/guide/developing/tools/monkeyrunner_concepts.html

源碼中也自動了Monkey的相應的一些文件：/work/project/Yeagle_turquoise_3.0/development/cmds/monkey$目錄下的一些文件

1四、官方給予的cts技術性相關疑問的聯繫：http://source.android.com/compatibility/contact-us.html

For CTS Technical Questions

If you have specific issues with the Compatibility Test Suite that require you to disclose information you'd prefer not to be public, you can contact an email address we've set up specifically this purpose: cts@android.com. This email address is for cases that require disclosure of confidential information only, so general questions will be directed back to the public android-compatibility list. Note also that this list is for specific technical questions; general inquiries will also be directed back to the android-compatibility list.

1五、其餘開源的andorid腳本：http://code.google.com/p/android-scripting/

http://code.google.com/p/robotium/