# Test Modes Lint's unit testing machinery has special support for “test modes”, where it repeats a unit test under different conditions and makes sure the test continues to pass with the same test results -- the same warnings in the same test files. There are a number of built-in test modes: * Test modes which make small tweaks to the source files which should be compatible, such as * Inserting unnecessary parentheses * Replacing imported symbols with fully qualified names * Replacing imported symbols with Kotlin import aliases * Replacing types with typealiases * Reordering Kotlin named arguments * Replacing simple functions with Kotlin expression bodies * etc * A partial analysis test mode which runs the tests in “partial analysis” mode; two phases, analysis and reporting, with minSdkVersion set to 1 during analysis and set to the true test value during reporting etc. * Bytecode Only: Any test files that specify both source and bytecode will only use the bytecode * Source Only: Any test files that specify both source and bytecode will only use the source code These are built-in test modes which will be applied to all detector tests, but you can opt out of any test modes by invoking the `skipTestModes` DSL method, as described below. You can also add in your own test modes. For example, lint adds its own internal test mode for making sure the built-in annotation checks work with Android platform annotations in the following test mode: [](https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:lint/libs/lint-tests/src/test/java/com/android/tools/lint/checks/AndroidPlatformAnnotationsTestMode.kt) ## How to debug Let's say you have a test failure in a particular test mode, such as `TestMode.PARENTHESIZED` which inserts unnecessary parentheses into the source code to make sure detectors are properly skipping through `UParenthesizedExpression` nodes. If you just run this under the debugger, lint will run through all the test modes as usual, which means you'll need to skip through a lot of intermediate breakpoint hits. For these scenarios, it's helpful to limit the test run to **only** the target test mode. To do this, go and specify that specific test mode as part of the run setup by adding the following method declaration into your detector class: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin override fun lint(): TestLintTask { return super.lint().testModes(TestMode.PARENTHESIZED) } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now when you run your test, it will run *only* this test mode, so you can set breakpoints and start debugging through the scenario without having to figure out which mode you're currently being invoked in. !!! Warning Don't forget to remove that override when you're done! ## Handling Intentional Failures There are cases where your lint check is doing something very particular related to the changes made by the test mode which means that the test mode doesn't really apply. For example, there is a test mode which adds unnecessary parentheses, to make sure that the detector is properly handling the case where there are intermediate parenthesis nodes in the AST. Normally, every lint check should behave the same whether or not optional parentheses are present. But, if the lint check you are writing is actually parenthesis specific, such as suggesting removal of optional parentheses, then obviously in that case you don't want to apply this test mode. To do this, there's a special test DSL method you can add, `skipTestModes`. Adding a comment for why that particular mode is skipped is useful as well. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin lint().files(...) .allowCompilationErrors() // When running multiple passes of lint each pass will warn // about the obsolete lint checks; that's fine .skipTestModes(TestMode.PARTIAL) .run() .expectClean() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ## Source-Modifying Test Modes The most powerful test modes are those that make some deliberate transformations to your source code, to test variations of the code patterns that may appear in the wild. Examples of this include having optional parentheses, or fully qualified names. Lint will make these transformations, then run your tests on the modified sources, and make sure the results are the same -- except for the part of the output which shows the relevant source code, since that part is expected to differ due to the modifications. When lint finds a failure, it will abort with a diff that includes not just the different error output between the default mode and the source modifying mode, but the source files as well; that makes it easier to spot what the difference is. In the following screenshot for example we've run a failing test inside IntelliJ, and have then clicked on the Show Difference link in the test output window (Ctrl+D or Cmd-D) which shows the test failure diff nicely: ![Figure [fqn]: Screenshot of test failure](fully-qualified-error.png) This is a test mode which converts all symbols to fully qualified names; in addition to the labeled output at the top we can see the diffs in the test case files below the error output diff. The test files include line numbers to help make it easy to correlate extra or missing warnings with their line numbers to the changed source code. ### Fully Qualified Names The `TestMode.FULLY_QUALIFIED` test mode will rewrite the source files such that all symbols that it can resolve are replaced with fully qualified names. For example, this mode will convert the following code: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin import android.widget.RemoteViews fun test(packageName: String, other: Any) { val rv = RemoteViews(packageName, R.layout.test) val ov = other as RemoteViews } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ to ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin import android.widget.RemoteViews fun test(packageName: String, other: Any) { val rv = android.widget.RemoteViews(packageName, R.layout.test) val ov = other as android.widget.RemoteViews } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This makes sure that your detector handles not only the case where a symbol appears in its normal imported state, but also when it is fully qualified in the code, perhaps because there is a different competing class of the same name. This will typically catch cases where the code is incorrectly just comparing the identifier at the call node instead of the fully qualified name. For example, one detector's tests failed in this mode because it was looking to identify references to an `EnumSet`, and the code looked like this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin private fun checkEnumSet(node: UCallExpression) { val receiver = node.receiver if (receiver is USimpleNameReferenceExpression && receiver.identifier == "EnumSet" ) { ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ which will work for code such as `EnumSet.of()` but not `java.util.EnumSet.of()`. Instead, use something like this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin private fun checkEnumSet(node: UCallExpression) { val targetClass = node.resolve()?.containingClass?.qualifiedName ?: return if (targetClass == "java.util.EnumSet") { ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As with all the source transforming test modes, there are cases where it doesn't apply. For example, lint had a built-in check for camera EXIF metadata, encouraging you to import the androidx version of the library instead of using the built-in version. If it sees you using the platform one it will normally encourage you to import the androidx one instead: ``` src/test/pkg/ExifUsage.java:9: Warning: Avoid using android.media.ExifInterface; use androidx.exifinterface.media.ExifInterface instead [ExifInterface] android.media.ExifInterface exif = new android.media.ExifInterface(path); --------------------------- ``` However, if you explicitly (via fully qualified imports) reference the platform one, in that case the lint check does not issue any warnings since it figures you're deliberately trying to use the older version. And in this test mode, the results between the two obviously differ, and that's fine; as usual we'll deliberately turn off the check in this detector: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~java @Override protected TestLintTask lint() { // This lint check deliberately treats fully qualified imports // differently (they are interpreted as a deliberate usage of // the discouraged API) so the fully qualified equivalence test // does not apply: return super.lint().skipTestModes(TestMode.FULLY_QUALIFIED); } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!! Tip In Kotlin code, you may have code that checks to see if a node is a `UCallExpression`. But note that if a call is fully qualified, the node will be a `UQualifiedReferenceExpression` instead, and you'll need to look at its selector. So watch out for code which does something like `node as? UCallExpression`. ### Import Aliasing In Kotlin, you can create an import alias, which lets you refer to the imported class using an entirely different name. This test mode will create import aliases for all the import statements in the file and will replace all the references to the import aliases instead. This makes sure that the detector handles the equivalent Kotlin code. For example, this mode will convert the following code: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin import android.widget.RemoteViews fun test(packageName: String, other: Any) { val rv = RemoteViews(packageName, R.layout.test) val ov = other as RemoteViews } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ to ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin import android.widget.RemoteViews as IMPORT_ALIAS_1_REMOTEVIEWS fun test(packageName: String, other: Any) { val rv = IMPORT_ALIAS_1_REMOTEVIEWS(packageName, R.layout.test) val ov = other as IMPORT_ALIAS_1_REMOTEVIEWS } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ### Type Aliasing Kotlin also lets you alias types using the `typealias` keyword. This test mode is similar to import aliasing, but applied to all types. In addition to the different AST representations of import aliases and type aliases, they apply to different things. For example, if we import TreeMap, and we have a code reference such as `TreeMap`, then the import alias will alias the tree map class itself, and the reference would look like `IMPORT_ALIAS_1`, whereas for type aliases, the alias would be for the whole `TreeMap`, and the code reference would be `TYPE_ALIAS_1`. Also, import aliases will only apply to the explicitly imported classes, whereas type aliases will apply to all types, including Int, Boolean, List, etc. For example, this mode will convert the following code: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin import android.widget.RemoteViews fun test(packageName: String, other: Any) { val rv = RemoteViews(packageName, R.layout.test) val ov = other as RemoteViews } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ to ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin import android.widget.RemoteViews fun test(packageName: TYPE_ALIAS_1, other: TYPE_ALIAS_2) { val rv = RemoteViews(packageName, R.layout.test) val ov = other as TYPE_ALIAS_3 } typealias TYPE_ALIAS_1 = String typealias TYPE_ALIAS_2 = Any typealias TYPE_ALIAS_3 = RemoteViews ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ### Parenthesis Mode Kotlin and Java code is allowed to contain extra clarifying parentheses. Sometimes these are leftovers from earlier more complicated expressions where when the expression was simplified the parentheses were left in place. In UAST, parentheses are represented in the AST (via a `UParenthesizedExpression` node). While this is good since it allows you to for example write lint checks which identifies unnecessary parentheses, it introduces a complication: you can't just look at a node's parent to for example see if it's a `UQualifiedExpression`; you have to be prepared to look “through” it such that if it's a `UParenthesizedExpression` node, you instead look at its parent in turn. (And programmers can of course put as (((many unnecessary))) parentheses as they want, so you may have to skip through repeated nodes.) Note also that this isn't just for looking upwards or outwards at parents. Let's say you're looking at a call and you want to see if the last argument is a literal expression such as a number or a String. You *can't* just use `if (call.valueArguments.lastOrNull() is ULiteralExpression)`, because that first argument could be a `UParenthesizedExpression`, as in `call(1, true, ("hello"))`, so you'd need to look inside the parentheses. UAST comes with two functions to help you handle this correctly: * Whenever you look at the parent, make sure you surround the call with `skipParenthesizedExprUp(UExpression)`. * If you are looking at a child node, use the method `skipParenthesizedExprDown`, an extension method on UExpression (and from Java import it from UastUtils). To help catch these bugs, lint has a special test mode where it inserts various redundant parentheses in your test code, and then makes sure that the same errors are reported. The error output will of course potentially vary slightly (since the source code snippets shown will contain extra parentheses), but the test will ignore these differences and only fail if it sees new errors reported or expected errors not reported. In the unlikely event that your lint check is actually doing something parenthesis specific, you can turn off this test mode using `.skipTestModes(TestMode.PARENTHESIZED)`. For example, this mode will convert the following code: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin (t as? String)?.plus("other")?.get(0)?.dec()?.inc() "foo".chars().allMatch { it.dec() > 0 }.toString() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ to ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin (((((t as? String))?.plus("other"))?.get(0))?.dec())?.inc() (("foo".chars()).allMatch { (it.dec() > 0) }).toString() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ By default the parenthesis mode limits itself to “likely” unnecessary parentheses; in particular, it won't put extra parenthesis around simple literals, like (1) or (false). You can explicitly construct `ParenthesizedTestMode(includeUnlikely=true)` if you want additional parentheses. ### Argument Reordering In Kotlin, with named parameters you're allowed to pass in the arguments in any order. To handle this correctly, detectors should never just line up parameters and arguments and match them by index; instead, there's a `computeArgumentMapping` method on `JavaEvaluator` which returns a map from argument to parameter. The argument-reordering test mode will locate all calls to Kotlin methods, and it will then first add argument names to any parameter not already specifying a name, and then it will shift all the arguments around, then repeat the test. This will catch any detectors which were incorrectly making assumptions about argument order. (Note that the test mode will not touch methods that have vararg parameters for now.) For example, this mode will convert the following code: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin test("test", 5, true) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ to ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin test(n = 5, z = true, s = "test") ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ### Body Removal In Kotlin, you can replace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin fun test(): List { return if (true) listOf("hello") else emptyList() } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ with ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin fun test(): List = if (true) listOf("hello") else emptyList() ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note that these two ASTs do not look the same; we'll only have an `UReturnExpression` node in the first case. Therefore, you have to be careful if your detector is just visiting `UReturnExpression`s in order to find exit points. The body removal test mode will identify all scenarios where it can replace a simple function declaration with an expression body, and will make sure that the test results are the same, to make sure detectors are handling both AST variations. It also does one more thing: it toggled optional braces from if expressions -- converting ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin if (x < y) { test(x+1) } else test(x+2) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ to ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin if (x < y) test(x+1) else { test(x+2) } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (Here it has removed the braces around the if-then body since they are optional, and it has added braces around the if-else body since it did not have optional braces.) The purpose of these tweaks are similar to the expression body change: making sure that detectors are properly handling the presence of absence of `UBlockExpression` around the child nodes. ### If to When Replacement In Kotlin, you can replace a series of `if`/`else` statements with a single `when` block. These two alternative do not look the same in the AST; `if` expressions show up as `UIfExpression`, and `when` expressions show up as `USwitchExpression`. The if-to-when test mode will change all the `if` statements in Kotlin lint tests with the corresponding when statement, and makes sure that the test results remain the same. This ensures that detectors are properly looking for both `UIfExpression` and `USwitchExpression` and handling each. When this test mode was introduced, around 12 unit tests in lint's built-in checks (spread across 5 detectors) needed some tweaks. ### Whitespace Mode This test mode inserts a number of “unnecessary” whitespace characters in valid places in the source code. This helps catch bugs where lint checks are improperly making assumptions about whitespace in the source file, particularly in quickfix implementations, or when for example looking up a qualified expression and just taking the `asSourceString()` or `text` property of a PSI element or PSI type and checking it for equality with something like `java.util.List`. For example, some of the built-in checks which performed quickfix string replacements based on regular expression matching had to be updated to be prepared for whitespace characters: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~diff +++ b/lint/libs/lint-checks/src/main/java/com/android/tools/lint/checks/WakelockDetector.java @@ -454,7 +454,7 @@ public class WakelockDetector extends Detector implements ClassScanner, SourceCo LintFix fix = fix().name("Set timeout to 10 minutes") .replace() - .pattern("acquire\\(()\\)") + .pattern("acquire\\s*\\(()\\s*\\)") .with("10*60*1000L /*10 minutes*/") .build(); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ### CDATA Mode When declaring string resources, you may want to use XML CDATA sections instead of plain text. For example, instead of ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~xml <?xml version="1.0" encoding="UTF-8"?> <resources> <string name="app_name">Application Name</string> </resources> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ you can equivalently use ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~xml <?xml version="1.0" encoding="UTF-8"?> <resources> <string name="app_name"><![CDATA[Application Name]]></string> </resources> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (where you can place newlines and other unescaped text inside the bracketed span.) This alternative form shows up differently in the XML DOM that is provided to lint detectors; in particular, if you are iterating through the `Node` children of an `Element`, you should not just look at nodes with `nodeType == Node.TEXT_NODE`; you need to also handle `noteType == Node.CDATA_SECTION_NODE`. This test mode will automatically retry all your tests that define string resources, and will convert regular text into `CDATA` and makes sure the results continue to be the same. ### Suppressible Mode Users should be able to ignore lint warnings by inserting suppress annotations (in Kotlin and Java), and via `tools:ignore` attributes in XML files. This normally works for simple checks, but if you are combining results from different parts of the code, or for example caching locations and reporting them later, this is sometimes broken. This test mode looks at the reported warnings from your unit tests, and then for each one, it looks up the corresponding error location's source file, and inserts a suppress directive at the nearest applicable location. It then re-runs the analysis, and makes sure that the warning no longer appears. ### @JvmOverloads Test Mode When UAST comes across a method like this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin @JvmOverloads fun test(parameter: Int = 0) { implementation() } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ it will “inline” these two methods in the AST, such that we see the whole method body twice: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin fun test() { implementation() } fun test(parameter: Int) { implementation() } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If there were additional default parameters, there would be additional repetitions. This is similar to what the compiler does, since Java doesn't have default arguments, but the compiler will actually just generate some trampoline code to jump to the implementation with all the parameters; it will NOT repeat the method implementation: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~kotlin fun test(parameter: Int) { implementation() } // $FF: synthetic method fun `test$default`(var0: Int, var1: Int, var2: Any?) { var var0 = var0 if ((var1 and 1) != 0) { var0 = 0 } test(var0) } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Again, UAST will instead just repeat the method body. And this means lint detectors may trigger repeatedly on the same code. In most cases this will result in duplicated warnings. But it can also lead to other problems; for example, a lint check which makes sure you don't have any code duplication would incorrectly believe code fragments are repeated. Lint already looks for this situation and avoids visiting duplicated methods in its shared implementations (which is dispatching to most `Detector` callbacks). However, if you manually visit a class yourself, you can run into this problem. This test mode simulates this situation by finding all methods where it can safely add at least one default parameter, and marks it @JvmOverloaded. It then makes sure the results are the same as before.