SIP-52 - Binary APIs

By: Author Nicolas Stucki

History

Summary

The purpose of binary APIs is to have publicly accessible definitions in generated bytecode for definitions that are package private or protected. This proposal introduces the @publicInBinary annotation on term definitions and the -WunstableInlineAccessors linting flag.

Motivation

Provide a sound way to refer to private members in inline definitions

Currently, the compiler automatically generates accessors for references to private members in inline definitions. This scheme interacts poorly with binary compatibility. It causes the following three unsoundness in the system:

• Changing any definition from private to public is a binary incompatible change
• Changing the implementation of an inline definition can be a binary incompatible change
• Removing final from a class is a binary incompatible change

You can find more details in https://github.com/lampepfl/dotty/issues/16983

Avoid duplication of inline accessors

Ideally, private definitions should have a maximum of one inline accessor, which is not the case now. When an inline method accesses a private/protected definition that is defined outside of its class, we generate an inline in the class of the inline method. This implies that accessors might be duplicated if a private/protected definition is accessed from different classes.

Removing deprecated APIs

There is no precise mechanism to remove a deprecated method from a library without causing binary incompatibilities. We should have a straightforward way to indicate that a method is no longer publicly available but still available in the generated code for binary compatibility.

- @deprecated(...) def myOldAPI: T = ...
+ private[C] def myOldAPI: T = ...

Related to discussion in https://github.com/lightbend/mima/discussions/724.

No way to inline reference to private constructors

It is currently impossible to refer to private constructors in inline methods.

class C private()
object C:
  inline def newC: C = new C() // Implementation restriction: cannot use private constructors in inline methods

If users want to access one of those, they must write an accessor explicitly. This extra indirection is undesirable.

class C private()
object C:
  private def newCInternal: C = new C()
  inline def newC: C = newCInternal

Proposed solution

High-level overview

This proposal introduces the @publicInBinary annotation, and adds a migration path to inline methods in libraries (requiring binary compatibility).

@publicInBinary annotation

A binary API is a definition that is annotated with @publicInBinary. This annotation can be placed on def, val, lazy val, var, object, and given definitions. A binary API will be publicly available in the bytecode.

This annotation cannot be used on private/private[this] definitions. With the exception of class constructors.

Removing this annotation from a non-public definition is a binary incompatible change.

Example:

class C {
  @publicInBinary private[C] def packagePrivateAPI: Int = ...
  @publicInBinary protected def protectedAPI: Int = ...
  @publicInBinary def publicAPI: Int = ... // warn: `@publicInBinary` has no effect on public definitions
}

will generate the following bytecode signatures

public class C {
  public C();
  public int packagePrivateAPI();
  public int protectedAPI();
  public int publicAPI();
}

In the bytecode, @publicInBinary definitions will have the ACC_PUBLIC flag.

Binary API and inlining

A non-public reference in an inline method is handled as follows:

• if the reference is a @publicInBinary the reference is used;
• otherwise, an accessor is automatically generated and used.

Example:

import scala.annotation.publicInBinary
class C {
  @publicInBinary private[C] def a: Int = ...
  private[C] def b: Int = ...
  @publicInBinary protected def c: Int = ...
  protected def d: Int = ...
  inline def foo: Int = a + b + c + d
}

before inlining the compiler will generate the accessors for inlined definitions

class C {
  @publicInBinary private[C] def a: Int = ...
  private[C] def b: Int = ...
  @publicInBinary protected def c: Int = ...
  protected def d: Int = ...
  final def C$$inline$b: Int = ...
  final def C$$inline$d: Int = ...
  inline def foo: Int = a + C$$inline$b + c + C$$inline$d
}

-WunstableInlineAccessors

In addition we introduce the -WunstableInlineAccessors flag to allow libraries to detect when the compiler generates unstable accessors. The previous code would show a linter warning that looks like this:

-- [E...] Compatibility Warning: C.scala -----------------------------
  |  inline def foo: Int = a + b + c + d
  |                            ^
  |            Unstable inline accessor C$$inline$b was generated in class C.
  |
  | longer explanation available when compiling with `-explain`
-- [E...] Compatibility Warning: C.scala -----------------------------
  |  inline def foo: Int = a + b + c + d
  |                                    ^
  |            Unstable inline accessor C$$inline$d was generated in class C.
  |
  | longer explanation available when compiling with `-explain`

When an accessor is detected we can tell the user how to fix the issue. For example we could use the -explain flag to add the following details to the message.

Specification

We must add publicInBinary to the standard library.

package scala.annotation

final class publicInBinary extends scala.annotation.StaticAnnotation

@publicInBinary annotation

• Only valid on def, val, lazy val, var, object, and given.
• If a definition overrides a @publicInBinary definition, it must also be annotated with @publicInBinary.
• TASTy will contain references to non-public definitions that are out of scope but @publicInBinary. TASTy already allows those references.
• The annotated definitions will be public in the generated bytecode. Definitions should be made public as early as possible in the compiler phases, as this can remove the need to create other accessors. It should be done after we check the accessibility of references.

Inline

• Inlining will not require the generation of an inline accessor for binary APIs.
• The user will be warned if a new inline accessor is automatically generated under -WunstableInlineAccessors. The message will suggest @publicInBinary and how to fix potential incompatibilities.

Compatibility

The introduction of the @publicInBinary do not introduce any binary incompatibility.

Using references to @publicInBinary in inline code can cause binary incompatibilities. These incompatibilities are equivalent to the ones that can occur due to the unsoundness we want to fix. When migrating to binary APIs, the compiler will show the implementation of accessors that the users need to add to keep binary compatibility with pre-publicInBinary code.

Other concerns

• Tools that analyze inlined TASTy code will need to know about @publicInBinary. For example MiMa and TASTy MiMa.

Alternatives

Add a @binaryAccessor

This annotation would generate an stable accessor. This annotation could be used on private definition. It would also mitigate migration costs for library authors that have published unstable accessors.

• Implementation https://github.com/lampepfl/dotty/pull/16992

Make all private[C] part of the binary API

Currently, we already make private[C] public in the binary API but do not have the same guarantees regarding binary compatibility. For example, the following change is binary compatible but would remove the existence of the private[C] definition in the bytecode.

class C:
-  private[C] def f: T = ...

We could change the rules to make all private[C] part of binary compatible to flag such a change as binary incompatible. This would imply that all these methods can be accessed directly from inline methods without generating an accessor.

The drawback of this approach is that that we would need to force users to keep their private[C] methods even if they never used inline methods.

Related work

• Initial discussions: https://github.com/lampepfl/dotty/issues/16983
• Initial proof of concept (outdated): https://github.com/lampepfl/dotty/pull/16992
• Single annotation proof of concept: https://github.com/lampepfl/dotty/pull/18402
• Community migration analysis: Gist
• Kotlin: PublishedApi plays the same role as @publicInBinary but its interaction with (inline definitions)[https://kotlinlang.org/docs/inline-functions.html#restrictions-for-public-api-inline-functions] is stricter as they do not support automatic accessor generation.