Library Author Guide

Language

This document targets developers who want to publish their work as a library that other programs can depend on. The document steps through the main questions that should be answered before publishing an open source library, and shows how a typical development environment looks like.

An example of library that follows the recommendations given here is available at https://github.com/scalacenter/library-example. For the sake of conciseness, this example uses commonly chosen technologies like GitHub, Travis CI, and sbt, but alternative technologies will be mentioned and adapting the contents of this document for them should be straightforward.

Choose an Open Source License

The first step consists in choosing an open source license specifying under which conditions the library can be reused by other people. You can browse the already existing open source licenses on the opensource.org website. If you don’t know which one to pick, we suggest using the Apache License 2.0, which allows users to use (including commercial use), share, modify and redistribute (including under different terms) your work under the condition that the license and copyright notices are preserved. For the record, Scala itself is licensed with Apache 2.0.

Once you have chosen a license, apply it to your project by creating a LICENSE file in the root directory of your project with the license contents or a link to it. This file usually indicates who owns the copyright. In our example of LICENSE file, we have written that all the contributors (as per the Git log) own the copyright.

Host the Source Code

We recommend sharing the source code of your library by hosting it on a public Git hosting site such as GitHub, Bitbucket or GitLab. In our example, we use GitHub.

Your project should include a README file including a description of what the library does and some documentation (or links to the documentation).

You should take care of putting only source files under version control. For instance, artifacts generated by the build system should not be versioned. You can instruct Git to ignore such files by adding them to a .gitignore file.

In case you are using sbt, make sure your repository has a project/build.properties file indicating the sbt version to use, so that people (or tools) working on your repository will automatically use the correct sbt version.

Setup Continuous Integration

The first reason for setting up a continuous integration (CI) server is to systematically run tests on pull requests. Examples of CI servers that are free for open source projects are Travis CI, Drone or AppVeyor.

Our example uses Travis CI. To enable Travis CI on your project, go to travis-ci.org, sign up using your GitHub account, and enable your project repository. Then, add a .travis.yml file to your repository with the following content:

language: scala

Push your changes and check that Travis CI triggers a build for your repository.

Travis CI tries to guess which build tool your project uses and executes a default command to run the project tests. For instance, if your repository contains a build.sbt file in the root directory, Travis CI executes the sbt ++$TRAVIS_SCALA_VERSION test command, where the TRAVIS_SCALA_VERSION variable is, by default, set to an arbitrary Scala version (2.12.8, at the time these lines are written), which could be inconsistent with the scalaVersion defined in your build.sbt file.

To avoid this potential inconsistency, you want to use one Scala version definition as a single source of truth. For instance, the sbt-travisci plugin lets you define the Scala version in the .travis.yml file, and then forwards this version to your sbt build definition. Alternatively, you can override the default command run by Travis CI to use the Scala version defined by the scalaVersion settings of your build.

The latter approach is the one used in this guide. Override the command run by Travis CI by adding the folliwng lines to your .travis.yml file:

jobs:
  include:
    - stage: test
      script: sbt test

Travis CI will now execute the sbt test command, which uses the Scala version from the build definition.

Last, an important thing to setup is caching, to avoid the CI server to re-download your project dependencies each time it runs. For instance, in case you use sbt, you can instruct Travis CI to save the content of the ~/.ivy2/ and ~/.sbt/ directories across builds by adding the following lines to your .travis.yml file:

# These directories are cached at the end of the build
cache:
  directories:
    - $HOME/.ivy2/cache
    - $HOME/.sbt
before_cache:
  # Cleanup the cached directories to avoid unnecessary cache updates
  - rm -fv $HOME/.ivy2/.sbt.ivy.lock
  - find $HOME/.ivy2/cache -name "ivydata-*.properties" -print -delete
  - find $HOME/.sbt        -name "*.lock"               -print -delete

For reference, here is our complete .travis.yml example file.

Publish a Release

Most build tools resolve third-party dependencies by looking them up on public repositories such as Maven Central or Bintray. These repositories host the library binaries as well as additional information such as the library authors, the open source license, and the dependencies of the library itself. Each release of a library is identified by a groupId, an artifactId, and a version number. For instance, consider the following dependency (written in sbt’s syntax):

"org.slf4j" % "slf4j-simple" % "1.7.25"

Its groupId is org.slf4j, its artifactId is slf4j-simple, and its version is 1.7.25.

In this document, we show how to publish the Maven Central repository. This process requires having a Sonatype account and a PGP key pair to sign the binaries.

Create a Sonatype Account and Project

Follow the instructions given on the OSSRH Guide to create a new Sonatype account (unless you already have one) and to create a new project ticket. This latter step is where you define the groupId that you will release to. You can use a domain name that you already own, otherwise a common practice is to use io.github.(username) (where (username) is replaced with your GitHub username).

This step has to be performed only once per groupId you want to have.

Create a PGP Key Pair

Sonatype requires that you sign the published files with PGP. Follow the instructions here to generate a key pair and to distribute your public key to a key server.

This step has to be performed only once per person.

Setup Your Project

In case you use sbt, we recommend using the sbt-sonatype and sbt-pgp plugins to publish your artifacts. Add the following dependencies to your project/plugins.sbt file:

addSbtPlugin("org.xerial.sbt" % "sbt-sonatype" % "2.4")
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "1.1.0")

And make sure your build fulfills the Sonatype requirements by defining the following settings:

// used as `artifactId`
name := "library-example"

// used as `groupId`
organization := "ch.epfl.scala"

// open source licenses that apply to the project
licenses := Seq("APL2" -> url("https://www.apache.org/licenses/LICENSE-2.0.txt"))

description := "A library that does nothing useful"

import xerial.sbt.Sonatype._
sonatypeProjectHosting := Some(GitHubHosting("scalacenter", "library-example", "julien.richard-foy@epfl.ch"))

// publish to the sonatype repository
publishTo := sonatypePublishTo.value

Put your Sonatype credentials in a $HOME/.sbt/1.0/sonatype.sbt file:

credentials += Credentials("Sonatype Nexus Repository Manager",
        "oss.sonatype.org",
        "(Sonatype user name)",
        "(Sonatype password)")

(Put your actual user name and password in place of (Sonatype user name) and (Sonatype password))

Never check this file into version control.

Last, we recommend using the sbt-dynver plugin to set the version number of your releases. Add the following dependency to your project/plugins.sbt file:

addSbtPlugin("com.dwijnand" % "sbt-dynver" % "3.1.0")

And make sure your build does not define the version setting.

Cut a Release

With this setup, the process for cutting a release is the following.

Create a Git tag whose name begins with a lowercase v followed by the version number:

$ git tag v0.1.0

This tag is used by sbt-dynver to compute the version of the release (0.1.0, in this example).

Deploy your artifact to the Central repository with the publishSigned sbt task:

$ sbt publishSigned

sbt-sonatype will package your project and ask your PGP passphrase to sign the files with your PGP key. It will then upload the files to Sonatype using your account credentials. When the task is finished, you can check the artifacts in the Nexus Repository Manager (under “Staging Repositories”).

Finally, perform the release with the sonatypeRelease sbt task:

$ sbt sonatypeRelease

Setup Continuous Publication

The release process described above has some drawbacks:

  • it requires running three commands,
  • it does not guarantee that the library is in a stable state when it is published (ie, some tests may be failing),
  • in case you work in a team, each contributor has to setup its own PGP key pair and have to have Sonatype credentials with access to the project’s groupId.

Continuous publication addresses these issues by delegating the publication process to the CI server. It works as follows: any contributor with write access to the repository can cut a release by pushing a Git tag, the CI server first checks that the tests pass and then runs the publication commands.

The remaining sections show how to setup Travis CI for continuous publication on Sonatype. You can find instructions for other CI servers and repositories in the sbt-release-early plugin documentation.

Setup the CI Server

You have to give your Sonatype account credentials to the CI server, as well as your PGP key pair. Fortunately, it is possible to securely give this information by using the secret management system of the CI server.

Export Your Sonatype Account Credentials

The SONATYPE_USERNAME and SONATYPE_PASSWORD environment variables are recognized by the sbt-sonatype plugin, as documented here.

With Travis CI, you will have to install the Travis CLI.

Then, run the following commands from your project root directory to add your Sonatype credentials as environment variables to your .travis.yml file in an encrypted form:

$ travis encrypt SONATYPE_USERNAME="(Sonatype user name)" --add
$ travis encrypt SONATYPE_PASSWORD="(Sonatype password)" --add

(Put your actual user name and password in place of (Sonatype user name) and (Sonatype password))

The --add option updates your .travis.yml file with entries like the following:

env:
  global:
    - secure: "dllL1w+pZT6yTBYwy5hX07t8r0JL5Cqer6YgYnXJ+q3OhSGUs7ul2fDUiqVxGIgUpTij1cGwBmoJOTbRk2V/be4+3Ua4ZNrAxjNF2ehqUcV5KdC3ufTTTXX0ZoL9MqEIb+GKzKtPqbzR4uly/5q5NbV7J1GeZRhummnx87POl6yH4kmXTpahig7vvnwN5dLanMshRb2Z8tO8kF4SnC31QuNBDQLnS89PEajHQu+LRAJloYvcikm+NeUj79m64CYg9JZdrHvZpIYKOMY1twT+lYoerqzG+asiNE1WrDs/We1RFVgcrKLpEThcvuIxuuPKhu24+0KteAX+7z/ulT0lndyBRfuuDjHV844LrNbjhnTB64V1uF7aEdaEZRLTsFQnFZqzpoqYqxzgfow9LN/kU5CMJX1R4wwf3YgR1VC9ZfjZnu0Pbt24g48I+72ZDNk3oRZoPsN9AtovwdZcg7TgU/iPcHNKSNhEZRP6ryhv/9aX3URLkfhnDaJmTXAnC3YCYt5cGo0FBUHARA+AHcas14Dx95bFSbH7EBivb2LiDmi44goRCWR4p+vNSBJ6Ak1NZz/+paai0pXDG6S/VdqwGSmmfjn7m9H3L5c8X5xNich9qtZbWz0fj2baZGq/doA8KE91JCzX11p/9fKNzbVivQZdsw3C3ZWDjkMZM+hl++0="

Export Your PGP Key Pair

To export your PGP key pair, you first need to know its identifier. Use the following command to list your PGP keys:

$ gpg --list-secret-keys
/home/julien/.gnupg/secring.gpg
-------------------------------
sec   2048R/BE614499 2016-08-12
uid                  Julien Richard-Foy <julien.richard-foy@epfl.ch>

In my case, I have one key pair, whose ID is BE614499.

Export your public and private keys into files, in a ci directory:

$ mkdir ci
$ gpg -a --export (key ID) > ci/pubring.asc
$ gpg -a --export-secret-keys (key ID) > ci/secring.asc

(Replace (key ID) with your key ID)

Add the ci/pubring.asc file (which contains your public key) to your repository. The secring.asc file (which contains your private key) should not be added as it is to the repository, so make sure it will be ignored by Git by adding it to the .gitignore file:

ci/secring.asc

Encrypt it with the travis tool:

$ travis encrypt-file ci/secring.asc ci/secring.asc.enc --add

As advised in the command output, make sure to add the secring.asc.enc to the git repository.

The --add option above adds a line like the following to your .travis.yml file:

before_install:
  - openssl aes-256-cbc -K $encrypted_602f530300eb_key -iv $encrypted_602f530300eb_iv -in ci/secring.asc.enc -out ci/secring.asc -d

Finally, add export your PGP passphrase to the .travis.yml file:

$ travis encrypt PGP_PASSPHRASE="(your passphrase)" --add

(Replace (your passphrase) with your actual passphrase)

Publish From the CI Server

On Travis CI, you can define a conditional stage publishing the library when a tag is pushed:

jobs:
  include:
    - stage: test
      script: sbt test
    - stage: deploy
      if: tag =~ ^v
      script: sbt publishSigned sonatypeRelease

The last step is to tell your build definition how to retrieve the PGP passphrase from the PGP_PASSPHRASE environment variable and to use the pubring.asc and secring.asc files as the PGP key pair. Include the following settings in your build.sbt file:

pgpPublicRing := file("ci/pubring.asc")
pgpSecretRing := file("ci/secring.asc")
pgpPassphrase := sys.env.get("PGP_PASSPHRASE").map(_.toArray)

Cut a Release

Just push a Git tag:

$ git tag v0.2.0
$ git push origin v0.2.0

Cross-Publish

If you have written a library, you probably want it to be usable from several Scala major versions (e.g., 2.11.x, 2.12.x, 2.13.x, etc.).

Define the versions you want to support in the crossScalaVersions setting, in your build.sbt file:

crossScalaVersions := Seq("2.12.8", "2.11.12")
scalaVersion := crossScalaVersions.value.head

The second line makes sbt use by default the first Scala version of the crossScalaVersions.

Modify the CI jobs to use all the Scala versions of your build definition by using the + prefix, when appropriate:

jobs:
  include:
    - stage: test
      script: sbt +test
    - stage: deploy
      if: tag =~ ^v
      script: sbt +publishSigned sonatypeRelease

Publish Online Documentation

An important property of documentation is that the code examples should compile and behave as they are presented. There are various ways to ensure that this property holds. One way, supported by tut and mdoc, is to actually evaluate code examples and write the result of their evaluation in the produced documentation. Another way consists in embedding snippets of source code coming from a real module or example.

The sbt-site plugin can help you organize, build and preview your documentation. It is well integrated with other sbt plugins for generating the documentation content or for publishing the resulting documentation to a web server.

Finally, a simple solution for publishing the documentation online consists in using the GitHub Pages service, which is automatically available for each GitHub repository. The sbt-ghpages plugin can automatically upload an sbt-site to GitHub Pages.

Create the Documentation Site

In this example we choose to use Paradox because it runs on the JVM and thus doesn’t require setting up another VM on your system (in contrast with most other documentation generators, which are based on Ruby, Node.js or Python).

To install Paradox and sbt-site, add the following lines to your project/plugins.sbt file:

addSbtPlugin("com.typesafe.sbt" % "sbt-site" % "1.3.2")
addSbtPlugin("com.lightbend.paradox" % "sbt-paradox" % "0.4.4")

And then add the following configuration to your build.sbt file:

enablePlugins(ParadoxPlugin, ParadoxSitePlugin)
Paradox / sourceDirectory := sourceDirectory.value / "documentation"

The ParadoxPlugin is responsible of generating the website, and the ParadoxSitePlugin provides integration with sbt-site. The second line is optional, it defines the location of the website source files. In our case, in src/documentation.

Add your documentation entry point in an src/documentation/index.md file. A typical documentation entry point uses the library name as title, shows a short sentence describing the purpose of the library, and a code snippet for adding the library to a build definition:

# Library Example

A library that does nothing.

## Setup

Add the following dependency to your `build.sbt` file:

@@@vars
~~~ scala
libraryDependencies += "ch.epfl.scala" %% "library-example" % "$project.version$"
~~~
@@@

@@@ index
* [Getting Started](getting-started.md)
* [Reference](reference.md)
@@@

Note that in our case we rely on a variable substitution mechanism to inject the correct version number in the documentation so that we don’t have to always update that part of the docs each time we publish a new release.

Our example also includes an @@@index directive, defining how the content of the documentation is organized. In our case, the documentation contains two pages, the first one provides a quick tutorial for getting familiar with the library, and the second one provides more detailed information.

The sbt-site plugin provides a convenient previewAuto task that serves the resulting documentation locally, so that you can see how it looks like, and re-generate the documentation when you edit it:

sbt:library-example> previewAuto
Embedded server listening at
  https://127.0.0.1:4000
Press any key to stop.

Browse the https://localhost:4000 URL to see the result:

Include Code Examples

This section shows two ways to make sure that code examples included in the documentation do compile and behave as they are presented.

Using a Markdown Preprocessor

One approach consists in using a Markdown preprocessor, such as tut or mdoc. These tools read your Markdown source files, search for code fences, evaluate them (throwing an error if they don’t compile), and produce a copy of your Markdown files where code fences have been updated to also include the result of evaluating the Scala expressions.

For instance, given the following src/documentation/getting-started.md file:

# Getting Started

First, start with the following import:

```tut
import ch.epfl.scala.Example
```

Then, do nothing with something:

```tut
Example.doNothing(42)
```

The tut tool will produce the following Markdown file:

# Getting Started

First, start with the following import:

```scala
scala> import ch.epfl.scala.Example
import ch.epfl.scala.Example
```

Then, do nothing with something:

```scala
scala> Example.doNothing(42)
res0: Int = 42
```

You can see that tut code fences have been replaced with scala code fences, and the result of evaluating their content is shown, as it would look like from a REPL.

To enable tut, add the following line to your project/plugins.sbt file:

addSbtPlugin("org.tpolecat" % "tut-plugin" % "0.6.10")

And apply the following changes to your build.sbt file:

+enablePlugins(TutPlugin)
-Paradox / sourceDirectory := sourceDirectory.value / "documentation"
+tutSourceDirectory := sourceDirectory.value / "documentation"
+Paradox / sourceDirectory := tutTargetDirectory.value
+makeSite := makeSite.dependsOn(tut).value

These changes add the TutPlugin, configure it to read sources from the src/documentation directory, configure Paradox to read the output of tut, and make sure tut is run before the site is built.

Embedding Snippets

Another approach consists in embedding fragments of Scala source files that are part of a module which is compiled by your build. For instance, given the following test in file src/test/ch/epfl/scala/Usage.scala:

package ch.epfl.scala

import scalaprops.{Property, Scalaprops}

object Usage extends Scalaprops {

  val testDoNothing =
// #do-nothing
    Property.forAll { x: Int =>
      Example.doNothing(x) == x
    }
// #do-nothing

}

You can embed the fragment surrounded by the #do-nothing identifiers with the @@snip Paradox directive, as shown in the src/documentation/reference.md file:

# Reference

The `doNothing` function takes anything as parameter and returns it unchanged:

@@snip [Usage.scala]($root$/src/test/scala/ch/epfl/scala/Usage.scala) { #do-nothing }

The resulting documentation looks like the following:

Include API Documentation

It can also be useful to have links to the API documentation (Scaladoc) from your documentation website.

This can be achieved by adding the following lines to your build.sbt:

enablePlugins(SiteScaladocPlugin)
SiteScaladoc / siteSubdirName := "api"
paradoxProperties += ("scaladoc.base_url" -> "api")

The SiteScaladocPlugin is provided by sbt-site and includes the API documentation to the generated website. The second line defines that the API documentation should be published at the /api base URL, and the third line makes this information available to Paradox.

You can then use the @scaladoc Paradox directive to include a link to the API documentation of a particular symbol of your library:

Browse the @scaladoc[API documentation](ch.epfl.scala.Example$) for more information.

The @scaladoc directive will produce a link to the /api/ch/epfl/scala/Example$.html page.

Publish Documentation

Add the sbt-ghpages plugin to your project/plugins.sbt:

addSbtPlugin("com.typesafe.sbt" % "sbt-ghpages" % "0.6.3")

And add the following configuration to your build.sbt:

enablePlugins(GhpagesPlugin)
git.remoteRepo := sonatypeProjectHosting.value.get.scmUrl

Create a gh-pages branch in your project repository as explained in the sbt-ghpages documentation.

Finally, publish your site by running the ghpagesPushSite sbt task:

sbt:library-example> ghpagesPushSite
[info] Cloning into '.'...
[info] [gh-pages 2e7f426] updated site
[info]  83 files changed, 8059 insertions(+)
[info]  create mode 100644 .nojekyll
[info]  create mode 100644 api/ch/epfl/index.html
…
[info] To git@github.com:scalacenter/library-example.git
[info]    2d62539..2e7f426  gh-pages -> gh-pages
[success] Total time: 9 s, completed Jan 22, 2019 10:55:15 AM

Your site should be online at https://(organization).github.io/(project). In our case, you can browse it at https://scalacenter.github.io/library-example/.

Continuous Publication

You need to grant the CI job write access to the Git repository hosting the documentation. This can be achieved by creating an SSH key that the CI job can use to push the website to GitHub.

Create an SSH key:

$ ssh-keygen -t rsa -b 4096 -C "sbt-site@travis" -f ci/travis-key

Make sure to not define a passphrase (just leave it empty and press enter), and to add the private key (the ci/travis-key file) to your .gitignore:

ci/secring.asc
ci/travis-key

Add the public key, ci/travis-key.pub, in the Deploy Keys section of your GitHub project’s settings page:

Make sure you “allow write access” by checking the box.

The private key has to be added to the repository, like we did with the PGP private key. Unfortunately, due to a limitation of Travis CI, you can not add several encrypted files. The workaround consists in creating an archive containing all the files to encrypt. In your case, you want to encrypt the PGP key and the SSH key into a single ci/secrets.tar file:

$ tar cvf ci/secrets.tar ci/secring.asc ci/travis-key
$ travis encrypt-file ci/secrets.tar ci/secrets.tar.enc --add

Make sure to add the ci/secrets.tar file to your .gitignore:

ci/secring.asc
ci/travis-key
ci/secrets.tar

Finally, update the .travis.yml file to unpack the archive and push the documentation website on releases:

jobs:
  include:
    - stage: test
      # Run tests for all Scala versions
      script: sbt +test
      name: "Tests"
      # Check that the documentation can be built
    - script: sbt makeSite
      name: "Documentation"

    - stage: deploy
      if: tag =~ ^v
      script:
        # decrypt PGP secret key and GitHub SSH key
        - openssl aes-256-cbc -K $encrypted_602f530300eb_key -iv $encrypted_602f530300eb_iv -in ci/secrets.tar.enc -out ci/secrets.tar -d
        - tar xvf ci/secrets.tar
        # load the key in the ssh-agent
        - chmod 600 ci/travis-key
        - eval "$(ssh-agent -s)"
        - ssh-add ci/travis-key
        # perform deployment
        - sbt makeSite +publishSigned sonatypeRelease ghpagesPushSite

(Replace the $encrypted_602f530300eb_key and $encrypted_602f530300eb_iv variables with the ones produced by the travis encrypt-file command)

As usual, cut a release by pushing a Git tag. The CI server will run the tests, publish the binaries and update the online documentation.

Welcome Contributors

This section gives you advice on how to make it easier to get people contributing to your project.

CONTRIBUTING.md

Add a CONTRIBUTING.md file to your repository, answering the following questions: how to build the project? What are the coding practices to follow? Where are the tests and how to run them?

For reference, you can read our minimal example of CONTRIBUTING.md file.

Issue Labels

We recommend you to label your project issues so that potential contributors can quickly see the scope of an issue (e.g., “documentation”, “core”, …), it’s level of difficulty (e.g., “good first issue”, “advanced”, …), or its priority (e.g., “blocker”, “nice to have”, …).

Code Formatting

Reviewing a pull requests where the substantial changes are diluted in code style changes can be a frustrating experience. You can avoid that problem by using a code formatter forcing all the contributors to follow a specific code style.

For instance, to use scalafmt, add the following line to your project/plugins.sbt file:

addSbtPlugin("com.geirsson" % "sbt-scalafmt" % "1.5.1")

In the CONTRIBUTING.md file, mention that you use that code formatter and encourage users to use the “format on save” feature of their editor.

In your .travis.yml file, add a first stage checking that the code has been properly formatted:

jobs:
  include:

    - stage: style
      script: sbt scalafmtCheck

Evolve

From the user point of view, upgrading to a new version of a library should be a smooth process. Possibly, it should even be a “non-event”.

Breaking changes and migration steps should be thoroughly documented, and a we recommend following the semantic versioning policy.

The MiMa tool can help you checking that you don’t break this versioning policy. Add the sbt-mima-plugin to your build with the following, in your project/plugins.sbt file:

addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.3.0")

Configure it as follow, in build.sbt:

mimaPreviousArtifacts := previousStableVersion.value.map(organization.value %% name.value % _).toSet

Last, add the following job to the “test” stage, in the .travis.yml file:

  - script: sbt mimaReportBinaryIssues
    name: "Binary compatibility"

This will check that pull requests don’t make changes that are binary incompatible with the previous stable version.

We suggest working with the following Git workflow: the master branch always receives pull requests for the next major version (so, binary compatibility checks are disabled, by setting the mimaPreviousArtifacts value to Set.empty), and each major version N has a corresponding N.x branch (e.g., 1.x, 2.x, etc.) branch where the binary compatibility checks are enabled.

Contributors to this page: