Fintrospect

overview

Fintrospect is a Scala web-framework with an intelligent HTTP routing layer, based on the Finagle RPC framework from Twitter. Via a shared contract, it provides a simple way to implement fast web service endpoints and HTTP clients which are:

• Type-safe : auto-marshalls all request parameters/bodies into the correct types (including primitives + JSON/XML etc...)
• Auto-validating : the presence of required and optional request parameters and bodies are checked before entering service-layer code
• Auto-documenting : runtime generation of endpoint documentation such as Swagger JSON or web sitemap XML. Generates JSON Schema for example object formats to be included in these API docs.
• Uniform : reuse the same contract to define both incoming or outgoing Finagle HTTP services. This also allows extremely low effort fake servers to be created

Additionally, Fintrospect provides a number of mechanisms to leverage these routes:

• Easily build type-safe HTTP responses with a set of custom builders for a wide variety of message formats:
  • JSON: Argo, Argonaut, Circe, GSON, Jackson, Json4S, Play JSON, Spray JSON
    • Auto-marshaling of case classes instances to/from JSON (for Argonaut/Circe/Json4S/Play).
    • Implement simple PATCH/PUT endpoints of case class instances (Circe only).
  • Native implementations of XML, Plain Text, HTML, XHTML
  • MsgPack binary format
• Serve static content from the classpath or a directory
• Template View support (with Hot-Reloading) for building responses with Mustache or Handlebars
• Anonymising headers for dynamic-path based endpoints, removing all dynamic path elements. This allows, for example, calls to particular endpoints to be grouped for metric purposes. e.g. /search/author/rowling becomes /search/author/{name}
• Interacts seamlessly with other Finagle based libraries, such as Finagle OAuth2
• Utilities to help you unit-test endpoint services and write HTTP contract tests for remote dependencies

broad concepts

The main concepts in play:

• RouteSpec: defines the self-describing contract of an HTTP endpoint
• ServerRoute: the binding of a RouteSpec to a Finagle Service implementing some business-logic, to create an inbound HTTP endpoint
• RouteClient: the binding of a RouteSpec to a Finagle Service representing an HTTP client, to create a simple function for making outbound HTTP calls
• ParameterSpec: defines the acceptable format for a request parameter (Path/Query/Header/Form-field). Provides the auto-marshaling mechanic for serializing and deserializing objects to and from HTTP messages
• BodySpec: similar to ParameterSpec, but applied to the body of an HTTP message
• RouteModule: defines a set of ServerRoute instances which are grouped under a particular request path. These modules can be combined and then converted to a Finagle Service and attached to a Finagle HTTP server. Each module provides an endpoint under which it's own runtime-generated documentation can be served (eg. in Swagger format)
• ResponseSpec: defines the characteristics of a possible response generated by a ServerRoute

regarding finagle

Since Fintrospect is built on top of Finagle, it's worth acquainting yourself with its concepts, which can be found here.

&tldr; Finagle Primer

1. Finagle provides protocol-agnostic RPC and is based on Netty
2. It is mainly asynchronous and makes heavy usage of Twitter's version of Scala Futures
3. It defines uniform Service and Filter interfaces for both client and server APIs that are effectively a single method...

   Service: def apply(request : Request) : Future[Response] Filter: def apply(request : RequestIn, service : Service[RequestOut, ResponseIn]) : Future[ResponseOut] 

4. Filters can be chained together and then applied to a Service, which results in another Service. This is useful to apply layers of functionality such as caching headers, retry behavior, and timeouts.

a note on style

• The code in this guide has omitted imports that would have made it read more concisely. The sacrifices we make in the name of learning... :)

installation

Fintrospect is intentionally dependency-lite by design - other than Finagle, the core library itself only has a single non-org.scala dependency.

To activate some optional features, additional dependencies may be required - these are shown in the subsequent sections depending on the use-case.

core

Add the following lines to build.sbt - the lib is hosted in Maven Central and JCenter:

resolvers += "JCenter" at "https://jcenter.bintray.com" libraryDependencies += "io.fintrospect" %% "fintrospect-core" % "15.1.0" 

message formats

For consuming request parameters in a type-safe way or using the relevant HTTP response builders add the following:

templating

For utilizing the templating engines listed below, add the following:

defining routes

A RouteSpec() call starts to define the specification of the contract (in terms of the required parameters) and the API follows the immutable builder pattern. Apart from the path-building elements (which terminate the builder), all of the "builder-y" calls here are optional, as are the descriptive strings (used for the auto-documenting features). Here's the simplest possible REST-like example for getting all employees in a notional system:

RouteSpec().at(Method.Get) / "employee" 

Notice that the request routing in that example was completely static? If we want an example of a dynamic endpoint, such as listing all users in a particular numerically-identified department, then we can introduce a Path parameter:

RouteSpec("list all employees in a particular group").at(Method.Get) / "employee" / Path.integer("departmentId") 

... and we can do the same for Header and Query parameters; both optional and mandatory parameters are supported, as are parameters that can appear multiple times.:

RouteSpec("list all employees in a particular group") .taking(Header.optional.boolean("listOnlyActive")) .taking(Query.required.*.localDate("datesTakenAsHoliday")) .at(Method.Get) / "employee" / Path.integer("departmentId") 

Moving onto HTTP bodies - for example adding an employee via a HTTP Post and declaring the content types that we produce (although this is optional):

RouteSpec("add employee", "Insert a new employee, failing if it already exists") .producing(ContentTypes.TEXT_PLAIN) .body(Body.form(FormField.required.string("name"), FormField.required.localDate("dateOfBirth"))) .at(Method.Post) / "user" / Path.integer("departmentId") 

... or via a form submission and declaring possible responses:

RouteSpec("add user", "Insert a new employee, failing if it already exists") .body(Body.form(FormField.required.string("name"), FormField.required.localDate("dateOfBirth"))) .returning(Created -> "Employee was created") .returning(Conflict -> "Employee already exists") .at(Method.Post) / "user" / Path.integer("departmentId") 

using routes

As can be seen above, there are several stages to defining a route. Here is the complete construction lifecycle:

1. Create a RouteSpec with a name and description
2. Add details of parameters, any body, media-types and possible responses
3. Finalize the RouteSpec with a call to at(). This creates an UnboundRoute.
4. Continue to add static or dynamic Path parameters to the URL structure (creating UnboundRoute<n> instances).

Once the final UnboundRoute has been created (with all of it's Path parts declared), it represents an HTTP contract, which can then be bound to:

1. an HTTP server Service if you wish to serve that contract to other systems.
2. an HTTP client Service if you wish to consume that contract from a remote system.

server routes & modules

A RouteSpec needs to be bound to a standard Finagle Service to receive requests, in order to create a ServerRoute. Since Finagle services are very lightweight, we can create a new instance of the Service for every request, and bind the RouteSpec to a factory method which receives the dynamic Path parameters and returns the Service. Other parameters can be retrieved directly in a type-safe manner from the HTTP request by using <--() or from() method on the parameter declaration.

validation

The presence and format validity of ALL parameters which are attached to a RouteSpec is verified by Fintrospect before requests make it to this bound Service, so no validation code is required. The response returned to the client is:

• Not Found 404: if there are any Path params which are missing or invalid (all are required)
• Bad Request 400: if there are any Header, Query, or Body params are missing (required only) or invalid

simple example

val holidays = Query.required.*.localDate("datesTakenAsHoliday") val includeManagement = Header.optional.boolean("includeManagement") def findEmployeesOnHoliday(departmentId: Integer) = Service.mk[Request, Response] { request => vale holidayDates: Seq[LocalDate] = holidays <-- request val includeManagementFlag: Option[Boolean] = includeManagement <-- request val response = Response(Ok) val baseMsg = s"Everyone from department $departmentId was at work on $holidayDates" response.contentString = baseMsg + (if (includeManagementFlag.getOrElse(false)) "" else ", even the management") Future(response) } val route = ServerRoute[Request, Response] = RouteSpec() .taking(holidays) .taking(includeManagement) .at(Method.Get) / "employee" / Path.integer("departmentId") bindTo findEmployeesOnHoliday 

modules

A Module is a collection of ServerRoute that share a common URL context, which is built up from the Root object. Add the routes and then convert into a standard Finagle Service object which is then attached in the normal way to an HTTP server.

def listEmployees(): Service[Request, Response] = Service.mk(req => Future(Response())) Http.serve(":8080", RouteModule(Root / "employee") .withRoute(RouteSpec("lists all employees").at(Method.Get) bindTo listEmployees) .toService ) 

Modules with different root contexts can also be combined with one another and then converted to a Service:

RouteModule(Root / "a").andThen(RouteModule(Root / "b")).toService 

self-describing Module APIs

A big feature of the Fintrospect library is the ability to generate API documentation at runtime. This can be activated by passing in a ModuleRenderer implementation when creating the RouteModule and when this is done, a new endpoint is created at the root of the module context (this location is overridable) which serves this documentation.

Bundled with Fintrospect are:

• Swagger (1.1 and 2.0) JSON, including JSON Schema models
• A simple JSON format
• Sitemap XML format

Other implementations are pluggable by implementing the ModuleRenderer trait - see the example code for a simple XML implementation.

val service = RouteModule(Root / "employee", Swagger2dot0Json(ApiInfo("an employee discovery API", "3.0"))).toService Http.serve(":8080", new HttpFilter(Cors.UnsafePermissivePolicy).andThen(service)) 

Note above the usage of the Finagle CorsPolicy filter, which will allow the services to be called from a Swagger UI - without it, the server will reject any cross-domain requests initiated inside a browser.

security

Module routes can be secured by adding an implementation of the Security trait - this essentially provides a filter through which all requests will be passed. An ApiKey implementation is bundled with the library which returns an 401 Unauthorized HTTP response code when a request does not pass authentication.

RouteModule(Root / "employee") .securedBy(ApiKey(Header.required.string("api_key"), (key: String) => Future(key == "extremelySecretThing"))) 

client routes

A RouteSpec can also be bound to a standard Finagle HTTP client Service and then called as a function, passing in the parameters which are bound to values by using the -->() or of() method. The client marshalls the passed parameters into an HTTP request and returns a Twitter Future containing the response. Any required manipulation of the Request (such as adding timeouts or caching headers) can be done in the standard way by chaining a Filter to the client Service. Note that Content-Type headers for posted HTTP bodies is already handled by the bound Body instance.:

val employeeId = Path.integer("employeeId") val name = Query.required.string("name") val client: RouteClient = RouteSpec() .taking(name) .at(Get) / "employee" / employeeId bindToClient Http.newService("localhost:10000") val response: Future[Response] = client(employeeId --> 1, name --> "") 

super-cool feature time: reuse of HTTP contracts

Because the RouteSpec objects can be used to bind to either a Server or a Client, we can be spectacularly smug and use them on both sides of an HTTP boundary to provide a type-safe remote contract, to either:

1. Auto-generate fake HTTP server implementations for remote HTTP dependencies. In this case, defining the RouteSpec as part of an HTTP client contract and then simply reusing them as the server-side contract of a testing fake - see the /clients example code
2. Use a shared library approach to define a contract and the data objects that go across it for reuse in multiple applications, each of which import the shared library. This obviously binary-couples the applications together to a certain degree, so utmost care should be taken, backed up with sufficient CDC-style testing to ensure that the version of the contract deployed is valid on both ends.

taking advantage of auto-marshalling

controlled mode

Some of the JSON libraries (Circe, Argonaut, Json4S, Play) supported by Fintrospect support auto-marshaling of Scala Case class instances directly to JSON without any custom conversion code needing to be written. This is supported by encode() and decode() methods present on the relevant Fintrospect JsonFormat format instance (e.g. io.fintrospect.formats.Circe.JsonFormat). Generally, these are very simple to use:

case class EmailAddress(address: String) import io.circe.generic.auto._ import io.fintrospect.formats.Circe.ResponseBuilder._ Status.Ok(Circe.JsonFormat.encode(EmailAddress("dev@fintrospect.io") 

The auto-marshaling functionality of these JSON libraries requires implicit parameters in order to make it work. This requirement is echoed in the signatures of the relevant Fintrospect encode() and decode() methods, but unless you're going to be providing custom encoder/decoder instances, you can get away with just importing the relevant implicit params from the parent lib, as in the example above.

full-auto mode

Fintrospect also contains filters which allow you to abstract away the HTTP Request/Response entirely. In this example, the Circe.Filters.AutoInOut filter converts the Service[Request, Response] to a Service[EmailAddress, ReversedEmailAddress], auto-converting the case class objects in and out of the request/response. The returned status code in the Response is 200, but this is overridable:

import io.circe.generic.auto._ import io.fintrospect.formats.Circe import io.fintrospect.formats.Circe.Auto._ case class ReversedEmailAddress(sserdda: String) val domainSvc = Service.mk[EmailAddress, ReversedEmailAddress] { email => Future(ReversedEmailAddress(email.address.reverse)) } val httpSvc: Service[Request, Response] = Circe.Auto.InOut(domainSvc) 

testing

routes

Provided trait Testing introspect Route can be used to unit test your routes, as in the simple example below:

object EchoRoute { val route = RouteSpec().at(Method.Get) / Path.string("message") bindTo( (message: String) => Service.mk { req: Request => Future(PlainText.ResponseBuilder.OK(message)) }) } class EchoRouteTest extends FunSpec with Matchers with TestingFintrospectRoute { override val route = EchoRoute.route describe("Echo") { it("bounces back message") { responseFor(Request("hello")).contentString shouldBe "hello" } } } 

test http server

The TestHttpServer is convenient for attaching routes during development, or can be used as a scaffold to provide automatically generated fake servers for downstream dependencies - simply complete the stub implementation of the server-side and fire it up. This works especially well if you are utilizing custom serialization formats (such as one of the auto-marshaling JSON libraries), as there is absolutely no marshaling code required to send back objects over the wire from your stub.

val route = RouteSpec().at(Get) / "myRoute" bindTo(() => Service.mk {r => Future(Response(Status.Ok))}) new TestHttpServer(9999, route).start()