Feorlen's Other Blog

Now that I know my YAML file works, because the generated curl command succeeds, it’s time to look at why the interactive version doesn’t.

The way this is supposed to go is that you click the little red dot on the right in your spiffy new doc section, enter your credentials, and then fill in the blanks to test right there from the documentation. Magic! But why would this fail when the similarly auto-generated curl command succeeds? It’s because the server restricts access to the resource the originating webserver is requesting. This is good for safety, but sometimes it’s what you need to do. It’s called cross-origin resource sharing (CORS) and it was mentioned in the Swagger docs. But until I ran into a problem, I wasn’t sure what I was supposed to do with this information.

Swagger UI sends an OPTIONS request to the server as the opening of a handshake that results in the resource being accessible when the subsequent actual request shows up. (That is a terrible description, hopefully someone who actually understands this can do better.) I saw those OPTIONS requests, but then nothing happened, there was no POST, and my new message in Zulip was never created.

I still can’t fully explain it, but basically some extra headers are required from the server to verify that the requesting domain should be allowed to request the desired resource. The generated sample back-end already took care of that and it wasn’t a problem. But for a real API like Zulip, in an environment not expecting what Swagger needs, it means changing something with Django. I know nothing about Django.

I did some research and found a Django app (I had no idea there was such a thing) that handled this: django-cors-headers. I installed and configured it and suddenly everything worked. The bad part is that means a 3rd party package must be installed on any Zulip server that wants to work with Swagger UI documentation.

I looked into what django-cors-headers does, but there is a lot more than just creating the necessary header. Much of it is the tests, administrative features and other things that make it a useful piece of software. If I continued to investigate I certainly could figure out the actual minimal necessary change I’d need on the server side. But why do that when there’s already a nice package available? Except not everyone has the luxury of just installing something. That is a question I will need to find an answer for before deploying this to production.

But now I have reached the end of my proof-of-concept experiment, and I have nice interactive documentation to demonstrate how my API works. There’s much work yet to cover all the endpoints and all the features, but this shows that it can be done.

The basic steps are these:

• Download the swagger-ui repo from github and drop the dist directory it contains in an appropriate location on your webserver
• Build a YAML file that describes your API, according to the OpenAPI/Swagger spec
• Update index.html to point to your file and make any other desired changes (like I did for language)
• Configure your webserver, if needed, to handle CORS correctly

This post builds on what we’ve learned already to create a working Swagger front-end that talks to a real server. (See Testing Swagger and Making Swagger work for real, part 1)

The previous example used generated server code for the back end so it worked nicely with the Swagger UI front end by design. The next step is to make it work with a real Zulip server, which does not.

For the codegen-based test, a curl command to create a new item looked like this:

curl -X POST --header 'Content-Type: application/json' 
--header 'Accept: application/json' 
-d '{ "id": 5, "name": "potato" }' 
'http://10.2.3.4:8080/api/items'

But creating a new message in Zulip looks like this:

curl -X POST 
--header 'Content-Type: application/x-www-form-urlencoded' 
--header 'Accept: application/json' 
--header 'Authorization: Basic MyAuthHash' 
-d 'type=stream&content=bar&to=test&subject=foo' 
'http://10.2.3.4:9991/api/v1/messages'

There are two differences to account for here. The first is that the Zulip endpoint is expecting url-encoded text, not JSON, in the POST data. The second is that it requires authentication. Both of these need to be correctly described in the YAML description of the API.

Building the YAML file

The store example specified that the endpoint both accepts and produces “application/json” data:

consumes:
  - application/json
produces:
  - application/json

and “in: body” in the parameter list means it expects them to be in the body of the POST in the format described by the newItem schema.

post:
  description: Creates a new item in the store.  Duplicates are allowed
  operationId: addItem
  produces:
    - application/json
  parameters:
    - name: item
      in: body
      description: Item to add to the store
      required: true
      schema:
        $ref: '#/definitions/newItem'

For Zulip, the endpoint still returns JSON, but is expecting to receive x-www-form-urlencoded data (just like an HTML form.)

consumes:
  - application/x-www-form-urlencoded
produces:
  - application/json

The individual parameters must be defined as “formData”, which doesn’t use a separate schema definition:

parameters:
  - name: type
    in: formData
    description: type of message to create
    required: true
    type: string
  - name: content
    in: formData
    description: content of message to create
    required: true
    type: string
  - name: to
    in: formData
    description: recipient of message to create
    required: true
    type: string
  - name: subject
    in: formData
    description: subject of message to create
    required: true
    type: string

Now that the incoming data is defined, we need to make it use Basic Auth to authenticate with the Zulip back end. The first part of that is to add a “securityDefinitions” section:

securityDefinitions:
  basicAuth:
    type: basic
    description: HTTP Basic Auth

and in the parameters specify that they require this security:

security:
  - basicAuth: []

Here’s the full YAML file:

swagger: '2.0'
info:
  version: '1.0.0'
  title: Sample API
  description: Some Stuff I wrote
  termsOfService: http://example.com
  contact:
    name: Feorlen
    email: nobody@example.com
    url: http://example.com
  license:
    name: Foo
    url: http://example.com
host: 10.2.3.4:9991
basePath: /api/v1
schemes:
  - http
consumes:
  - application/x-www-form-urlencoded
produces:
  - application/json
securityDefinitions:
  basicAuth:
    type: basic
    description: HTTP Basic Auth
paths:
  /messages:
    post:
      description: Creates a new Zulip message
      operationId: addMessage
      produces:
        - application/json
      parameters:
        - name: type
          in: formData
          description: type of message to create
          required: true
          type: string
        - name: content
          in: formData
          description: content of message to create
          required: true
          type: string
        - name: to
          in: formData
          description: recipient of message to create
          required: true
          type: string
        - name: subject
          in: formData
          description: subject of message to create
          required: true
          type: string
      security:
        - basicAuth: []
      responses:
        '200':
          description: message response
          schema:
            $ref: '#/definitions/messageResponse'
        default:
          description: unexpected error
          schema:
            $ref: '#/definitions/errorModel'
definitions:
  messageResponse:
    type: object
    required:
      - msg
      - result
      - id
    properties:
      msg:
        type: string
      result:
        type: string
      id:
        type: string
  errorModel:
    type: object
    required:
      - code
      - message
    properties:
      code:
        type: integer
        format: int32
      message:
        type: string

With this configuration, Swagger UI generates a curl command that can connect to the server, authenticate, and create a new post. The problem is, however, that the nice clickable demo that Swagger UI offers doesn’t.

I’ll get into that with the next post.

When we last left our intrepid explorer, there were bits of YAML all over the floor, but the shape of a functioning API doc could be seen emerging through the debris. Or at least I remember it that way. (Here’s the previous blog post.) With the basic Swagger functionality proven, the next step is to make it work in something like the desired real environment. It does, but getting there was a bit of a winding path through parts of webapp development I hadn’t looked at yet. That’s another long story, and I’ll get to it.

Before I do, I want to talk a little about file formats. As I make this project happen for real, I need to choose how I’m going to write the spec file that drives it. That doesn’t just affect me, but anyone who later needs to read or edit it. It’s not something to be taken lightly, and is a hotly contested topic.

Here’s a nice discussion illustrating the concerns: Tom Limoncelli on TOML vs JSON. (No, TOML isn’t named for Tom L., but he does find that point amusing. If you like servers, you should read his stuff sometime.)

The Swagger (or OpenAPI) specification says you can use either YAML or JSON. The Swagger editor works in YAML, so clearly they have an opinion. (You can import and export JSON.)

My choice is also YAML, for these reasons:

• It allows comments. You can annotate what something is, why you added it, or comment out work in progress.
• It is primarily whitespace-delimited. That makes it easier to visually scan than JSON’s braces, brackets, and quotes.
• That whitespace hierarchy, as annoying as it can be, is already familiar to Python developers. Zulip is mostly written in Python, so the developers maintaining it will already be accustomed to this.

Neither of these formats are really “human readable” if by that you mean “can be reliably maintained by average computer-using humans.” They can’t. They have strict format requirements and any deviation results in cryptic errors and non-obvious failures. I think YAML is better for humans than JSON, but even that isn’t saying much.

I started this experiment with YAML, and I see no reason to not stick with it. Despite much cursing in the process of building the file for the endpoint I was trying to model (and it isn’t even complete.) The tools are lousy. The failure modes are opaque. Some of that was my inexperience with the Swagger standard, but not all of it.

At one point I was making changes to my file in a text editor and pasting it into an empty Swagger editor window to see if it would validate. Because the errors didn’t make sense. Something, I’m still not sure what, introduced a tab character I couldn’t see. Between the four editors I had open, copying and pasting between them (one explicitly configured to not use tab characters) somehow it happened. Welcome to being a developer.

JSON, on the other hand, has so much visual clutter that I can’t read any non-trivial example without reformatting to add whitespace. Fans like that it’s more compact than YAML. (Whitespace characters are characters, tabs are forbidden, and more characters mean more bytes.) There are also more tools that work with it, because much of the web runs on data exchanged as JSON. But if I don’t have a reason to need maximum efficiency and performance, I’m going to choose the option that makes it easier on the human developer every time. (For the opposite end of this decision, see protocol buffers, a decidedly not human-readable format. It’s also not used for configuration files.)

So YAML it is. In the next post, I’ll talk about what I did with it.

Swagger is a language for defining an API, with an ecosystem of tools to generate both code and documentation. I’m experimenting with it to see how much I can use it to automate creating API docs for Zulip. It’s full power is in defining your API functionality in one file, and then generating both documentation and skeleton code for multiple programming languages. You still have to implement the actual behavior behind it, but it generates the structure to do that more easily.

As with most code generators, I’m not sure how useful this is going to be for an already existing system not designed with the same structure. But it’s useful enough as a doc tool if you are willing to create the spec by hand. Then you get some nice pretty pages with, in theory, the ability for 3rd party developers to test API samples right from the documentation.

My first pass is to set up some static docs, and then a basic generated API and docs with functional demos.

The simplest way to have docs is Swagger UI, a viewing front-end. I thought, from reading blog posts and tutorials from other users, that I had to modify Swagger UI itself to create docs from my own spec. That is a mess of installing a bunch of other things that I never got working. After some advice from the IRC channel, I learned I didn’t actually need to do that.

You can literally take a single directory out of the downloadable source from GitHub, add your custom YAML or JSON file defining your API, and stick it on a webserver. (For more, go to the Swagger docs and look for “Swagger UI Documentation”.) You don’t get nice interactive examples this way, but you do get a well-structured description of your API.

The default uses the Swagger pet store example, so in index.html you will need to change the line

url = "http://petstore.swagger.io/v2/swagger.json";

to be where you have your definition file. I put mine in the same directory, it can be either YAML or JSON. That’s it.

If you don’t want functioning examples (which won’t work in this simplistic demo) also change which methods are allowed for the “Try it out!” button. Remove all of them to disable the button altogether, or only some if you need to disallow testing certain types of requests.

supportedSubmitMethods: ['get', 'post', 'put', 'delete', 'patch'],

You can also specify which language you want Swagger UI to use for its static content (look in the lang directory), although I found from testing it appears you can only pick one. I haven’t figured out how to actually provide localized content in the user’s own language.

<!-- Some basic translations -->
<script src='lang/translator.js' type='text/javascript'></script>
<script src='lang/it.js' type='text/javascript'></script>

So that’s it for the super-simple option.

To generate a working API from your spec is a little more complicated, but not too much. In addition to creating your YAML or JSON file, you need to generate server code for it and install it on your own server. I had to install several additional packages, too.

For this, you need to start with Swagger Editor. You can install it locally, but I just used the web version.

I started with one of the samples available in the web version, a simplified pet store.

File-> Open Example -> petstore_simple.yaml

Edit the sample file as desired, the important things to note are the hostname and port of your server, and the base path where your API endpoint URLs will start. Here’s mine:

swagger: '2.0'
info:
  version: '1.0.0'
  title: Swagger test store
  description: A sample API that uses the swagger-2.0 specification
  termsOfService: http://feorlen.org
  contact:
    name: Feorlen
    email: foo@example.com
    url: http://swagger.io
  license:
    name: MIT
    url: http://opensource.org/licenses/MIT
host: 10.2.3.4:8080
basePath: /api
schemes:
  - http
consumes:
  - application/json
produces:
  - application/json
paths:
  /items:
    get:
      description: Returns all items from the system that the user has access to
      operationId: findItems
      produces:
        - application/json
        - application/xml
        - text/xml
        - text/html
      parameters:
        - name: tags
          in: query
          description: tags to filter by
          required: false
          type: array
          items:
            type: string
          collectionFormat: csv
        - name: limit
          in: query
          description: maximum number of results to return
          required: false
          type: integer
          format: int32
      responses:
        '200':
          description: item response
          schema:
            type: array
            items:
              $ref: '#/definitions/item'
        default:
          description: unexpected error
          schema:
            $ref: '#/definitions/errorModel'
    post:
      description: Creates a new item in the store.  Duplicates are allowed
      operationId: addItem
      produces:
        - application/json
      parameters:
        - name: item
          in: body
          description: Item to add to the store
          required: true
          schema:
            $ref: '#/definitions/newItem'
      responses:
        '200':
          description: item response
          schema:
            $ref: '#/definitions/item'
        default:
          description: unexpected error
          schema:
            $ref: '#/definitions/errorModel'
  /items/{id}:
    get:
      description: Returns a user based on a single ID, if the user does not have access to the item
      operationId: findItemById
      produces:
        - application/json
        - application/xml
        - text/xml
        - text/html
      parameters:
        - name: id
          in: path
          description: ID of item to fetch
          required: true
          type: integer
          format: int64
      responses:
        '200':
          description: item response
          schema:
            $ref: '#/definitions/item'
        default:
          description: unexpected error
          schema:
            $ref: '#/definitions/errorModel'
    delete:
      description: deletes a single item based on the ID supplied
      operationId: deleteItem
      parameters:
        - name: id
          in: path
          description: ID of item to delete
          required: true
          type: integer
          format: int64
      responses:
        '204':
          description: item deleted
        default:
          description: unexpected error
          schema:
            $ref: '#/definitions/errorModel'
definitions:
  item:
    type: object
    required:
      - id
      - name
    properties:
      id:
        type: integer
        format: int64
      name:
        type: string
      tag:
        type: string
  newItem:
    type: object
    required:
      - name
    properties:
      id:
        type: integer
        format: int64
      name:
        type: string
      tag:
        type: string
  errorModel:
    type: object
    required:
      - code
      - message
    properties:
      code:
        type: integer
        format: int32
      message:
        type: string

I edited this elsewhere and then pasted it back into the online editor. The right pane will tell you if you messed up anything required. Now generate server code based on this API definition. I’m using Python Flask, but there are many to choose from.

Generate Server -> Python Flask

Download the resulting zip file and put its contents on your server somewhere. Since I’m using port 8080, I can do this in my own home directory without running as root. (I’m making my changes on the server directly, but edit locally and then upload if you like.)

First, customize the generated code:

In app.py I added my correct host, some extra logging (“INFO” for less debugging), and changed the title. I haven’t figured out where this title is actually used, maybe I’ll find it eventually. It now looks like this:

#!/usr/bin/env python3                                                                              

import connexion
import logging

if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)
    app = connexion.App(__name__, specification_dir='./swagger/')
    app.add_api('swagger.yaml', arguments={'title': 'Sample Swagger server'})
    app.run(host='10.2.3.4', port=8080)

I also had to edit the generated implementations in controllers/default_controller.py to remove type hints because my version of Python doesn’t support them.

def add_item(item):
    return 'do some magic!'

Those are the basic changes. The code doesn’t do anything besides return a static message, but is otherwise functional. I haven’t figured out how to change the language for Swagger UI, but maybe that is possible.

Now install any necessary packages. I already have Python 3 and Flask, but I need Connexion to handle HTTP. To get that, I first have to install a version of pip that can handle packages for Python 3.

sudo apt-get install python3-pip
sudo pip3 install -U connexion

Next add execute permissions to the server script

chmod 744 app.py

and run the server

./app.py

And then the ui is available on port 8080 (note my base path is used here.)

I can interact with the api using this page, or coping the curl examples and executing them in a shell. Click on the example values to post the sample payload into the appropriate field.

curl -X GET --header 'Accept: text/html' 'http://10.2.3.4:8080/api/items'
do some magic!

curl -X POST --header 'Content-Type: applicheader 'Accept: application/json' -d '{ "id": 5, "name": "potato" }' 'http://10.2.3.4:8080/api/items'
"do some magic!"

The next step of this example is to actually implement the endpoint functionality. For my purposes, my next task is to figure out how to make the web UI work with an existing API that was not designed to use it.