6 min read
Website development as a sysadmin
by Daniel Ortiz
As a systems administrator for the past 15 years, I’m not very used to web development as this kind of work has been usually done by other colleagues in every company I’ve worked for. I wanted to create a web application myself using an external API, as an exercise to go out my comfort zone.
Some time ago I found that Marvel Comics has a REST API where you can get a lot of information about comics, characters, authors and other interesting information. This API is used in this exercise, get your own key.
My first reaction when I initially though about this problem was to start the project in python, writing the code to do the requests, authenticating to the API, parsing the responses and generating the HTML output. Everything in the same service. This is the “classical” way of developing and the code became dirty very fast as I am an inexperienced programmer. Doing the API authentication was also painful and I wanted a more generic and reusable code.
Being part of KrakenD, I decided to revisit this exercise using KrakenD to do the requests, authentication, parsing, filtering and merging the responses for me.
Check my Github repository for the code used in this post.
The backend: Adding automatic API authentication
The KrakenD API Gateway can filter the fields of all your requests’ responses using a declarative configuration in json but I also needed KrakenD to do the authentication against the Marvel API for me before doing these requests.
KrakenD supports a lot of middleware one of them being krakend-martian and, since one of the reasons for this project was to do some programming, I created a martian request modifier so each request would automatically add any data needed to authenticate against Marvel API, decoupling this custom authentication from the application.
In this case, Marvel requests need to include in every query string the following parameters:
ts: timestampapikey: the public API keyhash: an md5 of timestamp + private key + public key.
You can also see this modifier source code in github.
At the time I created this query string modifier we didn’t had plugins so I needed to recomplile KrakenD to add a custom modifier like this one.
Setting up KrakenD
To add the custom modifier to KrakenD it’s only needed to create a file like this one in the root of the KrakenD-CE repository:
marvel.go:
package main
import _ "github.com/taik0/marvelapi-martian_querystring"
and just follow the regular build steps (make prepare and make).
To configure KrakenD I needed to add an extra_config using the krakend-martian Namespace and the custom configuration for the modifier.
Check also the whitelist and mapping sections where I set a filter of some fields.
KrakenD configuration example
"endpoints": [
{
"backend": [
{
"extra_config": {
"github.com/devopsfaith/krakend-martian": {
"fifo.Group": {
"aggregateErrors": true,
"modifiers": [
{
"querystring.MarvelModifier": {
"private": "private_api_key",
"public": "public_api_key",
"scope": [
"request"
]
}
}
],
"scope": [
"request",
"response"
]
}
}
},
"host": [
"http://gateway.marvel.com"
],
"url_pattern": "/v1/public/characters?name={name}",
"whitelist": [
"data.results",
"attributionHTML"
],
"mapping": { "data": "characters"}
}
],
"endpoint": "/character/{name}",
"method": "GET"
}
],
"extra_config": {
"github_com/devopsfaith/krakend-gologging": {
"level": "INFO",
"prefix": "[KRAKEND]",
"stdout": true,
"syslog": false
}
},
"name": "My lovely gateway",
"port": 8080,
"timeout": "3s",
"version": 2
The frontend: API2HTML
To generate the frontend of the application I created some simple logic-less mustache templates and configured api2html to query the KrakenD endpoints and generate the HTML output without even the need of a web server.
In a nutshell, Api2html takes data from an API and injects it in the mustache templates so you can build websites without coding controllers, models or setting web servers. Besides being simple, the service uses the Go standard libraries and will offer a lot of throughput.
Mustache Template
The HTML template needs to live inside api2html and is used to display the frontend. It looks like this:
<html>
<head>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0-beta.3/css/bootstrap.min.css" integrity="sha384-Zug+QiDoJOrZ5t4lssLdxGhVrurbmBWopoEl+M6BdEfwnCJZtKxi1KgxUyJq13dy" crossorigin="anonymous">
</head>
<body>
{{#Data.characters.results}}
<div class="container">
<a href="/character/{{id}}"
<div class="row">
<div class="col-4">
<img class="rounded-right" src="{{thumbnail.path}}/landscape_xlarge.{{thumbnail.extension}}" alt="{{name}}">
</div>
</div>
<div class="row">
<div class="col-4">
<h5 class="mt-0">{{name}}</h5>
{{description}}
</div>
</div>
{{/Data.characters.results}}
<div class="row">
<div class="col-4">
{{{Data.attributionHTML}}}
</div>
</div>
</a>
</div>
</body>
</html>
api2html configuration
Notice that the backend URL it’s KrakenD service, not Marvel API.
{
"templates":{
"character": "character.mustache"
},
"pages":[
{
"name": "character",
"URLPattern": "/character/:character",
"BackendURLPattern": "http://localhost:8080/character/:character",
"CacheTTL": "3600s",
"Template": "character"
}
]
}
Running the service
The docker-compose will spin up both services, api2html for the frontend, and KrakenD for the backend:
13:27 $ docker-compose up
Starting krakendmarvelexample_krakend_1 ... done
Starting krakendmarvelexample_api2html_1 ... done
Attaching to krakendmarvelexample_krakend_1, krakendmarvelexample_api2html_1
krakend_1 | Parsing configuration file: /etc/krakend/krakend.json
krakend_1 | [KRAKEND] 2018/05/11 - 11:28:42.180 ▶ WARNIN building the etcd client: unable to create the etcd client: no config
krakend_1 | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
krakend_1 | - using env: export GIN_MODE=release
krakend_1 | - using code: gin.SetMode(gin.ReleaseMode)
krakend_1 |
krakend_1 | [GIN-debug] GET /__stats/ --> github.com/devopsfaith/krakend-ce/vendor/github.com/gin-gonic/gin.WrapH.func1 (4 handlers)
api2html_1 | [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
api2html_1 | - using env: export GIN_MODE=release
api2html_1 | - using code: gin.SetMode(gin.ReleaseMode)
api2html_1 |
api2html_1 | reading ./static/500 : open ./static/500: no such file or directory
api2html_1 | using the default 500 template
api2html_1 | [GIN-debug] GET /character/:character --> github.com/devopsfaith/api2html/engine.(*Handler).HandlerFunc-fm (4 handlers)
api2html_1 | handler without layout character
api2html_1 | reading ./static/404 : open ./static/404: no such file or directory
api2html_1 | using the default 404 template
api2html_1 | [GIN-debug] PUT /template/:templateName --> github.com/devopsfaith/api2html/engine.Factory.New.func1 (4 handlers)
api2html_1 | [GIN-debug] Listening and serving HTTP on :8080
The docker-compose exposes the service on the 8081 port:
curl -i http://localhost:8081/character/Wolverine
...
[GIN] 2018/05/11 - 11:29:15 | 200 | 849.266342ms | ::1 | GET /character/Wolverine
Or in your browser:
Check the documentation to see a lot more of api2html (like generating templates with templates!)
Now if you want to create your own site run this command:
docker run -it --rm -v $PWD:/tmp -w /tmp devopsfaith/api2html skel create blog
and a more complete example will be generated in example/blog.
Conclusion
One of the most important lessons I learned building this, apart from golang coding and some application building, is that you need to know the tools you are using as much as possible since we tend to over-complicate things and rebuild pieces that someone already did (exclude from this exercises/katas to understand better a technology) and knowing what a tool can and can’t do helps a lot!
So read documentation, test, play, build something, read a bit more and try again!
