Using Kong API Gateway with konga and Grafana Monitoring on Production

Raihan Taqi Fataya
5 min readSep 21, 2021

--

Hi, This is my second trivial stack using available open source. and this is our story to use Kong API Gateway on Production Level.

Why we use Kong ?

Begin with a third party who wants connect to our API, we can’t expose our API to Public. we uses almost private API that only our network can connect within it. Following this we create Kong as Gateway to our APIs. Also, Kong uses NGINX that we’re so familiar about it.

How to use Kong ?

There’s two way of using kong Standalone Deployment and Hybrid Deployment ( using Control Plane and Data Plane ) with database mode or not we already test both and using database mode on Production. if your APIs dont have much traffic we recommends you to use standalone mode with n1 standard since it’s the lowest possible that GCP promises 99.99 reliability. then if you have lots of APIs coming, uses Hybrid mode instead, separates data plane from control plane also it’s possible to create horizontal Autoscaling and HA. There’s also an option using Kong Gateway as Kubernetes Ingress if you considered it.

If you uses GCP don’t forget to create firewall filter that allows these ports:

  • :8000 on which Kong listens for incoming HTTP traffic from your clients, and forwards it to your upstream services.
  • :8443 on which Kong listens for incoming HTTPS traffic. This port has a similar behavior as the :8000 port, except that it expects HTTPS traffic only. This port can be disabled via the configuration file.
  • :8001 on which the Admin API used to configure Kong listens.
  • :8444 on which the Admin API listens for HTTPS traffic.

After you install Kong and bootstrap config with

kong migrations bootstrap [-c /etc/kong/kong.conf]

you need to consider to install LUA and LUAROCKS.

LUAROCKS is package manager that kong uses to install external Plugins that not available on Official Kong Plugins. to use it you must install lua first. we uses LUA5.3 and LUAROCKS to install kong-path-allow and also kong-OIDC ( OpenID Connect ) plugin that only available on Kong Enterprises, we uses it to connect our KEYCLOAK.

Simple Kong Command using Kong Admin API that will helps you

Create Service

curl -i -X POST http://localhost_admin_api:8001/services --data name=service_name --data url='your_url.com'

Create Routes on Service

curl -i -X POST http://localhost_admin_api:8001/services/service_name/routes --data 'paths[]=/service_name_paths' --data 'name=routes_name'

Create Consumers

curl -d "username=your_username&custom_id=your_custom_id" http://localhost_admin_api:8001/consumers/

Create Plugins Key Authentication on Routes with Consumers

curl -X POST http://localhost_admin_api:8001/routes/routes_name/plugins \
--data "name=key-auth" \
--data "config.key_names=apikey"
HOW TO CHECK APIKEY and CONSUMERIDcurl -X POST http://localhost_admin_api:8001/consumers/routes_name/key-authHOW TO HIT APIs WITH API KEYcurl https://localhost_admin_api:8000/routes?apikey=xxxxSAMPLEAPIKEYxxxx
  • also Basic Authentication plugins has the same steps

What now after install kong and additional plugins ?

of course we need Admin Dashboard and Monitoring Dashboard.

Kong Enterprise has Vitals, but Kong OSS have a lot of options, you can also develop your own dashboard apps if you can afford the sweat and time. In this matter we use Konga from Pantsel Konga — More than just another GUI to Kong Admin API (pantsel.github.io)

We use it for a long time and deploy it on Kubernetes.

apiVersion: apps/v1
kind: Deployment
metadata:
name: konga
namespace: your_namespace
spec:
selector:
matchLabels:
app: konga
replicas: 1
template:
metadata:
labels:
name: konga
app: konga
spec:
containers:
- name: konga
image: pantsel/konga
env:
- name: DB_ADAPTER
value: postgres
- name: DB_HOST
value: "your_db_host"
- name: DB_PORT
value: "db_port"
- name: DB_USER
value: "your_user"
- name: DB_PASSWORD
value: "your_db_password"
- name: DB_DATABASE
value: "your_database"
- name: TOKEN_SECRET
value: your_konga
- name: NODE_ENV
value: Production
ports:
- containerPort: 1338 //1338 for prods
protocol: TCP
volumeMounts:
- name: tz-jakarta
mountPath: /etc/localtime
volumes:
- name: tz-jakarta
hostPath:
path: /usr/share/zoneinfo/Asia/Jakarta
nodeSelector:
cloud.google.com/gke-nodepool: your_namespace
---
apiVersion: v1
kind: Service
metadata:
labels:
app: konga
name: konga-svc
namespace: your_namespace
spec:
ports:
- protocol: TCP
port: 443
targetPort: 1338
selector:
app: konga
type: NodePort
---
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "your_nginx_class"
nginx.ingress.kubernetes.io/whitelist-source-range: "0.0.0.0/0"
nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
name: konga
namespace: your_namespace
spec:
tls:
- hosts:
- your_host.com
secretName: your_tls
rules:
- host: your_host.com
http:
paths:
- path: /
backend:
serviceName: konga-svc
servicePort: 443

and then you can access it on your_host.com, dont forget to add your admin API url to new connection

Konga with multiple Connection to Admin APIs

How to Monitor our APIs ? unlike Kong Enterprise Dashboard has built in Vitals, we can still create separates Grafana Dashboard using Kong Incoming HTTP/HTTPS Traffic Metric to monitors.

Of course we need Prometheus Plugin to Expose metrics related to Kong and proxied Upstream services in Prometheus exposition format, which can be scraped by a Prometheus Server.

I added Prometheus Plugin Globally

curl -X POST http://localhost_admin_api:8001/plugins/ \
--data "name=prometheus"

and you can check it on localhost_admin_api:8001/metrics then scrape it using Prometheus, this is our example to scrape Prometheus Metrics on K8S and remote write to our Cortex Metrics

kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus
namespace: default
data:
prometheus.yml: |-
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'localhost_admin_api'
static_configs:
- targets: ['localhost_admin_api:8001']
metrics_path: /metrics
remote_write:
- url: https://my_cortex/api/prom/push
basic_auth:
username: my_username
password: my_password
  • if you use hybrid deployment uses status_listen = localhost:8100/metrics on Data Plane instead of Admin APIs port / Control Plane
Our Kong Dashboard, Can filter Service and Route

You can use this officialy dashboard from Kong Kong (official) dashboard for Grafana | Grafana Labs

As long you scrape correctly it should be done with Official Kong Dashboard, or you can modified or create manually using existing kong metrics.

What we lack from this stack.

How to calculate, rate and overview for Monetization. it possible to calculate from each consumers from success rate to hit APIs. but it need separate development/3rd party apps to overview for monetization.

Cheers.

As always I Aplogize for my bad English, because I live in 3rd world.

Thanks for Reading.!

--

--