Skip to main content

Nodejs SSA learnings

nodejs-akamai-page.MD

Overview

In almost every blog when people talk about deploying something on K8S they use node mostly because setting up http server is not that easy. I wish that it could be this easy for our case, (which it wasn’t). Manly because the way we wanted the application to work, plans for SEO and multiple data pipelines for business including Amplitude for client and Pubsub for application and business metrics.
Making all these calls from nodejs was easy for developers just 1 more promise. Everything went smoothly as long as functional testing was required. We were ready to launch and somebody from SRE/Devops team asked: “Have you done load testing?”. Everything was superb up to this point where load testing results are required to make service live.

Load Testing

We started to put load to our services using apache benchmark. Very soon we realised our application is not scaling as much as we expected. To our surprise application was able to handle only 3 requests/second.
With drastic results and tight timelines, it soon became extremely difficult at various technical and emotional levels. Teams started to blame each other, stating either infra issues or code issues. It was SRE teams’ first encounter with NodeJs single server side pages which made them uncomfortable. SRE team lead and principal developer started to spend a lot of time to figure out the bottleneck.
Initial thoughts were around event loop time and CPU bound operations. Code was making some call to message strings which showed up on cpu head graph. We fixed that in a hope that everything will be alright, well it didn’t. We have to go back and look the problem at different level.

Code Proxy

Developer wrote the code keeping 2 folders client and server. Client code was supposed to build and ship to client while server code was suppose to run on Kubernetes and facilitate SEO. In order to achieve that developers took library/vendor kind of approach where common code was kept. Downside of this was API call code was also common except for the fact that server to server call destination is different from client i.e browser to server call. To solve this developers asked SRE team to provide 1 DNS for both. However if you are in GCP(GKE service) and calling another GKE service URL with public IP , RTT is as high as 150ms. If you care to access the same over private DNS RTT is 3ms.
With this in mind developers made changes to API module and made sure that it understands which context it is running at. With above change application performance increased from RPS(Request per second) 3 to 7.

Making calls is expensive

Well, compare to 3 - 7 was good, however not what we wanted. This new system was replacement of old monolith which was service Global traffic of 12k r/s. Considering that number of instances required to handle the traffic was beyond comprehension, for at least that time. We instrumented our code with NewRelic to debug deeper. Very soon we saw that DNS resolution is taking as high as 90ms for our containers.

Solution

Since nodejs delegates tasks like network calls, dns resolution to LIVUB, consider increase LIBUV_THREADPOOL_SIZ to maximum i.e. 128.
On top of that if possible reduce calls to other services. Possibly use facade pattern to another java like service which can provide you all the data at single call.
Because of LIBUV application performance is inversely propositional to api calls you make
Scale
Since NodeJs app was not scaling well with CPU usage because low cpu usage and limit of libuv thread pool size. Kubernetes was not able to scale our application that time, since it was not supporting scale based on custom metrics. We have to upgrade our Kubernetes to version 1.9 and above to achieve that. Since our application was already desined to export prometheus metrics, we utilised that to export queries_per_second[Queries per second] metric and made below changes in deployment configuration,

Deployed custom controller on GKE

First thing first, we need to upgrade to Kubernetes version 1.9 to support metrics based autosacling and code change to add 1 custom metric queries_per_second. With below in mind we experimented with GKE Autoscaler document and found to provide the results that we want on staging environment. In a nutshell autoscaler requires 3 things:
  • Deploy custom metric stackdriver adaptor
  • Export metric to stackdriver
  • Deploy HPA settings to scale deployment based on cutom metrics exported by step-2

Deploy adaptor and rolebindings

GKE requires to use clusterrolebindings to allow adaptor to access cluster resources. In order to do that one need to run below command with clusteradmin privilege.
kubectl create clusterrolebinding cluster-admin-binding \
 --clusterrole cluster-admin --user "$(gcloud config get-value account)"
Deploy the adaptor in cluster
kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml

Export app metrics to stackdriver

Final content of deployment file was modified to include side car to push data to stackdriver. We have followed the recommendation to setup this however now Kubernetes also allows us to scaled baesd on third party metrics. Side car is running with Image[image: gcr.io/google-containers/prometheus-to-sd:v0.2.3].
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "7"
  creationTimestamp: 2018-05-22T11:47:43Z
  generation: 164
  labels:
    app: browser2.0
    env: production
    harness-app: browser2.0
    harness-env: production
    harness-revision: "201"
    harness-service: browser2
    revision: "49"
    run: viu-browser2
    service: browser2
  name: browser2.0.browser2.201
  namespace: viu-browser2
spec:
  replicas: 2
  selector:
    matchLabels:
      harness-app: browser2.0
      harness-env: production
      harness-revision: "201"
      harness-service: browser2
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: browser2.0
        env: production
        harness-app: browser2.0
        harness-env: production
        harness-revision: "201"
        harness-service: browser2
        revision: "49"
        run: viu-browser2
        service: browser2
    spec:
      containers:
      - env:
        - name: GLUSTER_ENABLED
          valueFrom:
            configMapKeyRef:
              key: gluster_enabled
              name: viu-browser2-browser-vuclip-config
        - name: PUB_SUB_ENABLED
          valueFrom:
            configMapKeyRef:
              key: pub_sub_enabled
              name: viu-browser2-browser-vuclip-config
        - name: NR_ENABLED
          valueFrom:
            configMapKeyRef:
              key: nr_enabled
              name: viu-browser2-browser-vuclip-config
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: NODE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: HOST_URL
          value: https://browser.vuclip.com
        - name: NEW_RELIC_LICENSE_KEY
          value: <Nr license key>
        - name: NEW_RELIC_APP_NAME
          value: browser2
        - name: NEW_RELIC_ENABLED
          value: "true"
        - name: SERVICE_NAME
          value: viu-browser-1
        - name: SERVICE_8000_CHECK_HTTP
          value: /metrics
        - name: SERVICE_8000_CHECK_INTERVAL
          value: 15s
        - name: SERVICE_8080_CHECK_TIMEOUT
          value: 1s
        - name: NODE_OPTS
          value: --max-old-space-size=800 
        - name: NODE_ENV
          value: production
        - name: SERVICE_TAGS
          value: env=prod,kubernetes,stack=viu-browser,type=app,service=browser2,job=browser2
        image: us.gcr.io/ppp-prod/viu-browser2:v435
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 3
          successThreshold: 1
          timeoutSeconds: 1
        name: viu-browser2
        ports:
        - containerPort: 8000
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8000
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 3
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 1717986918400m
          requests:
            cpu: 800m
            memory: 1717986918400m
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/general-shared-dir
          name: general-shared-dir
        - mountPath: /mnt/secrets-config
          name: secret-configs
      - command:
        - /monitor
        - --source=:http://localhost:8000/metrics
        - --stackdriver-prefix=custom.googleapis.com
        - --pod-id=$(POD_ID)
        - --namespace-id=$(POD_NAMESPACE)
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_ID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        image: gcr.io/google-containers/prometheus-to-sd:v0.2.3
        imagePullPolicy: IfNotPresent
        name: prometheus-to-sd
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      volumes:
      - emptyDir: {}
        name: general-shared-dir
      - name: secret-configs
        secret:
          defaultMode: 420
          items:
          - key: keyfile.p12
            mode: 511
          secretName: browser2

HPA setting

Below is custom metric with 2 components,
  • Based on CPU
  • Based on custom metric queries_per_second
  • Setup theshhold for scaling where CPU is setup for 30% average usage among pods, custom metric is setup to support 13 request/second among pods of deployment browser2.0.browser2.201
Highlighting the block for same:
- type: Pods
    pods:
      metricName: queries_per_second
      targetAverageValue: 13
Complete HPA file
apiVersion: "autoscaling/v2beta1"
kind: "HorizontalPodAutoscaler"
metadata:
  finalizers: []
  name: "browser2.0.browser2.production-40"
  namespace: "viu-browser2"
  ownerReferences: []
spec:
  maxReplicas: 60
  minReplicas: 40
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: "Deployment"
    name: "browser2.0.browser2.201"
  metrics:
  - type: "Resource"
    resource:
      name: "cpu"
      targetAverageUtilization: 30
  - type: Pods
    pods:
      metricName: queries_per_second
      targetAverageValue: 13
Below is the this is how our SRE monitoring dashboard looks like to understand this better:
Grafana dashboard

Akamai …

With above configrations we have a stable application now. We have applied a few caching in code in order to avoid network calls. Even with all of this our application response time was a little off, so we made sure that content is served with local caching where akamai helps us to do that. We have setup a GCS bucket to store js, css and images of every build and we push that to GCS. Akamai is configured to serve static JS from this bucket.

TODO

  • Make sure application can serve Dynamic content via akamai.
Labels:

Comments

Post a Comment

Popular posts

Hipchat and Icinga

Hipchat Notify 2.0 Hipchat notification with API 2.0 to be used with ICINGA/Nagios Table of Contents Table of Contents Author Audience Introduction Ruby Script Script used on server Service notification Host notification Change in command.conf for Icinga server Example notification Roadmap Author Shubhamkr619@gmail.com Audience System Engineers and operation engineers Introduction Change the default mail notification of Icinga server to hipchat notification using ruby code. This will allow a single place of management of all the notification and alerts across organization. Let that be service,host or business level alerts all can be managed and monitored using hipchat and hubot will give certain advantage over traditional alerting system. Proactive and reactive alerting Managed monitoring Single place of all the alerts Better communication and collaboration Integration with multiple tools in CI cycle Jenkins Chef ...
Post a Comment
Read more

Add user in Sudoer file

The command sudo provides other users to run a command as a root. To use sudo user must have their level of privilege in /etc/sudoer file. In some cases this file is set for read only and others can edit that file and add their user name in Sudoer for this case such as Fedora 12 users need to provide this syntax as a root on their terminal. Syntax for adding a user in Sudoer file is: echo "username ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers And to gain the writing privilege you can use the following code in your terminal : chmod 755 /etc/sudoers This code will be enough to give permissions to the user and his user group. If you are an ubuntu user you can easily add your user in this file like: vi /etc/sudoers And you will get a text file. Afterward you need to press 'i' for edit and edit by adding your username just below "root ALL=(ALL) ALL" "username ALL=(ALL) ALL"
Post a Comment
Read more

A few useful sql functions

Start mysql in Ubuntu without having root privilege:- If you want to use mysql in Ubutu you can use following command which will use a root level privilege   $ mysql -u root -p Enter password: Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 147 Server version: 5.1.49-1ubuntu8.1-log (Ubuntu) When it demands to enter the password fill it with 'root' and hopefully you'll get logged in .   Last_insert_id():- (with no argument) returns the first automatically generated value that was set for an AUTO_INCREMENT column by the most recently executed INSERT statement to affect such a column. For example, after inserting a row that generates an AUTO_INCREMENT value, you can get the value like this:   mysql> SELECT LAST_INSERT_ID(); Database():- Database() method returns the current selected database and you can use it in your communication and your queries. The syntax is :   mysql>select Database (); User():- It a...
Post a Comment
Read more

Read a File line by line in Shell

I've used a few techniques for this kind of reading but gets failed until I've read this fine blog . http://www.bashguru.com/2010/05/how-to-read-file-line-by-line-in-shell.html this helps a lot and i have developed a trick for me. Refer to the following example to examine the script : #!/bin/bash #SCRIPT: method2.sh #PURPOSE: Process a file line by line with redirected while-read loop . FILENAME=$1 count=0 while read LINE do let count++ echo "$count $LINE" done < $FILENAME echo -e "\nTotal $count Lines read" I have passed the file in the end of the loop so i didn't need  to use pipe that is why i process the file in fastest manner and gives  the required output in a quick manner. However i can use one more thing the File Descriptors. The File Descriptors are similar to the normal file pointers however in  more analogue way.If you have used C then you will have familier with the  the default file openers in C "st...
Post a Comment
Read more

Enter your email address: