Sensefull Logging | Nest js Recipies

6 min readSep 20, 2020

An Application without logs is very dangerous. and it becomes more dangerous when we do not monitor the logs. This post will be a mixture of my experience of facing such issues and one way to resolve from thousands of way available.

let’s start with the problem …

Some Problems

Recently there was one Project which is a mobile app talking to different other mobile apps (all are from the same development team ) and there was two APIs that were monolithic so the problem was we were encountering a lot of issues in the apps and we were actually not able to find the root cause unless the developer debugs it. so there was a tight dependency on app developers even to trace the problem whether it is servers, APIs, or app.

That was the first time we realize we should have a logging system that everyone has access to and anyone (the internal team ) can actually just see at some platform and find out what might went wrong . and notify the concerned person to resolve

There was one more case we were actually not notified or trace whether our hosted servers are up or not unless we use the app or make a request calls. that can also make a very bad impression on the client when you have a public-facing app

Let’s say there are 100 issues in the app which the end-user is facing out of that 100 users only 30 -40 % of users will take time to write to your support team about their experience. out of the rest 60% of users, 30% of users may uninstall the app and may give a negative recommendation to others. from a business perspective that is a very big loss.

Solution

As you might have already guessed from the title almost all these issues can be caught with a simple logging system. ill just saw what I think was best for that use cases your feedback is most welcomed

So I used Grafana Loki
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Grafana is used to monitor everything from Prometheus & Graphite metrics, logs, applications, power plants, beehives, sourdough starters, and custom data sources.

We Will also be using Nats Streaming server

Setup Grafana And Loki

I will be using a simple docker system

Step 1: Create Network
we require a docker network on which our different applications can work

docker network create smartcodehubnetwork

Step 2: Setup Nats Server

docker run -d --network smartcodehubnetwork -p 4222:4222 -p 8222:8222 -p 6222:6222 --name nats-server -ti nats:latest

Step 3: Setup Loki Grafana and promtail Instances

docker container run -d --name lokiinstance --network smartcodehubnetwork -p 3100:3100 grafana/loki:latest docker container run -d --name promtailinstance --network smartcodehubnetwork -v /var/log:/var/log grafana/promtail:latest docker container run -d --name grafanainstance --network smartcodehubnetwork -p 3060:3000 grafana/grafana:latest

now you can actually go to localhost:3060 to see the default grafana screen use admin as username and password. Change the password and rock on your grafana is up

Step 4: Connect Grafana with Loki

Go to setting icon (configuration on the dashboard ) then go to data source click on the add data source and select Loki and add the name as per the application and in URL add http://lokiinstance:3100

Note: as we can see we can access the services by name when they are on same network

click on save&test if it shows green once done you are all set up to see logs from different part of the system

Step 5: Create a microservice for logging

ill be using nest js as I like it its is typescript well structured and developer friendly but you can use anything

You can read about creating nest microservice here.

main.ts

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';
import { AppModule } from './app.module';
import { hostname } from 'os';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(
    AppModule,
    {
      transport: Transport.NATS,
      options: {
        url: `nats://nats-server:4222`,
      },
    },
  );
  app.listen(() => console.log('Microservice is listening'));
}
bootstrap();

as you can see online URL: nats://nats-server:4222 we are again using nats by name as our logging service will also run on the same network

appcomponent.ts

import { Controller, Get } from '@nestjs/common';
import { GrafanaLoggerService } from './app.service';
import { EventPattern } from '@nestjs/microservices';
@Controller()
export class AppController {
  constructor(private readonly appService: GrafanaLoggerService) {}

  @EventPattern('log')
  async handlelog({ data }: Record<string, any>) {
    this.appService.log({ ...data });

    // business logic
  }
  @EventPattern('error')
  async handleerror({ data }: Record<string, any>) {
    this.appService.error({ ...data });

    // business logic
  }
}GrafanaLoggerService.ts

import {
  Injectable,
  Scope,
  Inject,
  Logger,
  InternalServerErrorException,
} from '@nestjs/common';
import { WINSTON_MODULE_PROVIDER } from 'nest-winston';
import { Logger as ILogger } from 'winston';
import winston from 'winston';
@Injectable({ scope: Scope.TRANSIENT })
export class GrafanaLoggerService extends Logger {
  constructor(
    @Inject(WINSTON_MODULE_PROVIDER) private readonly logger: ILogger,
  ) {
    super();
  }

  log({ lable, message, ...meta }) {
    this.logger.info({
      message: message,
      level: 'debug',
      labels: { name: lable },
      ...meta,
    });
  }

  error({ lable, message, ...meta }) {
    this.logger.error({
      message: message,
      level: 'error',
      labels: { name: lable },
      ...meta,
    });
  }
}

Now for actually logging, we will be using Winston and Winston-Loki as transporter so our app module will look like

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';

import { WinstonModule } from 'nest-winston';
import * as winston from 'winston';
const LokiTransport = require('winston-loki');
import { ConfigModule, ConfigService } from '@nestjs/config';
import { GrafanaLoggerService } from './app.service';

const {
  combine,
  timestamp,
  label,
  printf,
  prettyPrint,
  logstash,
  json,
  simple,
} = winston.format;
@Module({
  imports: [
    WinstonModule.forRootAsync({
      imports: [
        ConfigModule.forRoot({
          isGlobal: true,
        }),
      ],
      useFactory: (configService: ConfigService) => ({
        // options
        format: winston.format.combine(
          winston.format.colorize({
            all: true,
            level: true,
            colors: {
              info: 'green',
              error: 'red',
            },
          }),

          json(),
        ),
        transports: [
          new LokiTransport({
            host: 'http://lokiinstance:3100',
            json: true,
          }),
        ],
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [AppController],
  providers: [GrafanaLoggerService],
})
export class AppModule {}

once this is done we will create a docker image out of it and start the container in the same network

create a docker file

dockerfile



FROM node:14-alpine AS builder
WORKDIR /app
COPY ./package.json ./
RUN npm config set registry http://registry.npmjs.org/ 

RUN  npm install 
COPY . .
RUN npm run build

FROM node:14-alpine
WORKDIR /app
COPY --from=builder /app .
RUN ls
CMD ["npm","run", "start:prod"]

once you have the docker file in the root of your microservice

docker build -t logger-service .
docker run -d --network smartcodehubnetwork --name logger-service logger-service:latest
docker logs logger-service

Step 6: Connect client to the microservice

Again ill be using nests to create a sample API project which we will connect to the logging service and emit some messages

inside your appmodule

 ...
 
 imports: [
    ClientsModule.register([
      {
        name: 'LOGGER_SERVICE',
        transport: Transport.NATS,
        options: {
          url: 'nats://nats-server:4222',
        },
      },
    ]),
  ],
  .....

once this is done inside the app component you can now inject the service and use it as follows

import { Controller, Logger, LoggerService, Inject, Get } from '@nestjs/common';
import { LoggerService } from './InterfaceLayer/Logger/ILoggerService';
import { ClientProxy } from '@nestjs/microservices';

@Controller()
export class AppController {
  constructor(  @Inject('LOGGER_SERVICE') private client: ClientProxy) {
  }
  @Get()
  async getOne() {
      this.client.emit('log', {
          data: {
            lable: "Smartcodehub",
            message,
            ...requestData,
          },
        })

    return 'Hello';
  }
}

and hurray you have a very simple yet very helpful logging system set up correctly you can just refresh the API page which will emit the message and you can see it in real-time

Thanks a lot for reading this article if it helps do let me know and do share your feedback if you have also faced such issues in your development experience

Originally published at https://blog.smartcodehub.com on September 20, 2020.

Sensefull Logging | Nest js Recipies

Solution

Setup Grafana And Loki

Written by Neeraj Dana

No responses yet