Observability Recipes

We discuss the guiding principles for observability, including its components and features, and then discuss recipes to apply it to useful applications.

What Is Observability?

Observability is the ability to derive a valid conclusion of what is happening currently to the system and why it is happening.

Guiding Principles for Observability 

1.     Context and sequential flow of each end-tend-end request is most important. We need to be able to see what is having an issue, which other parts might/are affected and what are the commonalities of issues when  things go wrong.

2.     Must be able to cut the data in many ways and correlate the different aspects of a request (e.g. ability to filter for each user, their session, each server node and any of them combined with  the other attributes)

3.     Use questions to drive features required for observability instead of relaying on what we can see.

Observability Components


What is means?


Metrics are numeric values to help evaluate a service's overall behavior over time.

They compromise of a set of data points that can be used to derive system's performance.

Typical examples are:

  • uptime
  • response time
  • # request per second
  • CPU/RAM utilisation


An event is a collection of data points about what it took to complete a unit of work. they are records of selected significant points that happened with metadata to provide context.

Typical examples are:

  • change of a workflow status
  • batch job completion


Logs are important for troubleshooting and trying to understand a problem. they provide detail data and context so one can re-create and diagnose a problem

Typical examples are:

  1. application logs
  2. serer logs
  3. error logs
  4. debug logs


Traces are important for showing a step-by-step journey of how a request or action as it moves through the system. these give specific insight into the flow and help one to identify errors, find bottlenecks so they can be optimised and rectified.


Data needs to be connected in a visual and easy to comprehend approach that allows data to be correlated and derive connections from the different data points and events that is happening in the system. This provides context that  are otherwise not easily identifiable by looking at individual metrics alone.

Observability Recipes

Breaking Observability components into recipes, and starting with what questions to answer. We can then list out the data points require to answer them. 

Using this approach, we can easily map out what the gaps are for an existing system, or use it as an implementation pattern for a new system.

An example is as follows: 










System Health

  • Are all components of my system up and running?
  • Are my servers reaching max capacity?
  • Resource details

o    Total/Consumed/Avail. CPU/RAM/Storage

  • Network statistics

o    latency

o    throughput

o    packet loss

Service start/stop status



Systems health dashboard

Application performance

  • How fast/slow are my application services running?
  • Why has system performance degrade over the past few months?
  • response time (min/max/avg)
  • response payload size (min/max/avg)

events with metrics as payload or having relevant entries in logs


Core components/service list with top 10 slow transactions


User experience

  • what are the browsers that my users are using?
  • What are their on-page performance?
  • How are the users using the system?
  • Applicaton
    • End user response time (min/max/avg)
    • Page load complete time
  • HTML
    • DOM size
    • Ajax request response time (min/max/avg)
  • User profile
    • IP address
    • country
    • browser type
  • User journey
    • entry page
    • exit page
    • time spent on page

Exception management

  1. Are there any errors in my application services?
  2. What is a functionality/page broken or not working?
  • Server request
    • errors/min
    • page views with JavaScript errors
  • error events with following:
  • Standardised error code
  • Date/time of error
  • unique user/session id
  • Standardised error code
  • date/time of error
  • unique
  • user/session id
  • user friendly error message
  • stack trace

detailed object call graph of a request

Dashboard provides:

  • summary view of all error/exception stats
  • list view of error events
  • ability to drill down to each call for troubleshooting


Besides the above recipes, these a couple that should be considered as well:

1.     Release management

o    Why did the release of feature "x" failed?

o    What went wrong during the release?

o    Why did a release take so long to deploy into production?

2.     Security monitoring

o    Are there are security breaches?

o    Are there any abnormal user behavior?

o    Are there any new vulnerabilities to my current system?

We ZippyOPS, Provide consulting, implementation, and management services on DevOps, DevSecOps, Cloud, Automated Ops, Microservices, Infrastructure, and Security

Services offered by us: https://www.zippyops.com/services

Our Products: https://www.zippyops.com/products

Our Solutions: https://www.zippyops.com/solutions

For Demo, videos check out YouTube Playlist: https://www.youtube.com/watch?v=4FYvPooN_Tg&list=PLCJ3JpanNyCfXlHahZhYgJH9-rV6ouPro

Relevant Blogs:

Decoding Disaster Recovery Scenarios in AWS 

Removing the Bastion Host and Improving the Security in AWS 

A Guide to AWS Instance Scheduler 

Unexpected AWS Charges You Should Be Monitoring Closely

Recent Comments

No comments

Leave a Comment