“If I’m not back in five minutes, just wait longer.” — Ace Ventura: Pet Detective

I am writing this blog after a break of two years due to milestones on the personal front which kept me busy.

The arrival of GenAI has lead to an unending stream of predictions for the future. From job loss to universal basic income…from SaaSpocalypse to having opensource projects overrun by agents…from getting cures for all diseases to getting wiped out by the Terminators.

Its important not to lose track of the present for an uncertain future. I chose to focus on the now and use GenAI to improve myself.

• The projects
  • Coin club
  • Care club
  • Campfire club
  • Career club
  • Core club
• Tech stack
  • Frontend
  • Backend
  • Architecture
• Lessons learnt using GenAI for coding
• Final thoughts

The projects

Those who know me well often refer to me as the spreadsheet guy since I have been documenting my plans, comparing products, daily diary etc on Google spreadsheets for more than a decade. I wanted to try to consolidate some of them into my own apps which I could customise.

I originally had 2 apps in mind to organize my life a bit - a finance management app and another for a daily diary. As my idea grew to cover Maslow’s hierarchy of needs, it became 5 apps to address different facets of life. But I lacked the frontend skills to develop them as it was atleast a decade since i worked on the UI (AngularJS days). I found it hard to find time to learn a new framework. Working with the Cursor IDE on a React project at work gave me the confidence that I needn’t hold back anymore.

I wanted to have a suite of apps. The word “Club” sounded like a good idea to keep them together. Keeping the tradition of old super hero names(Clark Kent, Bruce Banner, Peter Parker, Wonder Woman etc), I decided to follow alliteration while naming each club.

Coin club

A finance management app. Some of the features it includes are:

• Recurring bills with reminders
• Transactions (independent as well as those linked to bills)
• Investment (eg: mutual funds)
• Networth
• Finance advisor (business rules and potentially GenAI agent based)
• Analytics

Care club

A health care app focused on physical and mental well being. I really needed this after my kid started to fall sick regularly for a couple of months. Some of the features it includes are:

• Recurring medicine consumption list for reminders (similar to bills above)
• Medicines consumed (similar to transactions above)
• Historical tracking of medicines consumption(similar to transactions in Coin club)
• Daily diary for tracking my daily progress and encouraging mental wellness
• Analytics

Campfire club

A social connections app to help keep track family and friends. I need it especially to stary connected with my large extended family. Some of the features it includes are:

• Multiple social trees (Family tree, Friends tree etc)
• Profiles for each family and friend with reminders (birth, anniversaries etc)
• Conversation tracker for each person
• Reminders to connect with specific people
• Analytics

Career club

A career tracking app for those working as well as those searching for jobs. Some of the features it includes are:

• Career tracking of past and present jobs with goals for the future
• Certifications
• Daily work log
• “To do” list for work
• Job hunting which works with some popular sites for analysis
• Analytics

Core club

An app which brings the previous apps together. Some of the features it includes are:

• Maslow’s hierarchy of needs mapped to the other apps with scoring
• Reminders across all apps
• Motivation quotes (random)
• Weather reporting
• (More to come)

Tech stack

Frontend

Backend

Architecture

Its a simple client-server architecture with each app accessing two Postgres databases:

1. A shared database to store data like user login credentials for a SSO like experience across apps.
2. A database to store app specific data like app settings, transactions etc.

I have used business rules based high level insights in some places followed by local LLM calls for providing some deep insights. Details of this to come up in app specific blog posts.

No vector database as of now since the LLM calls are considerably lightweight with no expected context issues.

Lessons learnt using GenAI for coding

While I began developing the apps over weekends sometime in May 2025, it took me quite a while to get them into a state which I was happy with. Since it was easy to create prototypes, my initial 1.0 version of Coin Club was light weight with just a frontend and a Google spreadsheet as database. It worked well initially but Google Drive API limits forced me to adopt caching techniques and the frontend started to become complex. Eventually I decided to move to a container based setup and have a standard SQL database. I thus had my 2.0 version sometime in October 2025.

Plenty of lessons learnt in this journey till now.

Topic	Details	Screenshots
Mere prototypes	The initial versions worked but could never be deemed as production grade. I had to use multiple prompts to get the code refactored into an organized form (eg: Controller -> Service -> Model, all configurations in a central location etc). I didnt want to use agents.md or cursor.md files back then.
Code Hallucination	I had asked for frontend visualisation libraries to show my connections in a tree form. The LLM suggested a few options but some of them simply didnt exist.
Suggesting the most complex option	I had asked for options to integrate the different apps together. Module Federation was overkill for my scenario but I tested the LLM by asking if it would be a good option for me. The LLM was eager to please and agreed with my suggestion 😅.
Migration not following conventions	I had asked for some changes in the database table. The LLM went with direct python scripts with SQL DDL statements instead of creating ORM migration files.

Final thoughts

The apps are not fully ready yet. Some of the dashboards and some of the analytics pages are pending. Coin club is at the highest maturity since it was the first app in this suite. Core club is barely started since it was the last one taken up. I will be creating a blog post for each app once its ready along with opensourcing the app on Github.com. Expect Coin club to be released in the next few weeks. Its been fun to develop the apps using GenAI though I miss not having written major chunks of it. Its a bit like sliding down a slope at a high speed enjoying the breeze but missing out on the scenary. I did hit the brakes a few times to review the code but its not the same…

I miss the old days of development but can’t help remembering that change is the only thing thats constant in our line of work.

• The current state of documentation
• Benefits of good documentation
• How to create intuitive documentation
  • Manifesto
  • Folder structure
    • Eg - Cloud product
    • Eg - Process
  • Important documents
    • README.md
    • Onboarding sessions guide
    • Working on the first story guide
    • Architecture
    • Features watch list
    • Specialist path guides
  • Automation
• Final thoughts

“I have some paperwork to catch up. If I am not back in two days, organize a search and rescue team!” - Stanley Parker from The Better Half comic.

This post is focused on creating an intuitive documentation which scales for a large product. So it focuses on the balanced approach - between the low and high documentation levels below.

The current state of documentation

• The quality of documentation of a product can be considered as a barometer of the developer’s empathy. However it remains one of the most ignored parts of the software world.
• As the famous line in the Matrix movie series goes “There is a difference in knowing the path and walking the path”. A lot of people merely preach about good documentation while creating a lacklustre README.md file in the repo.
• Projects done by a lot of consultancies add documentation at the fag end of the work as an afterthought ensuring not much time is given to developing it.

Benefits of good documentation

Good intuitive documentation can be helpful in many ways. It can:

• Enable faster handover of projects to client tech teams.
• Reduce dependency on core team members.
• Enable easier maintenance of documentation.
• Simplify onboarding for new team members.
• Provide better estimation of the time needed for onboarding a new team member.
• Achieve easier compliance of a product’s vision.
• Guide engineers in their day to day work.

How to create intuitive documentation

My playbook for creating intuitive documentation :

1. Creating a simple manifesto/vision.
2. Creating a intuitive folder structure.
3. Creating specific documents to aid onboarding.
4. Automating creation of documentation where it adds value.

Manifesto

Translating the vision of the core engineers for the rest of the team is easier with a simple manifesto. Here is an example:

• Engineers should update the technical documentation.
  • Good documentation comes from empathy ie the desire to share the knowledge and enable a smooth onboarding experience.
  • Involve technical writers and analysts only if the audience are not engineers.
• New documentation should ALWAYS be reviewed.
  • PR if its a code repo, Automated review process if Atlassian’s Confluence or a simple people process to ensure engineers always tag reviewers on their documentation.
• Simplify learning experience for engineers.
  • Keep it as visual as possible. Use the 4+1 architectural view model OR C4 model to show different perspectives.
  • Use a layered approach to explain complex concepts and avoid overwhelming the audience. Use pre-read or post-read sections in a document to recommend a user journey.
  • Use draw.io to store the diagrams in the VCS to access the diagrams without a license.

Folder structure

Having an intuitive folder structure or hierarchy is extremely essential for navigating through the documentation with ease. Dont worry about the documents given in them. I have explained that in the documents section next. Lets go through few examples below.

Note : I have considered a documentation repo consisting of markdown files. You can assume similar structure for other tools like Confluence or Google drive or Sharepoint.

Eg - Cloud product

📁 azure
   ├ 📁 architecture
   │    ├ 📁 architecture-decision-records
   │    │    ├ 📁 2023
   │    │    │  ├ 📄 1-adr-some-feature.md
   │    │    │  ├ 📄 2-adr-some-feature-again.md
   │    │    ├ 📁 2024
   │    ├ 📁 latest
   │    │    ├ 📄 integrations.md
   │    │    ├ 📄 automations.md
   │    │    ├ 📄 org-hierarchy.md
   │    │    ├ 📄 network-traffic.md
   │    │    ├ 📄 tech-stack.md
   │    │    ├ 📄 well-architected-framework.md
   ├ 📁 guides
   │    ├ 📄 1-onboarding-sessions.md
   │    ├ 📄 2-working-on-the-first-story.md
   │    ├ 📄 3-troubleshooting.md
   │    ├ 📁 specialized
   │    │    ├ 📄 1-k8s-engineer-path.md
   │    │    ├ 📄 2-cloud-engineer-path.md
   ├ 📁 development
   │    ├ 📁 ai
   │    ├ 📁 compute
   │    ├ 📁 data
   │    ├ 📁 devops
   │    │    ├ 📁 iac
   │    │    │    ├ 📁 styleguide
   │    │    │    │    ├ 📄 resource-naming-conventions.md
   │    │    │    │    ├ 📄 variable-naming-conventions.md
   │    │    │    ├ 📁 testing
   │    │    │    │    ├ 📄 conventions.md
   │    │    │    ├ 📁 features-watch-list
   │    │    │    │    ├ 📄 2023-01-10.md
   │    │    │    │    ├ 📄 yy-mm-dd.md
   │    │    ├ 📄 versioning.md
   │    ├ 📁 monitoring
   │    ├ 📁 networking
   │    ├ 📁 security
   │    │    ├ 📄 k8s-security-matrix.md
   │    │    ├ 📄 personas.md
   │    │    ├ 📄 security.md
   ├ 📁 rfc
   │    ├ 📁 ai
   │    ├ 📁 compute
   │    │    ├ 📄 k8s-production-best-practices.md
   │    ├ 📁 data
   │    ├ 📁 devops
   │    ├ 📁 monitoring
   │    ├ 📁 networking
   │    ├ 📁 security
   ├ 📄 README.md
   ├ 📄 .gitignore

• Above folder structure is for Azure cloud but it can be extended for multi-cloud. Have a folder for each cloud and a common folder for documentation which is cloud agnostic.
• Important to have year based folders for easier segregation of ADR like documents which will keep getting created for the lifetime of a product.
• Essential to have the latest state of the architecture in one place as it provides a high level technical overview of the entire product.
• The guides folder has onboarding journeys which have to be seen by every new team member.
• The development folder contains implementation details. The folder above contains some of the main pillars of a cloud setup.
• The rfc folder is optional. I used it to seperate research documentation from actual implementation. It helped me easily transfer implementation documentation to clients.

Eg - Process

├ 📁 process
|   ├ 📁 agile
|   |   ├ 📄 ceremonies.doc
|   |   ├ 📄 definition-of-ready.doc
|   |   ├ 📄 definition-of-done.doc
|   |   ├ 📁 sprints
|   |   |   ├ 📁 2023
|   |   |   |   ├ 📁 sprint1
|   |   |   |       ├ 📄 sprint1-review.doc
|   |   |   |       ├ 📄 sprint1-retro.doc
|   |   |   ├ 📁 2024
|   ├ 📁 releases
|   |   ├ 📁 2023
|   |   |   ├ 📄 release-2023-01-19.doc
|   |   |   ├ 📄 release-2023-02-02.doc
|   |   ├ 📁 2024

The files above represent word or txt files in a file storage tool. This is useful as :

• It makes it easy for the non-devs (Product Owners, Scrum masters etc) to update it directly.
• The files here do not necesarily need a review.

You can have them in a repo as markdown too if you need a consistent user experience though its probably overkill.

Important documents

There are some documents which can immensely help the overall team. Let me share few of them.

README.md

The README.md file should contain the highlights of the app and absolute basics needed to use it. Anything more and you lose the audience. Keep the other important stuff in some sort of documentation repo or wiki.

Onboarding sessions guide

• This doc helps to estimate the onboarding experience if it involves live or recorded video sessions.
• It also helps to highlight the main topics in a sequential flow needed for a successful onboarding.

Eg:

Working on the first story guide

This doc jots down the exact steps to implement a story and have it deployed to prod. It involves issue tracking process, team review process, git changes, release process etc. It helps to remove ambiguity from the whole process.

Eg:

Architecture

• integrations.md - Lists all the 3rd party integrations in place with links to further details.
• automations.md - Lists all the automations in place with links to further details. Eg: Resource cleaning cron for sandbox cloud accuounts.
• tech-stack.md - Lists all the technologies in place with details on why they are used. Also lists best external or internal resources to learn more about them.
• well-architected-framework.md - An extremely important document which views the entire product from the prism of an industry recognized framework. This makes it easy to pitch the product to potential clients or internal tech leadership.

Eg:

Features watch list

• This document aids product maintenance. Taking the example of a cloud based product, you can view the new features of a specific cloud provider for the last one year(depending on the frequency of the exercise) and prioritise specific ones. Same applies for any tech - whether its a cloud product or a frontend framework.
• It can have different sections of “General Availability” and “Public preview” with top 10 features in each that you want to add to your product. The ones in “Public Preview” are prioritised ONLY if its absolutely needed.

Eg:

Specialist path guides

This set of documents define the personas which are needed for the team. It can contain a list of certifications or custom list of skills/tech or some combo of the two so that it sets up the new team member for success in the team and maybe even his/her career.

Eg of certifications for an Azure cloud engineer :

Eg of knowledge needed to master a specific k8s cluster setup :

Automation

Some of the documentation can be auto generated as long as they follow specific conventions or templates.Eg: API docs, Change logs, Release docs etc.

• Automate generation of the tech aspects of the README.md of a repo as also tag creation using simple bash.
• Tools like Docusaurus which offer an easy to use opinionated documentation structure and website. Do note that they merely focus on the high level folder structure and setup aspects.
• Tools like Slate provides beautiful static documentation for your API.
• Tools like Spotify’s Backstage help you create a developer portal and catalog of your services. Plenty of addons allow you to add a great number of automations and customisations.

Final thoughts

After all this - you might say “This looks to be good doc strategy. But whats the point if the team doesnt update it ?”. You resolve it through processes. Make reviews part of the process. Dont approve the PR if the docs arent reviewed.

Additionally - while automation can help offboard some of the document generation but an interesting document will still need imagination and ingenuity from a human who cares (or generative AI?).

Feel free to share your experiences. Every bit of knowledge helps :blush:.

• Background
• Preparation
  • Udemy course by Stephane Maarek
  • Practice tests
• Certification exam
  • Online exam
    • Process before the online exam
    • Giving the online exam
  • Test center exam
    • Process before the exam
    • Giving the exam
• Final thoughts

Background

“Surpass your limits. The path will open up for you” - Yami Sukehiro from the anime “Black Clover”.

I have been working on Azure cloud infrastructure for the past 3 years which enabled me to go really deep into it. Azure certs allowed me to explore further. I am now interested in refreshing my past experience with the other 2 major clouds. Certifications in them seems to be the logical next step of my journey :smile:.

I decided to pursue the “Architect” cloud certification path as it matched my role at work and I had already achieved this path for Azure cloud. So here is my journey to get the AWS Solutions Architect Associate cert.

I had studied for this exam back in 2017 when I was an Application Architect. But I didnt give it as I used to fall asleep at the S3 buckets part of the syllabus :sweat_smile:. That changed after I went deep into the Azure cloud as a cloud engineer and I began practically exploring the details :smile:.

Preparation

• The AWS exam guide for this cert lists the weightage of the 5 main topics but that isnt very helpful as there can be overlap of resources between them. Eg : Design High-Performing Architectures vs Design Cost-Optimized Architectures.
• The important part of this exam guide is the AWS Resources section at the end of the document. It is a non-exhaustive list of which resources are in syllabus and which are not. So at the end of your preparation, its good to know atleast the purpose of each resource in the list.

Udemy course by Stephane Maarek

• I prepared with Stephane Maarek’s course due to its popularity.
• It has a duration of 27 hours and is well detailed with focus on edge cases too. Stephane has gone the extra mile to try by covering plenty of topics from the exam point of view and it gets regularly updated.
• One practice test is given here with 65 questions.
• I went through the course at 1.5x speed to save some time. Took me a month balancing it with my work. I revised the syllabus by going through the excellent notes (PDF) provided in the course.
• Do note that the syllabus is HUGE ! There are a thousand things that can be asked and you will likely not be ready for it (Eg: Selecting the right EBS volume based on desired IOPS). So just doing this course will not be enough. You have to spend time on the practice tests as well as explore related AWS documentation to increase your coverage.

Practice tests

• After studying for the certification, I decided to go for the single practice test given in the above course as well as enroll for the 6 practice tests by Stephane Maarek and Abhishek Singh.
• As expected, the practice tests were tough. You dont remember the tiny details unless you see it in an exam :sweat_smile:. But the act of giving many practice tests allows you to memorize your studied content better to help when you give the actual exam.
• I didnt do well in my 1st attempt but it was near the 72% cutoff that is needed to pass the exam. So I decided to identify and improve on my gaps in knowledge after every exam. After giving 7 practice tests, I decided to go for round 2 of the same tests again. This time I crossed the 90s for most of them. I was able to complete each exam with 30 mins to spare.
• Something which helped me - I realized that some of the answers given in “Review questions” section at the end of a practice test were outdated. Eg: Kinesis Firehose output destinations. Clicking on the official AWS links helped me view the latest information.

At this point in time I was confident of clearing the exam.

Certification exam

Online exam

Process before the online exam

• I scheduled the exam through the AWS Certification page for Saturday 6:45 am.
• I checked in at 6:15 am and downloaded the OnVue software. I took the compatibility test and it passed. I took pics of my surroundings and uploaded it to the tool for verification.

Giving the online exam

• The online proctor then started the exam for me….and the tool crashed 😖. Guess the new version didnt agree with my Mac.
• I restarted my Mac and connected with another proctor who started the exam again…and it crashed again 😩.
• The proctor opened a ticket/case for me and gave me the ticket number. He then asked me to contact the Pearson Vue helpline for resolving this issue. I visited their website and started the live chat online option. I provided the ticket number to which the person said that the issue was in progress and I should try again after 2-3 days. In a couple of hours, I received an email stating that the exam was canceled and I would be getting the fees reimbursed this week.
• I had 2 options to give the exam
  1. Schedule the exam after 3-4 days and hope that they fix the issue with the next release of the online tool. I had chosen this option 2 yrs ago when the same issue happened during the lockdown days of the pandemic.
  2. Book the exam to be given at a Pearson Vue test center.
• I decided to try the offline test center option.

Test center exam

Process before the exam

• I scheduled the exam through the AWS Certification page for the next day ie Sunday.
• The test center was at a drive of 30 min from my place. The checkin process was super smooth and barely took 5 min. I just had to deposit my stuff (mobile, wallet etc) in their locker and confirm my identity through an original id proof. So simple and quick compared to the 20-30 min online process.
• The desktop computer had an empty erasable page nearby with a pen to help me take notes if a complex question needs it. You don’t get this option online :smile:.

Giving the exam

• It was an exam of 140 min with 65 questions and I needed 72% to pass.
• Similar to the practice tests, I paced myself to complete 20 questions every 30 min so that I wouldnt face the pressure of time.
• As expected, the exam was tough. Less overlap with the questions in the practice tests (max 5 questions I guess). Questions on disaster recovery were tough as I had to read a lot of content before examining the lengthy options. Spending too much time on them would definitely be disasterous for me 😅.
• I completed the whole set with 25 mins to spare. I used this time to review the 8 questions I had flagged. I ended the exam with 2 min left.
• The results of the exam are supposed to be given usually within 1-5 days. However I received the “pass” result in 2.5 hours via email. I visited the “Exam History” tab of the AWS Certification page to view my score. I had got 79%.

• I was happy with the result though I bit dissapointed with the marks. I was expecting something between 80-90%. I then remembered the fine print on the AWS exam guide:

  The exam includes 15 unscored questions that do not affect your score. AWS collects
  information about candidate performance on these unscored questions to evaluate
  these questions for future use as scored questions. These unscored questions are
  not identified on the exam.
  

• Due to this, there was no way to correctly predict my marks. I might have done well with those 15 questions and not so well in the ones that mattered. Oh well, atleast I got the shiny certification logo :smile:.

Final thoughts

• Its a fun exam to study for. One of the tougher ones out there for sure.
• Like most other cloud certs, you will need a lot of memory + understanding + some logic to clear this exam.
• The exam is easier if you have worked a lot in AWS recently. A lot of the finer details automatically become embedded in your memory.
• Create your own notes on complex topics like data transfer (the difference between the various Kinesis tools, DMS, Glue etc) so that you have a clear understanding. Makes it easier to remember.
• Practice tests only help in time preparation and make you understand the type of questions which can be asked. Don’t expect much overlap with the actual exam questions.
• This information will be lost if you don’t actively work on the cloud in the near future. Refer to my blog post on Certification strategy for more on this.

I wish you the best of luck if you plan on giving this exam :thumbsup:.
Feel free to share your experiences. Every bit of knowledge helps :blush:.

• Need for Alerting
• Need for an Alerting strategy
• Industry frameworks
  • The Four Golden Signals
  • The RED method
  • The USE method
• Cloud alert types
• Prioritisation
• Channels
• Recipients
• Use case
• Best practices
• References

Need for Alerting

Alerting is an essential step of monitoring. Monitoring provides you visibility into the health of your systems. The benefits of alerting are :

• An alert can contain enough contextual information to help us quickly get started on diagnostic activities.
• Alerting can be used to invoke remediation functions such as autoscaling.
• Alerts can also enable cost-awareness by watching budgets and limits.

ref: https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-overview

Need for an Alerting strategy

A robust alerting strategy provides you a plan to structure your alerts and rules to manage various anomalies in the system. An alerting strategy defines the organization’s standards for:

• The types of alert rules that you’ll create for different scenarios.
• How you’ll categorize and manage alerts after they’re created.
• Automated actions and notifications that you’ll take in response to alerts.

Industry frameworks

Reviewing the popular methodologies of USE, RED, and the Four golden signals, you’ll find that they have requests, latency, and errors in common. Security and cost can be additions to this if they are measurable.

The Four Golden Signals

Rob Ewaschuk described the “four golden signals” as the most important metrics to focus on at a high level. It was developed by Google’s SRE teams.

• Latency
  • The time it takes to service a request.
• Errors
  • The rate of requests that are not successful.
• Traffic
  • A measure of how much demand is placed on your system.
• Saturation
  • How full your service is.
  • A measure of your system fraction, emphasizing the resources that are most constrained (Eg: In a memory-constrained system, show memory; in an I/O-constrained system, show I/O).

The RED method

For request-driven applications (like microservices), Tom Wilkie defined the RED Method:

• Rate
  • The number of requests per second a service is processing.
• Errors
  • The number of failed requests per second.
• Duration
  • Distributions of the amount of time each request takes.

The USE method

Brendan Gregg proposed the USE Method for characterizing the performance of system resources:

• Utilization
  • The average time that the resource was busy servicing work.
• Saturation
  • The degree to which the resource has extra work that it can’t service, often queued.
• Errors
  • The count of error events.

Cloud alert types

Metric alerts

• They are useful when you want to be alerted about data that requires little or no manipulation.
• Metric data is stored in the system already pre-computed, so metric alerts are less expensive than log alerts. Eg: database size alert.

Activity alerts

• Activity alerts provide auditing of all control plane actions that occurred on resources.
• Use activity alerts to be alerted when a specific event happens to a resource. Eg: Activity log alert on Azure. Eg: Create/Update/Delete Network Security Group in Azure.

Log alerts

• Log alerts allow you to perform advanced logic operations on your data.
• If the data you want to monitor is available in logs, or requires advanced logic, you can use the robust features of the querying language for data manipulation using log alerts.

Prioritisation

Prioritisation allows the right alerts to show up as notifications and reduce the noise. One way of specifying the priority is by using a severity level that indicates how critical a situation is.

Channels

• Push to an app(Eg: Slack or PagerDuty).
• SMS messages.
• Voice messages.
• Email.

Recipients

• Many orgs out there have alerts going to a single cloud team. While this makes for an easy setup, it greatly increases the load as also the signal to noise ratio for the team.
• Discussing the different personas for the cloud is a topic better off as a seperate blog post. However let us talk about them from an alerts perspective. Some of the different personas possible within the Cloud Operations team:
  • Cloud Admins
  • Platform team
  • Site Reliability Engineers
  • FinOps team
  • SecOps
    • Cloud Security Leads
    • Cloud Security Engineers
• Based on the real world teams which exist, the right “owners” can be set for the different alerts. Its the responsibility of the “owners” to follow through with the process and respond to the alerts. Eg: Budget alerts can go to the FinOps team.

Use case

• An effective alerting strategy should be in line with the architecture of the cloud setup. If you take the below Hub spoke setup as an example, you can see which resources are critical for the uptime of the application.
• Examine each resource in this flow and apply the dimensions we discussed above. So examine each resource from left to the right - ie from the Firewall and VPN to the network peering to eventually the Virtual machine and database. Thus for each resource, you can:
  • Identity alerts for each resource. Plenty of docs available for this.
  • Prioritise the alerts. SLA can be used to determine this.
  • Assign the right channel based on priority. Eg: Assign the critical ones to something like PagerDuty or Slack or Teams to ensure timely response.
  • Ensure the right people are assigned ownership of the alerts. SLA can be used to determine response process and urgency.
• Here is an example of alerts for a virtual machine with suitable priority:
  • VM availability metric falls below 1 (Machine unavailable) - CRITICAL.
  • Percentage CPU is greater than 80% - WARNING.
  • Available Memory Bytes is less than 1 GB - WARNING.
  • Data Disk IOPS Consumed Percentage is greater than 80% - WARNING.
  • OS Disk IOPS Consumed Percentage is greater than 80% - WARNING.

Best practices

The following points should be considered when configuring alerts:

• Having well-defined owners is vital to optimizing operational effectiveness. Alerts can be set for non-technical notifications too.
  E.g. A budget owner should be made aware of capacity issues so that budgets can be adjusted and discussed.
• Instead of having teams actively monitor the systems and dashboard, send reliable alert notifications to the owners.
• Alerts should be configured for specific resource types adjusted to maximize signal to noise ratios.
  E.g. Only send a notification when a resource becomes unhealthy as per the defined requirements of the application health model or due to a cloud platform-initiated event.
• Consider transient issues when setting an appropriate threshold for resource unavailability.
  E.g. Configuring an alert for a virtual machine with a threshold of 1 minute for unavailability before an alert is triggered.
• Use an automated alerting solution instead of having people actively look for issues.
• Add a group email address rather than specific persons so as to reduce the need to update the notification settings with every team member’s changes.

Takeaway - Use the steps given above to formulate a solid alerting strategy for your cloud setup. A “horses for courses” approach works really well to ensure a steady flow of alerts which isnt noisy.

References

• Azure Alerts overview
• Amazon CloudWatch alarms
• Alerting for operations
• Best practices alerts

• Background
• Preparation
  • Official study material
  • Udemy Practice tests by Bryan Krausen
• Scheduling the exam
• Pre-exam process
• Exam
• Final thoughts

Background

“They give me questions I dont know. I give them answers they dont know” - Anonymous

The HashiCorp Certified: Terraform Associate exam is ideal for Infrastructure and Ops engineers. I cleared the exam today ie June 8, 2023. I had previous given the 002 version of this exam around 2 yrs ago. You can read about my journey on other certifications here.

My experience as a Cloud Architect and hands-on cloud engineer gave me the confidence to go for this certification.

Preparation

Official study material

• HashiCorp recommends those with Terraform experience to study with the review guide while those new to Terraform are better off with the study guide. Same content in both but the study guide has a better structure for learning from scratch.
• Those with Terraform experience should be able to breeze through the material in a weekend while those new will probably take a week or so.

Udemy Practice tests by Bryan Krausen

• I relied on a set of 5 practice tests.
• Similar to the actual certification exam, each test was hour long with 57 questions.
• I would give each exam + review the answers at the end to improve my score the next time. I was able to complete each test in half the time (30 min) every time.
• My scores were
  1. 82%
  2. 84%
  3. 87%
  4. 84%
  5. 89%
• I was confident of doing well in the exams now.

Scheduling the exam

I scheduled the exam through the HashiCorp website which eventually took me to PSI page.

Pre-exam process

I did the pre-exam prep below around 30 min before the exam :

• I had to download a software which tested my machine for compatibility.
• Take a pic of my identity card.
• Take 15 sec videos of my surroundings.
• Take 15 sec video of my hands and ears.
• After another live 360 degrees scan from the Proctor, The exam was ready to begin.

Exam

• It was an exam of 1 hour with 57 questions and I needed 70% to pass.
• Similar to the practice tests, I started off on a quick note. I completed 20 questions every 10 min.
• It took me 30 minutes to complete 57 questions.
• I had flagged around 9 questions whose answer I wasnt sure of. Some of them had tricky language.
• I had to fill a survey on myself and nature of the exam after this (no impact on the exam results).
• I immediately received the congratulatory message page for clearing the exam along with the details of my performance. I got 86%.

The certificate was added to Credly in 48 hours.

Final thoughts

This was the easiest online exam I have given. Focusing on the cli commands is the main ask. Highly recommended for all engineers who work with Infrastructure as Code (IaC) as its a quick win.

I wish you the best of luck if you plan on giving this exam :thumbsup:.
Feel free to share your experiences. Every bit of knowledge helps :blush:.

• Introduction
• Do service meshes only work with Kubernetes?
• Features
• Service mesh cons
• Service mesh vs Event mesh
• Service mesh architecture types
  • Sidecar proxy
  • Host based proxy
  • eBPF based
• Comparison of different service meshes
• References

Introduction

• At a high level, a service mesh ensures communication between applications.
• Specifically, a service mesh is a tool for adding observability, security, and reliability features to applications by inserting these features at the platform layer rather than the application layer (where libraries like Twitter’s Finagle, Netflix’s Hystrix, and Google’s Stubby were used).
• The term was introduced in 2016 by William Morgan, Buoyant CEO which eventually led to the creation of the Linkerd 1.0 service mesh. You can read his followup blog post here.

Do service meshes only work with Kubernetes?

• Service meshes are usually associated with Kubernetes. K8s network architecture and layered approach are well suited for service meshes.
• Some service meshes like Linkerd 2.x work only with Kubernetes. Others like Istio, Consul and Cilium can work with applications deployed on Virtual Machines too though the setup will likely involve some additional effort.

Features

• Observability
  • Organizations can get observability support (e.g., metrics, logs, and traces) as well as dependency or service graphs for each of their services (microservice or not), as they adopt a service mesh.
• Security
  • A service mesh can help in setting up a zero trust security model.
  • Authentication, authorization and encrypting traffic between services(mTLS) can be taken up by a service mesh.
  • Most service meshes provide a certificate authority (CA) to manage keys and certificates for securing service-to-service communication.
• Reliability
  • Resiliency features typically include circuit-breaking, latency-aware load balancing, eventually consistent service discovery, retries, timeouts, and deadlines.
  • Service meshes also safeguard service reliability by enforcing a timeout on long-running requests. It can ensure services don’t get overloaded by utilizing techniques like circuit breaking.

Service mesh cons

• Some service meshes can be quite resource heavy(eg: Istiod uses 1 vCPU and 1.5 GB of memory).
• Additional network hops for the traffic.
• Operational complexity can significantly rise for some service meshes wth high learning curve.
• With increase in maturity in k8s and in network CNIs, a number of features are already present and your dependency on service mesh may not be as as much as before.

Service mesh vs Event mesh

Event mesh and service mesh complement each other in the enterprise by providing two different but effective communication options.

• Event mesh connects not only microservices but also legacy applications, cloud-native services, devices, and data sources/sinks. These can operate both in cloud and non-cloud environments.
• While event mesh is asynchronous, service mesh supports more traditional synchronous request-reply messaging.

Service mesh architecture types

Sidecar proxy

• The most popular pattern for implementing a service mesh is the sidecar pattern.
• It involves deploying a network proxy for every service instance which handles all communication between the services. This is part of the service mesh data plane which is controlled by the mesh control plane.
• Many service meshes like Istio, Consul, Cilium use Envoy as proxy.

Host based proxy

• A host based proxy approach involves using a shared agent running on each node/vm of a cluster as proxy.
• It supposed to be a leaner alternative to the sidecar approach as Envoy based sidecars require a fair bit of resources to run (eg: Istiod uses 1 vCPU and 1.5 GB of memory).
• There are service meshes out there who use host based proxy along with sidecar based proxy for better security and division of responsibilities. Eg: Istio’s new Ambient mesh (this mode is not ready for production yet though).

Istio’s Ambient mesh which uses both host based(ztunnel) and sidecar proxy(waypoint):

eBPF based

Isovalent’s Cilium service mesh architecture:

• Extended Berkeley Packet Filter (eBPF) is a feature of the Linux kernel that allows applications to do certain types of work in the kernel itself. eBPF can be used to replace iptables rules, and accelerate the data plane by shortening the data path.
• There are efforts being made with eBPF to have an improved performance with sidecar free service meshes. Currently Cilium is a service mesh which uses eBPF and a node based proxy(Envoy). Istio is also experimenting on this with Merbridge.
• The Linux kernel has decades of features and safeguards in it. It looks difficult to have all proxy features of a service mesh in the kernel with eBPF(especially layer 7 features) but many organisations are looking into using eBPF for optimisation. We need to wait and watch to see how the winds blow here.

Comparison of different service meshes

The below table compares 4 prominent service meshes. For others you can look at https://servicemesh.es/(very detailed with a bit of outdated info and lesser meshes) or https://layer5.io/service-mesh-landscape.

References

• https://linkerd.io/what-is-a-service-mesh/
• https://konghq.com/learning-center/service-mesh/what-is-a-service-mesh
• https://isovalent.com/blog/post/addressing-bandwidth-exhaustion-with-cilium-bandwidth-manager/
• https://medium.com/elca-it/service-mesh-performance-evaluation-istio-linkerd-kuma-and-consul-d8a89390d630
• https://www.toptal.com/kubernetes/service-mesh-comparison

This is part of my Kubernetes series of posts.

• Using a Kubernetes cluster
  • Cluster setup
  • Kubectl
  • Imperative approach
  • Declarative approach
• Kubernetes Architecture
  • Control Plane
  • Worker Nodes
• Kubernetes objects
  • Namespace
  • Pod
  • Deployment
  • Service
  • Labels and selectors
  • ConfigMap
  • Secret
• Related tools
  • Helm
  • k9s
• Whats next ?

Using a Kubernetes cluster

Cluster setup

Before going into the internal k8s objects, you will need a k8s cluster to try out commands given in the next section. Use any of the below options given in an ascending order of complexity:

1. Use a free online k8s cluster at KillerCoda.
2. Use a free online k8s cluster at Kubernetes.io.
3. Setup a k8s cluster on your local machine through one click by using Docker Desktop.
4. Setup MiniKube on your local machine.
5. Provision a managed Kubernetes cluster on any of the cloud providers and use it. Eg: Azure Kubernetes Service.
6. Install k8s using a Platform as a Service (PaaS) offering like OpenShift.
7. Install k8s on a barebones virtual machine by installing KubeAdm and using KubeAdm to create the cluster.

Kubectl

Install Kubectl if you plan to run CLI commands from your local machine against the k8s cluster. Add the below alias in your ~/.bashrc file as you will be likely use kubectl a lot atleast in the initial days of trying out k8s.

alias k='kubectl'

Depending on the type of cluster you have installed above, there are different commands to setup a connection to the cluster. If you are connecting from your local terminal, the credentials will likely be stored in the ~/.kubeconfig file. Read this article for more details on kubeconfig.

Imperative approach

Kubernetes objects can quickly be created, updated, and deleted directly using imperative commands built into the kubectl command-line tool. For more details, you can visit the official docs on it. Eg:

kubectl get nodes

Declarative approach

• Kubernetes objects can be created, updated, and deleted by storing multiple object configuration files in a directory and using kubectl apply to recursively create and update those objects as needed. Such config files are called manifests.
• Kubernetes resources are usually created by posting a JSON or YAML manifest to the Kubernetes REST API endpoint.
• The yaml file usually consists of the following attributes:
  • apiVersion - This is the k8s API version which supports the specific resource. We can get the right api version using either the command kubectl api-resources. Alternatively you can always copy the right yaml from the official documentation.
  • kind - This indicates the resource to access.
  • metadata - This is an metadata object having attributes like name, labels, annotations etc.

ns.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: test1

kubectl apply -f ns.yaml

Kubernetes Architecture

Read this section only if you are really curious on what makes k8s work. Not necessary if you just want to deploy your app on the cluster.

Control Plane

• The Control Plane is what controls the cluster and makes it function.
• In a managed k8s cluster, you will not be able to access this part as its controlled by the clodu provider.
• You can remember the different parts of the control pane with the acronym CASE:
  • Controller
    • It replicates apps, keeping track of worker nodes, handling node failures, and so on.
  • API Server
    • This is the endpoint which every resource in k8s communicates with. You can see the mediator design pattern in play here.
    • The API server itself communicates on its own to some parts of the cluster too.
  • Scheduler
    • It schedules your apps by assigning thr suitable node to it.
  • etcd
    • This is the persistent data store which stores the cluster state.
    • Its a popular open source tool.

Worker Nodes

• Nodes are the compute resources which power the cluster. But generally, when we talk about nodes, we consider them as worker nodes ie distinct from nodes which power the control plane.
• The worker nodes are the machines that run your containerized applications (right side of the diagram)
• The different parts of the worker node are :
  • Kubelet
    • This talks to the API server and manages containers on this node.
  • Kube proxy
    • This is the k8s service proxy which load balances network traffic between application components.
  • Container runtime
    • This runs the containers.
    • Towards the end of 2020, Docker as an underlying runtime was deprecated in favor of runtimes like containerd that use the Container Runtime Interface (CRI). Don’t worry if your app image was created via Docker though. It will continue to work in k8s as its an OCI (Open Container Initiative) image.

Kubernetes objects

This is a basic list of k8s resources. There are plenty of resources not mentioned here (jobs, cronjobs, daemonsets, crd etc) but which are needed for advanced scenarios and thus out of scope of this guide.g

Namespace

• Similar to namespaces in other programming languages - Namespaces are a way to divide cluster resources between multiple users or environments. However they don’t provide any isolation for the running objects. For isolation, you will have to use more complex resources like network policies.
• The default namespaces in a k8s cluster are:
  • default - This is the default namespace. Not recommended for production.
  • kube-node-lease - It holds Lease objects associated with each node. It helps the control plane detect node failure.
  • kube-public - Its reserved for cluster usage in case that some resources should be visible and readable publicly throughout the whole cluster.
  • kube-system - For objects created by the Kubernetes system.
• Visit this link for the official docs on namespaces.

Imperative cmds:

kubectl create namespace test1
kubectl create namespace test2
kubectl get namespace
kubectl get namespace test2
kubectl delete namespace test2

Yaml for declarative approach:

apiVersion: v1
kind: Namespace
metadata:
  name: test1

Pod

• It is the basic building block in k8s.
• A pod is a group of one or more related application containers that will always run together on the same worker node and in the same Linux namespace.
• Visit this link for the official docs on pods.

Imperative cmds:

# Single namespace cmds
kubectl get pods                              # List all pods in the namespace
kubectl get pods -o wide                      # List all pods with more details
kubectl get pods --namespace custom-namespace # List all pods of another namespace
kubectl get pods -n custom-namespace          # List all pods of another namespace
kubectl get pod pod1 -o yaml                  # Get a pod's YAML
kubectl explain pods                          # get the documentation for pod manifests
kubectl edit pod pod1                         # Edit a running pod
kubectl delete pod pod1 pod2                  # Delete 2 pods
# Commands on all namespaces
kubectl get pods --all-namespaces             # List all pods in all namespaces
kubectl delete pods --all-namespaces          # Delete all pods in all namespaces

Yaml for declarative approach:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx:1.14.2
      ports:
        - containerPort: 80

Deployment

• It is a higher-level resource meant for deploying applications and updating them declaratively.
• A Deployment provides declarative updates for Pods. You can decide how many instances of a pod you want by setting the replicas. More on using this in the “Services” section.
• Visit this link for the official docs on deployments.

Imperative cmds:

# Single namespace cmds
kubectl create deployment nginx-deployment --image=nginx
kubectl get deployments
kubectl get deployments --namespace custom-namespace
kubectl get deployments -n custom-namespace
kubectl get deployments nginx-deployment
kubectl get deployments nginx-deployment -o yaml
kubectl explain deployments
kubectl edit deployments nginx-deployment
kubectl delete deployments nginx-deployment nginx-deployment2
# Commands on all namespace cmds
kubectl get pods --all-namespaces             # List all pods in all namespaces
kubectl delete deployments —all

Yaml for declarative approach:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.14.2
          ports:
            - containerPort: 8080 # Port on which nginx is exposed outside the container. Change it as you see fit

Service

• A k8s service is a resource you create to make a single, constant point of entry to a group of pods providing the same functionality.
• Each service has an IP address and port that never change while the service exists.
• Connections to the service are load-balanced across all the backing pods.
• A simple way to make a service accessible externally is to set the service type to “LoadBalancer”. This makes the service accessible through a dedicated load balancer usually provisioned from the cloud infrastructure k8s is running on.
• More details can be acessed [here].
• Visit this link for the official docs on services.

Imperative cmds:

# Create a service for a deployment nginx, which serves on port 80 and connects to the containers on port 8000
kubectl expose deployment nginx --type=LoadBalancer --name=nginx --port=80 --target-port=8080
# Alternate way to creat a service but not as flexible as above cmd.
kubectl create service loadbalancer nginx --tcp=80:8080
kubectl get services nginx
kubectl describe services nginx
kubectl edit service nginx
kubectl delete service nginx

Yaml for declarative approach:

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx # Same label as the previous deployment to link it.
  ports:
    - name: nginx-tcp
      protocol: TCP
      port: 80 # port on which service is exposed
      targetPort: 8080 # Should map to the container port on the deployment or pod.

Labels and selectors

• Labels are key/value pairs that are attached to objects, such as pods.
• The set of pods that a service targets is defined with a label selector (as seen in service yaml).
• Via a label selector, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes.

Imperative cmds:

kubectl label pods nginx env=debug              # add label
kubectl label pods nginx env=debug —overwrite   # over write label
kubectl label pods nginx env-                   # Remove a label
kubectl delete pods,services -l name=myLabel    # Delete pods and services based on label.
kubectl get pods --show-labels                  # Show labels for all pods

ConfigMap

• A ConfigMap is used to store non-confidential data in key-value pairs. Pods can consume ConfigMaps as environment variables, command-line arguments, or as configuration files in a volume.
• A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.
• Visit this link for the official docs on configmaps.

Imperative cmds:

kubectl create configmap test-config —from-literal=timer=25
kubectl get configmap test-config -o yaml
kubectl delete configmap test-config

Yaml for declarative approach:

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
data:
  # property-like keys; each key maps to a simple value
  player_initial_lives: "3"
  ui_properties_file_name: "user-interface.properties"

---
apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: registry.k8s.io/busybox
      command: ["/bin/sh", "-c", "env"]
      envFrom: # Define all of the ConfigMap's data as container environment variables.
        - configMapRef:
            name: special-config

Secret

• Secrets are similar to ConfigMaps but are specifically intended to hold confidential data.
• The values for all keys in the data field have to be base64-encoded strings. If the conversion to base64 string is not desirable, you can choose to specify the stringData field instead, which accepts arbitrary strings as values.
• Kubernetes Secrets are, by default, stored unencrypted in the API server’s underlying data store (etcd). Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace.
• In order to safely use Secrets, take at least the following steps:
  • Enable Encryption at Rest for Secrets.
  • Enable or configure RBAC rules with least-privilege access to Secrets.
  • Restrict Secret access to specific containers.
  • Consider using external Secret store providers.
• Visit this link for the official docs on secrets.

Imperative cmds:

kubectl create secret generic test-secret —from-literal=username=testing
kubectl get secret test-secret
kubectl get secret test-secret -o yaml # Use `echo "value" | base64 -d` to get decoded secret.
kubectl delete secret test-secret

Yaml for declarative approach:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  password: $(echo -n "s33msi4" | base64 -w0)
  username: $(echo -n "jane" | base64 -w0)

---
apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: registry.k8s.io/busybox
      command: ["/bin/sh", "-c", "env"]
      envFrom: # Define all of the Secret data as container environment variables.
        - secretRef:
            name: mysecret

Related tools

There are a LOT of tools in the k8s ecosystem. But we will focus on 2 of them in the initial part of our k8s journey.

Helm

• Helm is the package manager for Kubernetes.
• You can install various software into the k8s cluster with Helm. From infrastructure like databases(redis, mysql etc) to applications(cert-manager, grafana etc)
• Helm installs charts into Kubernetes, creating a new release for each installation. And to find new charts, you can search Helm chart repositories.
• You can search the Artifact Hub to view the various helm charts available.
• While Helm templating is a popular option for configuration management, Kustomize is a better alternative for templating.

helm search hub wordpress # search software in the Artifact hub.
helm install happy-panda bitnami/wordpress
helm status happy-panda
# upgrade helm chart with different values.
helm upgrade -f panda.yaml happy-panda bitnami/wordpress
helm get values happy-panda
helm  search repo -l
helm uninstall happy-panda
helm list # see all of your currently deployed releases

k9s

• k9s is a terminal based UI to interact with your Kubernetes clusters.
• With k9s, you dont need to remember cli commands to navigate through your cluster.

Whats next ?

This mindmap was created for Azure Kubernetes Service(AKS) which is a managed cloud service. But many of the branches here are applicable for k8s in general. The k8s ecosystem is like the rabbit hole from “Alice in Wonderland”. It goes on. It can be a long journey to master k8s but its fun !

Image source : https://stanislas.io/2021/09/08/mindmap-azure-kubernetes-service-september-21/

The target audience for this post are those absolutely new to Kubernetes - even if you are from a non technical background. The other parts of my Kubernetes series are meant for a technical audience.

• Where did Kubernetes come from ?
• What is Kubernetes?
  • World before containers
  • What are containers ?
  • Container Orchaestrators
• Why is Kubernetes needed ?
• When can Kubernetes be a bad idea ?
• Who is using Kubernetes ?

Where did Kubernetes come from ?

• Kubernetes was announced by Google in mid-2014. It was inspired by Google’s internal tool Borg which has been in use for more than a decade.
• Kubernetes is often abbreviated as k8s, counting the eight letters between the “K” and the “s”.
• Kubernetes means “helmsman,” “pilot,” or “governor”. Sticking to its ship analogy, most tools in the Kubernetes ecosystem have been named after some ship related concept or part.
• The creators of Kubernetes (and Borg) were inspired by the popular Star Trek series:
  • The orginal tool was named after the Borg - the hive-mind alien species which are a recurring antagonist in the series.
  • Kubernetes was originally called “Project 7” after the Star Trek ex-Borg character Seven of Nine and gave its logo a seven-spoked wheel.

What is Kubernetes?

World before containers

• In the old ages, an application would become “live” by installing software on machines. It involved downloading files, libraries and dependencies.
• Difficulties in this approach:
  • Takes a lot of effort.
  • Takes a lot of time based on software size and how many machines are involved(ie scale).
  • Compatibility of the software setup with different machines.
• You could reduce effort involved by via automation scripts or via other tools (Eg: Chef, Puppet, Ansible). But other issues remained.

What are containers ?

• A container is a software package which bundles an application’s code together with the related configuration files, libraries and dependencies.
  • Simple eg : Conceptually its similar to a large zip file having everything needed for the application to run.
  • Real world eg: you can compare it to your large travel bag which has everything you need to function in another place.
• Though containers existed for many years before, Docker made it famous in 2013 by simplifying its usage with an ecosystem of related tools. You can refer to my blog posts on Docker if you want to learn more.

Container Orchaestrators

• Container Orchestrators are tools to manage, scale, and maintain containers.
  • Simple eg : Its an automated tool which takes your “zip file” and deploys it to any number of suitable machine for making it functional, scalable and accessible.
  • Real world eg 1 : You can compare it to a hotel which enables you to function by taking care of you (and your travel bag!). It provides you security, support, scalability (more rooms!), ease of maintenance, easy access to the rooms (networking!) etc.
  • Real world eg 2 : You can compare it to an orchaestrator who knows and manages the playing or various music instruments in the suitable sequence to create a beautiful symphony.
• Popular container orchestrators are Kubernetes, tools built on top of Kubernetes(Rancher, OpenShift, VMware Tanzu) and cloud managed services (GKE, AKS, EKS), HashiCorp Nomad, Apache Mesos and other container hosting services on the cloud (AWS Fargate, ECS, Azure Container Instance, Google Cloud Run) etc.
• Kubernetes is also known as the operating system of the cluster.

With a ship on top of it, the famous Marina Bay Sands hotel of Singapore can be a great way to remember Kubernetes.

Why is Kubernetes needed ?

Kubernetes is complex but it needs to be as it has MANY responsibilities ! For more details, you can refer to the Robusta.Dev post on it.

Some advantages of Kubernetes:

• Simplifies application development and deployment.
  • With the automated Container Orchestration, app developers can focus on just making the application work in a container.
• Health checking and self healing.
  • If a container(ie an instance of the app inside it) fails and crashes, a new container(a new app instance) is brought up in its place.
• Automatic scaling.
  • Kubernetes supports load balancing between differences instances of the application container.
  • We can define how many instances we start off. We can also define the rules for scaling which will lead to automatic provisioning of more containers and load balancing of requests between them. Eg rule : Provision 1 more node when CPU utilisation exceeds 80%.
• Better utilisation of hardware.
  • We configure how many virtual machines support the Kubernetes cluster.
  • Kubernetes keeps track of how many containers are provisioned in each virtual machine and ensures they are equally distributed among the machines (or as per configured rules).
• Infrastructure as code(IaC).
  • Kubernetes supports using a declarative style ie manifests(yaml or json files) for configuration.
  • IaC enables tracking of changes as its part of the VCS(Version control system).
  • IaC is an important step forward for the DevOps and NoOps initiative.
• Helps in avoding vendor lock-in as k8s provides a high level of abstraction over various services due to its loosely coupled archtiecture.

When can Kubernetes be a bad idea ?

Below are the scenarios for which k8s can be a bad idea. For real-world incidents, please visit the infamous k8s.af.

• You just want to host a small web application.
  • Kubernetes was built for scale. Its power is clearly seen when you have a bunch of microservices accessed by a large audience while supported by hundreds of virtual machines.
  • Its overkill if you just have a few microservices and arent planning to scale as much. Serverless or other cloud managed services or a simple vistual machine are a better fit for hosting such an app.
• Your organisation lacks Infrastructure engineers.
  • K8s can be simple or complex depending on which level you wish to explore.
  • Even the most self-running automated tool can run into problems. K8s comprises integration between many open source tools and concepts. Infra engineers who will manage the k8s cluster, need to know all about the internals so that they can debug any issue swiftly.
• Application architecture not conducive to k8s.
  • Some legacy applications are not meant to be deployed into containers in their current form(eg: Monoliths).
• Unrealistic expectations by stakeholders.
  • Some stakeholders go for a shift to k8s with a lack of awareness. They either have a blind belief in the hype or expect immediate returns.
  • A well built k8s setup requires some upfront investment - whether its research on k8s, well thought of strategies on logging, storage, monitoring and scaling, cost etc. The returns will take time.

Who is using Kubernetes ?

Many organisations are using k8s. Some prominent ones:

• Mercedes and its 900 k8s clusters
• OpenAI
• US Department of Defense
• CERN
• Tinder
• Adidas

Stay tuned for more posts on Kubernetes !

Layoffs in the past year or so have become rampant with most big organisations joining the trend. People from diverse bands have faced the axe - from freshers to highly experienced individuals.

Our office time forms a significant chunk of our lives. Its understandable that calls for empathy are being made in social media by those who have been laid off. We often forget that the employer-employee relationship is a symbiotic one. Each one helps the other to achieve goals. Any one side can end this professional relationship.

My earlier post on “5 career tips for a software engineer” focuses on how you can improve yourself to be a successful software engineer in the industry. This post focuses on adapting to external factors which you do not control. Like driving on roads, you can recognise the patterns and do a better job navigating them.

• What makes you expendable
  • You are not a significant contributor to the crown jewels of the org
  • Your tech skills have become redundant
  • Your value proposition is not known eg: new hires
• Tips for surviving in the tech industry over a long time

Visualisation from Layoffs.fyi:

What makes you expendable

You are not a significant contributor to the crown jewels of the org

• The crown jewels of an organisation are the products which deliver the most impact(usually money) or have a big potential to do so in the future.
  • Eg: For Google, the crown jewels based on revenue could be Ads, Search, Gmail, YouTube, Chrome, Maps, Android and Cloud along with some internal tools which are big enablers (eg: Borg).
• For a lot of people, the goal is to get into their “dream” company. The role or type of work takes secondary place.
• During a consolidation or recession, the ones who usually get affected are contract workers, those working on non-essential services, those working on R&D of concepts with less impact etc. Even those working on minor tweaks on a big stable product can get laid off.

Your tech skills have become redundant

• The tech industry is one of those workstreams where you have to keep updating yourself even when it comes to core tenets. The idiom “you snooze, you lose” is very apt here.
• Chances of your tech skills being redundant are higher in some cases :
  • You are a generalist problem solver with skillsets matching a lot of others. Eg : Lots of folks out there advertise themselves as a Java engineer or a full stack engineer but hard to distinguish yourselves from others with just this skillset.
  • You rely on your people skills more than you rely on your tech skills. Its harder to quantify your achievements in an industry where the word “manager” is frowned upon.
  • Lesser opportunities for utilizing a niche skillset and you arent adapting to the new trends.
  • Its easy to replace you with a cheaper alternative in some other location (usually a developing country).
  • You have met your goals in the current org but arent moving on to the next one due to inertia.

Your value proposition is not known (eg: new hires)

• The best people I worked with are those who actively take responsibilities while going deep into a technology to become the go-to person for the tech/tool. People depend on them for solving their problems.
• If you arent known OR its easy to get people/automation to do your work, then your importance greatly diminishes in the eyes of your employer.
• If you are new to the job then you come in with a couple of big disadvantages during a recession:
  • You are still trying to make an impact in the initial probation period of your new job.
  • The organisation doesnt really know your worth and relies on the fact that they did fine without you before.

Tips for surviving in the tech industry over a long time

There is no guarantee for success but there are some practices which have worked well for a lot of techies.

• Join an organisation only if you get to do significant work on a core product.
• Aim to switch jobs during a recession ONLY if your current job is at risk.
  • As they say - “A bird in the hand is worth two in the bush”. You dont want to be the new person in an organisation and inherit the reduced job security which comes with it.
• Big tech consultancies or regulated companies can be safer bets during troubled financial times.
  • Consultancies might not be able to match salaries at product companies but they usually have a backlog of client work which can ride out a year of financial turmoil.
  • Consultancies also tend to avoid mass layoffs as they seek to retain some bench strength.
  • Additionally regulated industries like banking, healthcare, insurance, utilities, automative etc with inhouse tech teams are usually starved of talent and need people at most times.
• Join startups at the initial parts of your career to gain maximum learning over a short duration of time.
  • That prepares you for most stressful situations on the job and everything else feels easier after it.
  • Stay clear of startups who dont have a solid revenue plan and rely solely on VC money to stay afloat.
• Keep building your knowledge and your digital profile.
  • Dont just wait to post on LinkedIn during times of need. Share your knowledge at regular times.
  • More on this in my other post - “5 career tips for a software engineer”.
• Switching to a different region can be a good option for some.
  • Eg: Asia looks to be more resilient to economic downturns compared to others. The reduced cost of living and bigger opportunities to make an impact are factors working in favour of Asia.
• Help others.
  • You can choose to believe in karma or in the act of debt. It can be something small like writing a LinkedIn testimonial or something big like helping out with career advice. You never know when a random act of kindness comes back to assist you in your time of need.

• Why do stories matter ?
• What are the elements of a good story ?
  • Know your audience
  • An eye catching start
  • Clean takeaways
  • Well prepared content and presentation
• How can software engineers use storytelling regularly ?
  • Onboarding documentation
  • Sprint reviews
  • Backlog User Stories
  • Knowledge sessions
  • Profile pitch

Why do stories matter ?

Stories have been used from the begining of the human civilisation for community building. The most interesting chapters in our history books are those having a great story. The best books, movies, series and advertisements are those having a compelling story. Many of us have fond memories of our elders narrating stories with us hanging to every spoken word.
The common thread running through all of them - stories help to build an emotional connect.

At the end of the day - our software world involves people. Automation can help in optimisation and scaling but stories can get people to care.

What are the elements of a good story ?

Know your audience

You cannot please everyone. Identifying your core audience (eg: majority) is important and it allows you to connect with them after understanding their perspective.
Eg: For an audience of experienced k8s practioners, you will do better to go deep on concepts while for those new to k8s, you should explain from a generic tech perspective.

An eye catching start

All of us suffer from information overload. A crisp headline saves the time of the audience by communicating the core message in a single line. But do not use a bait-and-switch technique to give a misleading heading.

Clean takeaways

Well laid out takeaways help the audience to save time again while ensuring they dont easily forget the core message.

Well prepared content and presentation

Steve Jobs used to rehearse a lot for his picture perfect presentations. This helps refining the content well while internalising it. Practicing in front of a mirror helps you to take good care of your body language and verbal pitch.

How can software engineers use storytelling regularly ?

Onboarding documentation

• Storytelling can be the most powerful way to improve the onboarding journey for new folks. Its often the most ignored part though. Rather than point a new dev to handful of links with bland info, we can make the documentation experience interesting enough for devs to actually read it well.

Some tips :

• For those starting the document, add pre-requsities(info or links) to ease the learning curve.
• For those ending the document, add next articles to read. This helps create a sequential plan similar to a book reading experience.
• Use visuals/images to simplify concepts. Use elements of the 4+1 architectural view model OR C4 model to show different perspectives.
• For more improvements, place yourself in the role of a new dev and think of the documentation you would find interesting to read.

Eg: You can see an initial part of a “Fundamentals” section in a getting started guide I had created for an Azure Kubernetes setup.

Sprint reviews

Instead of focusing on what each person did during the sprint review, the sprint goal should be used to set the storyline. The major stories can be demoed in a flow which makes sense to the audience.

Some tips :

• Whether its a ppt or a markdown file, limit the content of each slide/story to just 3 parts - title, business impact, implementation summary.
• Focus on a quick demo of what was done. Use a recorded video to fast forward the demo if its time consuming.
• Ask for audience feedback at the end so that you know what worked and what didnt.

Backlog User Stories

Lots of guides are out there on how to write a good user story. Eg: Atlassian guide on user story.

Some tips :

• Ensure the story title resembles the format - “As a [persona], I [want to], [so that].”. This ensures the user, the summary and the business impact is always given.
• Attach images wherever possible as it helps the developer visualise things.

Knowledge sessions

• Learnings from the way Steve Jobs presented are a great way to make effective presentations where one way communication is involved.

Some tips :

• Involve the audience. Ask questions. Give an activity.
• Less content and more visuals on your presentation slide. Dont let the audience get distracted with reading the content. Let them focus on the visual shown and your words instead.
• Add emotions to the equation. Usually humour keeps the audience invested.
• Keep it conversational.
• Focus on takeaway for the audience and not on highlighting what you know.

Eg : For introducing stakeholders to a cloud setup, I used a block diagram to show the network architecture and focused on it. I created other diagrams for showing security, identity etc to be used in further sessions. This ensured I didnt lose the audience.

Profile pitch

• Most people pitch their profiles in elevator pitches, interviews, introductions etc.
• You have to focus on the most important aspect about yourself in a short time of a minute or so.

Tips :

• Focus on who you are, what do you do, what are your skills and finally how you can help the person/audience.
• Do your research on the person/audience to know their requirements really well.
• Avoid rambling and speaking in a fast monotonous way.

Story telling can elevate every conversation but it requires you to do a fair bit of homework. If you want to avoid a boring exchange of information then its definitely worth investing the time and effort for pracising the art of storytelling.