Introduction
Have you ever wondered what it’s like to be an on-call engineer, responsible for ensuring the seamless operation of vital systems in the tech industry? Well, let me take you on a journey into the life of an on-call engineer at SadaPay. As an on-call engineer myself, I’ve experienced firsthand the exhilarating challenges and the immense rewards that come with this role.
In this blog post, I want to provide you with a personal and inside look at what it truly means to be an on-call engineer at SadaPay. From tackling urgent bug fixes to navigating through critical system failures and security incidents, I’ll share with you the highs and lows, the triumphs and lessons learned, and the incredible camaraderie that makes this profession so unique.
Get ready to dive into the dynamic and fast-paced world of on-call engineering, where every decision counts and the thrill of problem-solving never fades. Welcome to my world as an on-call engineer at SadaPay.
Managing On-Call Engineer Duties at SadaPay 🤯
At SadaPay, the management of on-call engineer duties is meticulously structured to ensure smooth operations and timely incident resolution. The on-call responsibilities at SadaPay encompass two main categories: squad on-calls and primary on-calls.
Squad on calls revolve around incidents related to the services owned by each specific engineering squad. These rotations are organized on a week-by-week basis. When an incident occurs in production or an outage affects a particular service, the engineer assigned to the squad on-call rotation for that week is immediately paged. It becomes their responsibility to swiftly investigate the issue and take necessary actions to resolve it, ensuring minimal impact on our users. In addition, documenting an incident report is crucial for the continuous improvement of our systems and services.
On the other hand, primary on-calls present a more challenging and intriguing aspect of on-call engineering at SadaPay. Every engineer at SadaPay is required to undertake the role of a primary on-call engineer at least once a year. This duty extends 24/7, requiring constant vigilance. As a primary on-call engineer, the first task is to acknowledge any incoming incidents promptly.
Following that, a thorough investigation is conducted to determine the root cause and assess the criticality of the situation. To support smooth operations, SadaPay has established a support playbook, which is continuously updated to incorporate recurring incidents and aid future on-call engineers. While it is impossible to have knowledge of all the services running within an organization, in cases where an incident falls outside an engineer’s expertise, they have the ability to page the relevant squad for further investigation.
Nonetheless, as the primary in charge, it remains the on-call engineer’s responsibility to ensure that the necessary steps are taken to prevent service outages and mitigate any potential impact on end-users. This level of dedication and attention to detail is essential, as it directly affects SadaPay’s reputation, financial stability, and overall user experience.
By adhering to these well-defined duties and responsibilities, on-call engineers at SadaPay play a pivotal role in maintaining the smooth operation of our systems. The ability to swiftly respond, investigate, and take appropriate action during critical incidents ensures the utmost reliability and seamless user experience. Through these collective efforts, SadaPay continues to uphold its commitment to providing exceptional services to our valued customers.
My First Experience as an On-Call Engineer: Navigating Responsibilities and Gaining Insights
Undertaking my first-ever on-call engineer rotation was a mix of nervousness and excitement. The weight of the responsibility to ensure the availability and health of our systems loomed large. Initially, as I started my rotation, things went smoothly, with minimal alert frequency. Thanks to our well-documented support playbook, navigating and understanding the alerts were not much of a hassle.
During my on-call period, several planned maintenance activities were scheduled by a third party. This required me to take additional steps to inform our users about potential issues they might face during that period. Throughout the day, I juggled squad work while wearing the hat of an on-call engineer. During the night, I stayed awake for the most part, ensuring I didn’t miss acknowledging any incidents.
However, there were instances where I couldn’t acknowledge incidents during the night. Fortunately, we have a Backup On-Call engineer in place. They are paged if the primary on-call engineer fails to acknowledge an incident within a certain timeframe. In such cases, I delved into logs and traces on Dynatrace, our chosen third-party observability tool, to understand the root cause of failures and take necessary actions.
As part of our continuous improvement efforts, I made note of areas where our on-call process could be enhanced for the next person. Some of the improvements I suggested included refining incident routing based on specific endpoints to prevent unnecessary paging of the primary on-call engineer and documenting sample messages in our customer comms playbook for planned maintenance situations.
Overall, my experience as an on-call engineer was challenging yet rewarding. It was undoubtedly stressful, as staying vigilant and active at all times was necessary. However, it provided me with invaluable insights into our entire system architecture that I hadn’t been exposed to before. Additionally, it sharpened my skills in navigating logs and utilizing Dynatrace effectively. This one-week exercise strengthened my muscle memory and equipped me with the confidence to swiftly locate traces and gather essential details.
Being an on-call engineer is an immersive experience that pushes you to think on your feet and make crucial decisions. It fosters a deep understanding of the interconnectedness of our systems and the importance of maintaining their reliability. My first foray into this role taught me the significance of continuous improvement and the value of collaboration within our engineering teams. As I reflect on my journey, I’m grateful for the opportunity to contribute to the smooth operation of our systems and the growth it has provided me both professionally and personally.
Reflections ðŸ’
As I reflect on my first on-call experience, I recognize the areas where I can improve and strive to become an even better on-call engineer in the future.
Firstly, I plan to further enhance my technical knowledge by delving deeper into the intricacies of our systems. This will allow me to quickly diagnose and resolve issues, minimizing any potential impact on our users. I will also continue to refine my problem-solving skills, honing my ability to think creatively and strategically in high-pressure situations.
Collaboration has proven to be a valuable asset during my on-call rotation. Therefore, I aim to strengthen my collaboration skills further by actively seeking opportunities to work closely with other engineering teams. Sharing knowledge, insights, and best practices will not only benefit me personally but also contribute to a more cohesive and efficient on-call process.
Continuous improvement is at the core of SadaPay’s culture, and I fully embrace this mindset. I will actively contribute to enhancing our support playbook, incorporating new learnings and documenting recurring incidents to aid future on-call engineers. By identifying areas for improvement and implementing effective strategies, we can optimize incident response times and mitigate potential risks more efficiently.
As I look ahead to my next on-call rotation, I am eager to apply the lessons learned from this experience. I am committed to maintaining a proactive approach, staying updated with the latest technologies, and fostering a mindset of continuous learning.
Conclusion
In conclusion, while the life of an on-call engineer comes with its challenges, it also offers tremendous opportunities for growth and development. I am excited to embrace these opportunities, build upon my strengths, and continuously improve as an on-call engineer at SadaPay. Together with my dedicated team, we will strive for excellence, drive innovation, and deliver the highest level of reliability to our valued users.
If you are interested in learning about microservices, head over here.
