Ironic, isn’t it? Each new platform, architecture, and technology woos devs with promises of simpler, more modular and more workable ways to build apps. Which is all fine, until they hit the observability wall.
Debugging microservices and serverless architecture is complex, frustrating and most of all, s-l-o-w. Replicating issues locally is tricky at best, and more often, undoable. Adding log lines can take hours in some architectures, even assuming you can hit the right spot. Pushing new code is worse. It’s enough to set devs searching for the nearest server.
That’s where Rookout comes in. In these real-life stories, engineers and developers describe how Rookout helped accelerate their dev and debugging processes by shining a light where they needed it most.
Troubleshoot fast, without pushing new code
With over 2000 instances in its production environment, Backblaze engineers faced strict limitations on how often they could push new code. Issues couldn’t be reliably replicated in staging, and the engineers often lacked the log info they needed to zoom in on the issues. So how could they quickly discover root causes, when pushing a log line could take hours – and even then, it would often miss the right spot?
With Rookout, they easily collect data for fast troubleshooting, in staging as well as production. Adding debugging statements at different sites in the code lets them see where code is hit and then find the real issues, fast. They find Rookout so handy, that it has become part of the daily workflow for most Backblaze engineers.
Streamline the monolith-to-Kubernetes journey
Maverick, a large convenience store chain, recently took a collective deep breath and embarked on replacing its monolith architecture with Java-based microservices running on a Kubernetes stack. Initially, during development and staging, developers depended on an APM solution to find where problems might lie, then added log lines and re-released code to pinpoint root causes – a 5-hour rebuild-test-redeploy process.
One developer figured that a better way had to exist and decided to try Rookout. With Rookout’s non-breaking breakpoints set on the fly, Maverick developers can now grab variables and full stack trace data in seconds, not hours, and from live systems. Now they can pinpoint origins of problems, without changing and pushing new code, and without slowing or stopping the stack.
Increase dev velocity
The Behalf dev team loves to move at warp speed. But when it came to surfacing and reproducing issues for debugging, they were overwhelmed by challenges that slowed the process and led to delays. In development, their hybrid debugging approach worked well for individual microservices and for end-to-end flows. But tracing microservices into components with which they may interact meant deep dives into logs, new cloud environments, and collaborative efforts that derailed additional devs.
During production and staging, debugging slowed to a snail’s pace: adding a log line required a new release cycle. Even attaching a remote process could mean wait time, context switching and configuration changes. Each interruption slowed the team more, introduced more idle time – and made the team even more anxious to break free.
Rookout took the wait out of troubleshooting for Behalf with non-breaking breakpoints that can be added in real-time, even in production and staging. Adding log lines quickly and streaming them to Sumo Logic made for richer logs that added even more gas to the troubleshooting process. The result? The average time for debugging microservices with interactions dropped from around 2 hours per developer to just a few minutes.
Share the good news!
Now that you’ve heard about how Rookout’s next-gen developer workflow takes the friction out of debugging and logging, share the good news! Developers, DevOps engineers, and their managers will thank you.