Tips for debugging your production web app

Running the app in development or staging environment is a lot easier than production environment. There are so many unexpected factors which make your application unstable in production, such as a high number of concurrent users, lots of queries to database, etc. And when something go wrong on production app, we need to quickly identify the bug and fix it ASAP. The use of monitoring tools definitely can help you capture the errors occurring in your app. Sometimes, however, they do not provide sufficient information for you to analyse the error. Therefore, I believe it is necessary for you to have a basic knowledge of debugging your application in production server. It will definitely save your day at some point in the future.

1. Application-level errors

This is usually the logic or syntax error in your application, and often caused by end-users. The best way to check for these errors is by looking at the production.log file or your Monitoring tools (Airbrake). But if you are using some background processes like Delayed_job or Sidekiq, they often have their own logfile. And most of the time they should reside in the log folder.
These also are considered as the easiest errors to debug and fix because they are so obvious and your app still might continue to work even with the errors.
Just a note. By default, in order to not consume too much server resources on logging, the default log level of production environment in Rails in :info, which mean they will not provide you more details info like you have in development environment. Therefore, in some cases, you might want to change the log level in production environment to :debug to see all the details of each requests (but it is not recommended as many people forgot to turn it back to :info after debugging and this slows down the server)

2. Service-level errors

Your application is not running alone, it requires multiple other components to work, such as database, web server, etc. These errors usually do not reside in log folder. Here are the location of several well-known services often used. Usually you can find their log files in the /var/log directory, each with their own folder. For example, MySQL log file is in /var/log/mysql/, Nginx is in /var/log/nginx, etc. In case your server is responding status code like 404, 403 or 500, it is a good idea to check Nginx or Apache logfile because you would know whether or not the request has been successfully processed by the web server (Nginx or Apache) before coming to your web application server (Passenger, Puma, Unicorn, Thin, etc).

Just a story of my own, the customer reported me an issue in a application which did not occur very often. He said that it sometimes happen but he did not know how to replicate the issue. He said the server went down and no one can access it. That was difficult because I had no idea what was happening. I checked production.log and found nothing, no errors at all. I checked Nginx log file and it just said that server was not responding. And after struggling for a while, I checked MySQL log file and found that the MySQL service was killed accidentally, further investigation found that not enough memory was allocated to MySQL process. After upgrading server memory, the issue was resolved.
This story illustrates how difficult it is to debug an issue and you really need to be calm in order to resolve any issue. Don't be panic because it just makes you feel hopeless in resolving it. Debugging is an art and it takes time to master

A few side notes

If the application is running on Heroku, you do not have access to any log file as you want. Sometimes heroku logs is not enough. It is strongly recommended to install logging tool on any Heroku application (LogEntries)

Your logfile can become big very soon because it often logs any information it collects. That's why many framework will discard old information in the log file and only keep new information. Hence, it is good to debug as soon as the issue arises because you can easily find the root cause. Or you can utilize logrotate, an utility found on Unix which can automatically create archived log file for you and you can trace back the history anytime you want.

Conclusion

Debugging is unavoidable in the development of any application. It is not always easy, especially if you are very close to the deadline or the boss is staying behind you. Just accept it as a part of your life and make it a challenge for yourself and I am sure you will learn a lot from that.

Happy coding!