Heroku's logging infrastructure – the Logplex system

The Logplex system forms the foundation of the Heroku logging infrastructure. It collates and distributes log messages from the application and other parts of the Heroku infrastructure. These entries are made available through a public API and the Heroku command-line tool. Due to the distributed nature of the Heroku platform, trying to manually access log messages across different application components and making sense out of those is practically infeasible. You cannot get a unified view of the entire system behavior if you collate these messages manually and try to troubleshoot problems. The Logplex integrated logging system provides an alternative in this case.

The Logplex system acts like a conduit that routes messages from log entry sources, that is, the producer of the log messages (for example, an application running on the dyno, the Heroku platform itself) to log entry drains, that is, consumers of log messages (for example, archival systems or post processors doing mining of information).

The Logplex system does not provide storage for the entirely generated log data but only keeps the most recent data that is good enough to extract relevant information from the application run. Typically, this data is of the size of about 1,500 consolidated log messages. If you have a business need to store more than the predefined limit, you would need to choose alternative storage services, for example, Syslog drains to archive all messages.

Logplex is designed with the ability to work with external tools and services such as Loggly. If any of these external tools or services is unable to consume messages generated by Logplex, they can lose some of these messages. However, Logplex does insert a warning entry anytime it has to drop messages due to the consuming service downtime.