KITT is a data visualization tool, created to aid in troubleshooting a multi-layered web application.
Background
While supporting a multi-layered web application, we had to keep an eye out for the service’s KPIs and check whenever the daily values fell before the thresholds. At the time, this application had no central logging, and was running on several hundred servers, which made it nearly impossible to efficiently track which requests failed, at which layer, and why.
The difficulty was also increased due to the fact that the particular web application relied heavily on externally-provided content which was not always reliable.
Enter KITT
KITT stands for “KPI Investigation Tool”, the extra T was added for style, but also predicting that some would eventually call it “the KIT Tool”.
It was created with a specific goal in mind: aggregate all accesslog entries belonging to the KPI probes and display them in an easy-to-navigate interface. To that effect, every incoming request at the first layer was tagged with a unique identifier, which carried through to all the lower layers and allowed for simple log collection and processing.
Once the data was collected in KITT, we could check easily which requests had failed, at which layer of the application’s stack, and for what reason (error, timeout, caused internally or by an external partner, etc.), this sped up immensely our daily KPI checking work.
Detail of a failed probe, with a failed subrequest marked in red.
Further uses
We found other uses for KITT, once it was live.
Rather than just collecting and showing KPI probe logs, we made it collect also logs from our own controlled tests, which allowed us to be faster at troubleshooting. With time it was even adapted for different web applications and was an essential tool.
Technology
KITT posed some interesting engineering problems, particularly at the database design level, since the amount of data was considerable.
It was written in Perl, running as mod_cgi in apache. Storage was handled by a single MySQL instance.