Multi-importer is a data-extractor script suite.

Code snippet for the HTTP client setup phase

Background

Multi-importer was created to address a need for detailed daily reporting aggregated by service, for a group of distinct services.

At the time, the primary source of data was Splunk, but there were other sources that were also required for the reports, like Pingdom and team calendars.

The suite is configuration-heavy, it was designed so as to avoid any code changes whenever new queries against existing sources were needed, and to minimize code changes when new data sources showed up.

Technology

The script suite was written in Perl, with LWP::UserAgent as the core HTTP client library, Rose::DB::Object providing ORM mapping, and Rose::DB for database connection handling.

The extracted data was stored locally in a MySQL database for historic purposes.