Multi-importer is a data-extractor script suite.
Multi-importer was created to address a need for detailed daily reporting aggregated by service, for a group of distinct services.
At the time, the primary source of data was Splunk, but there were other sources that were also required for the reports, like Pingdom and team calendars.
The suite is configuration-heavy, it was designed so as to avoid any code changes whenever new queries against existing sources were needed, and to minimize code changes when new data sources showed up.
The script suite was written in Perl, with
LWP::UserAgent as the core HTTP client library,
Rose::DB::Object providing ORM mapping, and
Rose::DB for database connection handling.
The extracted data was stored locally in a MySQL database for historic purposes.