Curation Module for The ICONIST

The programm collects product data of many different providers. Therefore an adapter Concept is used. Coming from different provider formats a unified representation is stored in the database. Hundreds of thousands of different products are beeing imported and kept up-to-date. The import is triggered via cron job automatically but it can be started through the webinterface, too.

A locking mechanism prevents that the same source is beeing multiply imported at the same time. The import state can be determined at any time, even when the import is run by another process. It can be queried with the web interface and as the admin with the terminal. The importer recognizes updates of the particular products as well as the deletion of products by the provider.

The editors choose products for a further selection process and for the enrichment of the data. The web interface offers a plenty of filters to master the big amount of data. Besides very simple filtering options like “shop” or “price range” a keyword search and a filter for categories are provided. The differing categories from provider to provider can be mapped to a common category tree in the curating module. The category tree can be created in the web interface. The mapping of foreign category to local categories is done by an editor.

Two concepts to represent the category tree are used in the database: an adjacency list, that means every category knows its parent category. With this the implementation of the tree management is easy. Besides that the informatio of the Nested-Set Model are stored with the categories. So querying the tree is performant and easy.

For the curation process I wrote a small JavaScript application that reduced loading times distinctly by using AJAX and simplified the server routines a lot. E.g. the management of the application state takes place in the client and needed no server side implementation using sessions.