Guest

Playing with Bayesian Classifier

Daniel Fisla — Sat, 04 Jun 2011 17:50:44 GMT

Introduction

Just released source code for a simple Bayesian classifier based on ci-bayes project. The classifier offers two working modes: basic db mode, and direct db mode.

Details

In 'basicdb' mode the classifier still uses memory hash table structures to store feature/token and category counts, but also implements classifier and classification listener interfaces for persisting information in the back-end database. This allows for initializing the memory data structures from database instead of having to rebuild them each time before use.

In 'directdb' mode the classifier does not store any information in memory, instead works directly on back-end database tables. This mode was designed to work on very large data sets, large number of classification categories, where it is not feasible/desirable to store all data structures in memory.

Future enhancements, building on functionality in directdb mode, will include integration with memcached direitbuted cache implementation to minimize the number of calls to the database. Alternatively, one can use ehcache as a secondary cache for the Hibernate O/R backend with only configuration changes required.

Source Code

Hosted at Google Code: code.google.com/p/itmatter-bayes/ licensed under LGPLv3.

Google Pagerank for Wikipedia Topics

Daniel Fisla — Tue, 28 Sep 2010 15:14:59 GMT

Working on semantic extensions for the jamwiki software, I spend my time loading and parsing the Wikipedia dumps on regular basis. My goal is to use NLP Parsers and NLP co-reference linker to extract information on entities (people, places, dates, etc...). In addition to, linkage with freebase and dbpedia entities.

Out of the 7M+ plus pages in Wikipedia dumps about 2.2M pages fall into the "topic" page type. The other page types would be category pages, templates, redirects and so forth.

I realized that to take the 2.2M topics, which on average consist of anywhere from 50 to 100+ sentences, I am looking at 100-200M NLP POS sentence parse trees. To represent them as semantic graph/network results in full Million nodes and over 1Billion edges (unverified estimates). I/O architecture is key here and distributed hash table approach such as implemented by memcache turns out to be the best apprach to take.

To make this a more realistic exercise, I decided that I needed a way to limit the initial input size of the topics selected for analysis. To do this, I needed some kind of topic rank as my input set selection criteria. I searched the internet for top 100K or 200K topics but could not find a complete list - mostly just papers describing the scoring methodologies and their outcomes.

Naturally, I came across ideas of other great minds and I decided to read up on the Google Pagerank white paper and implemented the algorithm as described here: http://www.sirgroane.net/google-page-rank/

It is exciting to watch the Pagerank algorithm work its way and compute the Google PR for each topic based on other topics liking into it. My challenge right now is to find a way to parallelize the algorithm as it would take about a week or two using my current hardware configuration to compute a sufficiently converged PRs on the 122M set of topic links - I am far to impatient to wait that long. :-)

As described in the white paper the average of pagerank values for all topics should converge to 1.0 over time. While running the computation I decided to take some sample average values across all topics:

0.18007095210131

0.19245859487535

0.22674132451718

0.26364032294520

One can see that the average page rank value is starting to converge, the rate of convergence is determined by the initial dampening value - for details please refer to the Google white paper.

I will post the actual source code and final results on the Wikipedia data set in the near future. If anyone is interested in the source code and the resulting data set please contact me at dfisla at itmatter.com.

IM X Conference

Daniel Fisla — Sat, 17 Apr 2010 17:18:44 GMT

At our annual Information Management Conference (Adastra IM X) I had the pleasure to run into Peter Hora from Semanta, Roman Stanek from GoodData, including the usual keynote speakers such as Dr. Peter Aiken. As usual, the presentations were great, but the side discussions were even better!

Open Source Projects

Daniel Fisla — Fri, 16 Apr 2010 17:59:22 GMT

Company blog of Daniel Fisla - the founder of ITMATTER Inc. - Software architect/technologist.

As a founder of ITMATTER Inc. I am involved in numerous open source software projects and plan to use this blog to communicate short insights into current and future projects of interest.

Current list of projects consists of:

1. DataCleaner - Open Source - Data Quality Solution - http://datacleaner.eobjects.org

2. JAMWiki Java Wiki Engine - Open Source - MediaWiki compatible Wiki Solution - http://jamwiki.org

3. The Java Wikipedia API (Bliki engine) - Open Source - MediaWiki API and JAMWiki parser extension - http://code.google.com/p/gwtwiki/

4. ITMATTER/JAMWiki High Performance Extensions - Open Source - Extensions to JAMWiki Java Wiki Engine - http://jamwiki.org (dfisla branch)

5. uniBlogger.com - English Wikipedia and Wikinews mirror built on ITMATTER/JAMWiki High Performance Extensions - http://www.uniblogger.com

Distributed Data - I/O Bound

Daniel Fisla — Sat, 17 Apr 2010 15:38:31 GMT

As a part of the JAMWiki Java Engine High Performance Extensions project, I have done extensive research into minimizing disk access overhead. For validation purposes, I have loaded the Wikipedia articles into JAMWiki solution - about 22GB+ in size, 8M+ articles.

With respect to parsing Wikipedia articles and other related content, I have decided to pre-render a significant number of articles with content size over certain threshold (about 1Million articles in total - articles over 3KB -compressed blob data). This gave me a more or less a constant rendering time for large articles. From analysis where the article size was analysed, about 10% of all Wikipedia articles accounted for about 85% of total content size. Under such distribution, pre-rendering (caching) the top 10% of articles would greatly increase the request processing times.

First, I have evaluated EhCache where the idea was to push cached data into the edge nodes (web server tier) in order to avoid database access when retrieving topic, category, and other content. I have looked at versions from 1.6.0 up to latets 2.0.0 snapshot version of EhCache.

What became apparent is that this cache approach worked well for read operations, however did not work so well for frequent updates and pre-building of large persistent EhCache files. Any data structure to uses B-Tree indexes to store lookup values will not perform well under frequent updates. There are other data structures such as B+-Trees and Red-Black Trees that would do better (on average), however none of these can match the performance of dristributed hash tables.

PIM-Related Questions on the Minds of Business Leaders

Daniel Fisla — Sat, 17 Apr 2010 23:16:26 GMT

Interview with Miko Roller from Heiler Software AG.

I asked Adastra MDM/PIM consultant Daniel Fisla to elaborate on 10 topics about PIM based on his experience as a consultant in the industry. Mr. Fisla provides a valuable perspective coming from many years of interaction with companies and implementation of enterprise solutions including PIM.

For full article and PDF download visit Product Information Management Blog.

Speaking at Adastra IM 2009 Conference

Daniel Fisla — Sat, 17 Apr 2010 23:34:53 GMT

Had the opportunity to present about PIM: Integrating People, Processes and Technologies – Challenges and Solutions.

ABSTRACT

As Master Data Management gains more and more prominence in IT circles, the individual pieces that make up a comprehensive MDM strategy are sometimes not given the prominence they should.

Product Information Management is one of those important pieces. A subset of Master Data Management, Product Information Management (PIM) faces a unique challenge: the integration of people, products, and processes in face of rising amount and complexity of product related information. Attend this session to learn why PIM is such an important part of an enterprise MDM strategy, how PIM differs from other MDM components such as CDI, and what IT practitioners need to know when considering the implementation of PIM tools.

Successful Product Information Management Implementations: A Guide for PIM Tool Selection

Daniel Fisla — Sat, 17 Apr 2010 23:07:12 GMT

Abstract

This paper is the first in a series aimed at Master Data Management (MDM)-related challenges, and is geared towards a business audience and presents a high level overview of the approach and the methodology implemented by Adastra on numerous MDM and Product Information Management (PIM) projects. Leveraging this experience as well as extensive Business Intelligence and Extract-Transform-Load tool selections, we discuss the selection methods that should be applied to the PIM tool-selection process in order to minimize risk for organizations as they plan their PIM technology investment. Subsequent papers will address PIM implementation-specific challenges and solutions.

The Challenge

Given the rise of MDM, and PIM in particular, many organizations are challenged to find the software tools to meet their day-to-day and future operational needs. The larger software vendors have mainly acquired their technology by purchasing smaller vendors and are still in the process of integrating these solutions into their computing platforms. At the same time, smaller companies continue to challenge the established vendors by pushing innovation and staying on the leading edge of technology. The result is a complex array of multiple PIM vendors, products and solutions in an evolving marketplace. In this environment, a risk-adverse and proven approach to PIM tool selection is essential to ensuring project success.

The rest of the paper can be downloaded by registering at Adastra Canada web site.

Speaking at IRMAC

Daniel Fisla — Sat, 17 Apr 2010 23:31:22 GMT

Had the opportunity to speak at Information Resource Management Association of Canada - the same presentation as in MDM Toronto Summit 2008.

Speaking at MDM Toronto Sumit 2008

Daniel Fisla — Sat, 17 Apr 2010 23:26:51 GMT

MDM Challenges and Solutions from the Real World

For highlights see Adastra page.

Master Data – a core set of data critical to major business processes and functions

Master Data Management – organizational structures, business processes, culture and technical tools ensuring key (master) data in the enterprise is:

1. Reliable and Correct - reliable and stable data sources, managed and provided by reliable and stable systems
2. Unified - in content and understanding
3. Available - at the right place at the right time