In this thesis we present an algorithm for doing mixture modeling for heterogeneous data collections. Our model supports using both Gaussian- and Bernoulli distributions, creating possibilities for analysis of many kinds of different data. A major focus is spent to developing scalable inference for the proposed model, so that the algorithm can be used to analyze even a large amount of data relatively fast.
In the beginning of the thesis we review some required concepts from probability theory and then proceed to present the basic theory of an approximate inference framework called variational inference. We then move on to present the mixture modeling framework with examples of the Gaussian- and Bernoulli mixture models. These models are then combined to a joint model which we call GBMM for Gaussian and Bernoulli Mixture Model. We develop scalable and efficient variational inference for the proposed model using state-of-the-art results in Bayesian inference. More specifically, we use a novel data augmentation scheme for the Bernoulli part of the model coupled with overall algorithmic improvements such as incremental variational inference and multicore implementation. The efficiency of the proposed algorithm over standard variational inference is highlighted in a simple toy data experiment. Additionally, we demonstrate a scalable initialization for the main inference algorithm using a state-of-the-art random projection algorithm coupled with k-means++ clustering. The quality of the initialization is studied in an experiment with two separate datasets. As an extension to the GBMM model, we also develop inference for categorical features. This proves to be rather difficult and our presentation covers only the derivation of the required inference algorithm without a concrete implementation.
We apply the developed mixture model to analyze a dataset consisting of electronic patient records collected in a major Finnish hospital. We cluster the patients based on their usage of the hospital's services over 28-day time intervals over 7 years to find patterns that help in understanding the data better. This is done by running the GBMM algorithm on a big feature matrix with 269 columns and more than 1.7 million rows. We show that the proposed model is able to extract useful insights from the complex data, and that the results can be used as a guideline and/or preprocessing step for possible further, more detailed analysis that is left for future work.
