Data Dependant Learners Ensemble Pruning

doi:10.4304/jsw.7.4.919-926

Journal of Software, Vol 7, No 4 (2012), 919-926, Apr 2012

doi:10.4304/jsw.7.4.919-926

Data Dependant Learners Ensemble Pruning

Gang Zhang, Jian Yin, Xiaomin He, Lianglun Cheng

Abstract

Ensemble learning aims at combining several slightly different learners to construct stronger learner. Ensemble of a well selected subset of learners would outperform than ensemble of all. However, the well studied accuracy / diversity ensemble pruning framework would lead to over fit of training data, which results a target learner of relatively low generalization ability. We propose to ensemble with base learners trained by both labeled and unlabeled data, by adopting data dependant kernel mapping, which has been proved successful in semi-supervised learning, to get more generalized base learners. We bootstrap both training data and unlabeled data, namely point cloud, to build slight different data set, then construct data dependant kernel. With such kernels data point can be mapped to different feature space which results effective ensemble. We also proof that ensemble of learners trained by both labeled and unlabeled data is of better generalization ability in the meaning of graph Laplacian. Experiments on UCI data repository show the effectiveness of the proposed method.

Keywords

ensemble learning; generalization ability; data dependant kernel; kernel mapping; point cloud kernel

References

[1] A. Chandra, X. Yao, Evolving hybrid ensembles of learning machines for better generalisation. Neurocomput. 69, 7-9, pp. 686-700, 2006.G.

[2] Y. Zhang, S. Burer, W. N. Street, Ensemble pruning via semi-definite programming. JMLR. 7, pp. 1315-1338, 2006.

[3] Z.-H. Zhou, J. Wu, W. Tang, Ensembling neural networks: many could be better than all, Artificial Intelligence, 137, pp. 239–263, 2002.
http://dx.doi.org/10.1016/S0004-3702(02)00190-X

[4] D.D. Margineantu and T.G. Dietterich, Pruning adaptive boosting, In Proceedings of the 14th ICML, pp. 211-218, 1997.

[5] G. M. Muñoz and A. Suárez, Pruning in ordered bagging ensembles, In Proceedings of the 23rd ICML, pp. 609-616, 2006.

[6] Z. Lu, X. Wu, X. Zhu, J. Bongard, Ensemble pruning via individual contribution ordering, In Proceedings of the 16th ACM SIGKDD (KDD '10), pp. 871-880, 2010.

[7] H. Chen, X. Yao. Regularized Negative Correlation Learning for Neural Network Ensembles. IEEE Transactions on Neural Networks, Vol. 20, No. 12, pp. 1962-1979, 2009.
http://dx.doi.org/10.1109/TNN.2009.2034144
PMid:19923045

[8] R. Caruana and A. Niculescu-Mizil, An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd ICML, pp. 161-168, 2006.

[9] G. Brown, J. L. Wyatt, P. Tiňo, Managing diversity in regression ensembles, JMLR, 6, pp. 1621-1650, 2005.

[10] Zhi-Hua Zhou. Research Article: When Semi-Supervised Learning Meets Ensemble Learning. Front. Electr. Electron. Eng. China, 2010, 5(3).

[11] Min-Ling Zhang, Zhi-Hua Zhou. Exploiting Unlabeled Data to Enhance Ensemble Diversity. ICDM 2010.

[12] T. G. Dietterich. Ensemble methods in machine learning. In Proceedings of the 1st IWMCS, pp. 1-15, 2000.

[13] M. Belkin, P. Niyogi, V. Sindhwani, Manifold regularization: A geometric framework for learning from labeled and unlabeled examples, JMLR, 7, pp. 2399-2434, 2006.

[14] N. Li and Z.-H. Zhou. Selective ensemble under regularization framework. In Proceedings of the 8th IWMCS (MCS'09), Reykjavik, Iceland, LNCS 5519, 2009, pp.293-303.

[15] V. Sindhwani, P. Niyogi, M. Belkin. Beyond the point cloud:from transductive to semi-supervised learning. In: Proceed-ings of the 22nd ICML. Bonn, Germany: MIT Press, 2005, pp.824-831.

[16] D. Rosenberg, V. Sindhwani, P. Bartlett, P. Niyogi. A Kernel for Semi-supervised Learning with Multi-view Point Cloud Regularization. IEEE Signal Processing Magazine, 2009.
PMid:20046885 PMCid:2763329

[17] M. Belkin, P. Niyogi, V. Sindhwani . On Manifold Regularization. Artificial Intelligence and Statistics (AISTATS), 2005.

[18] X. Zhu, Z. Ghahramani, and J. Lafferty. Semisupervised learning using gaussian fields and harmonic functions. In Proceddings of the 20th ICML, pp. 912-919, 2003.

[19] Dietterich, T. G., Lathrop, R. H., & Lozano-P’erez, T. Solving the multiple-instance problem with axis-parallel rectangles. Artif. Intell., 89, 31-71, 1997.
http://dx.doi.org/10.1016/S0004-3702(96)00034-3

[20] T. Gartner, A.P. Flach, A. Kowalczyk, A. Smola. Multi-instance kernels. In Proceedings of the 19th ICML. 179–186, 2002.

[21] Zhi-Hua Zhou, Yu-Yin Sun, Yu-Feng Li. Multi-instance learning by treating instances as non-I.I.D. samples. In Proceedings of the 26th ICML. Montreal, Quebec, Canada, 2009: 1249-1256.

[22] Partha Niyogi. Manifold regularization and semi-supervised learning: Some theoretical analyses. Technical report, Department of Computer Science, University of Chicago, 2008.

[23] UCI Repository: http://archive.ics.uci.edu/ml/

[24] Chen, Y., & Wang, J. Z. Image categorization by learning and reasoning with regions. JMLR, 5, 913–939, 2004.

Full Text: PDF

Journal of Software (JSW, ISSN 1796-217X)

Username
Password
Remember me