Evolving nature of network traffic challenges existing models to fit and predict its behavior. In particular, real traffic modeling requires more flexible design that can adapt to long-range and short-range dependent traffic with dynamic patterns. Unfortunately, existing models cannot handle such requirements because various traffic behaviors such as periodic and self-similar are not taken into account. In this paper, Gaussian process regression (GPR) is adapted for traffic modeling and prediction. The connection between self-similarity as a traffic characteristic and GPR parameters has been driven and exerted to build of a new Hurst estimation method based on machine learning techniques. This led to propose self-similar covariance functions for enhancing prediction accuracy of GPR. The proposed GPR model has been applied for Hurst estimation as well as for traffic prediction on real traffic traces at different time-scales. The experimental results show the employment of self-similar covariance functions increases generalization ability of GPR for traffic modeling and prediction.