An excellent textbook Machine Learning: A Probabilistic Perspective by Kevin Murphy. More information as well as the first chapter of his book can be downloaded here.

For Probability Theory, there is for example 'Basic Probability Theory' by Robert Ash, buyable but also downloadable here (the 1970's version).

For matrix calculus, see www.cs.nyu.edu/~roweis/notes/matrixid.pdf