Pattinson made his Test debut in 2011 after playing only 6 first class games for Victoria. Despite the lack of first class experience Pattinson burst onto the scene with 5 wickets in the second innings as Australia beat New Zealand. This form continued into the Second Test and he was named man of the series after collecting 14 wickets at an average of 14.

He was a part of the 2014/15 and 2016/17 Sheffield Shield final winning teams scoring 80 in the first innings of the latter final. In 2017 he represented Nottinghamshire in English County Cricket hitting the winning runs in the one day cup final.

James continued his comeback from injury by receiving a Cricket Australia contract in 2019/20 and was also selected in the 2019 Ashes squad.