Fortunately for us, most of the pioneers of Deep Learning are good educators and builders. The result is abundantly available high quality and free learning resources. I will provide some guidance biased by my personal experience. I do not have a PhD and have picked up skills on the job and via the internet. Also, I consider myself a practitioner and not a researcher. Here are things that have helped me. Remember you can do these in parallel just like a GPU :)

Linear Algebra, Probability, and Calculus

I wish somebody would have told me this in high school.
Linear Algebra, Probability, and Calculus are the most important tools for expressing and executing ideas
Approach your study from a tools perspective.

The best way to start is read part 1 (http://www.deeplearningbook.org/contents/part_basics.html) of the Deep Learning Book.

(http://www.deeplearningbook.org/) Then make learning Linear Algebra, Probability, and Calculus part of your learning experience by continuously consuming material. Here are some more resources. No limit to how deep you can go. I am a novice and still learning…

• The Essence of Linear Algebra (https://www.youtube.com/watch?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab&v=kjBOesZCoqc):
  Amazing intuitions building. Just beautiful.

• The Essence of Calculus (https://www.youtube.com/playlist?list=PLZHQObOWTQDMsr9K-rj53DwVRMYO3t5Yr).

Just as above but for Calculus.

• Mathematical Methods for Computer Vision, Robotics, and Graphics Course Notes (https://graphics.stanford.edu/courses/cs205a-13-fall/assets/notes/cs205a_notes.pdf). This stuff is amazing.
  A little hard to digest and you will have to refer to supplementary material but totally worth it. I recommend going back to Khan Academy
  (https://www.khanacademy.org/math/linear-algebra) for supplementary materials.

• Mathematical Methods for Engineers 1 and 2 (https://ocw.mit.edu/courses/mathematics/18-085-computational-science-and-engineering-i-fall-2008/index.htm).

MIT course on applied mathematics. Some methods relevant to AI and others not.

        **Software Engineering for AI**

Praise be to the people who decided to use Python and not things like Matlab to share AI knowledge and techniques.

• Learn Python. Most of the AI libraries and sample code out there is implemented in Python. I came from a JAVA background and this book (https://www.amazon.com/Effective-Python-Specific-Software-Development/dp/0134034287)helped me immensely to understand the Pythonic ways.

• Once you are comfortable with the basics start getting comfortable with parallel programming in Python. You will have to use it to create training data in parallel or speed-up mundane data cooking tasks.
  This blog (http://sebastianraschka.com/Articles/2014_multiprocessing.html) will give you a good start. You will encounter the wonderful world of GIL :).

• Get good at using NumPy (http://www.numpy.org/) and Scipy (https://www.scipy.org/).
  These tools implement mathematics on computers. This introduction (http://cs231n.github.io/python-numpy-tutorial/) from Stanford is a great start. One subtle point is this: NumPy seems to have set the standards on how one should think about Linear Algebra on computers. So the concepts learnt here transfer to most other AI toolkits.

• Pick your Deep Learning toolkit and get good at it. For example, Tensorflow (https://www.tensorflow.org/)or Tensorflow + Keras (https://keras.io/). Many to choose from. The websites for these toolkits are filled with learning material and guides.

• Deep Learning is very compute intensive and the best way to solve this is by using GPUs. Most Deep Learning framework provide easy integration with GPUs. Learn about GPUs and how they make things faster. I recommend this book (https://www.amazon.com/CUDA-Example-Introduction-General-Purpose-Programming/dp/0131387685/ref=sr_1_1?ie=UTF8&keywords=cuda+by+example&qid=1494789351&s=books&sr=1-1)
  and this course (https://www.udacity.com/course/intro-to-parallel-programming--cs344).

Remember, you do not need to know everything but comfort with this material will help you run your networks faster and debug issues.

          **Deep Learning Materials**

IMHO the progression of digesting materials is lectures -> blogs/books -> papers. Some of my favorites.
Lectures

• Neural Networks for Machine Learning (https://www.coursera.org/learn/neural-networks)
• Convolutional Neural Networks for Visual Recognition (http://cs231n.stanford.edu/)
• Deep Learning for Natural Language Processing (http://cs224d.stanford.edu/)
• Oxford CS Deep NLP (https://github.com/oxford-cs-deepnlp-2017/lectures)
• Neural Networks (https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH)
  Blogs/Books
• Deep Learning Book (http://www.deeplearningbook.org/)
• Andrej Karapathy (http://karpathy.github.io/)
• Neural networks and deep learning (http://neuralnetworksanddeeplearning.com/)
• Colah (http://colah.github.io/)
• Sebastian Ruder (http://sebastianruder.com/)
• WildML (http://www.wildml.com/)
• Distill (http://distill.pub/)
  Papers
• Great list compiled by Terry Taewoong Um (https://github.com/terryum/awesome-deep-learning-papers)
• Create an account on Arxiv Sanity Preserver and well go insane (http://www.arxiv-sanity.com/)
  This field is moving so fast that the best way to “keep in the know” is to follow the leaders of the field. Best way to do that is social media. Some recommendations and most probably your social media of choice(Most of them are active on Twitter) will start showing you more relevant people :)
• Andrew Ng (https://twitter.com/AndrewYNg?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor)
• Yoshua Bengio (https://www.quora.com/profile/Yoshua-Bengio)
• Yann LeCun (https://twitter.com/ylecun?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor)
• Fie-Fie Li (https://twitter.com/drfeifei)
• Andrej Karpathy (https://twitter.com/karpathy)
• François Chollet (https://twitter.com/fchollet?lang=en)
  Infrastructure
  You will soon start lusting for a powerful GPU system. You will start paying more attention to new GPU release than to the new gadget release. I recommend building a GPU rig if you are serious about pursuing AI and playing with it. Following guides will be useful
• A Full Hardware Guide to Deep Learning - Tim Dettmers (http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/)
• Building your own deep learning box – Towards Data Science – Medium (https://medium.com/towards-data-science/building-your-own-deep-learning-box-47b918aea1eb)
• GTC 2017 (http://www.gputechconf.com/)