Generative Adversarial Network for Neural Decoding

Today I've been researching GAN's and how we might be able to use one for our problem. Right now I'm still focusing on fMRI data, but I hope to gain insight about developing models for calcium data as well. (Calcium imaging is a new neural imaging technique with far higher resolution.)

In Machine Learning most problems generalize to inferring latent variables from observable data (the feature space). Our case, inferring visual stimuli from fMRI data, is no exception. A discriminative model can accomplish this, but it can't do the reverse: that is, generate the data. A generative model, on the other hand, can do both.

A generative adversarial network simultaneously trains both a generative model G and a discriminative model D that compete with each other. While G trains on the training data, D's job is deciding whether data is from G or from actual training data. D and G participate in a so called zero-sum game, which means that the loss of one is the gain of the other and vice-versa.

Think of G as a animal of prey that evolves to be better and better at camouflage and D as a predator that becomes better at differentiating between prey and other surroundings. For our purposes, G trains on the neural imaging data, while D differentiates between actual neural imaging and imaging generated by G. 

To quote the original paper:

 The training procedure for G is to maximize the probability of D making a mistake. 

This is my idea for setting up a GAN for fMRI data:

Disregarding PCA for the moment, the generative model looks at the visual stimuli (the training labels if you will) and generates "fake" fMRI data which the discriminative model compares with real fMRI data. The problem is, that fMRI data has far too many dimensions so we use PCA for dimensionality reduction. 

PCA is basically singular value decomposition where the smaller singular values are thrown away. If you don't know what that means, think of it as a handy way to compress the fMRI data so we don't spend ages computing. Using PCA for dimensionality reduction on our GAN means that the generative model no longer generates actual fMRI data but the principal components of fMRI data. The discriminative model compares generated principal components with real principal components calculated from observed data.

I don't know if such a system will work or not. The challenge now is to implement such a network in Tensorflow and find out. If anyone has tips, suggestions or feedback please let me know. I'll keep you updated.