Project 2: Neural nets (multilayer perceptron networks)

The goal of this project is to implement the feed-forward algorithm for applying a multilayer perceptron to a data point and the backpropagation algorithm for training a multilayer perceptron. To simplify this problem, we will assume that this MLP does binary classification.

Write two functions in Python 3 with the following signatures:

classify(mlp, inputs)

train(M, data, targets)

Where

• mlp is something representing a multilayer perceptron
• inputs is an array-like with shape (x, d), containing x new data data points each of dimension d.
• M is an integer, the number of perceptrons in a hidden layer
• data is an array-like with shape (n,d), containing n data points each of dimension d.
• targets is a list of size n containing the target values for the data points in data.

The train function returns something that represents a multilayer perceptron with M hidden units in one hidden layer and a single output unit that returns 0 or 1 (or -1 or 1; or True or False). The number of input units is determined by the dimensionality of the training data. How the MLP is represented is up to you, but your train function must return something that can be passed to the first parameter of your classify function.

Test your implementation on at least two data sets and report on the performance.

Submit your code in two files, mlp.py and test_mlp.py so that the following code will work:

import mlp

clsfyr = mlp.train(30, X_train, y_train)

results = mlp.classify(clsyr, X_test)

Moreover, the following should work from the command line:

python3 test_mlp.py

Which should display information about the performance of your classifier.

Finally, include a file README that (briefly) describes how you tested your classifier and to what results, and anything else you think it would be good for me to know in order for me to give your submission the fairest grading.

Specific guidelines:

• You should implement these algorithms yourself. Do not use sklearn.neural_network.MLPClassifier or any similar available implementation. You may use library functions for pieces of your implementation as long as they do not violate the spirit of an assignment to implement feed-forward and backpropagation. (You may use sklearn.neural_network.MLPClassifier or something similar to generate results for comparison with those of your classifier.)
• Do not consult any resource that gives source code for these algorithms. Do not look at the source code for sklearn or similar libraries. The book Data Science from Scratch, which I placed in the lab as a reference book for this course, contains Python code for these two algorithms. I have removed it from the lab for the duration of this project.
• Any resource you consult besides class notes, our textbook, and the official documentation of the libraries used in lab activities must be cited in your README file. (This is a policy for our course, not just for this project.)
• In principle you may choose what activation function to use. However, the pseudocode presented in class assumes logistic sigmoid as the activation function. This make as difference because the weight updates in the algorithm are based on the gradient of the error function, which in turn depends on the derivative of the activation funciton. If you use a different activation function, you will need to use different formulas for updating the weights. See Subsection 5.3.2 in our textbook for formulas based on hyperbolic tangent as an activation function.
• If you would like to use the Perceptron class that I used for in-class examples, you can get that code from /homes/tvandrun/Public/cs394/perceptron.py. But that's up to you.

Update, March 25. It seems like those students who have put time into this project so far are having a lot of trouble getting backpropagation to work. So that we can make something of this project, here's what I am going to do and suggest:

• I am returning Data Science from Scratch to the lab. You can see that book's implementation of backpropagation on pg 219. Obviously, don't copy it. I highly recommend you do your own implementation first, and then compare (if you use the book at all). Additionally I am putting two other resources on backpropagation in the lab, one from Thomas Mitchel, Machine Learning and the other from Stephen Marsland, Machine Learning: An Algorithmic Perspective.
• As I said in class on Friday, start small. Try to train a single perceptrion. Try to train a two-unit XOR. Then go for something more substantial
• You may work together. The syllabus says it's ok.
• It may well be that some of you aren't able to get a working version, and that I will need to grade on progress. Either way, make sure you document what you did, to what success. If you want me to give you fair credit for the work you did on this project, tell me about it in your README file. Also, I'll probably end up reading more code that I normally would, so make sure your code is sufficiently documented.

To turn in:

Copy mlp.py, test_mlp.py, README, and any other files your code needs (such as data sets for testing) to

/cslab/class/cs394/[your userid]/mlp

Due Fri, March 29