This arithmetica practice is from the paper **An Agent-based Approach to Care in Independent Living posted at UCI.**

The paper introduces a fall detector based on a **neural network** and
a multi-agent architecture for requesting emergency services. It presented a multi-agent system for the care of elderly
people living at home on their own, with the aim to prolong their independence.
The system is composed of seven groups of agents providing a reliable, robust
and flexible monitoring by sensing the user in the environment, reconstructing
the position and posture to create the physical awareness of the user in the
environment, reacting to critical situations, calling for help in the case of an
emergency, and issuing warnings if unusual behavior is detected. The system
has been tested during several on-line demonstration. People used for recording of the data were wearing four tags (ankle left, ankle right, belt and chest). Each instance is a localization data for one of the tags. The tag can be identified by one of the attributes. *The goal for this practice is to correctly predict the activity the user is performing.*

**Source:**

- Creators: Mitja Lustrek (__mitja.lustrek '@' ijs.si__), Bostjan Kaluza (

- Jozef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenija

- Donor: Bozidara Cvetkovic (

- Jozef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenija

- Date received: October, 2010

B. Kaluza, V. Mirchevska, E. Dovgan, M. Lustrek, M. Gams, An Agent-based Approach to Care in Independent Living, International Joint Conference on Ambient Intelligence (AmI-10), Malaga, Spain, In press |

The evaluation of this dataset is done using Area Under the ROC curve (AUC).

An example of its application are ROC curves. Here, the true positive rates are plotted against false positive rates. An example is below. The closer AUC for a model comes to 1, the better it is. So models with higher AUCs are preferred over those with lower AUCs.

Please note, there are also other methods than ROC curves but they are also related to the true positive and false positive rates, e. g. precision-recall, F1-Score or Lorenz curves.

AUC is used most of the time to mean AUROC, AUC is ambiguous (could be any curve) while AUROC is not.

The AUROC has several equivalent interpretations:

- The expectation that a uniformly drawn random positive is ranked before a uniformly drawn random negative.
- The expected proportion of positives ranked before a uniformly drawn random negative.
- The expected true positive rate if the ranking is split just before a uniformly drawn random negative.
- The expected proportion of negatives ranked after a uniformly drawn random positive.
- The expected false positive rate if the ranking is split just after a uniformly drawn random positive.

Assume we have a probabilistic, binary classifier such as logistic regression.

Before presenting the ROC curve (= Receiver Operating Characteristic curve), the concept ofconfusion matrix must be understood. When we make a binary prediction, there can be 4 types of outcomes:

- We predict 0 while we should have the class is actually 0: this is called a True Negative, i.e. we correctly predict that the class is negative (0). For example, an antivirus did not detect a harmless file as a virus .
- We predict 0 while we should have the class is actually 1: this is called a False Negative, i.e. we incorrectly predict that the class is negative (0). For example, an antivirus failed to detect a virus.
- We predict 1 while we should have the class is actually 0: this is called a False Positive, i.e. we incorrectly predict that the class is positive (1). For example, an antivirus considered a harmless file to be a virus.
- We predict 1 while we should have the class is actually 1: this is called a True Positive, i.e. we correctly predict that the class is positive (1). For example, an antivirus rightfully detected a virus.

To get the confusion matrix, we go over all the predictions made by the model, and count how many times each of those 4 types of outcomes occur:

In this example of a confusion matrix, among the 50 data points that are classified, 45 are correctly classified and the 5 are misclassified.

Since to compare two different models it is often more convenient to have a single metric rather than several ones, we compute two metrics from the confusion matrix, which we will later combine into one:

- True positive rate (TPR), aka. sensitivity, hit rate, and recall, which is defined as
TPTP+FN $\frac{TP}{TP+FN}$. Intuitively this metric corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points. In other words, the higher TPR, the fewer positive data points we will miss. - False positive rate (FPR), aka. fall-out, which is defined as
FPFP+TN $\frac{FP}{FP+TN}$. Intuitively this metric corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points. In other words, the higher FPR, the more negative data points we will missclassified.

To combine the FPR and the TPR into one single metric, we first compute the two former metrics with many different threshold (for example

The following figure shows the AUROC graphically:

In this figure, the blue area corresponds to the Area Under the curve of the Receiver Operating Characteristic (AUROC). The dashed line in the diagonal we present the ROC curve of a random predictor: it has an AUROC of 0.5. The random predictor is commonly used as a baseline to see whether the model is useful.

If you want to get some first-hand experience:

- Python: http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html
- MATLAB: http://www.mathworks.com/help/stats/perfcurve.html

You cannot sign up from multiple accounts and therefore you cannot submit from multiple accounts.

Privately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.

You may submit a maximum of 5 entries per day.

You may select up to 2 final submissions for judging.

- Use of external data is not permitted. This includes use of pre-trained models.
- Hand-labeling is allowed on the training dataset only. Hand-labeling is not permitted on test data and will be grounds for disqualification.

At Arithmetica we value math and data. We provide an ability to practice and learn datascience.

- +1 (843) 882-7674
- Dublin, CA

**Contacts**

Email: info@arithmetica.io

Phone: +1 (843) 882-7674

Fax: +1 (843) 882-7674