Learn practical skills, build real-world projects, and advance your career
Created 3 years ago
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory
import os
for dirname, _, filenames in os.walk('/kaggle/input'):
for filename in filenames:
print(os.path.join(dirname, filename))
# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All"
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session
/kaggle/input/preprocess-choc/custom.css
/kaggle/input/preprocess-choc/__notebook__.ipynb
/kaggle/input/preprocess-choc/10 best RD_Feature
/kaggle/input/preprocess-choc/__results__.html
/kaggle/input/preprocess-choc/__output__.json
/kaggle/input/preprocess-choc/dfn.csv
/kaggle/input/preprocess-choc/__results___files/__results___8_1.png
!pip install libra
from libra import client #this is the only import you'll ever have to worry about :)
Requirement already satisfied: libra in /opt/conda/lib/python3.7/site-packages (1.1.1)
Requirement already satisfied: tensorflow==2.2.0 in /opt/conda/lib/python3.7/site-packages (from libra) (2.2.0)
Requirement already satisfied: keras-tuner in /opt/conda/lib/python3.7/site-packages (from libra) (1.0.1)
Requirement already satisfied: pillow in /opt/conda/lib/python3.7/site-packages (from libra) (7.2.0)
Requirement already satisfied: matplotlib in /opt/conda/lib/python3.7/site-packages (from libra) (3.2.1)
Requirement already satisfied: prince in /opt/conda/lib/python3.7/site-packages (from libra) (0.7.0)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from libra) (0.8.7)
Requirement already satisfied: nltk in /opt/conda/lib/python3.7/site-packages (from libra) (3.2.4)
Requirement already satisfied: spacy in /opt/conda/lib/python3.7/site-packages (from libra) (2.2.4)
Requirement already satisfied: xgboost in /opt/conda/lib/python3.7/site-packages (from libra) (1.1.1)
Requirement already satisfied: sklearn in /opt/conda/lib/python3.7/site-packages (from libra) (0.0)
Requirement already satisfied: jellyfish in /opt/conda/lib/python3.7/site-packages (from libra) (0.8.2)
Requirement already satisfied: colorama in /opt/conda/lib/python3.7/site-packages (from libra) (0.4.3)
Requirement already satisfied: download in /opt/conda/lib/python3.7/site-packages (from libra) (0.3.5)
Requirement already satisfied: opencv-python in /opt/conda/lib/python3.7/site-packages (from libra) (4.3.0.36)
Requirement already satisfied: transformers==2.11.0 in /opt/conda/lib/python3.7/site-packages (from libra) (2.11.0)
Requirement already satisfied: autocorrect in /opt/conda/lib/python3.7/site-packages (from libra) (2.0.0)
Requirement already satisfied: pprint in /opt/conda/lib/python3.7/site-packages (from libra) (0.1)
Requirement already satisfied: seaborn in /opt/conda/lib/python3.7/site-packages (from libra) (0.10.0)
Requirement already satisfied: keras==2.4.3 in /opt/conda/lib/python3.7/site-packages (from libra) (2.4.3)
Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from libra) (1.0.3)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from libra) (1.18.5)
Requirement already satisfied: textblob in /opt/conda/lib/python3.7/site-packages (from libra) (0.15.3)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (0.34.2)
Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (3.12.2)
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (2.10.0)
Requirement already satisfied: absl-py>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (0.9.0)
Requirement already satisfied: google-pasta>=0.1.8 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (0.2.0)
Requirement already satisfied: astunparse==1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.6.3)
Requirement already satisfied: keras-preprocessing>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.1.2)
Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (2.2.2)
Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (2.2.0)
Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.1.0)
Requirement already satisfied: wrapt>=1.11.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.11.2)
Requirement already satisfied: six>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.14.0)
Requirement already satisfied: scipy==1.4.1; python_version >= "3" in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.4.1)
Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (1.30.0)
Requirement already satisfied: gast==0.3.3 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (0.3.3)
Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow==2.2.0->libra) (3.3.0)
Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from keras-tuner->libra) (4.45.0)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from keras-tuner->libra) (2.23.0)
Requirement already satisfied: future in /opt/conda/lib/python3.7/site-packages (from keras-tuner->libra) (0.18.2)
Requirement already satisfied: terminaltables in /opt/conda/lib/python3.7/site-packages (from keras-tuner->libra) (3.1.0)
Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (from keras-tuner->libra) (0.23.1)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib->libra) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->libra) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->libra) (1.2.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib->libra) (2.8.1)
Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (1.0.0)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (2.0.3)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (1.0.2)
Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (0.7.1)
Requirement already satisfied: plac<1.2.0,>=0.9.6 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (1.1.3)
Requirement already satisfied: thinc==7.4.0 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (7.4.0)
Requirement already satisfied: blis<0.5.0,>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (0.4.1)
Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (1.0.2)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (46.1.3.post20200325)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from spacy->libra) (3.0.2)
Requirement already satisfied: sacremoses in /opt/conda/lib/python3.7/site-packages (from transformers==2.11.0->libra) (0.0.43)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from transformers==2.11.0->libra) (20.1)
Requirement already satisfied: sentencepiece in /opt/conda/lib/python3.7/site-packages (from transformers==2.11.0->libra) (0.1.91)
Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers==2.11.0->libra) (3.0.10)
Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers==2.11.0->libra) (2020.4.4)
Requirement already satisfied: tokenizers==0.7.0 in /opt/conda/lib/python3.7/site-packages (from transformers==2.11.0->libra) (0.7.0)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from keras==2.4.3->libra) (5.3.1)
Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.7/site-packages (from pandas->libra) (2019.3)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (1.0.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (1.14.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (1.7.0)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (3.2.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (0.4.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->keras-tuner->libra) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->keras-tuner->libra) (2.9)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->keras-tuner->libra) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->keras-tuner->libra) (2020.6.20)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->keras-tuner->libra) (2.1.0)
Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->keras-tuner->libra) (0.14.1)
Requirement already satisfied: importlib-metadata>=0.20; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from catalogue<1.1.0,>=0.0.7->spacy->libra) (1.7.0)
Requirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers==2.11.0->libra) (7.1.1)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (3.1.1)
Requirement already satisfied: rsa<4.1,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (4.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (0.2.7)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (1.2.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata>=0.20; python_version < "3.8"->catalogue<1.1.0,>=0.0.7->spacy->libra) (3.1.0)
Requirement already satisfied: pyasn1>=0.1.3 in /opt/conda/lib/python3.7/site-packages (from rsa<4.1,>=3.1.4->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0->libra) (3.0.1)
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/opt/conda/bin/python3.7 -m pip install --upgrade pip' command.
from libra import client
a_client = client('../input/preprocess-choc/dfn.csv')
print(a_client)
Creating client object
|- Reading in dataset
<libra.queries.client object at 0x7f140052c550>
a_client.neural_network_query('review_date', epochs=20) #lets say I want to run this query fairly quickly, so I limit the epochs / number of iterations
Reading in dataset
->Target column found: review_date
|- Establishing callback function
|- Training initial model
| Initial number of layers | Training Accuracy | Test Accuracy |
| 3 | 0.7962052226066589 | 0.7640449404716492 |
|- Testing number of layers
| Current number of layers | Training Accuracy | Test Accuracy |
| 3 | 0.7997189164161682 | 0.7724719047546387 |
| 4 | 0.8482080101966858 | 0.7921348214149475 |
| 5 | 0.8566408753395081 | 0.8033707737922668 |
| 6 | 0.8601546287536621 | 0.851123571395874 |
| 7 | 0.8594518899917603 | 0.8314606547355652 |
->Best number of layers found: 6
->Training Accuracy: 0.8601546287536621
->Test Accuracy: 0.851123571395874
|- Stored model under 'classification_ANN' key
import matplotlib.pyplot as plt
a_client.analyze()
plt.figure(figsize=(3,4))
Analyzing classification_ANN for further understanding
|- Making predictions for test data
|- Transforming dataset for plotting module
->Plotting ROC curves and creating confusion matrix...
|- Investigating potential issues with calculations
|- Gathering metrics for display:
->Accuracy on test set: 0.851123571395874
->Recall on test set: 0.851123595505618
->Precision on test set: 0.851123595505618
->F1 Score on test set: 0.851123595505618
<Figure size 216x288 with 0 Axes>
<Figure size 216x288 with 0 Axes>