Authors¶
- Soumil Nitin Shah
Soumil Nitin Shah¶
Bachelor in Electronic Engineering | Masters in Electrical Engineering | Master in Computer Engineering |
- Website : https://soumilshah.herokuapp.com
- Github: https://github.com/soumilshah1995
- Linkedin: https://www.linkedin.com/in/shah-soumil/
- Blog: https://soumilshah1995.blogspot.com/
- Youtube : https://www.youtube.com/channel/UC_eOodxvwS_H7x2uLQa-svw?view_as=subscriber
- Facebook Page : https://www.facebook.com/soumilshah1995/
- Email : shahsoumil519@gmail.com
- projects : https://soumilshah.herokuapp.com/project
Excellent experience of building scalable and high-performance Software Applications combining distinctive skill sets in Internet of Things (IoT), Machine Learning and Full Stack Web Development in Python.
Step 1: Imports¶
In [80]:
import os
import pandas as pd
import spacy
In [81]:
df=pd.read_csv("https://raw.githubusercontent.com/hanzhang0420/Women-Clothing-E-commerce/master/Womens%20Clothing%20E-Commerce%20Reviews.csv")
In [82]:
df.head(3)
Out[82]:
In [83]:
df.isna().sum()
Out[83]:
In [84]:
df.shape
Out[84]:
In [85]:
df = df[['Review Text','Recommended IND']].dropna()
In [86]:
df.head(6)
Out[86]:
In [87]:
# Negative Text
df.iloc[5]['Review Text']
Out[87]:
Loading Spacy Library¶
In [88]:
nlp=spacy.load("en_core_web_sm")
nlp.pipe_names
Out[88]:
In [89]:
# we created a simple Text cat we are adding that to spacy :D
textcat = nlp.create_pipe( "textcat", config={"exclusive_classes": True, "architecture": "simple_cnn"})
In [90]:
# Adding this to Pipe
nlp.add_pipe(textcat, last=True)
In [91]:
nlp.pipe_names
Out[91]:
In [92]:
# Adding the labels to textcat
textcat.add_label("POSITIVE")
textcat.add_label("NEGATIVE")
Out[92]:
In [93]:
textcat.labels
Out[93]:
Step 2:¶
- Pre procesing
In [94]:
# Converting review text to tuple
df['tuples'] = df.apply(lambda row: (row['Review Text'],row['Recommended IND']), axis=1)
In [95]:
df.head(1)
Out[95]:
& This is how data looks like
In [96]:
# This is how data looks like
df["tuples"][0]
Out[96]:
In [97]:
# Converting tuple to List
train = df['tuples'].tolist()
In [98]:
print(train[0])
print(len(train))
In [99]:
texts, labels = zip(*train)
In [100]:
texts[0]
Out[100]:
In [101]:
labels[0]
Out[101]:
- what we did is bascially created Flags if its Positive aka True and negative we gave False
In [102]:
cats = []
for y in labels:
if(bool(y)):
cats.append({"POSITIVE": True, "NEGATIVE":False})
else:
cats.append({"POSITIVE": False, "NEGATIVE":True})
In [103]:
TrainX = texts
TrainY = cats
In [104]:
n_texts=23486
In [105]:
len(TrainX)
Out[105]:
In [106]:
len(TrainY)
Out[106]:
In [107]:
train_data = list(zip(TrainX,[{'cats': cats} for cats in TrainY]))
- Data has to be in this format
In [108]:
train_data[0]
Out[108]:
In [109]:
len(train_data)
Out[109]:
Model¶
In [112]:
len(train_data)
Out[112]:
In [113]:
n_iter=10
In [114]:
from spacy.util import minibatch, compounding
# Disabling other components
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']
with nlp.disable_pipes(*other_pipes): # only train textcat
optimizer = nlp.begin_training()
print("Training the model...")
# Performing training
for i in range(n_iter):
print("Epoch : {} ".format(i))
losses = {}
batches = minibatch(train_data, size=compounding(4., 32., 1.001))
for batch in batches:
texts, annotations = zip(*batch)
nlp.update(texts, annotations, sgd=optimizer, drop=0.2,
losses=losses)
In [115]:
nlp.to_disk("sentiment")
In [116]:
nlp = spacy.load("sentiment")
In [117]:
# Testing the model
test_text = "I had such high hopes for this dress and really crappy worst product hate it wporst bad "
doc=nlp(test_text)
doc.cats
Out[117]:
