Machine Learning Tutorial 1 : Wage Prediction
Machine
learning has evolved over the past few years with advances in compute capacity
and availability of data coupled with state-of-the-art algorithms that have
been published by various research institutions.
It
refers to the process by which software programs learn from experience with
data and use this to make predictions or take certain actions.
This
tutorial is the first of many that I would be posting over the next couple of
weeks.
Here,
I would be demonstrating how to train an algorithm on data from US Census to
predict whether a person earns more than $50 000 dollars salary or not based on
social factors such as his education, occupation and many more.
Enough
with the Story! Let’s Get Started
Here
we would be using the awesome machine learning framework “Google TensorFlow”
And
run this experiment on Kaggle. If you
don’t have a Kaggle account, visit www.kaggle.com
to register, its free to run machine learning experiments.
Create
a new kernel and add this DataSet to it
This
full code for this tutorial is available just at the end.
Step
1: Import Packages
Import
all the necessary python packages that would be needed
import tensorflow as tf
import tensorflow.contrib as contrib
import pandas as pd
import tempfile
import numpy as np
import tempfile
STEP
2 : DEFINE COLUMNS
Next
create an array of Columns that would be used :
COLUMNS = ["age", "workclass", "fnlwgt", "education",
"education_num",
"marital_status", "occupation", "relationship", "race",
"gender",
"capital_gain", "capital_loss", "hours_per_week",
"native_country",
"income_bracket"]
CATEGORICAL_COLUMNS = ["workclass", "education", "marital_status",
"occupation",
"relationship", "race", "gender",
"native_country"]
CONTINUOUS_COLUMNS = ["age", "education_num", "capital_gain",
"capital_loss",
"hours_per_week"]
The
COLUMNS array contains a list of both continuous and categorical features
While
the two other arrays are subsets of the COLUMNS array, splitting the data into
continuous and categorical features.
It
is very important to define this boundaries as normal distributions (continuous
features) are represented differently
than Binomial distributions with a finite set.
Step
3: LOAD DATA SETS
Define
paths to the data files and load them as CSVs into Pandas Data frames
train_file = "../input/adult-training.csv"
test_file = "../input/adult-test.csv"
df_train = pd.read_csv(train_file, names = COLUMNS,
skipinitialspace = True,engine= "python")
df_test = pd.read_csv(test_file,names = COLUMNS,skipinitialspace = True, skiprows=1, engine = "python")
df_train.dropna(how="any",axis = 0)
df_test.dropna(how="any", axis = 0)
The
last part above is to remove missing values from the dataset. As seen above,
training would be done on the Train data set. Once our model is built, we have
to evaluate how well it would perform un unseen data, i.e. Data not used to
train it. For this purpose, we set aside the test data set, which would be used
to test the accuracy our model.
Step
4 : DEFINE TARGET COLUMN
We
have to define our most important COLUMN, which is the income_bracket column of
the dataset, it takes on two possible values
“>50K” and “<=50K” this narrows down our prediction task to a
binary classification.
LABEL_COLUMN = "label"
df_train[LABEL_COLUMN] = (df_train["income_bracket"].apply(lambda x:
">50K" in x)).astype(int)
df_test[LABEL_COLUMN] = (df_test["income_bracket"].apply(lambda x:
">50K" in x)).astype(int)
STEP
5 : DEFINE TENSORS
Vectors
are represented as Tensors in TensorFlow, any data point to be used as to be
converted to a Tensor. TensorFlow provides many built-in functions to do this.
First
lets convert the continuous columns
age = contrib.layers.real_valued_column("age")
age_buckets = contrib.layers.bucketized_column(age,boundaries=[18,
25, 30, 35, 40, 45,50, 55, 60,65])
education_num = contrib.layers.real_valued_column("education_num")
capital_gain = contrib.layers.real_valued_column("capital_gain")
capital_loss = contrib.layers.real_valued_column("capital_loss")
hours_per_week = contrib.layers.real_valued_column("hours_per_week")
It
is very important to consider that the age column cannot just be used as a
continuous number with a positive specific correlation to the target label. Salary
tends to increase with age but at a point, for example, retirement, salary
decreases with age. It is impossible for a single variable to be both
positively and negatively correlated, hence to represent this special
relationship, we have to treat age as a categorical column, by splitting it
into a bucketized column with age ranges.
Next
we convert categorical columns
workclass = contrib.layers.sparse_column_with_hash_bucket("workclass", hash_bucket_size= 100)
education = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=100)
marital_status = contrib.layers.sparse_column_with_hash_bucket("marital_status",hash_bucket_size=100)
occupation = contrib.layers.sparse_column_with_hash_bucket("occupation",hash_bucket_size=1000)
relationship = contrib.layers.sparse_column_with_hash_bucket("relationship",hash_bucket_size=100)
race = contrib.layers.sparse_column_with_hash_bucket("race",hash_bucket_size=100)
native_country = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=1000)
gender = contrib.layers.sparse_column_with_keys("gender",keys=["male","female"])
In
converting the categorical columns, we have to decide whether to use key values
or hash buckets.
Columns
with a small finite set, such as gender should be defined with its possible
values as keys
However,
occupation, country and others have a large set, hence you wouldn’t want to be
typing in all their possible values, hence you should use hash buckets and
specify their hash bucket size which would depend on the number of possible values.
Larger sets should have a value of 1000 or more, while smaller sets could be
100 or less.
Finally
We define cross columns
These
are Tensors that combine the interaction of two or more different tensors.
Often, there is great interdependence among the columns in a dataset. For
example, two different people with the same age and occupation but of different
races might earn different salaries. To represent this relationship, we created
cross columns combining related columns together.
education_occupation = contrib.layers.crossed_column(columns=[education,occupation],hash_bucket_size= int(1e4))
age_education_occupation = contrib.layers.crossed_column(columns=[age_buckets,education,occupation], hash_bucket_size= int(1e6))
native_country_occupation = contrib.layers.crossed_column(columns= [native_country,occupation], hash_bucket_size= int(1e4))
race_occupation = contrib.layers.crossed_column(columns = [race,occupation], hash_bucket_size = int(1e4))
Having
created all the needed Tensors, we combine them into a single array
wide_columns = [age,age_buckets,education_num,capital_gain,capital_loss,hours_per_week,workclass,education
,marital_status,occupation,relationship,race,native_country,gender,education_occupation,age_education_occupation
,native_country_occupation,race_occupation]
But
that’s not all, in this tutorial, we are using an approach that would combine
Linear Models with Deep Neural Networks to improve the accuracy of the model,
hence you need to define Tensors for the Deeep Neural Networks part.
deep_columns = [age,education_num,capital_gain,capital_loss,hours_per_week,
contrib.layers.embedding_column(workclass,dimension=8),
contrib.layers.embedding_column(education,dimension=8),
contrib.layers.embedding_column(marital_status,dimension=8),
contrib.layers.embedding_column(occupation,dimension=8),
contrib.layers.embedding_column(relationship,dimension=8),
contrib.layers.embedding_column(race,dimension=8),
contrib.layers.embedding_column(native_country,dimension=8),
contrib.layers.embedding_column(gender,dimension=8)
]
Step
6 : Input Builders
The Input builder functions perform
transformations on the data, it is used to pass in data into the Classifier
The
return type is a tuple consisting of the combined columns and the target label
def input_function(df):
continuos_cols = {k: tf.constant(df[k].values) for k in
CONTINUOUS_COLUMNS}
categorical_cols = {k: tf.SparseTensor(indices= [[i,0] for i in
range(df[k].size)],
values= df[k].values,
dense_shape= [df
[k].size,1])
for k in CATEGORICAL_COLUMNS}
label = tf.constant(df[LABEL_COLUMN].values)
feature_cols = dict(continuos_cols)
feature_cols.update(categorical_cols)
return feature_cols,label
STEP
7 : DEFINE AND FIT YOUR MODEL
Here
is the main part we are building up to, having prepared our data properly, we
are now ready to train the algorithm.
The
generated model would be saved in a folder, here we shall use a temp folder
#DEFINE MODEL DIR
model_dir = tempfile.mkdtemp()
#BUILD AND TRAIN MODEL
m = contrib.learn.DNNLinearCombinedClassifier(model_dir =
model_dir,linear_feature_columns=wide_columns,dnn_feature_columns=deep_columns,
dnn_hidden_units= [100,50],fix_global_step_increment_bug=True)
m.fit(input_fn= lambda: input_function(df_train),steps= 200)
Finally,
we evaluate the accuracy of the model by using the test data.
results = m.evaluate(input_fn = lambda: input_function(df_test),steps = 1)
for key in sorted(results):
print(key, end= " ")
print(results[key])
I had an accuracy of approximately 84% with this experiment.
Your result should be similar to these:
accuracy 0.837049 accuracy/baseline_label_mean 0.236226 accuracy/threshold_0.500000_mean 0.837049 auc 0.818777 auc_precision_recall 0.676786 global_step 200 labels/actual_label_mean 0.236226 labels/prediction_mean 0.186246 loss 0.776201 precision/positive_threshold_0.500000_mean 0.748438 recall/positive_threshold_0.500000_mean 0.467239
That’s it, we have just
developed a sophiscated model that can predict with about 84% accuracy, whether
a citizen earns more than $50,000 or not.
Drop your questions in the
comments section below. I would love to hear your feedback and help with any
questions you have. Any Bugs should be pointed out too.
Come back soon for more
tutorials.
Here is the link to the complete code
https://www.kaggle.com/johnolafenwa/wage-prediction
https://www.kaggle.com/johnolafenwa/wage-prediction
import tensorflow as tf
import tensorflow.contrib as contrib
import pandas as pd
import tempfile
import numpy as np
import tempfile
#WAGE PREDICTION USING A COMBINATION OF WIDE AND DEEP LEARNING ALGORITHMS
#DEFINE PATHS TO DATA FILES
train_file = "../input/adult-training.csv"
test_file = "../input/adult-test.csv"
#DEFINE COLUMNS
COLUMNS = ["age", "workclass", "fnlwgt", "education",
"education_num",
"marital_status", "occupation", "relationship", "race",
"gender",
"capital_gain", "capital_loss", "hours_per_week",
"native_country",
"income_bracket"]
LABEL_COLUMN = "label"
CATEGORICAL_COLUMNS = ["workclass", "education", "marital_status",
"occupation",
"relationship", "race", "gender",
"native_country"]
CONTINUOUS_COLUMNS = ["age", "education_num", "capital_gain",
"capital_loss",
"hours_per_week"]
df_train = pd.read_csv(train_file, names = COLUMNS,
skipinitialspace = True,engine= "python")
df_test = pd.read_csv(test_file,names = COLUMNS,skipinitialspace = True, skiprows=1, engine = "python")
#Remove NaN
df_train.dropna(how="any",axis = 0)
df_test.dropna(how="any", axis = 0)
#Set LABEL COLUMN
df_train[LABEL_COLUMN] = (df_train["income_bracket"].apply(lambda x:
">50K" in x)).astype(int)
df_test[LABEL_COLUMN] = (df_test["income_bracket"].apply(lambda x:
">50K" in x)).astype(int)
#CREATE CONTINUOS COLUMNS
age = contrib.layers.real_valued_column("age")
age_buckets = contrib.layers.bucketized_column(age,boundaries=[18,
25, 30, 35, 40, 45,
50, 55, 60,
65])
education_num = contrib.layers.real_valued_column("education_num")
capital_gain = contrib.layers.real_valued_column("capital_gain")
capital_loss = contrib.layers.real_valued_column("capital_loss")
hours_per_week = contrib.layers.real_valued_column("hours_per_week")
#CREATE CATEGORICAL COLUMNS
workclass = contrib.layers.sparse_column_with_hash_bucket("workclass", hash_bucket_size= 100)
education = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=100)
marital_status = contrib.layers.sparse_column_with_hash_bucket("marital_status",hash_bucket_size=100)
occupation = contrib.layers.sparse_column_with_hash_bucket("occupation",hash_bucket_size=1000)
relationship = contrib.layers.sparse_column_with_hash_bucket("relationship",hash_bucket_size=100)
race = contrib.layers.sparse_column_with_hash_bucket("race",hash_bucket_size=100)
native_country = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=1000)
gender = contrib.layers.sparse_column_with_keys("gender",keys=["male","female"])
#CROSS COLUMNS
education_occupation = contrib.layers.crossed_column(columns=[education,occupation],hash_bucket_size= int(1e4))
age_education_occupation = contrib.layers.crossed_column(columns=[age_buckets,education,occupation], hash_bucket_size= int(1e6))
native_country_occupation = contrib.layers.crossed_column(columns= [native_country,occupation], hash_bucket_size= int(1e4))
race_occupation = contrib.layers.crossed_column(columns = [race,occupation], hash_bucket_size = int(1e4))
#WIDE COLUMNS
wide_columns = [age,age_buckets,education_num,capital_gain,capital_loss,hours_per_week,workclass,education
,marital_status,occupation,relationship,race,native_country,gender,education_occupation,age_education_occupation
,native_country_occupation,race_occupation]
deep_columns = [age,education_num,capital_gain,capital_loss,hours_per_week,
contrib.layers.embedding_column(workclass,dimension=8),
contrib.layers.embedding_column(education,dimension=8),
contrib.layers.embedding_column(marital_status,dimension=8),
contrib.layers.embedding_column(occupation,dimension=8),
contrib.layers.embedding_column(relationship,dimension=8),
contrib.layers.embedding_column(race,dimension=8),
contrib.layers.embedding_column(native_country,dimension=8),
contrib.layers.embedding_column(gender,dimension=8)
]
def input_function(df):
continuos_cols = {k: tf.constant(df[k].values) for k in
CONTINUOUS_COLUMNS}
categorical_cols = {k: tf.SparseTensor(indices= [[i,0] for i in
range(df[k].size)],
values= df[k].values,
dense_shape= [df
[k].size,1])
for k in CATEGORICAL_COLUMNS}
label = tf.constant(df[LABEL_COLUMN].values)
feature_cols = dict(continuos_cols)
feature_cols.update(categorical_cols)
return feature_cols,label
#DEFINE MODEL DIR
model_dir = tempfile.mkdtemp()
#BUILD AND TRAIN MODEL
m = contrib.learn.DNNLinearCombinedClassifier(model_dir =
model_dir,linear_feature_columns=wide_columns,dnn_feature_columns=deep_columns,
dnn_hidden_units= [100,50],fix_global_step_increment_bug=True)
m.fit(input_fn= lambda: input_function(df_train),steps= 200)
results = m.evaluate(input_fn = lambda: input_function(df_test),steps = 1)
for key in sorted(results):
print(key, end= " ")
print(results[key])
If you can explain the meaning of the final return result, it would be great
ReplyDeleteHello, Everyone, I' m Adrik Vadim living in Kurgan City , I want to share with you all on here on how Mr Pedro help me with a loan of 15,000.000.00 Rubles to start up my food beverage delivery after all I have work in several hotels here in kurgan just to earn a living but unfortunate I was still having difficulties to pay a rent but I thank God now that I'm a self employ now with 5 workers working in my care. Just if you are looking for financial freedom I will advise you to contact Mr Pedro with this email below pedroloanss@gmail.com
ReplyDelete