Machine Learning Tutorial 1 : Wage Prediction

Machine learning has evolved over the past few years with advances in compute capacity and availability of data coupled with state-of-the-art algorithms that have been published by various research institutions.

It refers to the process by which software programs learn from experience with data and use this to make predictions or take certain actions.

This tutorial is the first of many that I would be posting over the next couple of weeks.
Here, I would be demonstrating how to train an algorithm on data from US Census to predict whether a person earns more than $50 000 dollars salary or not based on social factors such as his education, occupation and many more.

Enough with the Story! Let’s Get Started

Here we would be using the awesome machine learning framework “Google TensorFlow”

And run this experiment on Kaggle.  If you don’t have a Kaggle account, visit www.kaggle.com to register, its free to run machine learning experiments.

Create a new kernel and add this DataSet to it 


This full code for this tutorial is available just at the end.

Step 1: Import Packages
Import all the necessary python packages that would be needed

import tensorflow as tf
import tensorflow.contrib as contrib
import pandas as pd
import tempfile
import numpy as np
import tempfile

STEP 2 :  DEFINE COLUMNS
Next create an array of Columns that would be used :

COLUMNS = ["age", "workclass", "fnlwgt", "education", 

"education_num",
           "marital_status", "occupation", "relationship", "race", 

"gender",
           "capital_gain", "capital_loss", "hours_per_week", 

"native_country",
           "income_bracket"]
CATEGORICAL_COLUMNS = ["workclass", "education", "marital_status", 

"occupation",
                       "relationship", "race", "gender", 

"native_country"]
CONTINUOUS_COLUMNS = ["age", "education_num", "capital_gain", 

"capital_loss",
                      "hours_per_week"]

The COLUMNS array contains a list of both continuous and categorical features
While the two other arrays are subsets of the COLUMNS array, splitting the data into continuous and categorical features.

It is very important to define this boundaries as normal distributions (continuous features)  are represented differently than Binomial distributions with a finite set.

Step 3: LOAD DATA SETS

Define paths to the data files and load them as CSVs into Pandas Data frames

train_file = "../input/adult-training.csv"

test_file = "../input/adult-test.csv"

df_train = pd.read_csv(train_file, names = COLUMNS, 

skipinitialspace = True,engine= "python")
df_test = pd.read_csv(test_file,names = COLUMNS,skipinitialspace = True, skiprows=1, engine = "python")


df_train.dropna(how="any",axis = 0)
df_test.dropna(how="any", axis = 0)

The last part above is to remove missing values from the dataset. As seen above, training would be done on the Train data set. Once our model is built, we have to evaluate how well it would perform un unseen data, i.e. Data not used to train it. For this purpose, we set aside the test data set, which would be used to test the accuracy our model.

Step 4 : DEFINE TARGET COLUMN
We have to define our most important COLUMN, which is the income_bracket column of the dataset, it takes on two possible values  “>50K” and “<=50K” this narrows down our prediction task to a binary classification.

LABEL_COLUMN = "label"

df_train[LABEL_COLUMN] = (df_train["income_bracket"].apply(lambda x: 

">50K" in x)).astype(int)
df_test[LABEL_COLUMN] = (df_test["income_bracket"].apply(lambda x: 

">50K" in x)).astype(int)


STEP 5 : DEFINE TENSORS
Vectors are represented as Tensors in TensorFlow, any data point to be used as to be converted to a Tensor. TensorFlow provides many built-in functions to do this.

First lets convert the continuous columns

age = contrib.layers.real_valued_column("age")
age_buckets = contrib.layers.bucketized_column(age,boundaries=[18, 

25, 30, 35, 40, 45,50, 55, 60,65])
education_num = contrib.layers.real_valued_column("education_num")
capital_gain = contrib.layers.real_valued_column("capital_gain")
capital_loss = contrib.layers.real_valued_column("capital_loss")
hours_per_week = contrib.layers.real_valued_column("hours_per_week")


It is very important to consider that the age column cannot just be used as a continuous number with a positive specific correlation to the target label. Salary tends to increase with age but at a point, for example, retirement, salary decreases with age. It is impossible for a single variable to be both positively and negatively correlated, hence to represent this special relationship, we have to treat age as a categorical column, by splitting it into a bucketized column with age ranges.



Next we convert categorical columns

workclass = contrib.layers.sparse_column_with_hash_bucket("workclass", hash_bucket_size= 100)
education = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=100)
marital_status = contrib.layers.sparse_column_with_hash_bucket("marital_status",hash_bucket_size=100)
occupation = contrib.layers.sparse_column_with_hash_bucket("occupation",hash_bucket_size=1000)
relationship = contrib.layers.sparse_column_with_hash_bucket("relationship",hash_bucket_size=100)
race = contrib.layers.sparse_column_with_hash_bucket("race",hash_bucket_size=100)
native_country = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=1000)
gender = contrib.layers.sparse_column_with_keys("gender",keys=["male","female"])



In converting the categorical columns, we have to decide whether to use key values or hash buckets.
Columns with a small finite set, such as gender should be defined with its possible values as keys
However, occupation, country and others have a large set, hence you wouldn’t want to be typing in all their possible values, hence you should use hash buckets and specify their hash bucket size which would depend on the number of possible values. Larger sets should have a value of 1000 or more, while smaller sets could be 100 or less.



Finally We define cross columns

These are Tensors that combine the interaction of two or more different tensors. Often, there is great interdependence among the columns in a dataset. For example, two different people with the same age and occupation but of different races might earn different salaries. To represent this relationship, we created cross columns combining related columns together.

education_occupation = contrib.layers.crossed_column(columns=[education,occupation],hash_bucket_size= int(1e4))
age_education_occupation = contrib.layers.crossed_column(columns=[age_buckets,education,occupation], hash_bucket_size= int(1e6))
native_country_occupation = contrib.layers.crossed_column(columns= [native_country,occupation], hash_bucket_size= int(1e4))
race_occupation = contrib.layers.crossed_column(columns = [race,occupation], hash_bucket_size = int(1e4))



Having created all the needed Tensors, we combine them into a single array

wide_columns = [age,age_buckets,education_num,capital_gain,capital_loss,hours_per_week,workclass,education
                

,marital_status,occupation,relationship,race,native_country,gender,education_occupation,age_education_occupation
                ,native_country_occupation,race_occupation]



But that’s not all, in this tutorial, we are using an approach that would combine Linear Models with Deep Neural Networks to improve the accuracy of the model, hence you need to define Tensors for the Deeep Neural Networks part.

deep_columns = [age,education_num,capital_gain,capital_loss,hours_per_week,
                contrib.layers.embedding_column(workclass,dimension=8),
                contrib.layers.embedding_column(education,dimension=8),
                contrib.layers.embedding_column(marital_status,dimension=8),
                contrib.layers.embedding_column(occupation,dimension=8),
                contrib.layers.embedding_column(relationship,dimension=8),
                contrib.layers.embedding_column(race,dimension=8),
                contrib.layers.embedding_column(native_country,dimension=8),
                contrib.layers.embedding_column(gender,dimension=8)
                ]


Step 6 : Input Builders
  The Input builder functions perform transformations on the data, it is used to pass in data into the Classifier
The return type is a tuple consisting of the combined columns and the target label

def input_function(df):
    continuos_cols = {k: tf.constant(df[k].values) for k in 

CONTINUOUS_COLUMNS}

    categorical_cols = {k: tf.SparseTensor(indices= [[i,0] for i in 

range(df[k].size)],
                                           values= df[k].values,
                                           dense_shape= [df

[k].size,1])
                        for k in CATEGORICAL_COLUMNS}

    label = tf.constant(df[LABEL_COLUMN].values)

    feature_cols = dict(continuos_cols)
    feature_cols.update(categorical_cols)
    return feature_cols,label



STEP 7 : DEFINE AND FIT YOUR MODEL
Here is the main part we are building up to, having prepared our data properly, we are now ready to train the algorithm.
The generated model would be saved in a folder, here we shall use a temp folder


#DEFINE MODEL DIR
model_dir = tempfile.mkdtemp()


#BUILD AND TRAIN MODEL

m = contrib.learn.DNNLinearCombinedClassifier(model_dir = 
model_dir,linear_feature_columns=wide_columns,dnn_feature_columns=deep_columns,
                                              dnn_hidden_units= [100,50],fix_global_step_increment_bug=True)


m.fit(input_fn= lambda: input_function(df_train),steps= 200)


Finally, we evaluate the accuracy of the model by using the test data.

results = m.evaluate(input_fn = lambda: input_function(df_test),steps = 1)

for key in sorted(results):
    print(key, end= " ")
    print(results[key])


I had an accuracy of approximately 84% with this experiment.
Your result should be similar to these:

accuracy 0.837049 accuracy/baseline_label_mean 0.236226 accuracy/threshold_0.500000_mean 0.837049 auc 0.818777 auc_precision_recall 0.676786 global_step 200 labels/actual_label_mean 0.236226 labels/prediction_mean 0.186246 loss 0.776201 precision/positive_threshold_0.500000_mean 0.748438 recall/positive_threshold_0.500000_mean 0.467239



  
That’s it, we have just developed a sophiscated model that can predict with about 84% accuracy, whether a citizen earns more than $50,000 or not.

Drop your questions in the comments section below. I would love to hear your feedback and help with any questions you have. Any Bugs should be pointed out too.


Come back soon for more tutorials.


Here is the link to the complete code

https://www.kaggle.com/johnolafenwa/wage-prediction



import tensorflow as tf
import tensorflow.contrib as contrib
import pandas as pd
import tempfile
import numpy as np
import tempfile



#WAGE PREDICTION USING A COMBINATION OF WIDE AND DEEP LEARNING ALGORITHMS

#DEFINE PATHS TO DATA FILES

train_file = "../input/adult-training.csv"

test_file = "../input/adult-test.csv"

#DEFINE COLUMNS
COLUMNS = ["age", "workclass", "fnlwgt", "education", 

"education_num",
           "marital_status", "occupation", "relationship", "race", 

"gender",
           "capital_gain", "capital_loss", "hours_per_week", 

"native_country",
           "income_bracket"]
LABEL_COLUMN = "label"
CATEGORICAL_COLUMNS = ["workclass", "education", "marital_status", 

"occupation",
                       "relationship", "race", "gender", 

"native_country"]
CONTINUOUS_COLUMNS = ["age", "education_num", "capital_gain", 

"capital_loss",
                      "hours_per_week"]

df_train = pd.read_csv(train_file, names = COLUMNS, 

skipinitialspace = True,engine= "python")
df_test = pd.read_csv(test_file,names = COLUMNS,skipinitialspace = True, skiprows=1, engine = "python")

#Remove NaN

df_train.dropna(how="any",axis = 0)
df_test.dropna(how="any", axis = 0)

#Set LABEL COLUMN
df_train[LABEL_COLUMN] = (df_train["income_bracket"].apply(lambda x: 

">50K" in x)).astype(int)
df_test[LABEL_COLUMN] = (df_test["income_bracket"].apply(lambda x: 

">50K" in x)).astype(int)

#CREATE CONTINUOS COLUMNS

age = contrib.layers.real_valued_column("age")
age_buckets = contrib.layers.bucketized_column(age,boundaries=[18, 

25, 30, 35, 40, 45,
                                                        50, 55, 60, 

65])
education_num = contrib.layers.real_valued_column("education_num")
capital_gain = contrib.layers.real_valued_column("capital_gain")
capital_loss = contrib.layers.real_valued_column("capital_loss")
hours_per_week = contrib.layers.real_valued_column("hours_per_week")

#CREATE CATEGORICAL COLUMNS
workclass = contrib.layers.sparse_column_with_hash_bucket("workclass", hash_bucket_size= 100)
education = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=100)
marital_status = contrib.layers.sparse_column_with_hash_bucket("marital_status",hash_bucket_size=100)
occupation = contrib.layers.sparse_column_with_hash_bucket("occupation",hash_bucket_size=1000)
relationship = contrib.layers.sparse_column_with_hash_bucket("relationship",hash_bucket_size=100)
race = contrib.layers.sparse_column_with_hash_bucket("race",hash_bucket_size=100)
native_country = contrib.layers.sparse_column_with_hash_bucket("education",hash_bucket_size=1000)
gender = contrib.layers.sparse_column_with_keys("gender",keys=["male","female"])

#CROSS COLUMNS
education_occupation = contrib.layers.crossed_column(columns=[education,occupation],hash_bucket_size= int(1e4))
age_education_occupation = contrib.layers.crossed_column(columns=[age_buckets,education,occupation], hash_bucket_size= int(1e6))
native_country_occupation = contrib.layers.crossed_column(columns= [native_country,occupation], hash_bucket_size= int(1e4))
race_occupation = contrib.layers.crossed_column(columns = [race,occupation], hash_bucket_size = int(1e4))

#WIDE COLUMNS
wide_columns = [age,age_buckets,education_num,capital_gain,capital_loss,hours_per_week,workclass,education
                

,marital_status,occupation,relationship,race,native_country,gender,education_occupation,age_education_occupation
                ,native_country_occupation,race_occupation]

deep_columns = [age,education_num,capital_gain,capital_loss,hours_per_week,
                contrib.layers.embedding_column(workclass,dimension=8),
                contrib.layers.embedding_column(education,dimension=8),
                contrib.layers.embedding_column(marital_status,dimension=8),
                contrib.layers.embedding_column(occupation,dimension=8),
                contrib.layers.embedding_column(relationship,dimension=8),
                contrib.layers.embedding_column(race,dimension=8),
                contrib.layers.embedding_column(native_country,dimension=8),
                contrib.layers.embedding_column(gender,dimension=8)
                ]

def input_function(df):
    continuos_cols = {k: tf.constant(df[k].values) for k in 

CONTINUOUS_COLUMNS}

    categorical_cols = {k: tf.SparseTensor(indices= [[i,0] for i in 

range(df[k].size)],
                                           values= df[k].values,
                                           dense_shape= [df

[k].size,1])
                        for k in CATEGORICAL_COLUMNS}

    label = tf.constant(df[LABEL_COLUMN].values)

    feature_cols = dict(continuos_cols)
    feature_cols.update(categorical_cols)
    return feature_cols,label

#DEFINE MODEL DIR
model_dir = tempfile.mkdtemp()


#BUILD AND TRAIN MODEL

m = contrib.learn.DNNLinearCombinedClassifier(model_dir = 
model_dir,linear_feature_columns=wide_columns,dnn_feature_columns=deep_columns,
                                              dnn_hidden_units= [100,50],fix_global_step_increment_bug=True)


m.fit(input_fn= lambda: input_function(df_train),steps= 200)

results = m.evaluate(input_fn = lambda: input_function(df_test),steps = 1)

for key in sorted(results):
    print(key, end= " ")
    print(results[key])







Comments

  1. If you can explain the meaning of the final return result, it would be great

    ReplyDelete
  2. Hello, Everyone, I' m Adrik Vadim living in Kurgan City , I want to share with you all on here on how Mr Pedro help me with a loan of 15,000.000.00 Rubles to start up my food beverage delivery after all I have work in several hotels here in kurgan just to earn a living but unfortunate I was still having difficulties to pay a rent but I thank God now that I'm a self employ now with 5 workers working in my care. Just if you are looking for financial freedom I will advise you to contact Mr Pedro with this email below pedroloanss@gmail.com

    ReplyDelete

Post a Comment

Popular posts from this blog

Happy New Year