Hello!

I am doing the Quora Views challenge. In upcoming posts, I will chronicle my progress. In this post, I discuss the data munging after introducing the challenge.

Prompt

The questions the challenge wants to address are the following:

• Can you tell which questions can organically attract the most viewers?
• What about questiosn that eventually become viral?
• Which questions are timeless and can sustain traffic?

“Organically attract.” Hmm, what does that exactly mean?

The challenge is to predict the number of views per day in age of the question, given the following

1. quora question text,
2. topic data,
3. number of answers, and
4. number of people promoted to.

Training Data

The sample testcases folder contains two files input00.in and output00.out.

The input file has 10,000 questions as valid json objects. They are divided into two groups. The first group has 9,000 questions with the following fields:

question_key, question_text, context_topic, topics, anonymous, num_answers, promoted_to, __ans__

The last field is a (float) ratio of viewers to age of the question in days. In the second group, the questions have all but this field.

The output file has 1,000 rows of valid json objects. There are two fields in each line. A question_key key containing the same unique identifier as in the test/input file. And the predicted value keyed by __ans__.

So our data is split 9:1, training to test.

The Fields

This is copied from the prompt page:

• question_key (string): Unique identifier for the question.
• question_text (string): Text of the question.
• context_topic (object): The primary topic of a question, if present. Null otherwise. This object will contain name(string) and a count of followers(integer). The number of followers doesn’t exceed 106.
• topics (array of objects): All topics on a question, including the primary topic. Each topic object will contain a name and followers count.
• anonymous (boolean): Whether the question was anonymous.
• num_answers (integer): The number of visible non-collapsed answers the question has.
• promoted_to (integer): The number of people the question was promoted to.
• __ans__ (float): The ratio of viewers to age of the question in days.

Data Munging

After importing the input00.in file, each of its line is added to a list as a valid json object. Before converting the list data, the two integers marking the number of lines are removed. Finally, a dataframe df is generated from the list

json_data = open('../views/sample/input00.in') # Edit this to where you have put the input00.in file
import pandas as pd
import json

data = []
for line in json_data:
    data.append(json.loads(line))

data.remove(9000)
data.remove(1000)

df = pd.DataFrame(data)

The tail of df has a column __ans__ of NaN; as discussed earlier, they are missing because the last 1000 lines of our data are our test set and __ans__ is the target variable.

Notice that each row of context_topic and topics contains a valid json object and a list containing json objects, respectively. Recall that context_topic is the primary topic of the question and it’s not always present. On the other hand, topics is all topics on a question.

For now, the topics column is left as it is. But the context_topic column will be split to context_topic_followers and context_topic_name.

context_topic_df = df['context_topic'].apply(pd.Series).rename(columns='context_topic_{}'.format)

cleaned_df = pd.concat([df.drop(['context_topic'], axis=1), context_topic_df], axis = 1)

test = cleaned_df[9000:]

train = cleaned_df[:9000]