Pythonic 「Data Science」 Specialization

Why The "Data Science" Specialization

1. 溫習統計學的知識, 爲更深層次的學習作準備
   Andrew Ng 在 2015 GTC 的演講中說, deep learning 就是 black magic; 咱們理解50%, 但不知道另外的50%是如何work的. 我在臺下想, 對於那能夠理解的50%, 我好像都只懂了5%.
2. 參考"標準高效"的流程
   mine: emacs org mode + emacs magit + bitbucket + python. There must be some room for improvement.

How

課程用的是R. 我不想再學一門相似的語言了, 我會找出相對應的numpy 和 scipy solution.

Getting and Cleaning Data

Raw data 的來源

• Website APIs
• Databases
• Json
• Raw texts

Data analysis 流程

• Raw data --> Processing scripts --> tidy data (often ignored in the classes but really important)

  • Record the meta data
  • Record the recipes

• --> data analysis (covered in machine learning classes)

• --> data communication

什麼是乾淨的data

1. Each variable you measure should be in one column, 一個變量佔一列.
2. There should be one table for each "kind" of variable, generally data should be save in one file per table 爲何呢? 管理起來不會麻煩麼?
3. If you have multiple tables, they should include a column in the table thta allows them to be linked. 參見 dataframe.merge dataframe.join in pandas

The code book

代碼簿? (⊙o⊙)…

• Info about the variables (including units!)
  單位很重要! 沒有單位的測量是沒有物理意義的!
  但測量時候必需要考慮的有效位數在課程中卻沒有說起. 大抵是由於python 和 R 對於有效位數handle地很好? 不須要像C 裏邊同樣考慮 float 或者 double? 某些極端狀況下也會須要像sympy這樣的library吧.
• Info about the summary choice you made
• Info about the experimental study design you used

代碼簿的做用相似於wet lab中的實驗記錄本. 很慶幸很早就知道了emacs 的 org mode, 用在這裏很適合. 可是 Info about the variables 的重要性被我忽略了.

若是feature的數量不少, 並且feature自己意義深入, 就須要仔細挑選. 記得一次聽報告, 有家金融公司用decision tree 作portfolio, 算法自己稀鬆日常, 可是對於具體用了哪些feature, lecturer守口如瓶.

"There are many stages to the design and analysis of a successful study. The last of these steps is the calculation of an inferential statistic such as a P value, and the application of a 'decision rule' to it (for example, P < 0.05). In practice, decisions that are made earlier in data analysis have a much greater impact on results — from experimental design to batch effects, lack of adjustment for confounding factors, or simple measurement error. Arbitrary levels of statistical significance can be achieved by changing the ways in which data are cleaned, summarized or modelled."

Leek, Jeffrey T., and Roger D. Peng. "Statistics: P values are just the tip of the iceberg." Nature 520.7549 (2015): 612-612.

Downloading Files

我一般都是直接用wget, 可是那樣就不容易整合到腳本中. 幾個極可能會在download時候用到的python function:

# set up the env
os.path.dirname(os.path.realpath(__file__))
os.getcwd()
os.path.join()
os.chdir()
os.path.exists()
os.makedirs()

# dowload
urllib.request.urlretrieve()
urllib.request.urlopen()

# to tag your downloaded files
datetime.timezone()
datetime.datetime.now()

# an example
import shutil
import ssl
import urllib.request as ur

def download(myurl):
    """
    download to the current directory
    """
    fn = myurl.split('/')[-1]
    context = ssl._create_unverified_context()
    with ur.urlopen(myurl, context=context) as response, open(fn, 'wb') as out_file:
        shutil.copyfileobj(response, out_file)

    return fn

Loading flat files

pandas.read_csv()

Reading XML

Here is a very good introduction

Below are my summaries:

python 標準庫中自帶了xml.etree.ElementTree用來解析xml. 其中, ElementTree 表示整個XML文件, Element表示一個node.

The first element in every XML document is called the root element. 一個XML文件只能又一個root, 所以如下的不符合xml規範:

<foo></foo>
<bar></bar>

recursively 遍歷

# an excersice 
# find all elements with zipcode equals 21231
xml_fn = download("https://d396qusza40orc.cloudfront.net/getdata%2Fdata%2Frestaurants.xml")
tree = ET.parse(xml_fn)
for child in tree.iter():
    if child.tag == 'zipcode' and child.text == '21231':
        print(child)

JSON

• JSON stands for Javascript Object Notation
• lightweight data storage

JSON 的格式肉眼看起來就像是nested python dict. python 自帶的json的用法相似pickle.

Pattern Matching

Python makes a distinction between matching and searching. Matching looks only at the start of the target string, whereas searching looks for the pattern anywhere in the target.

Always use raw strings for regx.

Character sets
sth like r'[A-Za-z_]' would match an underscore or any uppercase or lowercase ASCII letter.

Characters that have special meanings in other regular expression contexts do not have special meanings within square brackets. The only character with a special meaning inside square brackets is a ^, and then only if it is the first character after the left (open- ing) bracket.

Summarizing Data

import pandas as pd
df = pd.DataFrame
# Look at a bit of the data
df.head()
df.tail()

# summary
df.describe()
df.quantile()

# cov and corr
# DataFrame’s corr and cov methods return a full correlation or covariance matrix as a DataFrame, respectively

# to calcuate pairwise correlation between a DataFrame's columns or rows
dset.corrwith(dset['<one col name>'])

# you can write your own analsis function and apply it to the dataframe, for example:
f = lambda x: x.max() - x.min()
df.apply(f, axis=1)

Check for missing values

df.dropna()
df.fillna(0)
# to modify inplace
_ = df.fillna(0, inplace=True)

# fill the nan with the mean
# 或者用naive bayesian的prediction
data.fillna(data.mean())

Exploratory Data Analysis

Analytic graphics

Principles of Analytic Graphics

1. Show comparisons
   If you build a model that can do some predictions, please come along with the performance of random guess.

2. Show causality, mechanism, explanation, systematic structure

3. Show multivariate data
   The world is inherently multivariate

4. Integration of evidence

5. Describe and document the evidence with appropriate labels, scales, sources, etc.

Simple Summaries of Data

Two dimensions

• scatterplots
• smooth scatterplots

> 2 dimensions

• Overlayed/multiple 2-D plots; coplots
• Use color, size, shape to add dimensions
• Spinning plots
• Actual 3-D plots (not very useful)

Graphics File Devices

• pdf: usefule for line-type graphics, resizes well, not efficient if a plot has many objects/points
• svg: XML-based scalable vector graphics; supports animation and interactivity, potentially useful for web-based plots
• png: bitmapped format, good for line drawings or images with solid colors, uses lossless compression, most web browers can read this format natively, does not resize well
• jpeg: good for photographs or natural scenes, uses lossy compression, does not resize well
• tiff: bitmapped format, supports lossless compression

Simulation in R

• rnorm:generate random Normal variates with a given mean and standard deviation
• dnorm: evaluate the Normal probability density (with a given mean/SD) at a point (or vector of points)

• pnorm: evaluate the cumulative distribution function for a Normal distribution

• d for density

• r for random number generation
• p for cumulative distribution
• q for quantile function

Setting the random number seed with set.seed ensures reproducibility

> set.seed(1)
> rnorm(5)