DAE Cheat Sheet - Categorical Data – Data with no intrinsic numerical value { Nominal: >2 - Studeersnel (2023)

Categorical Data – Data with no intrinsic numerical value {Nominal: >2 outcomes with no natural order (eg blond, brown grey)Ordinal: Two or more outcomes with natural order (eg bad, neutral, good) }Numerical Data – Data with intrinsic numerical value {Continuous: “numbers can be anything”Interval: No fixed zero point, ratios do not make sense (eg temp in Celsius)Ratio: Fixed zero point, ratios make sense (eg length and distances)Discrete: Data can only attain certain values, typically integers (eg numberof days with sunshine) }Reference Table: “store all data to look up easily”Demonstration Table: “Table used to illustrate a point (just enough data)”Plot Types: Dot plots / strip plots: God for showing actual values andstructure of numerical variables, not suitable for large data sets.Bar chart: Categorical dataHistogram: Numerical data with bins good for working with cumulativevalues but very sensitive to bin width, advised: √n, n data pointsScatter plots: Allows us to investigate relationships.Summary statistics: Types:-level: “show typical values” (location summary statistics)-spread: “show variance of values” (scale summary statistics)-relation: “How do different values relate” (association summary statistics)Location: Mean (x̄): average value, median: middle value when ordered (ifordered, take average of two middle values), mode: most frequentoccurring valueQuartiles: 1st: 25%, 2nd: 50% = median, 3rd: 75%, 0th percentile = smallestvalue in set, 100th percentile = largest.Percentile: (number of data points below / number of data points) * 100

Higher these values, more spread/variability in the dataOften MAD is defined as 1*MAD value to avoid systematic errors.Z-score: x’ = (x - x̄) / s, Negative x’ means value is below mean, positive x’means value is above mean. Mean value of z-scores of data set is 0 andstandard deviation is 1. A z-score of 2.5/-2 is considered an outlier, -1/1 isconsidered typical.

No relation: rxy ≈ 0, perfect relation: rxy ≈ -1 (neg corr) or +1 (pos corr)Advances statistical plots: Box and whisker plot: Used for indication ofsymmetry, whiskers show points 1 IQR from 1st/3rd quartile. Anythingfurther than a whisker is an outlier.Kernel density plots: Similar to histogram but shows shape of distribution,number of bins has no effect on it (no fixed bins) instead the frequencychosen does. Some distributions that are possible: unimodal (peak)/bimodal (2 peaks) and symmetric/ skewed (left/right).Violin plot: Combine box and whiskers plot with kernel density plot.ECDF: A function that for a given value x returns the fraction ofobservations smaller or equal to x.

Data visualizations != infographics (infographics are focused on narrativenot data, “Visualisation is the process that transforms (abstract) data into(interactive) graphical representations for the purpose of exploration,confirmation or communication.”Visualisations have 2 main goals: communication (inform humans) andexploration (when questions are not well defined)When not to visualize? When a table is better since precise values matterStrength’s computer: Data storage, Numerical calculations, searching, logic.Strength’s people: Cognition, common knowledge, CreativityHigh-level actions: Analysis, visualization for consuming (for end usersoften not technical) and producing (extends dataset, more interactive) data.Low-level actions: Searching: Target Known Target UnknownLocation Known Lookup BrowseLocation Unknown Locate ExploreGestalt theory: Objects close to each other appear as a group, objects thatare similar (colour, shape etc) are perceived as a group, objects following apattern are perceived as a group. Position and the arrangement of visualelements is the most important channel for visualisations.Humans are able to see by using cone cells (Containing pigments, perceivescolour) and the Visual cortex (Makes sense of everything).The human eye is sensitive to light but not same at all frequencies, mostsensitive to green least sensitive to purple and red.There are two chromatic channels Red-Green and Yellow-Blue, if somebodyis colour blind, they have degraded acuity for one of these channels. Avoidusing colours from only one of the channels. The default is using blue andorange, also change the shape/vary luminance.Key/Independent attribute: Index to look up value/dependent attributes.Visualizations use: Marks: points, lines, areas, complex shapes Channels(appearance of marks): position, colour, length, size, shape, curvatureArrangement types: Express (attribute mapped to spatial position along anaxis), Separate (Emphasize similarity /difference using categoricalattribute), Order (Emphasizes order), Align (Emphasize quantitativecomparison), Use (using existing structure like map for arrangement).When mapping colour use with Luminance, Saturation and Hue (colour),for ordered data we map to Luminance or Saturation. For categorical wemap to Hue. Never combine transparency with luminance or saturation.Always use Redundant encoding (use several channels/marks together,vary luminance and change shape). size, angle, shape and motion (add ch).Munzner’s reference model for constructing visualizations: what, why, howDesign strategies: Use position for most important aspects, colour forcategories, natural order + redundant encoding, consider readers context.Idioms = chart types, we should choose them carefully.Scatterplots: Data: 2 quantitative attributes, no keys, only values. Task: findtrends, outliers, distribution, correlation, clusters. Bar chart: Data:categorical attribute (key) 1 quantitative attribute (value). Task: compare,lookup values. Stacked bar chart: Data: 2 categorical attributes, 1quantitative attribute. Task: compare, lookup values, part-to-wholerelationship. Line chart: Data: 2 quantitative attributes, one key, one value.Task: find trend, connecting line emphasizes ordering of items. Heatmap:Data: 2 categorical attributes, 1 quantitative attributes (expression level)Task: find clusters, outliers. Histogram: Data: Table, Task: Find distribution(for shape). Boxplot same data but finds distribution for group comparisonand has 5 quantitative attributes (same for violin plot).

Predefined target (output variable) → supervised [else unsupervised]Information applicable to all our data → global methodInformation applicable to some of our data → local methodLinear Regression (Supervised, Global): Goal: Create a good linear modelto relate a independent variable (x) to a dependent variable (y). A goodmodel has low distance between regression equation and distance ofpoints.Residual = Actual – Expected (Random residuals indicate a good model)

Lower SSD means better linear model, higher R 2 means better model (means all y = ŷ). R 2 always increases when adding inputs we need tobeware of overfitting. Synergies of multiple inputs can improve predictionsfor example a cross term of 2 inputs (multiplying them) [x high + z highthen xz high, x low + z low then xz low, if mix then mix output, no pointusing xz]. To check if a function is linear regression no partial derivative ofthe model expression with respect to βj contains any βj’.Clustering (Unsupervised, Global): Goal: Partition all observations inmeaningful groups. K-mean clustering algorithm: 1) Pick k random pointsas initial centroids 2) Assign data points to their nearest centroid 3)Recompute centroids by finding mean of points assigned to them 4) Repeatsteps 2 + 3 until stable. Assessing quality: We use within-cluster distanceW(C) (The sum of the distances between a cluster and the points assignedto it summed over all clusters). It is crucial that we use the same units for xand y axis else use z-score. Pick k clusters such that 1 < k < n.Decision Tree (Supervised, Global): Goal: Learn a tree modern to makeyes/no decisions. A leaf is impure if 2 identical observations result in twodecisions. Can use accuracy (% correct predictions) to evaluate the tree. Confusion Matrix Tree Yes No Reality Yes TP FN No FP TNAccuracy = (TP + TN) / (TP + TN + FP + FN), Precision = TP / (TP + FP), p isradio of yes / total. Entropy H(p) = -plog 2 p – (1-p)log 2 (1-p), minimal entropyH(1)=H(0)=0 G, maximal entropy H(1/2) = 1 B. Minimize average entropy oftotal tree(av[wiH(pi) + wjH(pj)...]) compute entropy of each leaf.Association Rule Learning (Unsupervised, Local): Goal: Find highconfidence associations between frequently occurring subsets of items.Supp(X) = |X|/n, Supp(X⋂Y)=|X⋂Y|/n. Conf(X⇒Y)= |X⋂Y|/|X|, n is totalnumber of rows. Good rules have high support and confidence.Distances: Euclidean Distance: “As the crow flies” - good to use whenmovement is unrestricted, just a straight line from point A to point B.Network Distance: The distance one travels when following a network tocommute between a and b (i a road network). Good to use whennetwork is sparse, and we know it completely.Manhattan Distance: Sum of x distance and y distance from A to B. It isappropriate when restricted to a fixed grid.

Primary key: A minimal set of attributes (columns) that uniquely defineseach row of a table, any subset of this set is not a key, if two or morecolumns cannot be unique then they can be combined to a primary key.Schema: Logical structure of the database (i what tables do we include).Instance: The content of the database at a particular point in time (values).Entity: An object that exists and is distinguishable. All entities haveproperties, Entity Set: Set with entities that share property, eg.instructor_ID is the property, individual values is entity, entity set is table.A relationship is an association among several entities.Relationship set: A collection of relationships among entity sets, arelationship can contain attributes for example advisor assignment data.The ‘domain’ of an attribute is the set of allowed values for each attribute.ER diagram Syntax: Rectangle - entity set, diamond - relationship set, lines-link entity sets to relationship sets, underline: Primary key, attribute forrelationship sets – small rectangle linked to relationship s with dashed line.A relationship is represented as a table wit attributes for the primary keysfor the participating entity sets and attributes of the relationship sets.Data querying: We use SQL to query data bases. SQL data definitionlanguage allows to specify set of relations and information about relations.Note: NULL represents missing values. SQL is case insensitive.A typical SQL query has the form: SELECT, FROM, WHERE. With each on aseparate line. SELECT clause lists the attributes that we want to retrieve.FROM clause lists tables from which we query. WHERE defines a predicate(filter) over the values of attributes (what conditions to satisfy, horizontalselection). We can use the keyword DISTINCT to remove duplicate values,ALL specifies duplicate values do not need to be removed. The * clauseafter SELECT shows “all attributes” showing the table as it is. The SELECTclause can contain arithmetic expressions +, -, *, /. More filters in WHEREcan be done by using AND, OR and NOT (and parentheses). % representszero one or multiple characters eg ‘%Eindhoven%’ can mean Eindhoven-Woensel or BigEindhoven1. _ represents a single character. like is used incombination with % for example: branch_name like ‘%Eindhoven’ (showsrows in branch_name that has the word ‘Eindhoven’ somewhere).Tablename_name to select column from specific table for exampleborrower_number selects the loan_number column from borrowertable only. Rename a column by old_name as new_name. We can alsorename in the FROM clause. Common operations are a∪b (union), a⋂b(intersection), a\b (except). These automatically eliminate duplicates souse all to retain duplicates. We have the following aggregate functions(include duplicate values by default): count, min, max, avg, sum. group byis a function that groups together similar values. having can be used toprovide a condition after group by (having clause applied after forminggroups, where occurs before forming groups). TIPS: Customers who have aloan → depositor, customers who have an account → borrower

Primary data: collected by you. Secondary data: Collected by others. Youcan generate features using avg, min, max etc (aggregating). There can be agap between a high-level problem statement and the data, you need tobuild a bridge between them by picking the right features. Scientificmethod: A systematic approach to conducting empirical research to obtainsound answers to questions. Deduction: Using assumed to be truepremises together to reach a conclusion (Essentially logic/ math proofs).Induction: Conducting an experiment and using the results to reach aconclusion. When doing Induction, we need to ensure that our experimentis valid, reliable, reproducible. Occam’s Razor: Least complex thing isprobably true. Valid: The results obtained accurately describe the realworld. Reliable: Similar results are obtained in the different trails(essentially equal to ‘precise’). Internal validity: Are the conclusions validwithin the study? External validity: Can the conclusions be applied beyondthe context of the study? Verification by others: The results obtained canbe reproduced by others. Precision refers to errors introduced bymeasuring instrument (random errors) whilst accuracy refers to deviationsfrom real values (systematic errors). Convenience sampling: Refers togoing for the data that is the easiest to collect. It saves money and effortbut may lead to bias which threatens external validity. Random sampling:Each individual is equally likely to be included into the sample. Stratifiedrandom sampling: A strata is disjoint parts forming the whole targetpopulation (essentially an attribute that we use to split up the populationinto parts). We then either take an equal amount of random data fromeach part (proportionate) of pick unequal amounts from each(disproportionate). Voluntary sampling: Individuals select themselves.For Data cleaning check for: Incomplete data, out of range data andinconsistent data. When finding such data, we either chose to discard(leads to smaller data size and may introduce a bias), Impute: fill inestimated values in place of the faulty values (this can also lead to a biassince estimates != real values) or work in presence of missing data. Timeseries is a sequence of pairs (tn, xn) where tn is the observation time and xnis the observed value such that tn < tn+1. For equispaced time series (tn+1 - tn)is constant and it is called sampling frequency. Time series data oftencontains a lot of noise (spikes up and down) this is unwanted as it disturbsyou from seeing trends and may be the result of errors. Median Filter: Pickwindow size and place at start of data, compute median value and output,iterate through by moving window + 1 until end of data example – For[7,8,2,6,9,0,7] (t = 0..) with window size 3 we start with first 3 elements[7,8,2] median is 7 output 7 at t=1 continue to [8,2,6] median is 6 at t=continue... we get [7,6,6,6,7] (t = 1..). Mean Filter: Same as median filterexcept compute mean instead of median, more sensitive to outliers.Convolution Filter: Using an array of weights [w-1, w 0 , w+1] assuming awindow size 3 computing the outputted value is just done by w-1*vi +w 0 vi+1 + w+1vi+2, keep doing until end of array. Independent Variable:Used to determine how a change in this variable causes a change in thedependent variable. Confounding Variable: Variable not considered andthat can provide an explanation to the observed effect i., the wind inGA1. Distance between (xi, yi) and (xi+1, yi+1) is di,i+1 = sqrt((xi+1 – xi)^2 + (yi+1 –yi)^2). Speed at moment ti is done by p’(ti) = (pi+1 – pi-1) / (ti+1 – ti-1).Computing The Kernal For Differentiation: For weights [w 1 , w 0 , w 1 ], where

w 0 + 2w 1 = 1 we can compute: in this casedirect filer was [-w 1 , -w 0 , 0, w 0 , w 1 ]

Sum of all possible disjoint probabilities = 1. Binomial coefficient (n choosek) is (n k) = nCk = n!/k!(n-k)!. Note 0! = 1. The binomial probabilitydistribution X ~ (n, p) is:for n (trials) independent observations each with success probability p, X isnumber of successes in n observations, k is successful events. Cumulativeprobability formula:

E(X) = np [where E(X) is the mean)Var(X) = np(1-p) [to look at spread of data, bigger variance bigger spread,standard deviation squared]. Binomial Distribution: 𝑝̂=X/n. E(𝑝̂) = pVar(𝑝̂) = (p(1-p))/n. Probability Density Function:

Normal distribution is 𝑁(𝜇, 𝜎 2 ), 𝜇 shows mean of distribution, 𝜎represents how spread out the graph is (range). 𝜇 = 0 and 𝜎 2 = 1 isstandard normal distribution. If X ~ 𝑁(𝜇, 𝜎 2 ) then (X- 𝜇)/𝜎 ~ N(0,1). We canestimate in the normal distribution where 𝑋 is the sample mean (mean of asmaller subset of data): 𝑋 ∼ 𝑁 (𝜇, 𝜎

2𝑛) and thus 𝑋̅ −𝜇𝜎/√𝑛∼ 𝑁(0,1). ECDF curveis a function showing summations over probability of a binomialdistribution. Confidence Interval Formula: 𝑥 ± 𝑧𝛼/𝜎√𝑛 where 𝑥 = samplemean, n = number of observations and 𝑧𝛼/2 = normal quantile (commonvalues 𝑧0 = 1 , 𝑧0 = 1, 𝑧0 = 1). Larger standard deviation→ wider interval → less certainty about estimate. More data → tighterinterval → more certainty about estimate. Lower confidence → tighterinterval because less guaranteed coverage. Two more formulas:𝑥 ± 𝑡𝑛−1;𝛼/2√𝑛𝑠 this is for when 𝜎 is not known where s is the standard

deviation. This is for proportion: 𝑝̂ ± 𝑧𝛼/2√ 𝑝̂ (1−𝑝̂ ) 𝑛. Hypothesis Testing: If thesample size is too small there may also not be enough evidence to reject anull hypothesis, outcome too unlikely → reject hypothesis. We express ahypothesis using statistics, the null hypothesis (H 0 ) is the hypothesis thatthere is no difference regardless of the value of the dependent variable.The alternative hypothesis (Ha) is our hypothesis (independent anddependent variables are related). Null hypothesis is rejected when the p-value is below a threshold, this is usually 0, if it is not, we do not rejectH 0. They give 95 percent confidence interval if H 0 lies within interval do notreject H 0 else reject. Rejecting the null hypothesis does not directly provethe alternative hypothesis, since we use a sample its possible, we made amistake. If we incorrectly reject the null hypothesis (probability α) while H 0is true it’s called a type I error. If we don’t reject the null hypothesis whilstit is false (while Ha is true) it’s called a type II error (probability β). Thepower of a test is the probability of correctly rejecting H 0 so 1-β. One sidedhypothesis →single variable, two-sided hypothesis → two variables. Whenconducting these tests, we assume that the data sets are independent fromone another, that they come from same probability distribution, that theyhave a large sample size. To test if data comes from a normal distributionwe can do the Anderson-Darling test with a p-value of 0 instead of 0.If the sample size is > 50 then we do not need to check for normality.