Assess data skills better

Linear algebra is the branch of mathematics concerning linear transformations over vector spaces, and it is used in science and engineering to model natural phenomena. Algebra helps the Data Workers to understand the inner workings of ML algorithms, to formulate extensions to existing algorithms and to create new algorithms. Along with Calculus, Probability Theory and Optimization Theory, Linear Algebra lies at the mathematical foundations of Machine Learning.

Ability to prompt ChatGPT properly to assist in answering multi-step data science questions. Using ChatGPT to write and debug Python and SQL codes. Simulates real-life use of ChatGPT by data analysts and scientists.

Mathematics is at the foundation of Machine Learning. Though not directly a prerequisite for solving Data Science problems, a good theoretical and practical background in the following subject sheds light on Whats, Whys and Hows of Machine Learning: Set theory, basic arithmetic and algebra, analytic geometry, trigonometry, quadratic forms, polynomials, rational and real number systems.

Taking derivatives is the basis for backpropagation algorithm and gradient descent. Bayesian computation involves taking integrals. Estimation and inference involves law of large numbers and central limit theorem, hence infinite series. Limits are in almost every computation in Machine Learning. Deep neural networks are differentiable networks and use activation functions that need to be piecewise differentiable. Finally, functions are the overarching concept in Machine Learning. Thus, a Data Scientist should be well versed in Calculus.

Almost always, real-life data is not ready to be analyzed immediately. Data types in a dataset might be heterogeneous, data might be distributed over different databases and files. The most time consuming and probably the most important part of a Data Science project or Machine Learning pipeline is the data preparation stage. The primary job of a Data Scientist nowadays is to prepare data in the most optimal way as to ensure better data quality and better features. A Data Scientist should be familiar with the problems of real-data and the techniques to solve them.

Probably the maxim “a picture is a thousand words” is mostly relevant in the context of data analysis. Data Exploration and Visualization have two primary purposes: Understanding the data and explaining the findings. The former is an essential daily task for Data Scientists, and the latter is important for adoption of a model. Considering that there are more than one way of exploring or visualizing data, a Data Scientist should equip herself with the necessary knowledge and technologies. This not only requires the theoretical knowledge of representing data with graphs but also the visualization tools commonly used in practice.

The practical application of Machine Learning boils down to computations. As the Machine Learning has grown and the ML community has grown, the alternatives to running ML computations have emerged. Though commercial solutions still exist, open source alternatives are very competitive and commonly used. Python has become the de facto language of these open source alternatives. Though a perpetual commitment to a certain piece of technology should be avoided by Data Scientists, it is safe to say that Python will be here for a while, and every Data Scientist should be knowledgeable about Python programming skills and a set of important Python libraries such as Pandas, NumPy, Matplotlib and Scikit-learn.

The reusability of software components is important . Some of these components, libraries, packages, frameworks, etc. have become more popular and been more widely used than the others for a variety of reasons. Whatever the reasons might be, it is sound practice to use these open source libraries for the work and not rediscover everything from scratch. The practicing data scientist today should be well versed in scikit-learn for benefiting from ML algorithms on tabular data. They should have working knowledge of at least one of TensorFlow or PyTorch for deep learning computations.

Learning from data ultimately involves extracting structures, patterns, or dependencies between or among a dataset obtained via a variety of means. Almost exclusively, for the datasets of interest, there does not exist any physical, chemical or biological laws that explain the data with deterministic equations. The phenomenal success of deterministic physical equations in explaining physical phenomena paved the way for similar mathematical modelling endeavors in almost every discipline where data is generated as the result of natural or cultural processes. However, for most of the datasets of interest, there are not closed-form formulas explaining the data in a deterministic way. The probability theory is a tool to put a structure on these data generating processes so as to explain the structure of the data generating process and hopefully to make predictions. Hence, it sits at the foundations of Machine Learning as the goal of Machine Learning is to find robust patterns, structures, dependencies from a finite sample of data and generalize it for new cases. A strong background in Probability Theory is indispensable for the practicing Data Scientist.

Time series problems are routinely encountered by Data Scientists. Whether it is in the form of a demand forecasting problem or anomaly detection problem, there are many applications of time series modelling in many industries. Though traditionally being a field of econometrics and dynamical systems, the Machine Learning community is catching up with time series modelling problems with new algorithms such as Recurrent Neural Networks, LSTMs and Transformer Networks. Time series modelling poses extra challenges for Data Scientists such as nonstationarity, serial correlation, heteroscedasticity, structural breaks, model validation and model stability. The practicing Data Scientist in the field will definitely encounter many problems involving time series forecasting. Therefore they should be equipped with theoretical knowledge and practice of time series modelling.

The goal of statistical inference is to estimate population parameters that specify a statistical model from sample data. The important point here is that these estimations should be correct in a "statistical sense" so that the model can produce valid predictions. A Data Scientist needs to be knowledgeable about Statistical Inference for building correct models, interpreting and communicating model results.

A Data Scientist should get their hands dirty with the data; they have to understand and act on all the peculiarities of real-life datasets. The random variables in real-life datasets have heavy tails, are multi-modal, have many missing/incorrect values, consist of categorical and ordinal variables with high cardinality, and have nonlinear associations between variables. A good grasp of Descriptive Statistics has implications in preparing data in the right way for the subsequent inferential statistics phase by using the right transformations and algorithms to extract patterns and to communicate the findings of the data analysis in the right way.

An overarching goal in Machine Learning is to model the distribution of data generated by a process. If there is at least one data item that could be classified as the output of the system, one can then learn the dependency between the output(s) of the system in terms of other data fields generated by the process. This is called supervised learning and most of the practical applications of Machine Learning currently fall into this domain. The majority of interesting problems in Machine Learning are cast either as supervised learning problems or self-supervised learning problems. The spectrum of supervised learning covers problems from credit risk modeling to object classification, from predictive maintenance to predicting protein folding from aminoacid sequences.

An overarching goal in Machine Learning is to model the distribution of data generated by a process. Considering from a system point of view, if there is no candidate data item that could be designated as the output of the system, learning about this distribution is called unsupervised learning. Finding clusters of data where samples are concentrated, detecting outliers coming from a multivariate distribution, approximating the multivariate density of a set of random variables are unsupervised learning problems. A foundation in algorithms and techniques of unsupervised learning is a must for aspiring or practicing Data Scientists.

Though classified as a sub-discipline of Machine Learning, it has acquired a life of its own and Deep Learning models have generated SOTA solutions in problems of computer vision, speech understanding and natural language processing. The primary success of Deep Learning is end-to-end learning in multiple scenarios including unstructured complex data such as pictures, speech and text. Though it shares the basic nomenclature with Machine Learning, Deep Learning has introduced many new concepts such as various architectures, new layers of abstraction, training methods and tricks, architecture optimization, etc. It is indispensable that the practicing Data Scientist should become theoretically equipped and practiced enough to face many problems that will be solved by DL in the years to come.

Data Science has many applications in many walks of life. However, recognizing that a business problem –or part of it- could be cast as a Data Science problem and could be solved as such is not an easy feat. It requires domain knowledge and might require deep analysis of the problem.

Though unwittingly, A –good- Data Scientist routinely applies the Scientific method in their daily work: Formulates a problem, creates hypotheses, does experiments, gathers data, tests the compatibility of data with the hypotheses, repeats the cycle, and reports the conclusions. This is almost the definition of Data Science practice in disguise. The important thing is not to memorize these steps but to be truthful to its essence and to be rigorous in practice. Being lax in any one step of the scientific method would lead to falsities.

A data structure is a physical form of data type used to store a set of elements logically represented by abstract data types. Though Machine Learning libraries and tools employ the optimal algorithms and data structures for a certain computation scenario, there are cases where practicing Data Scientist might need to choose between alternatives. Therefore a solid grounding in algorithms and data structures is expected of Data Scientists in these situations.

The storage and access of data is abstracted away from Data Scientists. This means that, Data Scientists are consumers of database systems and they rarely need to know the inner workings of database systems. However, today's Data Scientists are encountering a heterogeneous array of data structures such as graph data, text data, image data, speech data, log files, click-stream data, etc. While it is not the primary job of Data Scientists to build database systems, they should be knowledgeable about the differences between database management systems and when a system should be preferred over the others.

SQL is a declarative programming language for managing and processing data held in relational database management systems(RDBMS). It is primarily used for processing structured data, i.e. it has a schema and incorporates relations among entities (e.g. customer, account) and variables (e.g. name, id, amount). As organizations started to use Machine Learning to extract intelligence from databases, SQL has been used for exploring data, preparing data, scoring data with ML models and serving the model artifacts on SQL databases. Data Scientists and Analysts should have sufficient knowledge of SQL in order to explore data and prepare it for further analysis.