AI as a technical skill set
instead of a consumer skill set
Jedediyah Williams, PhD
jedediyah@gmail.com
Belmont High School
jedediyah.github.io/csta2024
About Jed
Teaching → Astronomy → Robots → Teaching
Motivations for this talk
- The largely uncritical view of AI being presented to educators.
- A push to educate, not critical thinkers, but fateful consumers, i.e., "AI literacy" as a consumer skillset instead of a technical skillset.
-
We are responsible for teaching the consequences of our content.
Outline
- I'm going to gripe about how AI is garbage.
- Examples of AI from my classroom!
https://twitter.com/petergyang/status/1793480607198323196/photo/1
cheese not sticking to pizza
↓
... add about 1/8 cup of glue
how many rocks should I eat
↓
... at least one small rock per day
A common misconception is that
data + compute → solutions
If the problem isn't solved yet, it's just because you haven't added enough technology yet!
"However, two major discoveries of the twentieth century showed that Laplace's dream of complete prediction is not possible, even in principle...
It was the understanding of chaos that eventually laid to rest the hope of perfect prediction of all complex systems, quantum or otherwise." (Mitchell, 2019, p. 20)
"But even if it were the case that the natural laws had no longer any secret for us, we could still only know the initial situation approximately.
...
it may happen that small differences in the initial conditions produce very great ones in the final phenomenon.
...
Prediction becomes impossible."
(Poincaré, 1908, as cited in Mitchell, 2019, p. 21)
https://twitter.com/standupmaths/status/741251532167974912
"The lack of humility before nature that's being displayed here staggers me." - Malcolm, Jurassic Park
"Not only do many of the hiring tools not work, they are based on troubling
pseudoscience
and can discriminate"
Hilke Schellmann tried the
myInterview tool to check her "hiring score":
- Honest interview in English: 83%
- Reading a random wikipedia page in German: 73%
- Getting a robot voice to read her English: 79%
To what extent are we placing our students' futures into biased, pseudoscientific, noise machines?
"Our success,
happiness, and wellbeing are never fully of our own making. Others' decisions can profoundly affect
the course of our lives...
Arbitrary, inconsistent, or faulty decision-making thus raises
serious concerns..."
- Fairness and Machine
Learning, Barocas, Hardt, and Narayanan
https://www.nytimes.com/2019/08/16/technology/ai-humans.html
Anatomy of an AI system, Crawford
and Joler
Adversarial attack
"LLM chatbots have been designed in a way, known by psychologists and ethicists, to trick humans into believing they are intelligent."
"The hype is a lie"
- Tesla faked self-driving AI demo
- Google faked Duplex AI demo
- Google faked Gemini AI demo
- Google Deep Mind claims to discover 2.2 million compounds
- Amazon Fresh "walk out" turns out to be 1000 people in India
- Facebook chatbot M powered largely by humans
- Microsoft covered up vulnerabilities in DALL-E 3 image generator
- Rabbit R1 turns out to just be an Android app
- Humane AI Pin edited video to fix incorrect answers
- OpenAI reported 90th-percentile performance on Bar Exam.
"We can be impressed by their performance on some inputs,
but there are infinitely many inputs
where they must fail."
- Iris van Rooij (26:02)
Algorithms are brittle - Melanie Mitchell
AI and DS in my classroom
- Spreadsheets
- How to Lie with Statistics
- Data Modeling
- Game trees and state diagrams
- Robots with genetic path-finding algorithms
- Facial recognition algorithms
- import matplotlib.pyplot as plt
"Yes, train these young people to get these skills, but integrate into that not only the technical capacity but the critical capacity to question what they're doing and what's happening.
To me, it is not true empowerment unless people can have the power to question how these skills are going to be used."
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
- Modeling with data
- Teaching and scafolding modeling with data
- Critically analyzing data technologies
Many frameworks. Much overlap.
Fayyad et al (1996). The KDD Process for Extracting Useful Knowledge from Volumes of Data
(Knowledge Discovery in Databases)
Chapman et al (1999), Wirth (2000). "Towards a standard process model for data mining".
1. Obtain: pointing and clicking does not scale
2. Scrub: the world is a messy place
3. Explore: You can see a lot by looking
4. Models: always bad, sometimes ugly
5. INterpret: "The purpose of computing is insight, not numbers."
Mason and Wiggins (2010). "A Taxonomy of Data Science".
Schutt and O'Neil (2014). "Doing Data Science: Straight talk from the frontline".
GAIMME Guidlines for assessment & instruction in mathematical modeling education (2016).
Guidelines for Assessment and Instruction in Statistics Education (GAISE) Reports
(2020, based on 2007).
Estrellado et al (2020). Data Science in Education Using R, Section 3.2.
Zico Kolter (2021). Practical Data Science, Intrdouction
Common Core / Achieve the Core.
I like the video here!
Data
1. Get the data
Preprocess
2. Clean up the data
Explore
3. Explore the data
Model
4. Model it
Communicate
5. Share the results
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
• Design
∘ Turn a problem into a data-problem.
∘ Survey or experimental design
∘ Database infrastructure
• Acquire
∘ Survey or experiment
∘ Download the dataset! CSV, API, etc.
∘ Web scraping
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
• Wrangle
∘ Format
∘ Clean and organize
∘ Check data integrity
• Prepare
∘ Label
∘ Split into training and testing sets
∘ Normalize
Data Splitting
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
• Visualize
∘ Plot and familiarize with data
∘ Look for and compare features visually
∘ Consider appropriate models
• Inspect
∘ Exploratory data analysis
∘ Descriptive statistics
∘ Identify features analytically
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
• Model
∘ Try and compare multiple models
∘ Consider bias and variance
∘ Interpret model and performance
• Validate
∘ Assess model performance on independent test data
∘ Error analysis and stress-test
∘ Consider consequences
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
• Reflect
∘ Consider contexts, bias, and consequence
∘ Create audit plant
∘ Document - data and model
• Share
∘ Report documentation
∘ Inform policy
∘ Deploy in product
Data Modeling Process
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
Data Modeling Process
Environment
→
Data
→
Preprocess
→
Explore
→
Model
→
Communicate
→
A framework for critical analysis
Data
• Harmful data collection, lack of consent, insecure / lack of privacy, historical, representational, or measurement bias, ...
Preprocess
• Labor exploitation, labeling by non-experts, incorrect labeling, trauma experienced by labelers, ...
Explore
• Feature selection bias, bias in interpretation of data visualization, data manipulation, feature hacking, ...
Model
• Bias in model choice, model-amplified bias, environmental impact, learning bias, evaluation bias, peripheral modeling, ...
Communicate
• Biased model interpretation, ignoring variance, rejecting model, deploying harmful products, deployment bias, ...
Meta
• "Pernicious feedback loops", runaway homogeneity, susceptability to adversarial attack, lack of oversight or auditing, ...
"A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle", Harini Suresh and John V. Guttag
Critical Questions:
- What are the motivations for the project?
- What is the intended use?
- What is the unintended use or misuse?
- Where does the data come from?
- Who collects the data?
- Who owns the data?
- How is the data collected?
- How is the data stored?
- How old is the data?
- When will the data expire?
- How will the data be secured?
- What happens with the data when the company is sold?
- Who does the labeling?
- What labels will they decide to use?
- Are the labelers experts?
- Are the labels accurate?
- What biases are represented in the data?
- How is data included or excluded?
- How are outliers addressed?
- What subpopulations are represented?
- What subpopulations are over- or underrepresented?
- What portions of the data are inspected?
- What features are selected for modeling?
- What model is chosen?
- What features do we think are being modeled?
- What latent features are actually being modeled?
- What is the domain of the model?
- What are the consequences of error?
- What decisions will be made with the model?
- What biases are perpetuated?
- Where will the model be deployed?
- What could go wrong?
- Who is responsible when things go wrong?
- How can issues be reported?
- Will new data be fed back in to update the model?
How high does a bouncy ball bounce?
Data
Preprocess
Explore
Model
Communicate
How high does a bouncy ball bounce?
Data
• Data problem: What will be the bounce height \(h_{bounce}\) of my bouncy ball when dropped from rest from a given drop height \(h_{drop}\)?
• Record several slow-motion videos.
Preprocess
• Randomly choose a subset of videos as the training set.
• Parse the training set videos into a table.
Explore
• Create a scatter plot of \(h_{bounce}(h_{drop})\)
• Look for features! Notice and wonder. Consider models.
Model
• Find a best-fit model on the training data.
• Validate the model on the testing data.
Communicate
• Reflect on the process.
• Share out.
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Training Data
Testing Data
Data Splitting
Break models
"How high does a bouncy ball bounce?"
"How high does a bouncy ball bounce?"
becomes:
"How much can we minimize the error of a linear model
when predicting how high this particular bouncy ball will bounce
in this room on this surface
at this temperature and humidity when dropped from rest at a height of no more than two meters?"
Data Modeling Projects
- How high does a bouncy ball bounce?
- How far will the ball roll?
- What is the period of a pendulum?
- When will the water reach 40℃?
- When is high tide?
- How much daylight will there be on Jan 1?
- When will sun set on Feb 1?
- What is the best move in Hexapawn?
- What is the best move in Tic Tac Toe?
- Which NFL team will win Monday?
Here are some links to the machine learning projects:
Classify these fruit!
Classify these digits!
Conclusion
We have a lot of work to do.
Jedediyah Williams, PhD
jedediyah@gmail.com
Belmont High School
jedediyah.github.io/csta2024
About Jed
Motivations for this talk
- The largely uncritical view of AI being presented to educators.
- A push to educate, not critical thinkers, but fateful consumers, i.e., "AI literacy" as a consumer skillset instead of a technical skillset.
- We are responsible for teaching the consequences of our content.
Outline
- I'm going to gripe about how AI is garbage.
- Examples of AI from my classroom!
|
https://twitter.com/petergyang/status/1793480607198323196/photo/1
|
cheese not sticking to pizza ↓
... add about 1/8 cup of glue
|
|
|
how many rocks should I eat ↓
... at least one small rock per day
|
A common misconception is that
data + compute → solutions
If the problem isn't solved yet, it's just because you haven't added enough technology yet!
|
|
"However, two major discoveries of the twentieth century showed that Laplace's dream of complete prediction is not possible, even in principle...
It was the understanding of chaos that eventually laid to rest the hope of perfect prediction of all complex systems, quantum or otherwise." (Mitchell, 2019, p. 20) |
|
|
"But even if it were the case that the natural laws had no longer any secret for us, we could still only know the initial situation approximately.
... it may happen that small differences in the initial conditions produce very great ones in the final phenomenon. ... Prediction becomes impossible." (Poincaré, 1908, as cited in Mitchell, 2019, p. 21) |
https://twitter.com/standupmaths/status/741251532167974912
|
|
"Not only do many of the hiring tools not work, they are based on troubling pseudoscience and can discriminate" |
- Honest interview in English: 83%
- Reading a random wikipedia page in German: 73%
- Getting a robot voice to read her English: 79%
"Our success, happiness, and wellbeing are never fully of our own making. Others' decisions can profoundly affect the course of our lives...
Arbitrary, inconsistent, or faulty decision-making thus raises serious concerns..."
- Fairness and Machine Learning, Barocas, Hardt, and Narayanan
https://www.nytimes.com/2019/08/16/technology/ai-humans.html
Anatomy of an AI system, Crawford
and Joler
"LLM chatbots have been designed in a way, known by psychologists and ethicists, to trick humans into believing they are intelligent."
"The hype is a lie"
- Tesla faked self-driving AI demo
- Google faked Duplex AI demo
- Google faked Gemini AI demo
- Google Deep Mind claims to discover 2.2 million compounds
- Amazon Fresh "walk out" turns out to be 1000 people in India
- Facebook chatbot M powered largely by humans
- Microsoft covered up vulnerabilities in DALL-E 3 image generator
- Rabbit R1 turns out to just be an Android app
- Humane AI Pin edited video to fix incorrect answers
- OpenAI reported 90th-percentile performance on Bar Exam.
"We can be impressed by their performance on some inputs, but there are infinitely many inputs where they must fail."
- Iris van Rooij (26:02)
AI and DS in my classroom
- Spreadsheets
- How to Lie with Statistics
- Data Modeling
- Game trees and state diagrams
- Robots with genetic path-finding algorithms
- Facial recognition algorithms
- import matplotlib.pyplot as plt
|
|
"Yes, train these young people to get these skills, but integrate into that not only the technical capacity but the critical capacity to question what they're doing and what's happening. To me, it is not true empowerment unless people can have the power to question how these skills are going to be used." |
Data Modeling Process
Data Modeling Process
- Modeling with data
- Teaching and scafolding modeling with data
- Critically analyzing data technologies
Many frameworks. Much overlap.
|
|
|
|
|
|
|
|
Fayyad et al (1996). The KDD Process for Extracting Useful Knowledge from Volumes of Data
(Knowledge Discovery in Databases)
Chapman et al (1999), Wirth (2000). "Towards a standard process model for data mining".
2. Scrub: the world is a messy place
3. Explore: You can see a lot by looking
4. Models: always bad, sometimes ugly
5. INterpret: "The purpose of computing is insight, not numbers."
Mason and Wiggins (2010). "A Taxonomy of Data Science".
Schutt and O'Neil (2014). "Doing Data Science: Straight talk from the frontline".
GAIMME Guidlines for assessment & instruction in mathematical modeling education (2016).
Guidelines for Assessment and Instruction in Statistics Education (GAISE) Reports
(2020, based on 2007).
Estrellado et al (2020). Data Science in Education Using R, Section 3.2.
Zico Kolter (2021). Practical Data Science, Intrdouction
Common Core / Achieve the Core.
I like the video here!
Data |
1. Get the data |
Preprocess |
2. Clean up the data |
Explore |
3. Explore the data |
Model |
4. Model it |
Communicate |
5. Share the results |
Data Modeling Process
Data Modeling Process
• Design
∘ Turn a problem into a data-problem.
∘ Survey or experimental design
∘ Database infrastructure
• Acquire
∘ Survey or experiment
∘ Download the dataset! CSV, API, etc.
∘ Web scraping
Data Modeling Process
• Wrangle
∘ Format
∘ Clean and organize
∘ Check data integrity
• Prepare
∘ Label
∘ Split into training and testing sets
∘ Normalize
Data Splitting
Data Modeling Process
• Visualize
∘ Plot and familiarize with data
∘ Look for and compare features visually
∘ Consider appropriate models
• Inspect
∘ Exploratory data analysis
∘ Descriptive statistics
∘ Identify features analytically
Data Modeling Process
• Model
∘ Try and compare multiple models
∘ Consider bias and variance
∘ Interpret model and performance
• Validate
∘ Assess model performance on independent test data
∘ Error analysis and stress-test
∘ Consider consequences
Data Modeling Process
• Reflect
∘ Consider contexts, bias, and consequence
∘ Create audit plant
∘ Document - data and model
• Share
∘ Report documentation
∘ Inform policy
∘ Deploy in product
Data Modeling Process
Data Modeling Process
Environment
A framework for critical analysis
|
Data
|
• Harmful data collection, lack of consent, insecure / lack of privacy, historical, representational, or measurement bias, ...
|
|
Preprocess
|
• Labor exploitation, labeling by non-experts, incorrect labeling, trauma experienced by labelers, ...
|
|
Explore
|
• Feature selection bias, bias in interpretation of data visualization, data manipulation, feature hacking, ...
|
|
Model
|
• Bias in model choice, model-amplified bias, environmental impact, learning bias, evaluation bias, peripheral modeling, ...
|
|
Communicate
|
• Biased model interpretation, ignoring variance, rejecting model, deploying harmful products, deployment bias, ...
|
|
Meta
|
• "Pernicious feedback loops", runaway homogeneity, susceptability to adversarial attack, lack of oversight or auditing, ...
|
|
|
How high does a bouncy ball bounce?
How high does a bouncy ball bounce?
|
Data
|
• Data problem: What will be the bounce height \(h_{bounce}\) of my bouncy ball when dropped from rest from a given drop height \(h_{drop}\)?
• Record several slow-motion videos. |
|
Preprocess
|
• Randomly choose a subset of videos as the training set.
• Parse the training set videos into a table. |
|
Explore
|
• Create a scatter plot of \(h_{bounce}(h_{drop})\)
• Look for features! Notice and wonder. Consider models. |
|
Model
|
• Find a best-fit model on the training data.
• Validate the model on the testing data. |
|
Communicate
|
• Reflect on the process.
• Share out. |
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
Bounce Prediction Error
|
|
| Training Data | Testing Data |
Data Splitting
becomes:
"How much can we minimize the error of a linear model when predicting how high this particular bouncy ball will bounce in this room on this surface at this temperature and humidity when dropped from rest at a height of no more than two meters?"
Data Modeling Projects
- How high does a bouncy ball bounce?
- How far will the ball roll?
- What is the period of a pendulum?
- When will the water reach 40℃?
- When is high tide?
- How much daylight will there be on Jan 1?
- When will sun set on Feb 1?
- What is the best move in Hexapawn?
- What is the best move in Tic Tac Toe?
- Which NFL team will win Monday?