CURRENT ISSUE
ABOUT THE MAGAZINE
DEPARTMENTS
RECENT PHOTO GALLERIES
Not Exactly Rocket Science
When scientists struggle with a problem for over a decade, few of them think, “I know! I’ll ask computer gamers to help.” That, however, is exactly what
from the University of Washington did. The result: he and his legion of gaming co-authors have cracked a longstanding problem in AIDS research that scientists have puzzled over for years. It took them three weeks.
Khatib’s recruits played Foldit, a programme that reframes fiendish scientific challenges as a competitive multiplayer computer game. It taps into the collective problem-solving skills of tens of thousands of people, most of whom have little or no background in science. :
The goal of the game is to work out the three-dimensional structures of different proteins. Proteins are feats o they consist of long chains of amino acids that fold into very specific and complicated shapes. These shapes can reveal how proteins work, but solving them is fiendishly challenging. To do it, scientists typically need to grow crystals of purified protein before bouncing X-rays off them.
Foldit takes a different approach, using the collective efforts of causal gamers to do the hard work. And its best players can outperform software designed to do the same job. Best of all, you don’t need a PhD to play Foldit. Barely an eighth of the players work in science, and two-thirds of the top scorers have no biochemistry experience beyond high school. The co tutorial levels introduce the game’ colourful v and the interface is explained in simple language. While protein scientists concern themselves with “rotating alpha-helices” and “fixing degrees of freedom”, Foldit players simply ‘tweak’, ‘freeze’, ‘wiggle’ and ‘shake’ their on-screen shapes.
Foldit’s success relies on the fact that it doesn’t shallowly flirt with interactivity – it’s a true game. Its creator Seth Cooper designed it to “attract the widest possible audience… and encourage prolonged engagement”. It’s competitive: players are scored based on the stability of the structures they end up with and a leader board shows how they rank against other gamers. There’s also a social side: gamers can chat on online forums, work in groups to solve puzzles and share solutions on a wiki. And just like real game development, everything was tuned according to feedback from the players. Tools were added and refined, the difficulty of the tutorials was tweaked to stop frustrated beginners from leaving, and puzzles were matched to the skills of the players.
There’s the thrill of contributing to genuine scientific research, but that motivates less than half of the community. The rest do it for the achievement, the social aspects and largely, because the game was fun and immersive.
Foldit’s origins lie within , a piece of software designed to solve protein structures by simulating and testing thousands of different folds. Rosetta is an example of ‘ , where volunteers run the program on their home computers when they don’t need it. They effectively donate their computing power to speed up the laborious task of solving protein structures. But the volunteers wanted to use their biological computers – their brains – as well as their man-made ones. They suggested an interactive version of the programme and in May 2008, they got their wish with Foldit.
Last year, Cooper showed that Foldit’s gamers were better than the Rosetta programme at solving many protein structures. They used a wide range of strategies, they could pick the best places to begin, and they were better at long-term planning. Human intuition trumped mechanical number-crunching.
This year, Khatib wanted to see if the Foldit community could solve fresh problems. He entered the players into a twice-yearly contest called
(Critical Assessment of Techniques for Protein Structure Prediction), where structural biologists from all over the world compete to predict the structures of proteins that have almost been solved. They get the best predictions from Rosetta to begin with. Then, they’re on their own.
Khatib’s gamers, bearing names such as Foldit Contenders Group and Foldit Void Crushers Group, had varying degrees of success in the contest. In many of the categories, they did reasonably well but they couldn’t match the best groups. They weren’t as good at using the structures of similar proteins to tweak the ones they were working on. They could also head down dead ends if they started at the wrong place. In one case, their strategy of refining their starting structures to the best possible degree led to one of the “most spectacular successes” in the contest. But mostly, they focused too heavily on tweaking already imperfect solutions that other teams achieved better results by making large-scale changes.
Learning from that lesson, Khatib stepped in himself. He agitated the initial protein structures in many random ways, to create a wide variety of terrible answers that the gamers could then refine. In their attempts, they came up with the best-ranked answer to the most difficult challenge in the competition.
It was a success, and more would follow. After the competition, the players solved an even more important problem. They discovered the structure of a protein belonging to the Mason-Pfizer monkey virus (M-PMV), a close relative of HIV that causes AIDS in monkeys.
These viruses create many of their proteins in one big block. They need to be cut apart, and the viruses use a scissor enzyme –a protease – to do that. Many scientists are trying to find drugs that disable the proteases. If they don’t work, the virus is hobbled – it’s like a mechanic that cannot remove any of her tools from their box.
To disable M-PMV’s protease, we need to know exactly what it looks like. Like real scissors, the proteases come in two halves that need to lock together in order to work. If we knew where the halves joined together, we could create drugs that prevent them from uniting. But until now, scientists have only been able to discern the structure of the two halves together. They have spent more than ten years trying to solve structure of a single isolated half, without any success.
The Foldit players had no such problems. They came up with several answers, one of which was almost close to perfect. In a few days, Khatib had refined their solution to deduce the protein’s final structure, and he has already spotted features that could make attractive targets for new drugs.
“This is the first instance that we are aware of in which online gamers solved a longstanding scientific problem,” writes Khatib. “These results indi-cate the potential for integrating video games into the real-world scientific process: the ingenuity of game players is a formidable force that, if properly directed, can be used to solve a wide range of scientific problems.”
Update: Stephen Curry, who works on protein structures, had this to say about the paper: “Credit where it’s due: this is certainly an innovative approach to the problem of determining crystal structures of proteins. And I do like the idea of ‘citizen science’. Although it’s probably questionable how much science the gamers are understanding, the involvement in this sort of research, even if it is just at the level of playing a game, is undoubtedly a good thing.”
Curry also points out that a structure for this protein was published in 2003 using a different method called nuclear magnetic resonance. Khatib says that this is “quite inaccurate” and that people have struggled to use it to progress any further, but Curry says that they don’t say much about the differences between the old and new structures.
Likewise, Khatib doesn’t mention how closely related the M-PMV protease and the HIV ones are. “This information is crucial for deciding whether a structure of M-PMV protease is going to be any use as a template for the design of novel classes of drug targeted to HIV protease. If I had reviewed this paper, I would have asked for that information to be included because it is needed to make sense of observed differences in structure,” he says.
Reference: Khatib,
DiMaio, Foldit Contenders Group, Foldit Void Crushers Group, Cooper, Kazmierczyk, Gilski, Krzywda, Zabranska,
Pichova, Thompson, Popovi?, Jaskolski & Baker. 2011. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural and Molecular Biology
More on Foldit:
CATEGORIZED UNDER: , , ,
NEW ON DISCOVER
CITIZEN SCIENCE
Discover's Newsletter
Sign up to get the latest science news delivered weekly right to your inbox!
Not Exactly Rocket Science
Dive into the awe-inspiring, beautiful and quirky world of science news with award-winning writer Ed Yong. No previous experience required.
Discover Blogs
Select Category
Select Category
Amphibians
Ancient DNA
Animal behaviour
Animal communication
Animal defences
Animal intelligence
Animal movement
Animal senses
Anthropology and social science
Art and Culture
Bioinspiration
Biotechnology
Butterflies and moths
Carnivores
Cephalopods
Child development
Chimps and other apes
Climate change
Computer science
Consciousness and free will
Conservation
Convergent evolution
Cooperation
Crustaceans
Decision-making
Dolphins and whales
Drug resistance
Earth sciences
Echinoderms
Embodied cognition
Environment
Epigenetics
Evolutionary arms races
Exaptation
Eye evolution
Feathered dinosaurs
Game theory
Gender issues
Gene duplication
Genes and behaviour
Genetic disorders
Genetic modification
Horizontal gene transfer
Human evolution
Inside the brain
Invasive species
Invertebrates
Journalism
life history
Material science
Mathematics
Medicine & health
Mental Health
Microbiome
Molecular biology
Mosquitoes
Neuroscience and psychology
New drugs & treatments
Not Exactly Pocket Science
Not Exactly Rocket Satire
Palaeontology
Perception
Predators and prey
Rats and mice
Risk-taking
Sex and reproduction
Sexual conflict
South African wildlife
Speciation
Stem cells
Synthetic biology
Technology
Transitional fossils
Uncategorized
Video games
Year in review
Select Tag
Select Archive
Select Archive
December 2012
November 2012
October 2012
September 2012
August 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
April 2008
March 2008
February 2008
Recent Posts
Follow us:
More great sites from
 | 
FREE Newsletter
E-mail address:
Remember me
Forgot your password?
No problem.
to have it e-mailed to you.
Not Registered Yet?
Register now for FREE.
Registration only takes a few minutes to complete.Problem Solving Education - Teaching in Schools & Life
THINKING&SKILLS
Be&Creative-and-Critical
What,&Why,&and&How
Principles&and&Strategies
Liberating&Creativity
Creativity&for&Living
Creativity&in&Education
Research&about&Creativity
What&is&critical&thinking?
Why&teach&critical&thinking?
How&to&teach&it&effectively?
The&Ethics&of&Critical&Thinking
PROBLEM-SOLVING&SKILLS
Problems&and&Problem Solving
Creative-and-Critical&Thinking
Multiple&Intelligences&&&Styles
Problems&in&Design&&&Science
Problem&Solving&in&Education
You&can&explore&other&parts&of&our
website&for&Whole-Person&Education
(using&links&at&)&and
Education for
Problem Solving
Schools, for Life )
The sections in this page are:
&&&&&&&&&&&&&&&&&&
Problems and Problem Solving
What is a problem?& In
common language, a problem is an unpleasant situation, a difficulty.& But
in education the first definition in Webster's Dictionary — &a
question raised for inquiry, consideration, or solution& — is
a more common meaning.& More important, in life a problem is
any situation, in any area of life, where you have an opportunity to make a difference, to
& and problem solving is converting an actual current state into a desired future state.& Whenever you are thinking creatively-and-critically about ways to increase the quality of life (or to avoid
a decrease in quality), you are actively involved in problem solving.&&&& { by Craig Rusbult, editor of this links-page}& {more about
Problem-Solving Skills & Creative and Critical
An important goal of education
is helping students learn how to
while solving problems, by combining creative
thinking (to generate ideas) and critical
thinking (to evaluate ideas).& Both modes of thinking
are essential for a well-rounded productive thinker, according to scholars
in both fields:
Richard Paul (a prominent
advocate of )
says, &Alternative solutions are often not
given, they must be generated or thought-up.& Critical thinkers
must be creative thinkers as well, generating possible solutions in
order to find the best one.& Very often a problem persists, not
because we can't tell which available solution is best, but because
the best solution has not yet been made available — no one
has thought of it yet.& {}
Patrick Hillis & Gerard
Puccio (who focus on ) describe the combining of divergent generation and convergent
evaluation in a strategy of Creative Problem Solving that &contains
many tools which can be used interchangeably within any of the stages.& These
tools are selected according to the needs of the task and are either
divergent (i.e., used to generate options) or convergent (i.e., used
to evaluate options).& {}
Multiple Intelligences &
Learning Styles
We . & When we're solving a wide variety of problems, we can think productively
in a variety of ways, as described in a theory of
developed by Howard Gardner.& While we're trying to help students improve a wide variety of abilities,
we can develop teaching strategies that will be
effective for students
with different .& For example, we can help students understand-and-use...
Visual Logic:& We
can think logically & useful
include visually logical organizing techniques — concept maps, matrices
and diagrams (cluster, hierarchical, webbing, Venn,...), flowcharts,... — that
can encourage and facilitate creative-and-critical thinking.& [[== also art]]
Problem-Solving Process for Science and Design
We'll look at problem-solving process for Science (below) and , plus .
Problem-Solving Process for Science
Is there a “scientific method”?& We have reasons to say...
& & NO, because
there is not a rigid sequence of steps that is used in the same way by all scientists, in all areas of science, at all times.
& & and also YES, because expert scientists (and designers) tend to be more effective when they use flexible
strategies
& analogous to the flexible goal-directed improvising of a hockey player,
rigid choreography of a figure skater & to stimulate & coordinate their thinking-and-actions in productive ways, so they can solve problems more effectively.
Here are some models that can help students understand-and-do the process of science.& We'll begin with simplicity, before moving on to descriptions that are more thorough and accurate.
A simple model of science is PHEOC (Problem, Hypothesis, Experiment, Observe, Conclude).& When PHEOC, or a similar model, is presented & or is misinterpreted & as a rigid sequence of fixed steps, this can lead to misunderstandings of science, because
the real-world process of science is flexible.& An assumption that “model = rigidity” is a common criticism of all models-for-process, but
because all models emphasize the flexibility of problem-solving process in real life, and (ideally) in the classroom.& If a “step by step” model (like PHEOC or its variations) is interpreted properly and is used wisely, the model can be reasonably accurate and educationally useful.& For example,...
The 3-step model of
is useful for classroom instruction.
Science Buddies has
with a flowchart showing options for flexibility of timing.& They
say &Even though we show the scientific method as a series of steps, keep in mind that new information or thinking might cause a scientist to back up and repeat steps at any point during the process.& A process like the scientific method that involves such backing up and repeating is called an iterative process.&&& And they .
Lynn Fancher explains - in
- that &while science can be done (and often is) following different kinds of protocols, the [typical simplified] description of the scientific method includes some very important features that should lead to understanding some very basic aspects of all scientific practice,& including Induction & Deduction and more.
Other models for the problem solving process of science are more complex, so they can be more thorough & by including a wider range of factors that actually occur in real-life science, that influence the process of science when it's done by scientists who work as individuals and also as members of their research groups & larger communities & and thus more accurate.& For example,
Understanding Science (developed at U.C. Berkeley - )
describes a broad range of science-influencers,* beyond .& Because &the process of science is exciting& they want to &give users an inside look at the general principles, methods, and motivations that underlie all of science.&& You can begin learning in their
(with US 101, For Teachers, Resource Library,...) and an interactive
for &How Science Works& that lets you explore with mouse-overs and clicking.
* These factors affect the
process of science, and occasionally (at least in the short run) the results of science.& To learn more about science-influencers,...
& & Knowledge Building (developed by Bereiter & Scardamalia,
- ) describes a human
process of socially constructing knowledge.
by Henry Bauer & author of
(click &look inside&) & examines The Knowledge Filter and a Puzzle and Filter Model of &how science really works.&
Another model that includes a wide range of
factors (empirical, social, conceptual) is Integrated Scientific Method by Craig Rusbult, editor of this links-page.& Part
my PhD work was developing this model of Scientific Method, in a unifying synthesis of ideas from scholars in many fields, from scientists, philosophers, historians, sociologists, psychologists,
educators, and myself.& The model is described in two brief outlines (& ) and,
more thoroughly, in a
(with introduction, two visual/verbal representations, and summaries for ) and a
(examining the 9 aspects more deeply, with illustrations from history & philosophy of science), and even more deeply in my
(with links to the full text, plus a
“world record” Table of Contents, references, a visual history of
my diagrams for Science Process & Design Process, and ).
Problem-Solving Process for Design
Because &designing& covers a wide range of activities, we'll
three kinds of designing.
Engineering Design Process:& As with Scientific Method,
basic process of Engineering Design can be outlined in a brief models-with-steps& && & & & & & .&&&& {pages are produced by
& & and it can be examined in more depth:&
& & and in some of the models-with-steps above, and .
Problem-Solving Process:& similar thinking strategies, applied to a wider range of life, also have models-with-steps& && & & & & .
Design-Thinking Process:& similar thinking strategies, but with a strong emphasis on
empathy, as explained .
Later we'll look at more models, and how they can be used for education. ==[add links]
Problem-Solving Process for Science-and-Design
Science and Design:&
has separate models for
showing options for flexibility-of-timing
when using &Steps of the Scientific Method&) and for
to show their similarities & differences.& And they explain how both models describe
even though each model-framework has steps.
Above and below, you'll see separation (into Science and Design) versus integration (for Science-and-Design):& Above, Science Buddies has
separate models for
Science, and for Design.& Below is one
model that includes both together, with an integration of...
Science-and-Design:& While thinking about the problem-solving process we use for Science and for Engineering, I (Craig Rusbult, editor of this links-page) discovered
functional connections between 3 Elements & PREDICTIONS (made by imagining in a Mental Experiment) and OBSERVATIONS (made by actualizing in a Physical Experiment) and GOALS (for a satisfactory Problem-Solution) & when they are used in 3 Comparisons:& one Comparison is an evaluative Reality Check, used for S& two Comparisons are
evaluative Quality Checks, used for Engineering.
Science and Engineering are related, but are not the same, of course.& Therefore it's useful to distinguish between Science-Design (usually called Science) and General Design (which includes Engineering & much more).& This helps us think about problem-solving process for
The Wide Scope of Design
By using science and design, people try to make things better by .&& It can be educationally useful to
so it includes two kinds of design, with different problem-solving objectives:
& & in Science-Design (commonly called Science) we want to answer a question (a problem-question about “what happens, how, and why?”) by designing an explanatory theory, to “make your knowledge better” and help satisfy a human desire for understanding.
& & in General Design (commonly called Design) we want to solve a problem by designing a solution that is a better product, activity, or strategy, to “make things better” by helping satisfy other human needs-and-desires.
* In both kinds of Design, the objective is to solve a problem by &making things better& with improved understanding (in Science-Design) and (in General Design) by improving other aspects of life.& Together, these objectives include almost everything we do in life.&&&& {&&}
One part of
&almost everything& is... The Wide Scope of Science-Design:& In
all of life, not just in science, we use our explanatory theories about “how the world works” to understand “what is happening, how, why”
predict &what will happen& in the future.& When our
theories about the world are more thorough and accurate, this improved understanding will increase the accuracy of our theory-based predictions that, along with
good values & priorities, help us make
wise decisions,
personally and professionally, while pursuing our goals in life.& We use scientific thinking often in life, whenever we hear a claim, or make a claim, and
ask &what is the evidence-and-logic supporting this claim?&
Another part of
&almost everything& is... The Wide Scope of General Design:& General Design is used for “engineering” and much more.& In the
for K-12 Education, there is
a broad definition of engineering (it's &any engagement in a systematic practice of design to achieve solutions to particular human problems&) and technologies (which &result when engineers apply their understanding of the natural world and of human behavior to design ways to satisfy human needs and wants& and &include all types of human-made systems and processes&) in order to &emphasize practices that all citizens should learn & such as defining problems in terms of criteria and constraints, generating and evaluating multiple solutions, building and testing prototypes, and optimizing & which have not been explicitly included in science standards until now.& (from Appendix I, &Engineering Design in the NGSS&)&& /&& When we also include other kinds of General Design, and Science-Design, the scope of Design Thinking expands to &include almost everything we do in life.&
Problem-Solving Process
The basic process is simple:
& & first, to Define a Problem you Define your Objective (for what you want to “make better”) and Define your Goals (for a satisfactory Problem-Solution);
& & then, to Solve the Problem you creatively Generate Options (for a Problem-Solution) and critically Evaluate Options, in iterative Cycles of Design.
Later in this page you can see “more about process” in .& In all models, an important activity is...
Designing Experiments so you can
Use Experiments
What is an experiment?&& While you are Designing Experiments, you can stimulate your creative thinking & by
so you can more freely explore the wide variety of Options for Experiments & with a simple, broad, minimally restrictive definition:& an Experiment is any Situation/System that provides an opportunity to get
Information by making Observations (in a Physical Experiment) or making
Predictions (in a Mental Experiment), so an Experimental System is any Observation-Situation or Prediction-Situation.
How do you use experiments?& During a process of problem solving,
you often Design Experiments (they're “things happening” in
Experimental Systems) that
you think might provide useful Information, that
might help you solve the problem.& Then you USE Experiments in three ways:
& & 1. USE an Experiment (Mental or Physical) to make Information (Predictions or Observations);
& & 2. USE this Experimental Information to do Evaluation of an O
& & 3. USE this Experiment-Based Evaluation to guide Generation of other Options.
USES are described in more detail below, and you can see them in the diagram.& When you study
8 times you'll find
&using& or &Use& or &use&.& And when you move your mouse over the &1 2 3 3& boxes
added to it, you can see four isolation diagrams that show only the problem-solving actions
for USE #1 (&using& to make Information) and USE #2 (&Use& to do Evaluation) and USE #3 (&use& to guide Generation in one
Science Cycle & two Design Cycles).
1. for Experiment & Information,& you USE an Experiment & by “running it” physically or mentally & to make two kinds of Experimental Information.& How?
& & You imagine the Experimental System in a Mental Experiment so you can make PREDICTIONS,
& & or you actualize the Experimental System in a Physical Experiment so you can make OBSERVATIONS.
2. for Information & Evaluation,& you USE this Experimental Information (from #1) to do two kinds of Experiment-Based Evaluation, with...
& & & evaluative Reality Checks:& During a process of Science-Design or General Design, you can test your explanatory Theory(s) by comparing your Theory-based PREDICTIONS with Reality-based OBSERVATIONS.& This evaluative comparison is a Reality Check that will help you determine how closely “the way you think the world is” corresponds to “the way
the world really is.”
& & & evaluative Quality Checks:& Early in a
process of General Design, you Define your GOALS for a Solution, for the properties you want in a problem-Solution that is ideal, or at least is satisfactory.& Later, you
generate Options for a Solution.& You can test
the Quality of an Option by comparing your GOALS (for your desired properties, which define Quality)
PREDICTIONS (about expected properties of this Option) or with OBSERVATIONS (the observed properties of this Option).& These evaluative comparisons & when
you ask “how closely do the properties of this Option match the properties I want?” & are Quality Checks.
3. for Evaluation & Generation,& you USE this Experiment-based critical Evaluation (of an old Option in #2) to stimulate-and-guide your creative Generation (of a new Option in #3).& How?
& & & In Science-Design, if necessary & if you were “surprised” because (when you Evaluated in #2 using a Reality Check) your OBSERVATIONS didn't match your PREDICTIONS & you ask
(when you're Generating in #3)
“how can I revise my old Option {for a Theory} about how I think the world is, so it corresponds more closely to how the world really is.”
& && In General Design, you ask (based on Evaluation in #2 using a Quality Check) “what aspects of the old Option {for a Solution} need to be improved?”
and then (for Generation in #3) “how can I revise this old Option to improve it, to generate a better new Option?”
MORE & ==[later, here I will add a link - re: 8 ways & options for branching]
Problem Solving for Education & Teaching Skills in Schools
Educators should want to design instruction that will help students improve their
thinking skills.& An effective strategy for doing this is...
Goal-Directed Designing of Curriculum & Instruction
When we are trying to solve a problem (to “make it better”) by improving education, a useful two-part process is to...
& & & Define GOALS for
desired outcomes, for the
ideas & skills we wa
& & & Design INSTRUCTION with Learning Activities that will provide opportunities for experience with these ideas & skills, and will help students learn more from their experiences.
Basically, the first & is about WHAT to Teach, and the second & is HOW to Teach.
But before looking at
and &, here are some ways to combine them with...
Strategies for Goal-Directed Designing of WHAT-and-HOW.
Understanding by Design (UbD) is a team of experts in goal-directed designing:
Vanderbilt U has an excellent .
(the ) that includes pages about * and .&& /&& *&UbD &offers a planning process and structure to guide curriculum, assessment, and instruction.& Its two key ideas are contained in the title:& 1) focus on teaching and assessing for understanding and learning transfer, and&& 2) design curriculum &backward& from those ends.&
() says, &students succeed when educators start with the end goal in mind.& This backward design approach allows educators to create deliberate and focused unit design choice.&
(the ) offers
that include
& others) and & when you explore using the Navigation Bar (Articles, Books, Webinars,...) or just scroll down the page &
much more.
describes two key features of UbD:& &In backward design, the teacher starts with classroom outcomes and then plans the curriculum, choosing activities and materials that help determine student ability and foster student learning,& and& &The goal of Teaching for Understanding is to give students the tools to take what they know, and what they will eventually know, and make a mindful connection between the ideas. ...& Transferability of skills is at the heart of the technique. Jay McTighe and Grant Wiggin's technique.& If a student is able to transfer the skills they learn in the classroom to unfamiliar situations, whether academic or non-academic, they are said to truly understand.&
Other techniques include .
In two steps for a , you:& 1) Define GOALS (for WHAT you want students to improve);& 2) Design INSTRUCTION (for HOW to achieve these Goals).& Although the sections below are labeled
there is lots of overlapping, so you will find some &how& in the WHAT, and much &what& in the HOW.
1 & Define GOALS (decide WHAT to Teach)
What educational goals are most valuable for students?& Here are some options:
Ideas-and-Skills:& We
define goals for ideas (what students know) that are conceptual knowledge, and for skills (what they can do) that are procedural knowledge.& Our goals for ideas-and-skills include ideas, and skills that are applied in skills-with-ideas when creative-and-critical thinking skills interact with ideas in .
A Bigger Picture:& We want to help students improve their
and achieve a wide range of desirable outcomes that are COGNITIVE (for ideas-and-skills in many areas of school & life) and AFFECTIVE (for attitudes, motivations, emotions) and PHYSICAL (for nutrition, health & fitness,
physical skills) and for CHARACTER (for empathy, kindness, compassion,
ethics,...).
Because we have limited amounts of educational resources & of time, people, money,... & we must ask, “How much of these resources should we invest in each
kind of goal?”
Although the discussion below recognizes the wide-context “big picture” of educational goals, it will focus mainly on Cognitive Goals for Ideas-and-Skills, but with some discussion of goals for
and .& Even within this restricted
range (of goals that are mainly cognitive) we must make many decisions, including the following
choices (re: ideas & skills, science & design, performing & learning) about priorities:
Ideas versus Skills?
Most educators want to teach ideas AND
skills, but unfortunately a competitive tension often exists.& If we are
able to maximize a mastery of both, we should aim for an optimal combination of ideas and skills.& But what is optimal?& Many educators, including me, think the balance should shift toward more emphasis on skills and skills-with-ideas, aiming for an improvement in skills-ability that outweighs (in our value system) any decrease in ideas-ability.& This is possible because &ideas versus skills& is not a zero-sum game, especially for lifelong learning when we
to help students cope with a wide range of challenges in their futures.
The Difficulty of Designing Exams to Evaluate Skills:& We want to generate accurate information about student achievements with both ideas and skills.& But measuring ideas-knowledge is easy compared with the difficulty & expense of accurately measuring skills-knowledge.& This is an important
factor when educators (at the levels of classroom, school, district, state, and nation) develop strategies & make policy decisions for education, and there are .
Two Kinds of Inquiry Activities& (for Science and Design)
To more effectively help students improve their problem-solving skills, teachers can provide opportunities for students to be actively involved in solving problems, with inquiry activities.& What happens during inquiry?& Opportunities for inquiry occur whenever a gap in knowledge & in conceptual knowledge (so students don't understand) or procedural knowledge (so they don't know what to do, or how) & stimulates action (mental and/or physical) and students are allowed to think-do-learn.
Students can be challenged to solve
during two kinds of inquiry activity:
& & during Science-Inquiry they try to improve their understanding, by asking problem-questions and seeking answers.& During their process of solving problems, they are using Science-Design, aka Science, to design a better explanatory theory.
& & during Design-Inquiry they try to improve some other aspect(s) of life, by defining problem-projects and seeking solutions.& During their process of solving problems, they are using General Design (which includes Engineering and more) to design a better product, activity, or strategy.
& & But... whether the main objective is for Science-Design or General Design, a
skilled designer will be flexible, will do whatever will help them solve the problem(s).& Therefore a “scientist” sometimes does engineering, and an “engineer” sometimes does science.& A teacher can help students recognize how-and-why they also do these “”
during an activity for Science Inquiry or Design Inquiry.&
Due to these connections, we can ,& and combine both
to develop “hybrid activities” for Science-and-Design Inquiry.
Goal-Priorities:& There are two kinds of inquiry, so (re: Goals for What to Learn) what emphasis do we want to place on activities for Science-Inquiry and Design-Inquiry?
Two Kinds of Improving& (for Performing and Learning)
Goal-Priorities:& There are two kinds of improving,
so (re: Goals for What to Learn) what emphasis do we want to place on better Performing (now) and Learning (for later)?
When defining goals for education, we ask “How important is improving the quality of performing now, and (by learning now) of performing later&?”&& For example, a basketball team (coach & players) will have a
different emphasis in an early-season practice (when their main goal is learning well) and
end-of-season championship game (when their main goal is performing well).&&&& {we can try to optimize the “total value” of }
2 & Design INSTRUCTION (decide HOW to Teach)
Strategies for
One useful strategy for
Other educators also have developed strategies for goal-directed designing.& For example,
== can help
guide our selection-and-sequencing of
activities that include
== to help students achieve specific learning outcomes.&&With more detail,
& & &An integrative analysis of instruction can improve our understanding of the functional relationships between activities, between goals, and between activities and goals.& This knowledge
about the structure of instruction (as it is now, or could be later) can help us
coordinate & with respect to types of experience, levels of difficulty, and contexts & the activities that help students achieve goals for learning.& The purpose of a carefully planned selection-and-sequencing of activities is to increase the mutually supportive synergism between activities in a coherent system for teaching all of the goals,
to produce a more effective environment for learning.&
Strategies for Teaching
There is a wide variety of views, and thus controversy, when educators ask important questions:
should thinking skills play in education?&
If there is a “competition” of
limitations (of time, people, money) so we must make tough decisions,...& How much of our valuable resources should we invest in
thinking skills?
How can we effectively teach
thinking skills?” & there is a wide variety of views.
& What role
should thinking skills play in education?&
& Asking &How can we effectively teach
thinking skills?& leads to many sub-questions, including these:
& (e.g. reading, discussing,& learning by discovery and/& lecture & and more)
[ to be continued below ]
I.O.U. - Soon,
March 4-7, I will continue revising everything in this &brown box&
in simple ways (like checking-and-fixing links) and by developing the ideas in it more fully,
expressing them more clearly, and organizing them better.&
(note: Places with &==& are notes-to-myself about things that need to be fixed.& And other &fixers& will be obvious.)
An excellent overview is
by Kathleen Cotton. (the second half of her page is a comprehensive bibliography)
This article is part of
(available from
and ) where you can find many useful articles about thinking skills & other topics, by Cotton & other authors.
An excellent overview is
by Kathleen Cotton.* (the second half of her page is a comprehensive bibliography)
This article is part of
(available from
and ) where you can find many useful articles about thinking skills & other topics, by Cotton & other authors.
* [== it's excellent but fairly old, 1991, IOU - I will search and find more-recent overviews ]]
Another useful page —
(by Fennimore & Tinzmann) — begins with
principles and then moves into applications in Language Arts, Mathematics,
Sciences, and Social Sciences.
links-page for
== summarizes
and explores a
variety of ideas about effective teaching (based on
principles of constructivism,
meaningful reception,...) designed to stimulate active learning and
improve thinking skills.& Later,
a continuing exploration of the
web will reveal more web-pages with
useful &thinking skills & problem solving& ideas (especially
for K-12 students & teachers) and we'll share these with you, here
and in .[==this duplicates a sentence above]
Of course,
thinking skills are not just for scholars and schoolwork, as emphasized
in an ERIC Digest, .& And
you can get information about 23
the U.S. Dept of Education.&
== helping students learn how to develop/use non-violent solutions for social problems /reference/article/teach-young-children-problem-solving/
Infusion and/or Separate Programs?
Earlier, among the unresolved questions is & What is the difference?& With infusion, thinking skills are closely integrated with content instruction in a subject area.& In separate programs, independent from content-courses,
the explicit focus is to help students improve their thinking skills.
[blockquote] Of
the demonstrably effective programs, about half are of the infused
variety, and the other half are taught separately from the regular
curriculum. ...& The strong support that exists for both approaches...
indicates that either approach can be effective.& Freseman represents
what is perhaps a means of reconciling these differences [between
enthusiastic advocates of each approach] when he writes, at the conclusion
of his 1990 study: “Thinking skills need to be taught directly
before they are applied to the content areas. ...& I consider
the concept of teaching thinking skills directly to be of value especially
when there follows an immediate application to the content area.”
For principles and examples
of infusion, check the National Center for Teaching Thinking which
lets you see == (an introduction to the art of infusing thinking skills
into content instruction), and == (for different subjects, grade levels, and thinking skills). http://teach-think.org/resources/lessons-and-articles/ [== lessons designed to infuse Critical and Creative Thinking into content instruction]
(by Robert Swarz & David Perkins) also /category_s/535.htm
DIRECT TEACHING OF PRINCIPLES ?
explicit direct teaching of principles? -- yes, /science/article/pii/S0066
link to my #wyep
zb, CER for all (+ POE for Science)
USING MODELS ?
#NSM - using models?
During , students can learn principles of inquiry-process by using a model and/or semi-model and/or no model. website#dpomnsm??/wsepp?
COMBINING MODELS -- combining two (or more) Models-for-Process
Structure + Function -- structure for instruction, strategies for thinking
combining models -- Short-Term plus Long-Term -- link to home.htm%234a4b / dp-om.htm#seq/ws.htm#dpmo4aseq ? -- also
link to section above, with models for Science, and Engineering Design, plus my Science-and-Design
Also, for different kinds of models -- Robert Marzano's New Taxonomy of Educational Objectives has three systems (Self-System, Metacognitive System, Cognitive System) and a Knowledge Domain that includes Information, Mental Procedures,
Physical P&
and Models of Problem Solving & Learning (from educational researchers at CRESST) provide a framework for thinking about an
== that uses Design Process to improve the mutually supportive interactions between ideas and skills.&
GUIDING - MINI-ACTIVITIES, etc
During any
== (like ) a teacher's
interactions with students
== that are opportunities for thinking-and-learning.& To “guide” students a teacher can
ask questions, respond to
questions, give tips (to adjust the level of difficulty), model thinking skills, provide formative feedback, and
by directing attention to
“what can be learned” at appropriate times during the activity.& The main objectives of skillful guiding &
by wisely choosing the types, amounts, and timings of guidance & are to help students improve their current performing (so they can solve a problem now) and/or their current learning (so they can improve their future performing), to optimize the total value
(in ) of their educational experience.&&&& {an overview by Craig Rusbult}== cm-ei.htm%23dai / ws.htm%23hwmini
Effective feedback-for-learning can come from a teacher, or in other ways.& For example,...
ThinkSpace is & developed at ISU [Iowa State U] that encourages students to think critically about the solutions to complex, real-world problems. ... By using real-world scenarios, it allows students to work through electronic platforms and permits faculty to see how the students arrive at their final solutions.&& ThinkSpace & a case-study approach that simulates real-world problems and environments, thereby encouraging... innovative curriculum and instructional approaches to problem solving.&& It & present complex problems to students, which are completed through a series of intermediate tasks.& By receiving automated feedback on their work, students will be able to track their progress as they work to solve the problems.&
Some teachers are
THINKING SKILLS& (and PROCESS)
We can help students improve “thinking skills” that are general (useful for a wide range of problems) and specific (more useful for a certain kind of problem).
GENERAL Thinking Skills& (and Process)
are useful for solving all problems, in all areas of life. (@ first section)
Blooms Taxonomy:&
The basic principles of Bloom's Taxonomy (Original 1956, and Revised 2001) are
by Patricia Armstrong (for Vanderbilt U) who links to a
(by Mary Forehand, U of Georgia) that &is particularly useful because [in addition to its valuable information] it contains links to dozens of other web sites.&&&
(from U of Illinois) includes
by Barbara Shadden.
Domains of Learning:& Benjamin Bloom led a committee that proposed a Taxonomy of Learning Domains (Cognitive, Affective,
Psychomotor) along with Levels of Learning (e.g. for Cognitive, Bloom's levels are Knowledge, Comprehension, Application, Analysis, Synthesis, Evaluation);&
(by Vernellia Randall, U of Dayton School of Law) describes the domain-categories, &Cognitive is for mental skills (Knowledge), affective is for growth in feelings or emotional areas (Attitude), while psychomotor is for manual or physical skills (Skills).&
models for Evaluative Thinking - CER & POE --
left=website.htm%23trsci&right=ws.htm%23dpmo3a
Ideas plus Skills:& The importance of teaching ideas-and-skills (i.e.
skills-with-ideas) is emphasized by prominent educators, including NGSS, Marzano, CRESST. [==find more links, + NGSS, Common Core]
Ideas plus Skills:& Many prominent educators emphasize the importance of defining knowledge widely, to include
ideas-and-skills.
Marzano, etc (from my #cm and #is)
(NGSS) recommend instruction that
&integrates
the knowledge of scientific explanations (i.e., content knowledge) and the practices [i.e., procedural knowledge] needed to engage in scientific inquiry and engineering design.& (and I'm hoping NGSS won't increase )&&
Problem-solving methods (like Design Method and Scientific Method)
are just strategies for effectively combining familiar thinking
skills in order to achieve a goal, to solve a problem.&
Skills and Problem-Solving Methods are
closely related, as shown in an
== that compares four perspectives: Design Process (Rusbult), Dimensions of Thinking (Marzano, et al), Infusion of Thinking Skills (Swartz), Four Frames of Knowledge (Perkins).
== ?? [more generally,
is a sitemap for pages by Craig Rusbult.
METACOGNITION
As one part of
a teacher can help students reflect on their experiences, so .==& This metacognitive reflection is essential in a teaching strategy of using a process of inquiry (with reflections, discussions, explanations) to teach principles for inquiry. home.htm%23 / ws.htm#hw#dpinq
https://cft.vanderbilt.edu/guides-sub-pages/metacognition/
mindfulness -- use?? -- https://cft.vanderbilt.edu/guides-sub-pages/contemplative-pedagogy/
dp-mo.htm#==/ws.htm#mc
thinking strategies == home.htm#? / ws.htm%23mc / ws.htm#mctwo ---- dp-mo#emmc (emp + self-emp)/#mc ---- #mcreg/#mctwo
learning more from experience == mc-we.htm / dp-om.htm%23srl / ws.htm%23mclshow [do with 2 links, for mc-we and for #srl]
active.htm#mc ? for links == What is metacognition, and how is it useful? LINKS -- active.htm#metacognitio
EXPERIMENTAL Thinking Skills& (and Process)
Designing and Using Experiments
Although experimental skills are used generally (in
all problem solving), experimenting is typically
associated with Science.& For example,...
And from Craig Rusbult,
about goal-directed design, anomaly resolution, crucial experiments, heuristic experiments, vicarious experimentation, thought-experiments, and more.
Lester Miller's
uses a model
with 7 steps and a flowchart showing Hypothetico-Deductive Logic.& To illustrate the scientific skill of designing useful experiments, they use the historical question of Spontaneous Generation, and explain how the hypothesis of Francesco Redi & that flies (not meat by itself) are necessary to produce maggots & was supported by observations in the
experiments he designed and ran.&&&& {also: Katie Mayfield cleverly designed a
with more details about the history of &spontaneous generation& theories (pro & con) and the scientific resolution of this controversy due to the carefully designed experiments of Francesco Redi and (later) Louis Pasteur.}& {more about }
And in an overview of
by Kathleen Marrs (of IUPUI),& Section 2 & &Experimentation: The Key to the Scientific Method& & begins, &A key ingredient of the scientific process: the controlled experiment.&& She links to an example, with experiments to test explanatory theories about && Or, from U of Arizona, in an everyday situation you might be wondering
Basic experimental skills (examined in ERIC Digests) are
Mill's Methods:& These logical principles can help you Design Useful Experiments and Analyze Experimental Observations to Determine Causes.&&
SCIENCE Thinking Skills& (and Process)
POE (Predict, Observe, Learn) is near the other end of a spectrum for simplicity&complexity.& It's in
== which ends with a question, &What is missing in POE?&& One
== is that the main science-actions it's
missing are “Ask a Question” and “Design an Experiment”.& But a teacher can adapt instruction using basic POE (or basic CER) so students do Ask Questions and Design Experiments.& In fact, an obvious question when using CER (by asking “is that all the Evidence you have? how could you get more?”) will challenge a student to
(by ) so they can get more Evidence (old or new). // @ POE/CER Evaluative Thinking
Science Buddies - with links
about - physics & chemistry
Using Scientific Method
Dany Adams (Smith College)
helps students learn how to think more effectively by : &Because
the scientific method is a formalization of critical thinking, it
can be used as a simple model that... puts critical thinking at the
center of a straightforward, easily implemented, teaching strategy.
...& Explicitly discussing the logic and the thought processes
that inform experimental methods works better than hoping students
will ‘get it’ if they hear enough experiments described.&
mine -- put all in #4, link to that --
is a good starting place for the main activities of science:& Using Experiments to Make Observations, Making Predictions, Theory Design (by generation-and-evaluation),
Experimental Design (by generation-and-evaluation).
for teacher [certainly] and/or [by reading, or by teacher passing on] student -- The basic ideas of &scientific
method& (and associated scientific thinking skills) are outlined
in overviews from
(five &steps& and key elements)
Introduction to the Scientific Method [and its Key Elements] -
Hypothesis, Model, Theory & Law - Experimental Methods - and more (Books on Science and Politics,
left=website.htm%23trsci&right=ws.htm%23dpmo3a
&&Science: A Process Approach (SAPA) was a
that began in the 1960s.& Michael Padilla explains how SAPA
as &a set of broadly transferable abilities, appropriate to many science disciplines and reflective of the behavior of scientists.& SAPA categorized process skills into two types, basic and integrated.& The basic (simpler) process skills provide a foundation for learning the integrated (more complex) skills.&&&&& {}
definition of process skills - skills in doing process - coordinating by developing-and-using conditional knowledge, metacognitive - or as in SAPA
http://medical-/process+skills
These are the &process& skills -- &we use to guide and direct key parts of our organizing work with groups of people such as meetings, planning sessions, and training of our members and leaders. Whether it's a meeting (big or small) or a training session, someone has to shape and guide the process of working together so that you meet your goals and accomplish what you've set out to do.&
are &Skills used to manage and modify actions in the completing of daily living tasks, such as pacing oneself, choosing and using appropriate tools to complete a task, or organizing a task into a logical sequence for successful completion.&
Building Teamwork Process Skills in Students - http://teaching.berkeley.edu/news/building-teamwork-process-skills-students
& In the engineering disciplines,... project teams are an important part of our work culture, ...
While the concept of effective teamwork is highly valued, few instructional hours and resources are typically devoted [in typical undergraduate engineering education] to specifically developing this skill.
& Promotive Interaction: Members do real work, usually face to face
& Positive Interdependence: Members focus on a common goal, with complementary contributions
& Individual and Group Accountability: Everyone takes responsibility for their own work and the overall work of the team
& Teamwork Skills: Each member practices effective communication, decision making, problem solving, conflict management, leadership
& Group Processing: Team periodically reflects on how well the team is working
DESIGN Thinking Skills& (and Process)
@ General Design (vs Science-Design) so has Wide Scope
ENGINEERING DESIGN PROCESS -- EPICS-etc,
DESIGN THINKING -- dschool, nueva, etc --
emphasis on empathy!
WIDE SCOPE ==[here, I will describe differences between my broad definition of &design& and narrower definitions that emphasize
problem solving that is human-centered, and the i& also, differences in models-for-process]
Problem-Solving Objectives:& An objective is a
you want to solve, so you can &make things better.&& People
use creative-and-critical
to solve problems in a wide range of
design fields & such as engineering, architecture, mathematics, music, art, fashion, literature, education, philosophy, history, science (physical, biological, social), law, business, athletics, and medicine & when the objective is to design (to find, invent, or improve) a better product, activity, strategy (in General Design) and/or (in Science-Design) an explanatory theory.& These objectives include almost everything we do in life.
In some ways, science process (aka scientific method) is similar to design process but there is a new focus for action.& In
science the main goal is to understand nature, to construct a theory and
test its accuracy with reality checks that help
us decide if &the way we think the world is& corresponds to &the
way the way the world really is.&& It can be useful to think of science as
the designing of theories, and conventional design as
the designing of products or strategies.& [== the Science Question - Were you surprised? (why?)]
& & Building Educational Bridges (for Transfers of Learning & Transitions of Attitudes) between School and Life}
& & Building Educational Bridges between School and Life, for Transfers of Learning & Transitions of Attitudes}
If you're wondering &What
can I do in my classroom tomorrow?&, eventually there will be a section for &thinking
skills activities& in
the area for .
with teaching strategies that provide . ==[use structure+strategies only in edu-section?]
MOTIVATIONS and EQUITY
https://cft.vanderbilt.edu/guides-sub-pages/motivating-students/
The wide scope of
== (from life into school, and from school into life) to improve transfers of learning & transitions of attitudes, and problem-solving skills.& This will help us improve diversity-and-equity in education by increasing confidence & motivations for a wider range of students, and providing a wider variety of opportunities for learning in school, and success in school. etalk.htm%23br1/ws.htm%23mo
These bridges & from life into school, and back into life & will improve transfers-of-learning and transitions-of-attitudes.& This will help more students, with a wider diversity, improve their confidence & motivations and problem-solving skills, for better educational equity.
A motivated
student — perhaps inspired by an effective teacher — can
adopt == by
imagining the benefits of improved personal knowledge-and-skill in
the future. mo.htm#life / website#trlife ws.htm#trlife
https://cft.vanderbilt.edu/guides-sub-pages/diversity/ ?? [find others?]
Problem-Based
is a way to improve motivation, thinking, and
learning:& you can read a
using Problem-Based Learning for
two websites to explore (Samford
University -
the book-intro for
(use quotes==) -
for problem-based
learning -
and a comprehensive ==.
PBL -- Problems as Possibilities by Linda Torp and Sara Sage
Table of Contents http://www.ascd.org/publications/books/101064.aspx
Introduction -- http://www.ascd.org/publications/books/101064/chapters/Introduction-to-the-2nd-Edition.aspx
plus samples from the
chapters, and
from ACSD.
and [click the links] evaluation & more)
(Illinois Math & Science
[with links to mission,...] and https://www.imsa.edu/extensionprograms/problem-based-learning
PBL Network [== includes ==]) —
[[get others]] [service learning] projects field trips -- Vanderbilt U has information & resource-links about https://cft.vanderbilt.edu/guides-sub-pages/place-based-and-project-based-learning/
https://cft.vanderbilt.edu/guides-sub-pages/teaching-through-community-engagement/
https://cft.vanderbilt.edu/guides-sub-pages/case-studies/
-- https://cft.vanderbilt.edu/guides-sub-pages/problem-solving/
Science Fairs:&
(from ERIC Digests) and
Science Buddies - http://www.sciencebuddies.org/ explore their website (@ models for Science & Engineering)
http://www.sciencebuddies.org/science-engineering-careers
Wikipedia https://en.wikipedia.org/wiki/Science_Buddies
with links to specific tools https://en.wikipedia.org/wiki/Science_Buddies#Specific_tools
teachers, students, and parents
Their main objective is helping students develop science projects or design projects for exhibitions (like science fairs) in K-12, but their models & especially the &Detailed Help& & can be generally useful in the classroom. -- And for each model they
offer &Detailed Help for Each Step& to supplement the basic model-frameworks.
-- we can help students improve their
give tips for
and summarize
research about .
You can read about && (like those typically found in textbooks and on exams) and
general problem-solving strategies that are also useful outside school.& For
problem solving in everyday life (including business,...) a series of pages by Robert Harris provides a thorough overview of
if you scroll down to the section about &Tools for the Age of Knowledge& and you'll find An Introduction to Creative Thinking, Creative Thinking Techniques, Criteria for Evaluating a Creative Solution, Introduction to Problem Solving, Human-Factor Phenomena in Problem Solving, Problem Solving Techniques, Introduction to Decision Making, and (in other parts of his links-page) much more.
A model for
includes 9 aspects of Science Process:
1. use Empirical Factors for Theory Evaluation,
2. use Conceptual Factors for Theory Evaluation,
3. use Cultural-Personal Factors for Theory Evaluation,
4. Evaluate Theories (with critical thinking), and
5. Generate Theories (with creative thinking);
6. Design Experiments (by generating-and-evaluating);
7. do Science Projects (planning and coordinating);
8. be influenced by Thought Styles (cultural & personal),
9. use creative-and-critical Productive Thinking.
These two representations
& verbal & verbal/visual, on the left & right sides &
relationships
within and between four sub-categories: 1, 89.
Here is an
Inquiry Activity:& In the diagram, do you see...
& & symbolisms in the colors?&& (yellow & green & yellow-green,& red & blue)
& & three kinds of meanings for the arrows?
contains responses for these two inquiry-questions about colors & arrows.&
A DISCLAIMER:& The internet
offers an abundance of resources, so our main challenge is selectivity,
and we have tried to find high-quality pages for you to read.& But
the pages above don't necessarily represent views of the American Scientific
Affiliation.& As always, we encourage you to use your critical thinking
skills to evaluate everything you read.
This website for Whole-Person Education has TWO KINDS OF LINKS:
an ITALICIZED LINK
keeps you inside a page,
moving you to another part of it, and
&a NON-ITALICIZED LINK opens another page.& Both keep everything inside this window,&
so your browser's BACK-button will always take you back to where you were.
The area of
has sub-areas for
PROBLEM SOLVING in Education and Life [it's this page]
This links-page for Thinking Skills & Problem-Solving Methods in Education and Life,
by Craig Rusbult, is http://www.asa3.org/ASA/education/think/methods.htm
copyright & 2001 by Craig Rusbult, all rights reserved
All links are now being checked and fixed, in late-February 2017.