Tech Critique – Beyond Bits & Atoms

Critique an educational technology through the theoretical and design lenses we have examined so far. The critique involves an analysis of the design decisions of a learning technology — both the explicit and implicit intentions of the designer. The critique might directly inform your final project, and it might be used for the related work section in the paper.

First, we suggest you to have a more general view of the technology by looking at its context and intended users. Then, look at the different design aspects and evaluate them according to the dimensions we provided. Only after those two tasks, try to tie everything together to evaluate what kind of theoretical and design models the creators might have worked with. Note that some of these elements could be implicit or explicit, intentional or by omission. Then we will ask you to propose a redesign of one aspect the technology–only this last part needs to be written up. All the rest should just be in your presentation and slides.

Note also that we don’t want you talking about every single aspect of the product — pick some aspects that seem more relevant or interesting to you, and focus your critique and redesign on those.

Deliverables

Description and context

In your slides, pick 3-4 of those questions and answer in bullet-point form. Pick the ones that are most relevant to your product and critique.

What are the learning goals / desired activities? Are they explicitly stated or do you have to infer them?
How specific / broad is the target population? What assumptions are made about the target population?
Is the tool tmeant to be used individually or collaboratively? If collaboratively, is the group meant to be diverse (e.g. family, teacher-student, etc.) or more focused (e.g. students of same age)?
In which contexts (physical and social) is the tool meant to be used?
Is the tool meant as a developmental tool, i.e. used repetitively over longer time spans? Is there a growth possibility within the tool, or does a user have to move on to another tool if s/he becomes proficient with it?
What does the technology allow you to do with respect to the target domain (e.g. if the technology is about programming, then what are the affordances for programming? If the technology is about math, then what are the affordances for “doing” math?)
Are there affordances to customize designs, so that the characters, etc., can be personalized? Are there game mechanics involved to maintain motivation?
What kind of instructional scaffolds and supports are provided with the technology?
- Built-in scaffolds within the technology (e.g. feedback systems, structuring of activities, enabling features only with progression, etc.)
- External scaffolds and support provided with the technology (e.g. a forum to ask questions, automated help, web materials, etc.)
Does it come with a curriculum? Instructions?

Critique

Next, write a critique of 500-1000 words which addresses the following three sections. In each section, not all questions might be relevant to your technology and thus not all questions need to be addressed. Address the questions that are most relevant to your product.

1. Design of the Technology (presentation/slides only)

Here are the relevant dimensions along which you should evaluate the tool. They can of course be intermeshed; for example, some affordances might evoke cultural forms and result from the selective exposure for usability (e.g. a puzzle design). In general, they should be evaluated with the specified learning goals in mind. (Note that some dimensions might also not apply to your particular product. You don’t have to answer all the bullets — pick the ones that are relevant to your product)

Selective exposure (blackboxing) – to be evaluated with respect to learning goals.
What is foregrounded, what is backgrounded in hardware and software design? How do designers hide the complexity of the underlying technology to make learning goals easier to achieve? Do the “black boxes” (backgrounded elements) get unveiled as you progress as a learner? At what cost does this “blackboxing” come?
Selective exposure for usability (“embedded error correction”). How does the material and design communicate rules for its use? Does is correct user errors by itself?
Selective Exposure for power ( e.g., “tangibility mapping” for physical kits). How are cognitive and physical/UI actions/operations mapped to each other? Does the design make them more explicit? Is the interface/toolkit self-explanatory? Does it help the user figure out what the main valid operations are? What cognitive processes are offloaded onto the technology? How relevant are they to the core learning goals or desired patterns of activity? Does it free up time for more relevant processes?
Symbolic systems and representations
How is knowledge represented, i.e. what kind of symbolic systems do they use? E.g. in music, do they use the normal music annotation, or do they have other representations?
Are there multiple representations “of the same thing”?
Are representations mapped explicitly onto each other, or does the user have to make that step?
Cultural forms: To what extent do the designs tap into existing cognitive, physical and emotional resources and social patterns of activity?
Does the tool demand new sets practices, or does it fit into existing practices of an environment using it?
Do the designs of objects and /or patterns of activities and situations evoke cultural forms?
Fidelity of designs of objects and activities to source cultural forms?
Low floor – high ceiling – wide walls
Low floor: What are the “entry level” requirements to use that tool? What is the relation between the tool and support material? Can you just jump into it and get started?
High ceiling: Does the tool support use by novices and experts alike?
Wide walls: Does the tool support different forms of exploration and expression?
Salience of powerful ideas
How are the core ideas (related to the learning goals) embedded in the system? Does the learner have to reason with them in order to engage with the tool? Are they implicit in the interaction with the tool? Are they taught directly?

2. Synthesis (presentation/slides only)

The synthesis builds on the analysis of design aspects to extract ideas the designers might have had about the learner and learning processes. We want you to evaluate what models of learner, learning, knowing and teaching that the designer implicitly or explicitly worked off of. Use the previous analysis and the following questions as guides through this part (you don’t have to answer them all — pick the ones that are relevant to your product).

Is there an educational philosophy embedded in the product, implicitly or explicitly? (Refer to the literature). How mindful of learning theory is it?
Evaluate how well the underlying theories, design decisions and learning goals align: does the product fulfill its promise?
How are relevant cognitive processes distributed across the learner and the technology? E.g. does the technology make the “doing” of things so easy that the learner mainly has to engage cognitively at the conceptual level; or does a large effort go into the “doing” of things?
Does the technology supplant or augment relevant (wrt learning goals) cognitive processes?
What support does the technology provide for reflection about what is being learned?
What kind of agency does a learner have in engaging with the tool? What meaningful decisions can the learner make?
At what level is the technology “personalized” and ties into personal experiences?
How culturally flexible is the technology?
How does the platform address issues of cost and diversity of the audience?
Epistemological Pluralism: Does it support many different paths and ways of knowing and engaging? If so, at what level?

3. Conclusion: Design proposal (presentation/slides and handed in written document)

Then we ask you to pick 1-2 aspects of the product that you think could use a redesign, and make a proposal, addressing issues that might have come up in the analysis. For example, if you analyzed a programming environment for children than only allows for very simple programs (low ceiling), you could think of a way to make the “ceiling” higher. If you examined a physical toolkit for electrical circuits that exposes a lot of complexity to students, you could think about backgrounding some of that complexity. If you looked at a robotics kit that is too expensive, you could think of ways to make certain parts cheaper. The redesign should be something creative and solid—for example, “creating a webpage with instructions” is not a good redesign in the context of this assignment. You do not have to implement any of the changes, or even know how to do it technically. While the other parts are just for the presentation, this should be handed in as a ~500-700-word explanation and a series of sketches/wire-frames, drawn by hand or on the computer.

Assessment

	✓ –	✓	✓ +
Design analysis	BUT: the dimensions of analysis is not made explicit, the claims are vague, or the claims are often not supported by evidence.	The analysis is clear about which dimensions of the design space are being considered, even if theory is not cited directly. The analysis makes clear claims about the artifact, supported by evidence.	AND: The dimensions of analysis are well-chosen (perhaps based on context) to allow for an insightful critique. Specific non-obvious points are made, consistently justified with evidence.
Synthesis	BUT: the synthesis is not grounded in theory, or is ineffective in connecting the larger theoretical questions to the analysis of this technology.	The synthesis is explicitly grounded in a theoretical stance. Proposes questions based on this stance, and answers them using evidence from the analysis.	AND: The synthesis engages productively with the larger theoretical questions, illustrating or challenging general principles.
Design proposal	BUT: the proposal is vague or not clearly connected to the synthesis.	Proposes a specific design change based on the synthesis.	AND: the design proposal is generative, either exploring a theoretical possibility or informing the designer’s practice.