A Black Box Approach

Steve Lowe, 2010

Abstract

A black box approach to eLearning course design and review. This postgraduate research is in the domain of intelligent agents and virtual environments, taking a functional approach. Four texts have informed the study thus far: Richard Bartle, Designing Virtual Worlds; Stuart Russell and Peter Norvig, Artificial Intelligence, A Modern Approach; Valentino Braitenberg, Vehicles - Experiments in Synthetic Psychology; and David McFarland, Guilty Robots, Happy Dogs. Applying this kind of system thinking to eLearning might seem dehumanising... but when conducting blame-free autopsies in the context of Good to Great, the impersonal view can be empowering. People who might otherwise have to accept some personal responsibility for failure can instead look impassionately at their function. In design, black boxes with labels like "Magic Course Coordinator" and "Teach-by-Wire" help to predict outcomes. In review, black boxes with labels like "Zombified QA" help to understand what went wrong. The debate might continue amongst those who prefer to leave the person out of it, and those for whom e-learning will always be about people, people, people.

Foundations

At first I took a systems approach. I looked at courses as if they were some kind of engine with inputs, outputs, controls, feedforward and feedback. Making up the primary inputs were the raw materials: framework, descriptors, content, people, time and dollars. There were two outputs. At one output were the products: graduates, partial successes, drop-outs, and exhausted tutors; also there was waste including time, dollars, unread study notes, and unfulfilled promises. At the other output, at least in the CE's dreams, were large amounts of dollars spewing into the bank. The controls were inputs from TEC, Academic Board, and maybe others. Feedforward was supplied by the bums-on-seats model in the form of slack selection procedures, oversell from marketing, discounted fees, and canvassing by tutors. Feedback was supplied from success and retention figures, documented withdrawals, non-attendance, and student satisfaction reports. The feedback generally amounted to very little of any real use because, while success and retention figures tell you how much, they don't tell you why, or how to fix it. Non-attendance is the same, it tells you that your course is not the most important thing in the students' lives, but it doesn't tell you what could make it so. Student satisfaction reports are of limited value because they often don't ask the right questions, and students have a tendency to acquiesce, fudge and lie. There was a secondary feedback pipe that dribbled a very small amount of dollars back into the gaping maw of the intake pipe. The whole thing dripped oil, and looked as if it was about to explode.

Figure 1: The erroneous 'engine' model.

Then one day I was on a bus reading Russell & Norvig to pass the time. Now I realised that my engine model was wrong. An engine is a tight organisation of components each of which performs one specific function to contribute to a common goal. Consider a car engine: it has a carburettor that controls the mixture of petrol and air, a coil that makes an electrical impulse, a distributor that sends that impulse to the right sparking plug and so on. All those parts are bolted together, and the timing of their functions is orchestrated by the camshaft, everything working towards one goal.... to transport people and their luggage along the road. But courses are designed and run by a disparate collection of components, not originally designed for the work in hand, which come and go more or less as they please, and are loosely orchestrated by processes which are not clear to them. They form a dysfunctional robot crew that perform many and varied functions only some of which are directed towards the goal... creating graduates. Better, I thought, to look at the problem through the lens of IA: autonomous intelligent agents, sensors, percepts, actuators, actions, agent programs, agent functions, partially observable environments, stochastic reasoning, and goal conflict resolution.

Figure 2: The 'constant gardener' model.

According to Russell & Norvig (2003, p.32), "An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators." I thought: No agent is omniscient, because if it is we call it the "god object" or the "observer"; so far as agents do think, some agents may think they are omniscient, because all they know is the world that they can perceive through their sensors; agents may gather information by exploring, and stacking percepts in sequence as they come in. Later they pop their stack, and rational agents choose appropriate actions, and irrational agents choose appropriate or inappropriate actions by chance. Further, Russell & Norvig categorise agents and their environments as follows: simple reflex agents, model-based reflex agents, goal-based agents, utility-based agents and learning agents; environments, they say, are fully or partially observable, deterministic or stochastic, episodic or sequential, static or dynamic, discrete or continuous, and single agent or multiagent. As domain specific language is the first step on the staircase to understanding, I have been happy to adopt their language and to incorporate it into the terminology I use when attempting to describe these ideas. This study considers goal-based agents operating in a partially observable, stochastic, sequential, dynamic, continuous and multiagent environment. I dubbed them 'constant gardeners' after the Fernando Meirelles film of 2005.

Braitenberg (1984, p.14) invites us to, "consider the enormous wealth of different properties that we may give Vehicle 3c by choosing various sensors and various combinations of crossed and uncrossed, excitatory and inhibitory, connections." Even the simplest of robots, with no central processor or memory, Braitenberg demonstrates, can exhibit apparently intelligent behaviour and attracts the attribution of 'values', 'like', 'love', and even 'knowledge'. In other words, Braitenberg shows us before we have even completed chapter three that the simplest of systems can demonstrate seemingly complex behaviours. Or to put it another way, when observing a given behaviour we tend to assume a far more complex system than is necessarily the case. Further, most people tend to anthropomorhise... to attribute human characteristics to machine behaviours.

McFarland [2008, p.48] warns us, "If my dog wanders into my kitchen while I am cooking, I may interpret its behaviour in a number of ways. If I happen to know that my dog has not been fed that day, I may assume that she is hungry, can smell food, and so is performing behaviour appropriate for a hungry dog. If I know that my dog has been fed, I may assume that the dog wants my company. In reality I know nothing (for sure) about the dog's inner workings, but I am, nevertheless, interpreting the dog's behaviour." Prior to reading McFarland I had noticed that I was, like most people, too quick to attribute the behaviour of colleagues to perceived causes based on my own set of values. After reading McFarland I started observing my dog Poppy more closely, and thinking about behaviour and function in more detail.

Bartle [2004, p.131], the designer of MUD1, had a (then, circa 1990) unique opportunity to observe a very large sample of users. Out of this research came his Player Interest Graph, reproduced below:

Figure 3: The player interest graph (Bartle).

This model of player behaviours has proved very useful in my own observations of student and tutor behaviour in online learning environments. In recent months I have translated this back into the real world, looking at the behaviours of the key players in the wider real life system of online course development. I started by taking a purely behaviourist stance, but I shifted to more of a functionalist stance. Later, I narrowed my focus further, and started trying to open up the black boxes to inspect the programs running within. It isn't usually easy to gain access, and my techniques have included course review processes, participant observation, and face to face conversation, telephone conversations, email and long periods of reflection.

Jim Collins (2010, Internet) has to a great extent defined the contexts for this study. The institution recently engaged with the Good to Great programme, and now we are busy facing "The Brutal Facts", and conducting "Autopsies Without Blame", and "Pulling the Plug". It is my observation that people in the institution, many of whom have worked for two or more decades in public sector tertiary education, are having real difficulty coming to grips with these ideas. These people continue to sweep the facts under the carpet, point the finger, and sponsor unviable projects against all reason.

A black box approach may help when: facing the brutal facts, because they are explained in abstracted terms; conducting autopsies without blame, because it removes the person from the model; pulling the plug, because we are stopping the machine, not sacking the people.

Constructions

I took a sheet of A4 paper, in portrait, and called it "The Institution". The Institution couldn't exist in a void, so I took a sheet of A3 paper, also in portrait, and called it "Civilisation". On the Civilisation sheet at the top I wrote "Taxation", "Funding" and "Guidelines", and at the bottom, in the position normally reserved for Hades, I wrote "The Sea of Entrepreneurs".

Then I placed the A4 sheet reprepresenting the Institution World in the middle of the sheet of A3 representing the Civilisation World. This became my task environment (Russell & Norvig, 2003, p.38) and it was, I reflected, with all its external and internal politics, all its ministries and departments, all its walls and doors, all its employers and consultants, employees and contractors, most certainly partially observable, stochastic, sequential, dynamic, continuous and multiagent (Russell & Norvig, 2003, p.43). Any one agent operating in this world would only be able to perceive a tiny part of it, and any changes they made to it with their actuators would probably be small, if not totally insignificant.

Figure 4: The 'task environment'.

Then I built some black boxes out of matchboxes, and they were like objects-to-think-with.

Figure 5: The 'specialisations'.

There were specialisations of the constant gardeners. These specialisations had different sensors and actuators, but they were still a black box with inputs and outputs and running an internal script. I wrote scripts — the programs I thought the specialised agents might be running — on little scraps of paper which I folded up and put inside the black boxes. Three examples were:

• Resource Writer {read, paraphrase, write}

• Funnel Head Student {acquiesce}

• CEO {protect position, increase salary, look for better positions}

The CEO script was adapted from Tilden's parody of Asimov's Laws of Robotics, "Protect thy ass; Feed thy ass; Move thy ass to better real estate" (Hrynkiw & Tilden, 2002, p.11).

Then I left them for two weeks, until I had nearly forgotten about them, and only then did I got them out again and read them. This led to insights, which in turn made me want to go and see what was happening in the real world that I was modeling.

Applications

Two main applications were considered: Course review, and course design.

Course review

This study came about because of a course review. I was trying to make sense of a number of interconnected behaviours that had led one particular project down a strange and dangerous path. I knew many of the people involved personally, and I was trying to remove the subjective, and focus on the objective. These people were not working as one cohesive project team, but were working separately on what each perceived to be their job. In my mind I replaced them with black boxes, then I found I could think more freely about them. Messages passed between these autonomous agents like chinese whispers. Each operated in a very small and isolated part of the task environment, in one case with considerable physical separation from the others. The review findings included: lack of continuing business justification, lack of any definable project management, lack of visibility, lack of accountability, and lack of any realistic estimate of the likely outcomes.

Course design

The question now arises, can this model be useful as an instrument to inform future course design? Is the real goal to attain the cohesive 'engine' model with its inputs, outputs, feedforward, feedbacks and quality assurance controls? Should we skip trying to make the dysfunctional robot crew functional, and instead bolt them firmly to some rigid framework? They could run strictly controlled scripts with outputs that contribute only to predefined outcomes. Or should we accept the reality of our situation, and work to improve messaging between autonomous agents, to refine scripts, and to embed a common vision of the final output and outcomes? In deciding whether to stick with the autonomous agents we should consider the benefits of hill-climbing. If we build the engine, it will only ever obtain its optimum designed outputs, and it is very unlikely that it will ever exceed them. But, if we adopt the autonomous agent model, loosely guided by a common vision, and we accept and even encourage some random behaviour, these peturbations may cause us to eventually reach the greater summits. This Darwinian approach seems too good to pass up.

Experiments

I have been engaging in a series of thought experiments. Sometimes I move my matchbox toys around on the paper task environment, sometimes I just perform these actions in my imagination. Much like the descriptive names given to design patterns I think up catchy titles that are in themselves objects to promote thought. Examples would be Teach-by-wire, Zombified QA, and Magic Coco.

Teach-by-wire

In Teach-by-wire EduShark cruises in the Sea of Entpreneurs, keen to snap up any unsuspecting SM (senior manager) thrashing around outside the flags. Once done, EduShark rigs a wire that passes straight through the Institution connecting to a large cohort of the Institution's students without either Ed or the students ever coming on campus. Frequent direct transfers to EduShark's bank complete the picture. Teach-by-wire can be heavily defended by those involved, because it is seen as a maintenance-free income stream. If the wire is suddenly cut by some champion of True Quality or Capability activist the shock waves can bounce off the walls of the Institution for months, even years.

Zombified QA

In Zombified QA a Magic Coco shows the QA Agent only what it wants the QA Agent to see. At intervals the Magic Coco holds up a mask with a tiny pinhole in it for the Zombified QA to peep through. Meanwhile the Magic Coco's other actuator offers a bucket of platitudes. After several effective sessions of zombification the QA Agent finds its actuators manacled.

Figure 6: The 'zombified QA' scenario.

Magic Coco

In the Magic Coco pattern one sees an example of the true "Black Box". There are sensors that attend meetings, listen hard at conferences, and sniff the breeze. There are actuators that magically seem to get things done. What is not visible is how things get done. The script is entirely hidden from view, is seldom revealed, and can be near impossible to analyse without resort to CIA tactics like bugging the phone and reading emails.

Reflection

Sometimes when I reflect on this study I think I need to get more sleep. Other times, I feel I may be onto something really useful. One possible design goal could be to gain reliable operation by designing people right out of the system, creating a tutor-less self-coordinating online course. But it isn't as easy as that. If all the people were replaced by Intelligent Agents, and after programming the lids on all the little black boxes were screwed down tight, still there would be emergent behaviours. Russell & Norvig (2003, p.453), McFarland (2008, p.190) and Braitenberg (1984, p.68) are all quite clear on that point. Whether those emergent behaviours moved us nearer to or further from the design goals would be a matter of chance.

Terms

Anthropomorphise To endow with human qualities; To attribute human characteristics to something that is non-human.

Percept The input that an intelligent agent is perceiving at any given moment; In philosophy, psychology, and cognitive science, perception is the process of attaining awareness or understanding of sensory information.

Stochastic Being or having a random variable; A stochastic process is one whose behavior is non-deterministic, in that a system's subsequent state is determined both by the process's predictable actions and by a random element; model that contains a random component.

References

Bartle, R. (2004) Designing Virtual Worlds. Berkeley: New Riders.

Braitenberg, V. (1984) Vehicles, experiments in synthetic psychology. Cambridge: The MIT Press.

Collins, J. (2010) Homepage (Internet) URL: http://www.jimcollins.com/ Last accessed: 31st August 2010.

Hrynkiw, D. & Tilden, M. (2002) Junkbots, Bugbots & Bots on Wheels. New York: McGraw Hill / Osborne.

McFarland, D. (2008) Guilty Robots, Happy Dogs. Oxford: Oxford University Press.

Russell, S. & Norvig, P. (2003) Artificial Intelligence, A Modern Approach, 2nd Edition. Upper Saddle River: Pearson Education.

 

Version 20100925