Kyle Mathews :: Dreams With In

This was a comment I left on Jon Mott's blog post, Getting From Here to There. Jon is an Academic Technology Strategist here at BYU who shares my enthusiasm for using web2.0 tools in the classroom.

I think a critical tool to have in building and proselytizing learning 2.0 tools is a key metric. This metric would be used to guide building decisions and to measure success.

Researchers have shown that the distribution of many natural and social phenomenons follow what's called the power law. Power laws are known by other names such as the 20-80 rule (80% of wealth is controlled by 20% of the population), the long tail, Winner-Take-All, etc.

Here is an example power law graph from Wikipedia:

Many have written that almost all measures of web site usage follow the same pattern whether that be page views, in-bound links, etc.

This past semester I built an e-learning site for two sections of an upper-level university course at Brigham Young University. Now that the semester is finished, we've started to analyze the data we've collected on how students used the site.

And what did I notice when I started to graph the data? Why the power law of course.

So onto the graphs I made.

First is a graph of the number of comments each student made on the class website throughout the semester. This is a good measure of their engagement with the online community. You can see that some students left as many as 40-60 comments throughout the semester and others left as little as 1 or two:
Another graph which follows the power law. This time a graph of the number of visits to the class website per student. Again we see a wide disparity between students. One student visited the site over 400 times! Which in a four month semester means they visited the site on average 3 times a day! And on the other end, a number of students only visited the site 5-10 times the entire semester.

Why the Power Law?

So why does this happen? Why do some students participate on the class website so much more then other students? I'm not sure I know the answer to that. Why do some people write blogs, participate in Wikipedia, upload their photos to flickr, etc. and many others don't?

Do some students just not see the value of blogging? Are some students more comfortable interacting online for whatever reason? Or on another tact, does it matter if some students participate more then others? After all, in any community, there's always a core group which cares much more then others about the success of the community (read about the power law of participation). What type of participation in an online learning community meets the learning goals for a classroom?

There's a host of other ideas rumbling around in my head about power laws, the three types of networks (political, social, creative) and how all this relates to creating effective online learning/collaborative environments. But that will all have to wait for another day. In the meantime, here's some links.

Extra Credit Reading:

Read these two articles by Ross Mayfield, first this one, then this one.

Then two by Clay Shirky, "Power Laws, Weblogs, and Inequality" and "Social Software and the Politics of Groups."

And one more on building vibrant communities.

I've been putting some final touches on a website I built for a class running at BYU this semester on web analytics. You can visit the site here. I did a write-up about the site for Drupal's education working group. I discuss the design principles that guided my construction of the site.
A bit from the write-up:

From the Economist:

At some point in the decade after he moved from the farm in Nebraska where he grew up to the innovation hub that is the San Francisco Bay Area, Evan Williams accidentally stumbled upon three insights:

1. that genuinely new ideas are, well, accidentally stumbled upon rather than sought out
2. second, that new ideas are by definition hard to explain to others, because words can express only what is already known
3. and third, that good ideas seem obvious in retrospect.

For my research job at BYU, I'm reading quite a bit about social software, which is, as defined by Clay Shirky, software that supports group interactions. One article I read recently by Clay entitled, "Communities, Audiences, and Scale"is especially good and provided much of the inspiration for this post. The gist of the article is that audiences scale and communities don't. And understanding this principle I think is extra ordinarily important for designers of social software.

So what is the difference between audiences and communities? Audiences primarily consume content, communities primarily communicate with one another. TVs have audiences -- they have large numbers of people that watch their content. But there is very little communication between individual watchers of TV and between the watchers and the makers of TV content. On the other hand, a small group of men who gather to play cards on Friday nights is a community -- because they actively communicate with one another. Communities communicate with one another, audiences don't.

In addition, a group is a group and a community is a community because of the connections that form between its members. Two best friends have very strong connections. A high school basketball team has connections from running lines and sitting on the bus together for long road trips.

But connections get weaker and weaker as a group or community adds more members. Connections are created by communication. With two people, all communications happens between the two people. So the connections made are very strong. Add another person and the number of connections that need to be maintained increases from 1 to 3. Add another person making 4 in the group and the number of connections increases from 3 to 6. Add another member and the connections become 10 (read Clay for more of the math). Obviously as the group grows larger and larger, the bonds weaken between individual members of the group.

This idea is encapsulated in several common English expressions like "two's company, three's a crowd". Amongst young people, a person who tries to hang out with a couple feels like and is sometimes referred to as a "third wheel". It's possible to have 2-3 best friends but no more. The "gang" you hung out with in high school probably had 4-7 people but no more. Older people tend to have fewer friends then younger adults but with closer bonds. Clay Shirky references the research of primatologist Robin Dunbar who argues that, "humans are adapted for social group sizes of around 150 or less, a size that shows up in a number of traditional societies, as well as in present day groups such as the Hutterite religious communities."

This contradicts a fundamental assumption of most designers of social software that "more users is always a good thing." Different types of groups can maintain their identity to different sizes but at some scale all online communities start to lose the dense interconnections that make the community a community.

The line where an audience begins and a community ends can get rather murky with online social software. Consider for example the weblog. If three girlfriends use blogs to discuss their lives, this is obviously a community. On the other hand, a popular blogger such as Seth Goodin is not supporting a community on its blog but is operating a broadcast media platform much as CNN or Fox News.

I'm experiencing a good example of this murkiness. A Mailing list is an example of social software often used to support online communities. I am a student at BYU majoring in Information Systems. The ISys department at BYU provides a mailing list for the use of the 200+ students who are in the ISys major. The mailing list is quite popular -- I'd say it sees an average of 10-20 emails a day during the school year

But the funny thing I've observed in the year or so I've been on the list is that even though everyone has equal rights to email the list, I see the same 10-15 names over and over. The rest of the ISys students don't communicate via the mailing list but rather are audience members. So it seems the mailing list, as a community-building device, can't scale past a certain number of people.

So why does it matter if our social software isn't so social at times, or that not everyone can/will participate on a mass mailing list? We should care because we are social creatures. In our increasingly rushed and splintered world, technology can help fulfill our need for friends and community.

Joel Spolsky, in a post entitled "Building Communities with Software" speaks movingly of our need as humans for community and how social software can fulfill that need. He ends his post with this message, "Creating community, in any case, is a noble goal, because it's sorely missing for so many of us. Let's keep plugging away at it."

Social software done right can create a community for all its participants. I feel strongly about the need for excellent social software, in our schools, workplaces, and other organizations. Social software can help us learn, work, and live with greater effectiveness and joy.

In my next few posts I'll continue to explore the fundamental problems and opportunities with social software.

I recently finished up a semester at BYU. The end of another semester brings with it that joyful tradition, finals. Going through six finals gave me cause to think quite a bit about tests; what are they for and why are they useful?

I came up with three ways to measure the quality of a test:

1. Does it measure something valuable?
2. Is the test both reliable and valid?
3. Does it provide the correct incentives?

Does it measure something valuable?
This depends on the context of the test -- a juggling test is valuable for a circus school, a gum chewing test is valuable to aspiring gum gurus, etc. The important questions is, does the test measure one's understanding of or skill in the valuable part(s) of a subject matter.

Is the test both reliable and valid?
Reliable means the test is consistently administered and graded for all test takers. Valid means the test measures what it intends to. A juggling test which asks a student to hold two balls but not throw them would be a reliable test -- all test takers could be measured easily the same -- but wouldn't be valid as being able to hold two balls doesn't show that one can in fact juggle balls. A professor can prepare a test that measures all students fairly but wouldn't test the students on anything valuable (as defined above).

Does it provide the correct incentives?
I haven't worked this measure out satisfactorily in my mind yet but it's something like the following. People's behavior is shaped by incentives. Because tests are a major part of a class grade and most students care about grades, tests motivate students to study more then they would without the incentive. So far this is fine and dandy. I for one appreciate tests for this very reason as I know I've studied more, and learned more, over the years because of tests.

So why can this go wrong?
Students go to school to learn, presumably. There is of course the old yarn that education is the only thing that Americans pay for and hope to be cheated. Many students it seems care more about the grade then for learning. Because learning is hard, there is a strong incentive for students to try to 'game' the test. Some types of tests can be passed by cramming the night before the tests. Tests that only require you to memorize material don't promote true learning. So this poorer sort of test would provide the wrong incentive to students in that it pushes them toward the poorer sort of learning. A good test would push students toward the richer, and harder, sort of learning.

Now what types of tests provide the correct or wrong incentives I'm not quite sure. As I said, I haven't completely worked this measure out. I'm not even sure if it is a correct measure. If at some point I can work up more interest in this topic to clarify my thinking, I'll write another post. Or someone else can save me the work and write it themselves (or link to a page in the comments section).