Kyle Mathews :: Dreams With In

This was a comment I left on Jon Mott's blog post, Getting From Here to There. Jon is an Academic Technology Strategist here at BYU who shares my enthusiasm for using web2.0 tools in the classroom.

I think a critical tool to have in building and proselytizing learning 2.0 tools is a key metric. This metric would be used to guide building decisions and to measure success.

Researchers have shown that the distribution of many natural and social phenomenons follow what's called the power law. Power laws are known by other names such as the 20-80 rule (80% of wealth is controlled by 20% of the population), the long tail, Winner-Take-All, etc.

Here is an example power law graph from Wikipedia:

Many have written that almost all measures of web site usage follow the same pattern whether that be page views, in-bound links, etc.

This past semester I built an e-learning site for two sections of an upper-level university course at Brigham Young University. Now that the semester is finished, we've started to analyze the data we've collected on how students used the site.

And what did I notice when I started to graph the data? Why the power law of course.

So onto the graphs I made.

First is a graph of the number of comments each student made on the class website throughout the semester. This is a good measure of their engagement with the online community. You can see that some students left as many as 40-60 comments throughout the semester and others left as little as 1 or two:
Another graph which follows the power law. This time a graph of the number of visits to the class website per student. Again we see a wide disparity between students. One student visited the site over 400 times! Which in a four month semester means they visited the site on average 3 times a day! And on the other end, a number of students only visited the site 5-10 times the entire semester.

Why the Power Law?

So why does this happen? Why do some students participate on the class website so much more then other students? I'm not sure I know the answer to that. Why do some people write blogs, participate in Wikipedia, upload their photos to flickr, etc. and many others don't?

Do some students just not see the value of blogging? Are some students more comfortable interacting online for whatever reason? Or on another tact, does it matter if some students participate more then others? After all, in any community, there's always a core group which cares much more then others about the success of the community (read about the power law of participation). What type of participation in an online learning community meets the learning goals for a classroom?

There's a host of other ideas rumbling around in my head about power laws, the three types of networks (political, social, creative) and how all this relates to creating effective online learning/collaborative environments. But that will all have to wait for another day. In the meantime, here's some links.

Extra Credit Reading:

Read these two articles by Ross Mayfield, first this one, then this one.

Then two by Clay Shirky, "Power Laws, Weblogs, and Inequality" and "Social Software and the Politics of Groups."

And one more on building vibrant communities.

I've been putting some final touches on a website I built for a class running at BYU this semester on web analytics. You can visit the site here. I did a write-up about the site for Drupal's education working group. I discuss the design principles that guided my construction of the site.
A bit from the write-up:

From Jack Ricchiuto:

1. Get to know the strengths and passions of people in your first and second circles.
2. Make your strengths and passions more known to your 1st two circles.
3. Discover who in your 1st two circles would benefit from introductions.
4. Make high quality introductions at the best levels possible.
5. Engage your 2nd circle to introduce you to people in your 3rd circle.
6. Look for new opportunities to learn with and from your first two circles.
7. Create new opportunities to collaborate with your 1st two circles.
8. Increase your asset, positional, and generative value in your networks.
9. Help other people increase their value in their networks.
10. Built trust and help others build trust through promise making and keeping.

From the Economist:

At some point in the decade after he moved from the farm in Nebraska where he grew up to the innovation hub that is the San Francisco Bay Area, Evan Williams accidentally stumbled upon three insights:

1. that genuinely new ideas are, well, accidentally stumbled upon rather than sought out
2. second, that new ideas are by definition hard to explain to others, because words can express only what is already known
3. and third, that good ideas seem obvious in retrospect.

A year ago in just my second blog post, I explained why I write a blog. My answer then (and now) is I keep a blog as an intellectual journal of sorts. Blogging is my way of pulling together into a coherent form all the stray thoughts rolling around in my mind. Writing helps me sift the good thoughts from all the bad and fit them all together in a logical pattern.

I was reading a blog post earlier by Ross Mayfield. He was asked what were his best decision and worst mistake as an entrepreneur.

All his answers were good but I was especially struck by what he calls his biggest mistake:

Biggest Mistake -- Not taking bigger risks earlier

Maybe because in hindsight all risks are clear, but I always find myself regretting not taking bigger risks earlier. For example, open sourcing the Socialtext code was something we waited on until the company had strong footing. Partially because we thought there would be cannibalization, partially because we were understaffed to really engage with the community. But I believe if we bought this bullet earlier in the history of the company we would be reaping better rewards. As a planning exercise, now I always try to ask two questions: "How could we take more risk?" and "What risk can we take that creates the greatest amount of options?" I find there is always a way to do a little more, in particular by getting past instinct to control prevalent in so many entrepreneurs.

I was so struck because I'm in the middle of taking a big risk.

I am passionate about web development. I want to build web applications that enable people to connect, collaborate, and get more done in their lives. But at my current and soon to be past job, I'm not learning how to build these beautiful web applications.

So I decided to quit and employee myself. I decided that as no one is going to pay me to learn how to do web development, I'll just have to pay myself.

And now my mom, older sister, and I are going into business together. In a few months, we'll unveil Los-Detalles.com, the newest and greatest baby card announcement e-retailer. And in the months leading up the our launch, I will be learning web development.

Ross Mayfield learned it was a mistake to not take risks but that taking (calculated) risks opens up options where caution and control can never go. I'm trying to learn this same lesson. By taking this risk, the options I want to have will open to me.

I recently finished up a semester at BYU. The end of another semester brings with it that joyful tradition, finals. Going through six finals gave me cause to think quite a bit about tests; what are they for and why are they useful?

I came up with three ways to measure the quality of a test:

1. Does it measure something valuable?
2. Is the test both reliable and valid?
3. Does it provide the correct incentives?

Does it measure something valuable?
This depends on the context of the test -- a juggling test is valuable for a circus school, a gum chewing test is valuable to aspiring gum gurus, etc. The important questions is, does the test measure one's understanding of or skill in the valuable part(s) of a subject matter.

Is the test both reliable and valid?
Reliable means the test is consistently administered and graded for all test takers. Valid means the test measures what it intends to. A juggling test which asks a student to hold two balls but not throw them would be a reliable test -- all test takers could be measured easily the same -- but wouldn't be valid as being able to hold two balls doesn't show that one can in fact juggle balls. A professor can prepare a test that measures all students fairly but wouldn't test the students on anything valuable (as defined above).

Does it provide the correct incentives?
I haven't worked this measure out satisfactorily in my mind yet but it's something like the following. People's behavior is shaped by incentives. Because tests are a major part of a class grade and most students care about grades, tests motivate students to study more then they would without the incentive. So far this is fine and dandy. I for one appreciate tests for this very reason as I know I've studied more, and learned more, over the years because of tests.

So why can this go wrong?
Students go to school to learn, presumably. There is of course the old yarn that education is the only thing that Americans pay for and hope to be cheated. Many students it seems care more about the grade then for learning. Because learning is hard, there is a strong incentive for students to try to 'game' the test. Some types of tests can be passed by cramming the night before the tests. Tests that only require you to memorize material don't promote true learning. So this poorer sort of test would provide the wrong incentive to students in that it pushes them toward the poorer sort of learning. A good test would push students toward the richer, and harder, sort of learning.

Now what types of tests provide the correct or wrong incentives I'm not quite sure. As I said, I haven't completely worked this measure out. I'm not even sure if it is a correct measure. If at some point I can work up more interest in this topic to clarify my thinking, I'll write another post. Or someone else can save me the work and write it themselves (or link to a page in the comments section).

I've just made a new writing goal -- I will write 200 words a day -- 100 words in my journal and 100 words for my blog. This isn't much really, just 4 paragraphs or so. Some days if I want to write more, I can or course, but I'm never to write less.

(Later) I've changed my mind after thinking it over. I need a goal that is completely realistic. The idea for setting a word target for my writing came from a blog post on productivity over at Crooked Timber. The writer said that the main productivity killer for academics are distractions that pull them from writing. The key to writing productivity is to write at least something every day. Something is better then nothing and little things add up to big things. So even though 200 words for my journal/blog isn't much, I can foresee days where I won't reach my target. So I'm dropping the number a bit to 50 words for both categories - again with the caveat that I can go over -- just as long as I reach 50 words every day. And speaking of which, I've already written over 200 words -- and I think I'll post this as well so I've exceeded both my journal and blog writing goals. Wow, my goals are working already!

I'd been thinking about blogging for a long time but what finally pushed me over the edge was this post. The author, Scott McLemee, wrote about what he's learned from a pamphlet by C. Wright Mill "On Intellectual Craftsmanship". Here's the key quote,

What Mills calls “intellectual craftsmanship” involves more than the ability to produce work that can pass peer review. “Scholarship is a choice of how to live,” he writes, “as well as a choice of career.” It is (if I may be excused for borrowing another old Greek word) an ethos. That is, a structure of habits that sustains and embodies a quality of mind, a tendency of character.

“Whether he knows it or not,” Mills goes on to say, “the intellectual workman forms his own self as he works towards the perfection of his craft.” The notion of having a “career” is subordinate to — even a side-effect of — this process of inner shaping. “To realize his own potentialities, and any opportunities that come his way,” the scholar “constructs a character which has its core the qualities of the good workman.”

For Mills, there is a kind of bench where all of this crafting takes place. He calls it “the file.” I’m not sure this is the happiest of expressions. It’s simple enough, but Mills uses it in his own sense."

Mills suggests that aspiring young intellectuals keep what he calls, "the file," or the term I like, "intellectual journal." Here are collected reading notes, stray ideas, and complications or successes in research. It serves as a journal as you analyze and sort through ideas. Mill writes that the process of collecting and sifting through your notes and ideas leads to more systematic thinking as well as more directed and thorough learning.

Starting my junior and senior years of high school, I came to love writing. Not that it is entertaining or easy – it's about the hardest thing to do intellectually – but for the powerful and beneficial effect it has on my thinking. I find that when I write about an idea I've had or a book I've read, the writing serves to clarify my thinking and make explicit my reasoning.

My experience with the effects of writing fits with Mill's main assertion, that the keeping of the file or intellectual journal is critical to the “[construction of] a character which has its core the qualities of the good workman.” My writing refined and shaped my intellectual charecter.

A critical realization I've come to over the past several years is the importance of my habits. My life is directed more by what I choose to do every day then any so-called life changing decisions. I like to think of habits as bricks. A single brick is really quite insignificant but from thousands of bricks mortared together come sturdy houses and soaring cathedrals. And from habits we build, one brick at a time, our lives. My life, whether it will be useful and great or broken and poor, depends a great deal on my habits.

The great disconnect in most people's lives is between what they know they should do and what they actually do. Witness the vast number of people enslaved by addictions: alcohol, drugs, gambling, food, and so forth. What they lack is internal discipline or the character to live what they know is right. For whatever reason they have never built the character that allows their mind to rule their body. Instead, their bodies enslave their minds.

So how does this bring us back to Mill's file? For many years, there has been a disconnect between what I want to be intellectually, and what I actually do, my habits. I feel far too often I don't think as I should. I don't analysis, probe, collect, sift, sort, synthesize. In other words, as much as I want to, I don't have the character of an intellectual workman.

My hope is this blog will provide the motivation and discipline to write so as to help me develop the habits of the mind of an intellectual craftsman.

Update: Other people seem to be thinking along the same lines: here and here. Also, here is a link to a PDF of Mill's orginal essay, "On Intellectual Craftsmanship".