We Still Need Better Filters

Chris,

As I said in my post yesterday (http://www.winextra.com/2008/03/17/has-friendfe...) I think FriendFeed is used properly - especially now it has search - it could come closer to fitting your filter needs than you might be giving it credit for.

Chris, you can imagine our dilemma in the news media industry. There are all these incredible tools to share/push content, and that's a wonderful thing. There is probably more consumption of media than ever. But how to cut through the noise? How do we reach interested audiences/customers in a meaningful way?

Chris, I don't own a car but today's situation with FriendFeed, Twitter, Google Reader, etc, is like owning 10 cars to get to many places. Seems we move in tiny steps (RSS for example) but the killer app will show up someday. It'll need much better intelligence than anything I've seen today. Yahoo Pipes isn't bad, maybe things can move on from that idea.

/r

It's funny ... as an efficiency consultant, on the surface it seems that I should be particularly concerned with the creation of a better filtering system. But I'm not.

Osmosis, for lack of a better term, really works for me. On Twitter I follow more than 1,000 people but I never seem to miss the important things. I subscribe to over 1,000 RSS feeds and definitely don't read all of them, but sometimes there's a gem that pops up if I click "All Items" and read whatever the first 10 things are that pop up.

I think we need to let go of the fear that we're always "missing" things. Someone may write a fantastic motivational post, yes. But there are other fantastic motivational posts. And forget about your idea of a fantastic post versus my idea -- your own idea of a fantastic post may change 10 times over the course of a day.

I don't want a filter. The book "Impro" by Keith Johnstone had a hugely significant impact on my life, even though it's an instructional manual for improvisational actors (which I have never done and have no intention of doing) ... an obese woman was blocking my way in an aisle at Borders and I happened to pick up the book for lack of a better thing to do and I happened to turn to an extremely relevant page. What filter is going to give me *that*?

Dude, i totally agree! I think FB Beacon while not totally thought out was on its way there. I totally use my friends as a filter for what's good and what's not. With things like FB Beacon, SocialThing, or Friend Feed, I can see their behaviors, likes, and dislikes.

Amen. Aggregation is pretty much nailed. Filtering is next on the list.

I rely on the people I'm connected to via feeds, twitter, facebook, etc for filtering. However, the results are so-so. There are lots of folks I connect to because I value their opinions on specific topics, but find their off-topic data useless noise.

I'd love to be able to follow people only when they're yapping about specific stuff. Of course, I also want to make sure I get plenty of high-quality serendipitous stuff (NOT bad serendipitous stuff).

We're not there yet, but this is what we're trying to do with 9Neighbors -- to give people a way follow the content created by people in their community, but only when it's about their community.

I'd love to see it have folder sortability with "rules" we can write to sort them out. I'd love to have a "racing" channel, a channel for friends & family, coworkers, etc.

I hate to sound like retro guy here (I'm not, really) but this is where human editors become really, really useful. I know that I cannot possible follow every blog or every twitterer with something interesting to talk about, so I focus on getting a core set of people in Reader and Twitter who will tell me when something interesting is out there.

It's a very old media model, but it works. It's more useful, for example, to have Dwight Silverman, the tech editor at the Houston Chronicle, publish his linkpost every morning with a dozen interesting stories that to try to sort through a hundred sources.

Is it perfect? No, nothing is, and of course you are relying on somebody else's judgment to tell you "hey, you should know about this." But if you choose those people carefully it works well.

To Rick Burnes: I'm not sure aggregation is completely nailed yet. At least not for me. I'm still trying to figure out how to get twitter and my RSS feeds in Google Reader combined into one stream. I haven't figured out the value in Twitter because I only remember to head over there once a week. I don't really 'head over' anywhere anymore except for my news reader and email.

As for filters, I'm with john Whiteside. I've got my personal a-list of bloggers and when they recommend sites or other blogs I check them out and add them to the list.

The network is the filter. :-)

I used to worry that I was getting tunnel vision this way, but it's not working out like that. Jake Mckee got me drilling into adult LEGO, there's a marketing blogger who's into F1 racing and another who's follows my Premier League football team. The network of bloggers I follow are connected to a LOT of neat people and ideas. More than enough for me.

Technically, the network is the filter, but I find that we need some kind of tool that makes the filter more realistic. I can't just sift all the time. I need a way to go from "this is interesting" to "this is interesting to Chris Brogan and his pursuits."

It technically can be done. I have to think more on HOW, but it's out there. It's a two step somehow.

Does Google Reader already have an API?

I'm reminded of fishing.
Something I've done with Twitter a few times, is to setup a keyword, "track teacher" for example, and let it go for a month or two. For all the posts I get with the word "teacher" in them, (everything from college students' dreams of finding a job, to the bored students texting in the back of class), I slowly add people I think are worth keeping an eye on. Once I have a good sampling of content providers ;) I "untrack" that keyword and work on something else.
So, part of our filtering is to cast out a net, and then toss back the fish we don't want, keeping only the best *for us*. I'm not sure if a computer can do that for me, yet...

I believe that when all is said and done, actual humans and not automated software is the best way to "filter" the vast amount of information now at our disposal; in fact, I'll go so far as to say that in the coming mature Information Age, the biggest general job-class will be that of "editor," just as "managers" dominated the business world during the Industrial Age. In this sense, then, I think the key to comprehending this information better is not in filtering it but rather aggregating it in a smarter way than we currently do; in other words, don't remove any of the Twitter posts that a person currently receives, but invent a way so that similar posts get grouped together visually, or in another way so that dominant themes and memes emerge naturally.

As a good example, I in particular am an RSS junkie, and follow along now with something like 400 feeds a day; but just this week I actually deleted all my "hard news" feeds, in favor of a daily trip to Google News instead. I'm still getting the same amount of raw information there -- a quick check just now, for example, shows links to roughly 600 individual articles on the Google News front page -- just that Google sorts and groups this information in a much more intuitive and useful way than my RSS feeds did, so that at a glance I can see which seven links on the front page all pertain to the same topic. At that point, then, I can choose which human newspaper staff I trust and like the most, and go read their article in particular.

I would love to see the tech industry focus more on things like this -- on presenting both raw and edited information in a smarter and better way, not on how to best -remove- information. Leave the filtering up to actual humans, I say, who will always be infinitely better than any automated piece of software at determining what is "important" and what isn't.

The problem with filters is that someone always controls them. It is quite a dilemma, you can either muddle your way through the chaff to find the wheat that suits you, or you can trust that someone (or the masses) will do what's best for you in helping you choose the best wheat. The problem with that is that fields and fields of other grains could get excluded because digg doesn't agree with their position... or worse yet, you get the dumbed down stuff that the mass public likes.

I prefer to filter for myself and keep yet another layer of control OUT of the Internet.

What if you had tools to make your own filters? What if you had a way to let in the everything (per Marina's point), but a mechanism to further elevate the pieces that have even more value than others?

Could Yahoo Pipes come in handy with filtering the feeds of some of these sites?

A little bit. Yahoo Pipes might be the thing that STARTS the process. I think a tightened up app at the end of the day might make it better, but you know... why NOT start with Pipes and see if we can figure it out.

Now, I'm a Pipes hack, but if YOU are smarter (and you are), can you take a swing on this?

I'd like to see an example with two columns, something like "Wanted" and "Unwanted". Maybe a couple of weighting controls like importance 1 to 5. Then the filter logic, needs to be beefed (veged) up to do the work. Here's an example in the world of VoIP (over-simplified).

A company called Iotum makes an automatic call routing system. It reads your calendar and address book and it knows who people you know are to you. You give an importance rating to known customer, a mother-in-law, an ex. A call comes in from a number belonging to a person in your calendar and it sees that you have a lunch date in one hour. It deduces that this call is important, maybe a change of plans. Could this be done with filtering feeds instead of calls?

So assuming
Unwanted: iPhone, YouTube
Wanted: Voip, Coltrane

You wouldn't want to lose Voip just because a new service also mentioned the iPhone. Logic needs to be able to "deduce" that you're dead sick of YouTube and the iPhone, certain articles that mention both may still be of interest.

These things are a lot like programming. If someone can put it into logic, it can be done. The challenge is therefore how you are able to explain your interests (aka the interface).

/r

Hi Chris -

This is a great post and a huge problem for both personal and organizational productivity.

The problem: We all learn and absorb information, content, and relationships differently so we need different - and as you pointed out - personalized filters.

Because of this I think the right approach is multi-faceted. Some automated filtering (personalized), a way to tag and 'favorite' content and relationships for yourself, and the use of human editors. Mixing and mashing that into an interface that is personalized is the key. For people who want unfiltered access to all content that mentions 'monkeys' may be one persons idea of great filtering...other people may only work best with what their trusted colleagues recommend.

Some of the technology is available today - Yahoo Pipes was mentions and social networking is one kind of trusted filter - but the functionality is not pervasive across properties and we are still struggling with presentation and personalization of the display. Very few people have the patience to customize Netvibes...although I doubt your audience is representative :)

Great debate about a problem that needs better solutions.

How about 'search by date' as a feature of a given search tool? It would help by pre-sifting to recent (more relevant?) material.

I agree with Marina. I no longer worry about missing the good stuff. So maybe I don't catch a blog post right when it's published- if it has value for me chances are the information will bubble up to me if I keep my eyes open. Great information can come from so many directions it's impossible to stay on top of them all. Using filters and tracking keywords can help, but it's still going to be info overload.

You're right, Chris: comments _are_ better than the post ;-)

Before I toss my two cents in the kitty, I'll say I realize that it's some people's job to read _everything_, and that positions like that--along with rare circumstances where you're on a time-delimited project and filtering can save your bacon. I'm not ag'in' 'em, the Super-Dee-Duper Smart Filters of the Future, but I am for the selective usage and application of them.

Because part of the joy of life is serendipity, and that means dealing with a certain amount of dross.

I think Marina hit the nail on the head. Fundamentally, we need to release the idea that we can/should read everything. I also think there's a little value in randomness: sometimes, I discover something great that no way would have bubbled up in a filter, no matter how smart.

I also think Marina wins Comment of the Month Award. Tell the little lady what she's won, eh, Chris?

i'm going to go grab Socialthing, i was damn impressed with it.

Hello,

We're building large list of blogs (2000+) for clients & agencies interested in engaging or advertising in blogs and I'd like to share our experience:

1- Filtering is as you mentioned a key issue, it's hard to reach relevance, eliminate splogs and blogs that are just build to catch ads.

2- Finding blogs & expanding your list is also a real challenge. Topics like "iPods usage" or even "Teen parenting" represent significant relevance challenges. How do you filter out all the commercial blogs on iPods or filter out teen blogs.

Most of the so called blog search available are actually post search and don't really help qualifying the blog as a media i.e is this blog a personal finance blog or does it just have a post where "credit card" is mentioned.

Using a mix of topology, linguistic and predictive analytics we 've been able to significantly reduce the "time to build a list of relevant blog/web 2.0 properties" from 3-4 weeks and lots of manual processing to an *almost* automated process that takes a day.

Then, this is just the beginning of the journey....
We make this initial (clean) list available to marketing teams that can collaboratively enrich and fine tune it over time, getting to know bloggers, gauging their relevance and competence and engaging when it makes sense.

As Jason said, humans, and in our case, a collaborative group of humans are the one performing the filtering. They do this while listening, learning and participating in the conversation.

*Coming soon!*

I noticed a long time ago that, even when noise sources like flaming were removed, material that was truly operational got swamped. I'm grateful to some long-forgotten source on the web for providing me with the key term: "phatic" ... communicative gestures that are purely social in nature.

The mid-90s, when the web was just coming into its own, was a time of disappointment. All the forums and discussion venues ... more heat than light, even the best ones. Hundreds of thousands of people spending millions of their hours online generating ... what? As for social exchange, well, sure ... generating pages of yada-blah might but fun, but it isn't much more than that.
Around that time, late 90s, concept mapping started becoming productive ... systems like cMap and Compendium and Rationale. Absolutely noise free and about as attractive as a root canal.

I came up with my design after a happy coincidence: I happened to be reading at once both John Willinksy's work on OpenAccess and Jurgen Habermas' on "discourse ethics". I can recall precisely the moment the penny dropped for me.

"Participatory deliberation" is going to be a reality soon, but my worry is this: what business model can support real exchange? Drama makes for buzz ... fun makes for buzz ... can a source of solid unadulterated information make for buzz?
We'll see ... soon, I hope.

Ugh, fishbowl.

Go talk to your neighbors - explain to them why any of this should matter to them.

Regarding the notion of human filters being a "control" - well, a good filter (e.g., an editor) brings a depth of subject knowledge to the filtering process that most of us don't have.

If I were an expert on everything, I wouldn't need editors. I'm not, and none of us are.

The whole idea that human editors don't add value becomes really troubling when you apply it general news coverage. The fact that a certain set of stories is popular or grab my attention doesn't mean that they are accurate, complete, or useful to me when I start making political choices. I'm not suggesting that we place blind faith in someone to tell us what we need to know, but let's face it; if, say, the political stability of the middle east is an important topic to you when you vote, you are much better off finding out what acknowledged experts have to say - and what sources they recommend for keeping abreast of things - that seeing what story was the most popular on a news site. Or just diving in and deciding what you like, which all too often is what you want to hear.

And remember, it's not a binary choice. There are critical roles for both human-guided information sources and self-created filters that help you find specialized knowledge that you want or need.

Expertise is meaningful, and most of us only have it in a very limited number of subject areas.

Humans have to be part of the mix, but humans augmented with better tools is where the gravy is. Right? It's humans inside with tools to move the value up the chain, right?

[sorry, this is long]

Chris said:

"What if you had a way to let in the everything (per Marina’s point), but a mechanism to further elevate the pieces that have even more value than others?"

I'm intrigued by this, although I'm not exactly sure what you're after yet. Does this match what you're thinking Chris?

- All the news from my network: Google Reader, Twitter, Facebook, etc. etc. all aggregated to one point. Some sort of uber-aggregator.
- A way to say what's relevant to you right now. A list of search terms, keywords, tags? Always changeable, of course, as your interests change.
- a way for you to read something and flag it as 'relevant' or 'not relevant'. Something like Digg?
- A recommendation engine that crawls your uber-aggregator feed and meshes it with your relevant keywords, fine tunes with your Digg list and serves the most probable relevant content first.
- Go beyond your uber-aggregator to the wider web and recommend content that comes from outside your network. Add the good stuff / contributors to your network, of course.

So the filter is predictive as well as tuneable over time. It changes as your mood/interest changes. And it scans your network as well as the wider world.

Is that close?

Mark- you're spot on. I was describing this to a friend who's in the technology meets marketing space and she said, "Oh, like Digg," and I said "yes, but a nation of diggs." Hmm... know what?

REDDIT.

Where's Alexis?

Chris
I am a contributing editor at a non-profit social news site called NewsTrust.net. It's like Digg, except where "digg" is vauge - ranking a story on newstrust is all about whether or not you think the information is of high quality or not.

Would love you to check it out and give some feedback on where it might miss the spot on what you are picturing in your head.

As a previous commenter noted: This is an important question - how can you filter signal from noise. This is especially true when it comes to news/journalism - how can you filter good information from bad information.

Chris,

Great point.

A perfect place for this type of thing? Friendfeed or socialthing. All of the streams gather up there, why not let us form our own mashups and bubble up rules etc. Yahoo pipes maybe? But that's more complex than most folks want to go.

@Todd- I agree that either FriendFeed or Social Thing could make this work faster than I'll figure it out. I hope they can help with such magical things. : )