Thank you very much for your comments. I must admit I probably made it all a little confusing by condemning user-generated solutions first and then saying they are the cure :) What I meant was that when looking for relevant stories, one shouldn’t rely on user votes/tags/link etc, because you will get gamed or at least very biased results.

On the other hand, when all you need to do is filter out a source that is pure spam or not a blog, that’s where the amazing classifier which is the human brain comes up trumps. So if you only let the users report bad sources and do semantic search on the stories from the acceptable sources, that’s already a big step towards having a broad, yet good range of sources and stories.

Hope this clears our idea up a little, if you have any further questions or thoughts, let me know :)

Cheers,
Martin

Twingly does well for search but if you take a look at our site, blog search is really just one part of it, it’s more Blog Browsing (with bonus search!).
Anyways, good followup post and if you have any feedback please let me know, we love to get it. You can email or call me at 404-493-5121 to discuss. Cheers!

As I said on my post on the Blog Herald, I think the semantic Web has potential to really improve things in this area, but I am not certain it will really fix the problem as it doesn’t address a lot of the issues.

One thing I do find kind of odd is that you hail a user-generated solution to the problem, namely Wikipedia-style voting and recommendations, while condemning other user-generated solutions such as inbound links, tags and voting as non-answers. Perhaps I’m misreading those two paragraphs, I get the feeling I’m missing something, but I don’t see how one user-generated solution can be a fix for a mess created by three others.

We will see if the semantic Web can help. I’m going to remain a skeptic until I see the progress. Right now though, if blog search isn’t dead, it’s on life support. Doesn’t mean it won’t come back. If I’ve learned one thing from my love affair with horror movies, it’s the being dead really doesn’t mean all that much.

As for the “detection of topics”, word-count/total posts should be actually equal (in distribution) to word-count/total words, since total-words = total-posts*average-words-per-post, or, to put it bluntly, both are constant for a given corpus. What I was referring to is a measure like word-count/posts-where-word-occurred-count. This way if you have say 100 posts and both “learning” and “classroom” have a total count of 50, your current approach would rank them the same. But imagine if “learning” appears in 50 posts (i.e. always once in a post, never more) and “classroom” only in 10 posts (so average of 5 per post). This would hint to the conclusion that “classroom” is an important topic within the posts it comes up in, while “learning” is just a ‘background-term’ that’s parasitic to the field.

This is obviously a very simple idea, one can take this further to consider average counts per post where a word occurs, spread over time chunks etc. Hope you can catch my drift.

And yea, let’s hope people recognize the power of AI and NLP, as we need more ideas here. There are some pretty powerful statistical basics in the field, what we need now are some inventive applications to push it further.

You wrote one thing I didn’t quite understand, about detecting “topics.” FeedVis measures overall frequency, as in word-uses/total words for a particular word. I originally was doing word-uses/total posts, which I think is something like what you were saying. I was very surprised to see that these measures actually came out very similar. It would be interesting to look more into this.

I think we’re agreed that however it comes about, this kind of automatic metadata generation is the future: tags are nice, but no one is going to sit around tagging the entire internet. That’s what we’ve got computers for.

Thanks for linking to the cartoon – I’m glad it’s sparking some discussion!

What we have right now in texts is the standard layer – the _words_ themselves, which are never 100% precise, since words are employed to mean exactly what the writer wants them to and not as some universal concept. And then we have the covering tag-layer, which is also comprised of _words_ that are not unambiguous. So why not just have one layer, where you have the text, from which you take the key words to “tag” itself.

Again, I’m only suggesting this for textual content and this way the number of tags would not be reduced (the number of tags equals the number of all words now and will be the same after, ie. infinite).

Why do you think it would be useful to reduce the number of possible tags ?