Over the last year, there has been a flurry of stories on the blogosphere about the about the inability of big search players and small startups to create a viable blog search engine. It all gradually started with the ubiquitous niggle at Technorati, the “leading” player in the blog search market, a company that probably spends more time at thinking about the perfect ad placement coordinates than spam reduction and search relevance.. hmm, well, you see what I mean by “ubiquitous niggle”.

But then I just thought it was all meant to encourage Technorati, as we all have this innate need to see Google beaten at any race they’re in, even though we actually just need that to happen to be able to tell ourselves that Google is not the only option and we’re using it at free will (sorry, drifting offtopic..). Anyway, that seemed not to be the case, as the will to see Technorati improve slightly went over into a general sense of frustration, capped off in August 2008 with Mashable’s supposition that Blog Search is Broken.

The main notion in that post was that Technorati has the most potential, but just doesn’t seem to be getting any love from the users, while the newer options like MyBlogLog, BlogCatalog or Wikio are just trying to put collections of blogs and/or blog posts together in a directory-like manner, without offering much of the search dimension.

Since then, at least one startup has sprung to life which seems to be getting a lot of acclaim, namely Twingly. But while the company is doing really good progress, it doesn’t seem to have established itself as a major player yet, and thus we come to March 2009, where, as the legend goes, blog search has finally died. At least according to The Blog Herald’s latest analysis, according to which there is too much spam and irrelevant stuff whenever it comes to blog searching.

Now they are probably completely right on the problem aspect, i.e. there is really a lot of spam-blogs out there, as well as feeds that aren’t even blog feeds or some company’s great idea to index the whole page of a blog post, without even trying to cut out the actual text from it (Go*cough*ogle). As for the solutions, there is much more that needs to be done but also much reason to be optimistic even of the very near future to come up with a decent solution.

First, the problem or spam-blogs, feeds that don’t point to blogs and other basic irrelevancy. Think of just one approach here, the Wikipedia-approach, and you will be rewarded aplenty! Yes, spammers will try to add their spam-blogs, while others will be accidentally saved by the crawlers, but there are users to help us out there. A simple combination of user-voting and admin-monitoring can do wonders here in my opinion, there just needs to be a good base of blogs to start with to attract the crowd and then the snowball will be on its’ way.

A much bigger problem is that blog search engines are still focusing on the old approaches of link-based post ranking, tagging and also the new approach of user-voting. As I pointed out in previous posts, the first is just a tonic for top 100 blogs, the second highly misleading and the third just not applicable for a pure search site (it surely is for a news aggregator or such). What is amiss here are semantic technologies that will mine deeper into the meaning of the blog posts and provide the user with highly relevant stories based on their content and not some keyword-link rank.

Thus, to revive blog search, we have to see that it’s different from the initial Web problem of finding any relevant content in a haystack, it’s about finding the one that makes the most sense. Luckily for all concerned, there are many semantic startups launching and getting good support around the scene and it can’t be long before they spread over to such tasks as blog search. When that day comes (i.e. I stop talking and finally deliver thing thingy called Topify), Jonathan, you will be one of the first to get an invite and have your faith in blog search restored :)

Before I speak any further about a possible “competitor”, I would like to clarify my personal position in this whole mess a little: Although the Topify project aims to provide its’ users with a perfect topic exploration experience and blow all the competition away, *grin*, we also want to see more linguistic tools used on the Web, especially when it comes to dealing with natural text. Accordingly, we welcome and encourage every idea that tries to use NLP tools to make sense of human-generated texts. Again, as I said before, tagging is an amazing idea and currently the most efficient approach to finding relevant information on the Web, but I deeply believe that natural language can bring so much more to the scene.

That clarified, I would like to express my sincere joy at seeing a company try and use some kind of (albeit very simple) structural approach to “track hot topics“. FeedVis doesn’t do much more than count word ocurrency frequencies in blog posts and put these together to possibly detect the most important and/or currently trending topics in a thematically connected set of blogs. This doesn’t sound like much, but it’s a great start already, although probably with a couple of misconceptions.

What we see in the FeedVis “tag cloud” is in my opinion not precisely a set of topics, but rather a specialized base vocabulary of the selected area. If, when and to what degree the one word or the other is actually a topic is hard to say from such a generalized statistic though. The word “learning” might appear once in _every_ post, whereas some others, which are much lower down on the absolute count, can appear several times in a single post. For me, the first case would not be a topic, but a general theme-defined word which doesn’t really help us understand what the current trends in the area are. The second case, however, is a real topic, which gives strong clues about the main motifs in an area at the given time.

All in all, it’s nice to see a service like FeedVis, that goes beyond the common tagging approach and tries to extract some sense out of texts without relying on anything really. Will be exciting to compete with you once Topify is running. You get a head start anyway :)

Sorry for not writing for a such a long time, but the Topify team has been quite busy with the personal life in the last 1.5 months to make any sort of huge progress. We have successfully launched our alpha version during the so-called Developers Day last month and met a lot of positive and supportive feedback. The main part of the site that most people were missing is some sort of community-like area, which was planned to be done next, but…

..but apparently we haven’t thought everything through when we started developing the site. So due to this, we were, unfortunately, stuck with a site design that didn’t allow for too many Web 2.0 features and a CMS that is a bit behind in this area. So after some consideration, we decided to switch both. The design has been much improved and looks sleeker than ever, which is a real joy. We have also switched our CMS and after a short period of porting the old stuff, we’re back on track and developing the final features quicker than ever. This said, there is still work to be done before we can launch a stable public beta. I’m not going to do the silly mistake of estimating a launch date again, but, to be honest, I’m kind of tired of developing in stealth mode and just want to get the thing out into the wild as soon as it goes. Unleash the beast on you! Yes, you! and YOU! Don’t run away!

Yes, it’s finally this far! We are violently beating at our keyboards and mouse buttons to finish up what seems like an alpha version of an exciting product. We still have some features missing completely, but the main bulk of the Topify world – blog topic mining and analytics – is done and awaiting first tests and reviews.

We will launch this first private alpha at a public, yet very small developers meeting hosted by the Chair for Information Systems and Databases at the RWTH Aachen University where me and my partner are still happily employed at the moment. The meeting is called “Developers’ Day” and has been staged for a couple of years now, with a host of interesting topics on current developments in – surprise – development of different software. You can visit the official page of the meeting here and drop by if you’re in Aachen. I should have posted this earlier, but if you really want to come, just come, it won’t be a problem. :)

After this, we will try to implement the last part of the site – the personal area – as quick as possible, consider all the suggestions from the alpha testers and launch for the public! These are hard times for start ups obviously and, as our application feedback form the seedcamp competition suggests, the market we’re entering has already some strong players, but we hope to bring a fresh perspective to blogs and topics. So, for the 1324145th time, stay tuned and we hope you will like Topify.

The news is actually out for a while now, but I’ve been obviously not too eager to inform anyone.. that we haven’t been invited for a shortlist-interview to London on September 3rd. It is quite sad that we won’t be able to make the great experience of seedcamp this year, but either way, we have learned a lot about our product and the ways it can be presented while working on a prototype and the seedcamp application. So thank you for organizing the event and all the best of luck to all the participants!

Now if you listened carefully, you might have noticed the word “prototype” somewhere along the last lines :) We do indeed have a running prototype, which we’re currently in the process of improving to the point where it can be respectably presented to the public. We’ll try to add as many features as possible from the long list we have, but the main goal for us is to incorporate the main features like the “topifying” of blog posts and the corresponding search and exploration.

So the question now is where to launch Topify? There are a few options, even though none of these is really perfect and nothing is more important than getting off from the right note. We could launch at a workshop within the Computer Science Chair where me and my partner work at the moment. But this would be way too internal I guess. Another option would be to launch during the social meetup after the Web 2.0 Expo in Berlin proposed by Mike Butcher over at TechCrunch. Or at one of the Digital Island meetups in Ireland, which I personally think most suited for our cause in the very near future. A tough decision, but in the end it will all come down to just launching as soon as possible and letting you, the users, tell us where to go next! :)

So stay tuned!

Well, I hope you all know what seedcamp is and if not visit www.seedcamp.com to find out. To put it shortly, seedcamp is a week-long startup incubator where enterpreneurs get to meet experts from different fields and learn important lessons on how to manage to startup.

Why am I posting this? Well mostly because we have applied for seedcamp 2008! We were able to put together a live, but not yet fully functional, prototype web site for the application and hope that the judges will find our service exciting enough to invite us to London on the 3rd July for the shortlist interview. The decision will fall on August 25th and I’ll keep you all fully informed.

The good news for all of you is that we’re trying extra hard to get into seedcamp and have hence doubled and tripled our effort to get a beta site running as soon as possible. So expect more exciting news at the end of this or at the beginning of next month. :) Cheers.

In response to Rob Cottingham’s post about the death of tag clouds on ReadWriteWeb I would like to elaborate a little why it appears that tag clouds are disappearing. Just to make it clear, I’m not saying that they are actually disappearing, but I do think that there are several reasons why tag clouds aren’t as popular as they used to be.

As is the case with any other new and cool feature, tag clouds have enjoyed a significant amount of hype over the last couple of years. After all, is there an easier way to “visualize” your blog than a simple, yet in some mystical way sophisticated tag cloud? No! But is it really a well readable and that sophisticated visualization? No again.

Image via Wikipedia

The thing about tag clouds is that after the hype tuned down a little, people started actually thinking about making them acceptably usable. And let’s be honest, that’s not an easy task if you have a blog that’s not as narrow-themed as this one ;) But even if you just write on one single topic, there will be numerous aspects to it and hence numerous tags, which you would want to appear in the nice and shiny tag cloud. However, once the number of tags in a tag cloud reaches 5 or so, they become highly indistinguishable and hard to perceive. In a way, it’s like some amazingly cute but useless social network visualization with 50+ nodes, where all you can see is pretty dots and lines making up a shape, but not the pieces themselves.

So to make a point, tag clouds can be fun and usable at the same time, but like with everything else, they need some more work put into it. One way is to group them by the topic that the target user is interested in and display these groups accordingly. So if I were reading the above mentioned post on RWW I wouldn’t get the entire tag cloud, but only a portion of those that are most related to the tags of the entry (“tagging”, “delicious”, “tag clouds”) -> “semantic”, “metadata” etc.. As you see, there are (many more) ways to make tag clouds appear everywhere again, just make them more usable and you’ll see some of these ideas implemented in the upcoming launch of our Topify project pretty soon. ;)

So stay tuned and.. Tag clouds are dead! Long live tag clouds!

Yay! After months of pushing ideas, planning the concept and just being lazy as usual, the work on the Topify web site has finally kicked off “for real”. It’s not a lot yet, but we have a design and some good ideas, mostly aiming towards being user-friendly, intuitive and easily usable. The technical stuff is nearly complete as well, so it’s now a matter of combining and adjusting the algorithms and the web stuff. This might take some time still, since we’re only a 2-man-team with studies and part-time work to attend to as well, but we’re doing all things possible to bring this project to life as soon as possible! :) Cheers.

I apologize for not being able to write a lot these last few weeks. I’ve been mostly busy working on a chapter about the theory of public opinion for my final thesis. This was a very enlightening experience, since it opened my eyes on the different ways one can take when measuring the “voix publique” and the corresponding problems. The question I raised at the end of the chapter in my thesis is the one I ask here: do blogs represent the public opinion? This is indeed a very tricky question, but if one takes a look at what the public opinion actually is, there is hope still, that the question can be answered!

The concept of a public opinion comes from the 18th century France, where it was represented by a group of similarly educated, well-read intellectuals. Due to the similarity in the education and the philosophy of the members of the group, the critique they expressed tended to come in unison, hence the initial singular in “public opinion”. Later on, as more differently educated social groups appeared, the “one” public opinion dispersed into the opinions of these groups. By the 20th century, every person was considered bright enough to think on his own, so the notion of an “opinion expressed in social groups” was dropped for a more “public” definition of the term.

However, ever since the public opinion was considered to be a sum of all the personal opinions in the public, no one has been really able to grasp it anymore. Multiple institutions for research of public opinion exist all over the world, but all they do is question the people on the street (at home, on the Web, whatever.. keyword is “at random”). The biggest problem with this approach is that though not every person has an opinion to every subject, they’re still browbeaten by the interviewers to say something..

On the other hand, most people have opinions to certain subjects. And most of us wouldn’t mind sharing them, provided that our anonymity is guaranteed. These are thus exactly the thoughts and opinions that one should gather in order to get an idea of the so called “public opinion”. How do we find these people? Look on the Web! Find all posts that deal with a topic that you want to examine and see what people are saying. This is being done already by specialized brand-monitoring companies like Trackur, which was just recently reviewed on ReadWriteWeb. Unfortunately, however, such services still run for a fee and not for all the random subjects around us.

Thus I would argue that a service that gives its’ users an overview of the “hot” topics and the opinions about these should be of great value and interest to everyone from a common man to a scheming politician. It’s time to put a stop to the “A”-blogs dominating the agenda like the social groups of the 19th century. There is much more information out there that is just as exciting and valuable, we just tend to underestimate the long tail. To get an impression of the matter just click over to this post over at the Data Mining blog and take a look at the odds for a “normal” blogger getting a headline at a site like TechMeme compared to those of an “A”-blogger. Impressive huh? Now just try to imagine the diversity of the topics and opinions in the public blogs.. it’s IMMENSE!

Just a couple of weeks ago I was trying to set things straight in the rather troublesome relationship between tags and topics. The bottom line was that tags are just the same words we use in the content and are hence just as arbitrary and hard to generalize as the content itself. No perfect solution for this problem exists as yet, although I proposed to topify the content in order to keep the generalizing terms (tags.. topics..) as close to the content as possible.

Now I get reminded that there is a third member of the family that may be used in this way, but rather tends to go astray and entertain people. I’m of course talking about the ever present headlines that set the tone to a piece of content long before we actually get to read the whole thing. So what is the function of this, rather distinct, member of the meta-family?

One might have thought that titles should provide the reader a short taster of what is coming next, so that he or she would be able to filter and prioritize a piece of content among others. However, certain media actors don’t like to play fair, especially those who don’t have much of worthy content to offer. This is first and foremost the case with the so called Yellow Press, which has invented and perfected the art of indirect headline for the last century or so.

The idea of an indirect headline is to use a quality of human nature – a humans curiosity – to drive readers to some potentially utterly worthless content, as was the case in a blog entry in The Blog Herald that inspired this here post. The evil thing about this is that we can’t help it but take the bait over and over, even after getting disappointed time and time again.

So what can we do to tame this extravagant member of the meta-family? Well, frankly not much. One could check out the site’s tags to get some additional information, but the problem with tags is that they are becoming more and more like headlines as well. If you tag your content in detail, you risk losing readers who know for sure they don’t care about certain stuff. General tags on the other hand don’t allow you to filter and prioritize the content properly, so you have to check out at least the first few lines of the text. This is where, I think, a more intrinsic way of meta-describing content by topics e.g. can come in handy, from the user point of view anyway.