Diego has been writing about a project of his the “Conversation Engine”. I thought I would add a few thoughts to the fray that might help.

For discovery a simple RPC-Ping can be used, this can then add that site onto the spidering queue. To maintain the small scale test of the engine to begin with a whitelist of sites that are allowed to ping. Each page should be spidered multiple times over time to take advantage of trackback and pings. This could be coupled with trying to take the date/time out of the published posts. This may be a little difficult but still possible.

Scoble has lost alot of credibility with me today. Looking at the times he has posted he has been active on his weblog at 2 separate times in the last day and hasn’t posted the biggest IT related news (IMHO) of the day. The release of Firefox. I should be giving him the benefit of the doubt, but the KoolAid really could be getting to his brain.