Comments on: SecondCast #57 – “Search Furtherment” http://www.reznation.com/2007/04/secondcast-57-search-furtherment/ SecondCast, Metaverse Sessions, Second Life Future Salon, and Linden Lab Town Hall Meetings Sun, 22 Jun 2008 13:55:16 -0500 http://wordpress.org/?v=2.8.2 hourly 1 By: Kip Yellowjacket http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-38314 Kip Yellowjacket Fri, 27 Jul 2007 16:30:12 +0000 http://www.reznation.com/?p=106#comment-38314 also a very nice listen...would have been nice to hear some discussion on the inventory and how it might be improved...takes ages to get an inventory uncluttered...seems like being able to do that from a website would be a nice feature... also a very nice listen…would have been nice to hear some discussion on the inventory and how it might be improved…takes ages to get an inventory uncluttered…seems like being able to do that from a website would be a nice feature…

]]> By: baack to the future » Blog Archive » search, operators, and free as in beer http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-14999 baack to the future » Blog Archive » search, operators, and free as in beer Mon, 16 Apr 2007 04:50:03 +0000 http://www.reznation.com/?p=106#comment-14999 [...] In the wake of much bloggage, I wanted to voice my appreciation both for Ordinal Malaprop’s post, Searching, Publicity; How Well It Can Work, How Well It Should (and her follow-up) as well as to Hiro Pendragon and the SecondCast crew for a great discussion about search (rather, on ‘Search Furtherment’). [...] [...] In the wake of much bloggage, I wanted to voice my appreciation both for Ordinal Malaprop’s post, Searching, Publicity; How Well It Can Work, How Well It Should (and her follow-up) as well as to Hiro Pendragon and the SecondCast crew for a great discussion about search (rather, on ‘Search Furtherment’). [...]

]]> By: HatHead Rickenbacker http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-14609 HatHead Rickenbacker Sat, 14 Apr 2007 04:08:09 +0000 http://www.reznation.com/?p=106#comment-14609 Does the Grid Sheppard IM the owner of the goods to let them know they have goods that have been spidered by a bot? This would allow the person to make sure the data is accurate in the search system or give them a chance to opt out. Also, does the bot scan across parcel boundaries? This wasn't brought up in the podcast but could enable scanning of some land that the bot is banned from. Great podcast - peace! Does the Grid Sheppard IM the owner of the goods to let them know they have goods that have been spidered by a bot? This would allow the person to make sure the data is accurate in the search system or give them a chance to opt out.

Also, does the bot scan across parcel boundaries? This wasn’t brought up in the podcast but could enable scanning of some land that the bot is banned from.

Great podcast – peace!

]]> By: A conversation with Sheep Elite C.Prior « secondlifetribune http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-14593 A conversation with Sheep Elite C.Prior « secondlifetribune Sat, 14 Apr 2007 01:53:20 +0000 http://www.reznation.com/?p=106#comment-14593 [...] A conversation with Sheep Elite C.Prior <Tenshi>    so… what’s the deal? Why opt-out and not opt-in? I mean, I can see where you’d get less participation with an opt-in (possibly) but.. <Tenshi>    also, how do you feel about it? <CPrior>    right, so i had a good conversation about that on the second cast that went up a couple nights ago with christiano, lordfly, and hiro <CPrior>    as lordfly pointed out, opt-in systems just don’t work <CPrior>    ESC did an all-opt in search engine (slindex.com) previously and it didn’t go anywhere because no one used it– there was no value to get from the service initially <Tenshi>    Hm, I didn’t know there was a second cast episode out. I should go dig that up. <CPrior>    second411.com search, also all opt in, is now owned by mark barrett, who also works for esc <CPrior>    so this is something we had thought extensively about <Tenshi>    Do you think that the system is worth the problems of a few people hollering “foul”? <CPrior>    all opt-out systems like mark’s first projects, slbuzz, which clearly raised some red flags (mine own included, perhaps), are obviously a bit much <CPrior>    so the black and white sort of approach doesn’t yield much to work with <CPrior>    but we decided to make that compromise: if someone is trying to sell something, presumably they want people to be able to find and purchase it <CPrior>    i’ve had a heck of a week and am absolutely doing everything i can do address the concerns people have voiced <CPrior>    but fundamentally, yes, i do believe this is a step forward for finding things in virtual worlds — ie, both for the person looking to purchase something, and the content creator / business owner <CPrior>    if i decide i want to buy a hat, i don’t walk downstairs and wander around around manhattan hoping to bump into someone <CPrior>    i google it <CPrior>    and in SL, there were literally thousands of hidden treasures that I would need to somehow in conversation find out about and receive a landmark <Tenshi>    I honestly thought it was great until I realized that it did expose my private home location. That wasn’t so good. But I found the problems and fixed them, I think - unchecked the “for sale” bit, which I’m still not sure was on. <Tenshi>    Right, the SL search is just like Google. At least for me. <CPrior>    things have been found in SL through word of mouth only, for the most part, and that’s almost unbelievable to me <CPrior>    when you compare it to the way we use the web to find our way through RL these days <Tenshi>    that’s very true <Tenshi>    I can think of a few people that consistently link to Wikipedia, which isn’t always accurate. <CPrior>    as a comparison for the search service? <Tenshi>    No, as an add-on to your comment “the way we use the web to find our way through RL” <Tenshi>    How do you think this will add to the future of the Sheep? <CPrior>    the majority of the 14 months I’ve had with the sheep i was working on SLBoutique.com <CPrior>    and we were keenly aware that taking items and boxing them up, putting them in vendors, then flipping over to a web site to login and add pictures, descriptions, prices, etc., was a pain in the neck when all of this stuff could be seen in world <CPrior>    we wanted to build services on that data, but LL didn’t provide an API to get it <Tenshi>    As a retailer in Second Life, I have to agree with that. I didn’t want to do things twice, so I don’t utilize SL Boutique. <CPrior>    yeah it’s a pain, i completely appreciate that <CPrior>    so it’s taken a bit longer, but we now have the data, and (I can’t stress enough) want to build those APIs to make it available for everyone <CPrior>    and hopefully can help enrich services like SLB, even SLX, SLQ, …, by providing this base <Tenshi>    Groovy. [...] [...] A conversation with Sheep Elite C.Prior <Tenshi>    so… what’s the deal? Why opt-out and not opt-in? I mean, I can see where you’d get less participation with an opt-in (possibly) but.. <Tenshi>    also, how do you feel about it? <CPrior>    right, so i had a good conversation about that on the second cast that went up a couple nights ago with christiano, lordfly, and hiro <CPrior>    as lordfly pointed out, opt-in systems just don’t work <CPrior>    ESC did an all-opt in search engine (slindex.com) previously and it didn’t go anywhere because no one used it– there was no value to get from the service initially <Tenshi>    Hm, I didn’t know there was a second cast episode out. I should go dig that up. <CPrior>    second411.com search, also all opt in, is now owned by mark barrett, who also works for esc <CPrior>    so this is something we had thought extensively about <Tenshi>    Do you think that the system is worth the problems of a few people hollering “foul”? <CPrior>    all opt-out systems like mark’s first projects, slbuzz, which clearly raised some red flags (mine own included, perhaps), are obviously a bit much <CPrior>    so the black and white sort of approach doesn’t yield much to work with <CPrior>    but we decided to make that compromise: if someone is trying to sell something, presumably they want people to be able to find and purchase it <CPrior>    i’ve had a heck of a week and am absolutely doing everything i can do address the concerns people have voiced <CPrior>    but fundamentally, yes, i do believe this is a step forward for finding things in virtual worlds — ie, both for the person looking to purchase something, and the content creator / business owner <CPrior>    if i decide i want to buy a hat, i don’t walk downstairs and wander around around manhattan hoping to bump into someone <CPrior>    i google it <CPrior>    and in SL, there were literally thousands of hidden treasures that I would need to somehow in conversation find out about and receive a landmark <Tenshi>    I honestly thought it was great until I realized that it did expose my private home location. That wasn’t so good. But I found the problems and fixed them, I think – unchecked the “for sale” bit, which I’m still not sure was on. <Tenshi>    Right, the SL search is just like Google. At least for me. <CPrior>    things have been found in SL through word of mouth only, for the most part, and that’s almost unbelievable to me <CPrior>    when you compare it to the way we use the web to find our way through RL these days <Tenshi>    that’s very true <Tenshi>    I can think of a few people that consistently link to Wikipedia, which isn’t always accurate. <CPrior>    as a comparison for the search service? <Tenshi>    No, as an add-on to your comment “the way we use the web to find our way through RL” <Tenshi>    How do you think this will add to the future of the Sheep? <CPrior>    the majority of the 14 months I’ve had with the sheep i was working on SLBoutique.com <CPrior>    and we were keenly aware that taking items and boxing them up, putting them in vendors, then flipping over to a web site to login and add pictures, descriptions, prices, etc., was a pain in the neck when all of this stuff could be seen in world <CPrior>    we wanted to build services on that data, but LL didn’t provide an API to get it <Tenshi>    As a retailer in Second Life, I have to agree with that. I didn’t want to do things twice, so I don’t utilize SL Boutique. <CPrior>    yeah it’s a pain, i completely appreciate that <CPrior>    so it’s taken a bit longer, but we now have the data, and (I can’t stress enough) want to build those APIs to make it available for everyone <CPrior>    and hopefully can help enrich services like SLB, even SLX, SLQ, …, by providing this base <Tenshi>    Groovy. [...]

]]> By: John Hurliman http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-14490 John Hurliman Fri, 13 Apr 2007 13:20:05 +0000 http://www.reznation.com/?p=106#comment-14490 Interesting podcast, and the ESC search engine is an interesting beast in itself. To follow up on the "hard numbers" that people have been wondering about bots, I don't have printouts in front of me but I can give a pretty good estimate. Phoenix Linden is quoted in several places saying that 90% of the clientsimulator traffic is asset transfers, almost exclusively texture downloads. I know that of the 10% remaining, somewhere between 10-20% of it is what is called LayerData, which describes the terrain and provides real-time wind and cloud data to the clients. Most bots, including a search engine like this throttle those channels down to zero so you're at around 8-9% or less of the network traffic of a typical client. Of that, some amount is AgentUpdate packets, or status updates. The official viewer sends status updates (that describe avatar and camera movement) to the server many times per second, while bots don't need that real-time smooth movement feedback and typically send them only two to four times per second; much less than the official viewer. What makes up the bulk of that remaining traffic is object updates. This describes the prims you see around you, and it's not something you query for. Clients don't fly around the grid asking "what do I see now, what do I see now", the simulator just sends the information. This is one of the reasons that is impossible to restrict bots that crawl or scrape for information, because they aren't even requesting the data in the first place. The simulator sends it to them by default, and crawlers are just intelligently indexing that information, and going to every simulator in the metaverse. The only difference in this case is that it is requesting object properties, but that happens in the official viewer when you hover your mouse over any object, without even clicking a button. So the argument that the grid will go down the toilet if everyone is running a crawler bot doesn't have a strong factual basis. The exception here are directory searches, which have to hit a backend database and doing them en masse will have a real effect on grid load. So comparing landbots, or people that refresh the for sale listings every few seconds to a search engine crawler is not apples to apples. Another tough problem to deal with is that you can't ban bad bots. Banning bots doesn't work in World of Warcraft (even though they spend countless hours and dollars trying) and it can't work here. The protocol is completely documented, the client is open source, the lines are constantly blurring between enhanced clients and fully automated bots. Having code signed off by an authority, only allowing bots to create opt-in systems, etc. is a nice idea and would keep honest people honest, but that's it. If someone is pulling in $1000 a week camping or running a landbot and you say "hey you can't run that code because it doesn't meet community standards", they probably won't stop running the code. They typically run on transient rented servers so IP banning is out, accounts in SL take seconds to create and even if you charge for them they are making more than enough money to buy all the accounts they need, it's not something you can solve with community standards. The only long term solution is to harden the grid: rate limit certain actions, and make sure that no individual can put an undue burden on the system. Easier said than done though, in Second Life and the Internet as a whole. Interesting podcast, and the ESC search engine is an interesting beast in itself. To follow up on the “hard numbers” that people have been wondering about bots, I don’t have printouts in front of me but I can give a pretty good estimate. Phoenix Linden is quoted in several places saying that 90% of the clientsimulator traffic is asset transfers, almost exclusively texture downloads. I know that of the 10% remaining, somewhere between 10-20% of it is what is called LayerData, which describes the terrain and provides real-time wind and cloud data to the clients. Most bots, including a search engine like this throttle those channels down to zero so you’re at around 8-9% or less of the network traffic of a typical client. Of that, some amount is AgentUpdate packets, or status updates. The official viewer sends status updates (that describe avatar and camera movement) to the server many times per second, while bots don’t need that real-time smooth movement feedback and typically send them only two to four times per second; much less than the official viewer.

What makes up the bulk of that remaining traffic is object updates. This describes the prims you see around you, and it’s not something you query for. Clients don’t fly around the grid asking “what do I see now, what do I see now”, the simulator just sends the information. This is one of the reasons that is impossible to restrict bots that crawl or scrape for information, because they aren’t even requesting the data in the first place. The simulator sends it to them by default, and crawlers are just intelligently indexing that information, and going to every simulator in the metaverse. The only difference in this case is that it is requesting object properties, but that happens in the official viewer when you hover your mouse over any object, without even clicking a button. So the argument that the grid will go down the toilet if everyone is running a crawler bot doesn’t have a strong factual basis. The exception here are directory searches, which have to hit a backend database and doing them en masse will have a real effect on grid load. So comparing landbots, or people that refresh the for sale listings every few seconds to a search engine crawler is not apples to apples.

Another tough problem to deal with is that you can’t ban bad bots. Banning bots doesn’t work in World of Warcraft (even though they spend countless hours and dollars trying) and it can’t work here. The protocol is completely documented, the client is open source, the lines are constantly blurring between enhanced clients and fully automated bots. Having code signed off by an authority, only allowing bots to create opt-in systems, etc. is a nice idea and would keep honest people honest, but that’s it. If someone is pulling in $1000 a week camping or running a landbot and you say “hey you can’t run that code because it doesn’t meet community standards”, they probably won’t stop running the code. They typically run on transient rented servers so IP banning is out, accounts in SL take seconds to create and even if you charge for them they are making more than enough money to buy all the accounts they need, it’s not something you can solve with community standards. The only long term solution is to harden the grid: rate limit certain actions, and make sure that no individual can put an undue burden on the system. Easier said than done though, in Second Life and the Internet as a whole.

]]> By: firestarter http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-14442 firestarter Fri, 13 Apr 2007 03:43:15 +0000 http://www.reznation.com/?p=106#comment-14442 urgen bergen furgen een beta verjun. . . und rock with fire! great discussion. urgen bergen furgen een beta verjun. . . und rock with fire! great discussion.

]]> By: Search engine voor Second Life | MarcNext http://www.reznation.com/2007/04/secondcast-57-search-furtherment/comment-page-1/#comment-14400 Search engine voor Second Life | MarcNext Thu, 12 Apr 2007 23:08:08 +0000 http://www.reznation.com/?p=106#comment-14400 [...] Sheep Labs heeft een beta-versie van een search engine voor Second Life gelanceerd. Het is allemaal nog in een beginnend stadium, echt uitgebreid is het aantal zoekresultaten nog niet. Op het moment van schrijven bevat de database ruim 800.000 SL-producten. De lancering heeft een behoorlijke discussie in de SL-blogosphere teweeg gebracht, ondermeer op het gebied van landrechten in SL. In de nieuwste Secondcast van Rez Nation wordt op de mogelijkheden van de engine ingegaan. [...] [...] Sheep Labs heeft een beta-versie van een search engine voor Second Life gelanceerd. Het is allemaal nog in een beginnend stadium, echt uitgebreid is het aantal zoekresultaten nog niet. Op het moment van schrijven bevat de database ruim 800.000 SL-producten. De lancering heeft een behoorlijke discussie in de SL-blogosphere teweeg gebracht, ondermeer op het gebied van landrechten in SL. In de nieuwste Secondcast van Rez Nation wordt op de mogelijkheden van de engine ingegaan. [...]

]]>