Apologies if this has been threaded before, but I can't tell because there's no SEARCH SITE!

Can we please have a search function?

ben seems to be ignoring us on this one

Well, I've got to figure some things out first, like how to make it work well with so little memory in the machine, and so many posts in the database...

posts in the database: 1,075,053
size of search index: over 600 meg
free memory on server: less than 500 meg

this is a big part of the problem. I might end up with the search only indexing the posts from the last 30 days, or ??

Unfortunately, we aren't google

I don't guess there's a way to give that process a very low thread priority and then just have it crawl through the database? I've never tried such a thing, so I dunno...

You might run a script that creates a "condensed" version of every thread, that is nothing but extracted words with no duplicates (especially dropping all those quotes), and perform the search against this smaller dataset. Thereafter, new threads might go through this "condenser" on a weekly schedule...

Eh...tough problem. Just tossing up some weak thoughts.

Ben, many thanks for your reply and concerns, and for running prince.org.

OK, but the problem will get worse without a search as you no doubt have thought yourself already. We will get duplication of threads because we can't search for existing ones, further aggravating the capacity problems.
I've not managed a server, so forgive me if the following shows my ignorance, but anyway, here goes.

• Limit the search string to the title of the thread topic, this may have the added effect of contributors choosing descriptive titles for threads (could also produce a reverse effect I guess).
  
• Limit the Search to a particular Forum.
  
• Increase the memory of the server.
  
• Catalogue existing threads to make a searchable archive, limit the live searches, e.g. as you suggested by 30 days (although I'd prefer longer if possible).
  
• Are there search engine modules available to you from developers or php? (It is a php server no?)
  

Whatever is possible and practical, install it soon. Thanks.
[This message was edited Wed Oct 1 15:10:21 PDT 2003 by infinitetrio]

Sorry to keep going on about this an all, any progress?

Teller: it's not the indexing that is the problem, it's the searching/retrieval.

infinitetrio: some good suggestions. I don't think the title-search would be all *that* useful... but it would be a lot faster, I agree.

Definitely I am using search tools, not writing them myself! I use mnogosearch, which is quite nice. Our dataset is just too big. Limiting the search space is a main way to fix it. Maybe make them go back 3 months only would fix it.

If I get some time this weekend I'll try some experiments.

Just a dumb question here...what if you just add gobs of RAM? RAM is cheap.

I don't control the machine, my hosting company does--and 2 main problems: (a) I can only add another 512M before the machine is maxxed out, and (b) they charge me monthly for ram upgrades. Yes, I know it's stupid, but they do, and there isn't much I can do about it. It's not worth another $20/month or whatever, for another half gig. Sigh. If it was a one-time charge, of course I'd have maxxed it out a while back...

...is it nearly a month since my request?



...seek and ye shall NOT find.

infinitetrio said:

Um, did you read what Ben said above?

Yeah we know, a search facility that actually worked would be brilliant. We'd all find it very useful, not least of all myself. However, as Ben has mentioned, the solution is not trivial. If you or anyone knows of a solution that is both free and quick to implement, please let us know.

Otherwise I'm afraid there are no guarantees if and when there will be a resolution. Of course, this site is a free service and our resources (not least of all Ben's free time - and he is the person that manages the code for the site) is limited.

My concern acknowledges this issue is not trivial and you are right, we would all find a search function useful.

Somewhere, there's a thread about a voluntary donation idea via a PayPal button on the org to help fund the site. PayPal only require an account to link the pay button to, you probably know that it's straight forward to set up. I'd prefer it to be a voluntary donation, I'm skint most of the time but have no problem donating what I can if it makes a difference, surely others may feel the same. How about a special appeal for a 'Search Site' function? It seems from what Ben has said that financing is one of the problems, so let's solve that one somehow.

Cheers, see you tomorrow at Pivo Cafe

infinitetrio said:

I, too, would encourage revenue for the org...but Ben says he'll set up the org merchandise area when he gets back to the states...so buy a shirt!

teller said:

Really? Any idea when that will be, ben? I'm actually itching to get a prince.org shirt to show off my nerdiness...

Natsume said:

Seems like around March 2004 at the current rate, although it's possible sooner. My life is a little bit crazy right now

By the way, I *did* spend some effort on trying to reimplement search last week, to no success. Apparently the latest version of mnogosearch is horribly broken for database indexing. It dies with a bunch of nonsensical errors; BUT, I want to upgrade, because there are useful sounding bugfixes between that version, and the version I was using previously. Sigh.

ben said:

Thank ye kindly for the response, nonetheless. Looking forward to the new merch! And good look with sorting out your craziness...

ben said:

Why not use cafepress.com?? Unless your opposed to the use of third party companies Also Google can do a search box thing that searches your site Ive heard it improves google ratings at the same time
[This message was edited Wed Oct 29 11:06:44 PST 2003 by cloud9mission]

cloud9mission said:

Yes, as cloud9 said, you can get a google search box, and the more you google, the better it gets.

Also, you can just use google itself, it works very well.
I just did a google search with the string "prince.org You might run a script that"

("You might run a script that" is just a string of words from teller's post to this thread) and my first result was
www.prince.org/msg/3/63631 (this thread!).

I've been told that google will pretty much find anything, but only if it's been around for a while.

I've no idea how google manages to 'know' about these (meaning I know bugger all about search engines), but it does. I've done a bit of messing about, and google seems to be able to find anything thats been here 2 weeks or longer, any newer that that and it doesn't find it.
[This message was edited Thu Oct 30 6:33:34 PST 2003 by AsylumUtopia]