For my company, we're using a on-the-server search engine to let customers search our site. We're using an internal one to keep them on our site while they search. We've been using ht://dig but I'm not really pleased with the results. While looking around, I'm not finding different search engine projects all that easily. Does anyone have any suggestions for free/OSS search engine software for a LAMP (Linux/Apache/MySQL/PHP) server?
I was afraid of that. I mean, ht://dig works -- but it's just really archaic and a pain to integrate into a PHP-driven site. I've found a project called TSEP (http://tsep.sf.net) which even appears to be worked on regularly! However, I don't think anyone in charge of coding speaks English . . . and it doesn't work on install . . .
As far as buying Google's search applicance, my boss is a ridiculous cheapskate. If it's not free, he's not interested.
If I can get TSEP working, I'll share my results here.
Where are you trying to put the data? You could set something up with wget to do text files, but it would probably be very messy. You'd have to parse through the html page of the web page to get to your data.