I am relatively new to unix, and was wondering if there is a faster
way to search through text files.

I am currently searching through various text files that are usually
200 to 800 MB.  I am using grep, but was wonder if there were any
other utilities that would speed up the search, i.e. are there other
utilities that do a better job than grep?

Thanks,

Brent

[ Post a follow-up to this message ]

In article <3f7fedd0.0410211220.72154d@posting.google.com>,
brent.omalley@pegs.com (Brent OMalley) wrote:

> I am relatively new to unix, and was wondering if there is a faster
> way to search through text files.
>
> I am currently searching through various text files that are usually
> 200 to 800 MB.  I am using grep, but was wonder if there were any
> other utilities that would speed up the search, i.e. are there other
> utilities that do a better job than grep?

The distinction between grep and fgrep has all but gone away, but in the
old days, the response would be "use fgrep, it's faster".  If you're not
using a regex, fgrep will do just about the same job as grep.

If you're constrained by disk I/O or CPU when you search, the standard
wisdom would be to move the files to a faster disk (i.e a RAM disk) or
buy a faster CPU.  Note that SMP probably won't help here as grep is a
single-threaded task.

Or you could feed the text into an indexing program such as those used
by various web sites:

http://www.verity.com/products/ultraseek/cce.html
http://docs.rinet.ru/Nt4Web/ch15.htm
http://www.asindexing.org/site/webndx.shtml
http://homepage.mac.com/pauljlucas/software/swish/
http://swish-e.org/

Or you could write a program that reads the file into a mmap'd file and
search it ala GNU grep.

--
DeeDee, don't press that button!  DeeDee!  NO!  Dee...

[ Post a follow-up to this message ]

On 21 Oct 2004 13:20:48 -0700, Brent OMalley
<brent.omalley@pegs.com> wrote:
> I am relatively new to unix, and was wondering if there is a faster
> way to search through text files.
>
> I am currently searching through various text files that are usually
> 200 to 800 MB.  I am using grep, but was wonder if there were any
> other utilities that would speed up the search, i.e. are there other
> utilities that do a better job than grep?
>
Newer versions of Unix and Linux often use the UTF-8 character set by
default.  If your text files are written in multiple languages you may need
UTF-8, but otherwise grep can run much faster using ASCII (LANG=C) or an
8-bit character set (LANG=en_US.iso-8859-15).


--
aptitude is also Y2K-compliant, non-fattening, naturally cleansing, and
housebroken.

[ Post a follow-up to this message ]

Brent OMalley <brent.omalley@pegs.com> wrote:
> I am relatively new to unix, and was wondering if there is a faster
> way to search through text files.

> I am currently searching through various text files that are usually
> 200 to 800 MB.  I am using grep, but was wonder if there were any
> other utilities that would speed up the search, i.e. are there other
> utilities that do a better job than grep?

glimps might be a tool for static or relativly static data.


--
Peter Håkanson
IPSec  Sverige      ( At Gothenburg Riverside )
Sorry about my e-mail address, but i'm trying to keep spam out,
remove "icke-reklam" if you feel for mailing me. Thanx.

[ Post a follow-up to this message ]