I've written a sh script file on HPUX to check the connections to ftp
servers. When I first used ftp command without input redirection inside
a while loop shown as following, the while loop executed once and
exited. Later, some experts told me to use for loop instead of while or
redirect the input of ftp command from /dev/null in while loop. Those
changes worked.

Could you talk about /dev/null? Why is it so magic, is it a keyword?

grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}' | \
while read host ip user passwd; do
ftp -n -v "$host" </dev/null 2>&1 | \
grep "Connected to $host"
done

[ Post a follow-up to this message ]

lovecreatesbea...@gmail.com <lovecreatesbeauty@gmail.com> wrote:
> Could you talk about /dev/null? Why is it so magic, is it a keyword?
>
>    grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}' | \
>    while read host ip user passwd; do
>        ftp -n -v "$host" </dev/null 2>&1 | \
>        grep "Connected to $host"
>    done

ftp eats up your stdin. So ftp reads the remaining lines of the
grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}'
output. By redirecting stdin from /dev/null just for ftp, it cannot
mess up your global stdin any more.

--
Daniel

[ Post a follow-up to this message ]

"lovecreatesbea...@gmail.com" <lovecreatesbeauty@gmail.com> writes:

> Could you talk about /dev/null? Why is it so magic, is it a keyword?

type
man null

to learn about this. The file /dev/null shold always exist on a Unix system,
is always readable, and is always empty.


If you write to it, the write will succeed, and the data will be discarded.

It's used to create empty data and discard data.


Therefore when you use
ftp ..... </dev/null

the standard input will be connected to /dev/null, and if ftp reads
any data - it will get an end-of-file. It will then exit as there is no
more input.

Without it, the FTP program might read from your terminal, and wait
until the file is closed. Therefore it might not terminate.


--
Sending unsolicited commercial e-mail to this account incurs a fee of
$500 per message, and acknowledges the legality of this contract.

[ Post a follow-up to this message ]

lovecreatesbea...@gmail.com wrote:

> I've written a sh script file on HPUX to check the connections to ftp
> servers. When I first used ftp command without input redirection inside
> a while loop shown as following, the while loop executed once and
> exited. Later, some experts told me to use for loop instead of while or
> redirect the input of ftp command from /dev/null in while loop. Those
> changes worked.
>
> Could you talk about /dev/null? Why is it so magic, is it a keyword?
>
>     grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}' | \
>     while read host ip user passwd; do
>         ftp -n -v "$host" </dev/null 2>&1 | \
>         grep "Connected to $host"
>     done
>

You already got your question answered, but FYI you don't need to use
grep and awk since awk can search for or exclude patterns too. Also, the
default field separate for awk is a space, so you don't need to set it
to space. So, in this case, instead of:

grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}' |

all you need is:

awk 'NF{print $1, $2, $3, $4}' "$hosts" |

Regards,

Ed.

[ Post a follow-up to this message ]

On 2007-01-17, Ed Morton wrote:
> lovecreatesbea...@gmail.com wrote:
> 
>
> You already got your question answered, but FYI you don't need to use
> grep and awk since awk can search for or exclude patterns too.

Though you don't need grep, it may be faster to use grep as well as
awk. As the designers of the language write in "The AWK Programming
Language":

"If you have to search a big file to isolate a small
amount of data, use grep or egrep for the searching and
awk for the processing. If there are a large number of
substitutions [...], you might use a stream editor like
sed for that part. In other words, break the job into
separate pieces, and apply the most appropriate tool to
each piece."

Of all the varieties of awk that I have tried, only mawk can search
through a file as quickly as grep.

> Also, the
> default field separate for awk is a space, so you don't need to set it
> to space. So, in this case, instead of:
>
>     grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}' |
>
> all you need is:
>
>     awk 'NF{print $1, $2, $3, $4}' "$hosts" |


--
Chris F.A. Johnson, author        <http://cfaj.freeshell.org/shell>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence

[ Post a follow-up to this message ]

Chris F.A. Johnson wrote:

> On 2007-01-17, Ed Morton wrote:
> 
>
>
>    Though you don't need grep, it may be faster to use grep as well as
>    awk.

It may be faster still to write the whole thing in C, but that doesn't
mean it's a good idea to make that your initial approach. Keep it
simple. Optimise later IFF necessary.

> As the designers of the language write in "The AWK Programming
>    Language":
>              "If you have to search a big file to isolate a small
>              amount of data, use grep or egrep for the searching and
>              awk for the processing. If there are a large number of
>              substitutions [...], you might use a stream editor like
>              sed for that part. In other words, break the job into
>              separate pieces, and apply the most appropriate tool to
>              each piece."

Reasonable advice IF you ever actually came across a need to speed up
your program.

>    Of all the varieties of awk that I have tried, only mawk can search
>    through a file as quickly as grep.

Of all the varieties of awk that I have tried, I've never come across a
need to speed any of them up by preprocessing the input with grep.

Ed.

[ Post a follow-up to this message ]

Ed Morton wrote:

>
> Of all the varieties of awk that I have tried, I've never come across a
> need to speed any of them up by preprocessing the input with grep.
>
>     Ed.

At work I used egrep as a prefilter to nawk.  The reason: I had a very
large data file with hundreds of thousands of records, but my nawk
script was only interested in about ten percent of them.  [The records
were "variable  value", and I needed a very small subset of them.]

So by using egrep "^(var1|var2...)" file | nawk ...
the process was significantly faster.  I believe the reason is that the
nawk script didn't have to parse all the records just to ignore ninety
percent of them.

For one particular file, without egrep, it took 42 wall-clock minutes to
complete, but with the egrep, the time was about 5 minutes, IIRC.

Bill Seivert

[ Post a follow-up to this message ]

Bill Seivert wrote:
> Ed Morton wrote:
> 
>
>
> At work I used egrep as a prefilter to nawk.  The reason: I had a very
> large data file with hundreds of thousands of records, but my nawk
> script was only interested in about ten percent of them.  [The records
> were "variable  value", and I needed a very small subset of them.]
>
> So by using egrep "^(var1|var2...)" file | nawk ...
> the process was significantly faster.  I believe the reason is that the
> nawk script didn't have to parse all the records just to ignore ninety
> percent of them.
>
> For one particular file, without egrep, it took 42 wall-clock minutes to
> complete, but with the egrep, the time was about 5 minutes, IIRC.
>
> Bill Seivert

Maybe the filtering part of the nawk script was poorly written. For a
100000 line file of the form:

$ head -10 f100000

All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
All work and no play makes Jack a dull boy.
This is the 10th line.

where the number 10 appears only on every tenth line and I want to
process that line (i.e. 10% of the file), here's the "time" result for
the 4th consecutive execution of the specified command:

-----------
$ time awk '/10/{ for (i=1;i<=NF;i++) v[i]=$i }' f100000

real    0m0.470s
user    0m0.421s
sys     0m0.078s

--------------
$ time grep 10 f100000 | awk '{ for (i=1;i<=NF;i++) v[i]=$i }'

real    0m0.466s
user    0m0.420s
sys     0m0.092s
--------------

I'm not saying it's impossible for preprocessing with grep to produce
faster results, it just doesn't seem like it's worth worrying about
unless you have a specific problem with your awk scripts performance.

Regards,

Ed.

[ Post a follow-up to this message ]

"lovecreatesbea...@gmail.com" <lovecreatesbeauty@gmail.com> writes:
>     grep -v ^$ "$hosts" | awk -F" " '{print $1, $2, $3, $4}' | \
>     while read host ip user passwd; do
>         ftp -n -v "$host" </dev/null 2>&1 | \
>         grep "Connected to $host"
>     done

It was pointed in another thread that you don't need grep.  I'll go
one step further and say that you don't need awk either:

#v+
while read host ip user passwd dummy; do
[ -n "$host" ] || continue
ftp -n -v "$host" </dev/null 2>&1 | \
grep "Connected to $host"
done <"$hosts"
#v-

Not tested but I'm pretty sure it'll work on any POSIX shell.

--
Best regards,                                         _     _
.o. | Liege of Serenly Enlightened Majesty of      o' \,=./ `o
.o | Computer Science,  Michal "mina86" Nazarewicz   (o o)
ooo +--<mina86*tlen.pl>---<jid:mina86*chrome.pl>--ooO--(_)--Ooo--

[ Post a follow-up to this message ]