|
Home > Archive > Unix Shell > November 2006 > How do I send stdout to file and stderr to both file & tty?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
How do I send stdout to file and stderr to both file & tty?
|
|
| paintedjazz@gmail.com 2006-11-29, 7:22 pm |
| I use:
bash -x some_script > some_log 2>&1
but I don't want stderr going to the log file only.
How can I make stderr go to the log file and to the screen?
Is it possible?
Thx.
| |
| Stephane CHAZELAS 2006-11-29, 7:22 pm |
| 2006-11-29, 12:37(-08), paintedjazz@gmail.com:
> I use:
>
> bash -x some_script > some_log 2>&1
>
> but I don't want stderr going to the log file only.
>
> How can I make stderr go to the log file and to the screen?
> Is it possible?
[...]
With zsh:
bash -x some_script 2>&2 >& some_log
With other shells (including zsh):
: > some_log
bash -x some_script 2>&1 >> some_log | tee -a some_log >&2
--
Stéphane
| |
| Stephane CHAZELAS 2006-11-29, 7:22 pm |
| 2006-11-29, 20:47(+00), Stephane CHAZELAS:
> 2006-11-29, 12:37(-08), paintedjazz@gmail.com:
> [...]
>
> With zsh:
>
> bash -x some_script 2>&2 >& some_log
>
> With other shells (including zsh):
>
> : > some_log
> bash -x some_script 2>&1 >> some_log | tee -a some_log >&2
Or if your system has /dev/fd/<n>
{ bash -x some_script 2>&1 >&3 3>&- | tee /dev/fd/2 >&3 3>&-; } 3> some_log
--
Stéphane
| |
| paintedjazz@gmail.com 2006-11-30, 1:33 am |
|
Stephane CHAZELAS wrote:
> 2006-11-29, 20:47(+00), Stephane CHAZELAS:
>
> Or if your system has /dev/fd/<n>
>
> { bash -x some_script 2>&1 >&3 3>&- | tee /dev/fd/2 >&3 3>&-; } 3> some_log
Hi Stephane,
Many thanks but both of these commands don't quite do what I was hoping
for
since they both send stdout to the screen and I only want stdout to go
to the log file.
Stderr, on the other hand, should go to both or just the screen. Sorry
if I failed
to make that clear.
$ bash --version
GNU bash, version 2.05b.0(1)-release (powerpc-apple-darwin8.0)
Copyright (C) 2002 Free Software Foundation, Inc.
$ uname -sr
Darwin 8.8.0
Thx.
| |
| Kaz Kylheku 2006-11-30, 1:33 am |
| paintedjazz@gmail.com wrote:
> Many thanks but both of these commands don't quite do what I was hoping
> for
> since they both send stdout to the screen and I only want stdout to go
> to the log file.
> Stderr, on the other hand, should go to both or just the screen. Sorry
> if I failed
> to make that clear.
Simply put, redirect all verbiage into the file, but show the important
verbiage (error output) on standard output also, so that it shows up on
the tty, or can be separately redirected to another file if necessary.
In order to meet this requirement, is is necessary to open two separate
file descriptors, and connect them both to the log file. So that they
don't overwrite each other's output, they have to work in append mode.
Here is how:
The program's standard output needs to be duplicated into the standard
error position, so that error goes to the output.
First, we need to swap standard input and output. This has to be done
by a rotation through three descriptor positions:
command 3>&1 1>&2 2>&3 # x = y; y = z; z = x
Now, standard out is going to standard error, and standard error is
going to standard out!
So next, redirect standard error (which contains the output) into the
file, using append >>
command 2>> logfile 3>&1 1>&2 2>&3
Finally, use tee to also append the standard output (which contains
errors) to the logfile and show it as well:
command 2>> logfile 3>&1 1>&2 2>&3 | tee -a logfile
(We could use 2>&3- to close 3 after moving it to 2, to be a little
less sloppy).
Ah well, doesn't work for me.
The shell's >> and tee -a are not actually using proper append mode on
the file descriptor, and so are racing with each other to overwrite the
output. Apparently, append means open the file normally, but seek to
the end before writing.
It was worth a shot.
It feels liberating to just give up.
By the way, which one of the two is the culprit? I bet you it's tee.
I'm using bash. The documentation says that >> opens in append mode. We
can freaking prove it. One would be to run strace on bash while it runs
a script that appends, see if the O_APPEND mode is used. But that would
be boring, so let's be empiricists:
append ()
{
local i=0;
exec 3>> logfile
while [ $i -lt $1 ]; do
echo $2_$i >3;
i=$((i + 1));
done
}
append 10000 a & append 10000 b
In my resulting logfile, not a damn thing is missing. There are 10000
messages from both a, and b, with no missing numbers, and the thing is
consequently 22000 lines long.
It's that tee not appending like it should, piece of shit! So we have
to drop it from consideration, and declare ourselves stuck.
No wait, can we /write/ our own tee using the shell? You bet. All it
has to do is read standard input, which is easy: while read line ; do
.... done. And each line has to go to a file, and to standard output.
That's easy, we just need logfile opened in append mode, and whee,
right?
command 2>>logfile 3>&1 1>&2 2>&3 | (exec 3>>logfile ; while read line
; do echo $line ; echo $line >3 ; done )
The (...) stuff should be put into a function.
Anyway, one finishing touch: create the error log at the beginning, or
truncate existing one to zero:
> logfile ; command 2>> ...
Not sure how portable that all is, but it seems to work with bash
3.00.16 on a Fedora Core 4 Linux box.
| |
| Kaz Kylheku 2006-11-30, 1:33 am |
| paintedjazz@gmail.com wrote:
> Many thanks but both of these commands don't quite do what I was hoping
> for
> since they both send stdout to the screen and I only want stdout to go
> to the log file.
> Stderr, on the other hand, should go to both or just the screen. Sorry
> if I failed
> to make that clear.
Simply put, redirect all verbiage into the file, but show the important
verbiage (error output) on standard output also, so that it shows up on
the tty, or can be separately redirected to another file if necessary.
In order to meet this requirement, is is necessary to open two separate
file descriptors, and connect them both to the log file. So that they
don't overwrite each other's output, they have to work in append mode.
Here is how:
The program's standard output needs to be duplicated into the standard
error position, so that error goes to the output.
First, we need to swap standard input and output. This has to be done
by a rotation through three descriptor positions:
command 3>&1 1>&2 2>&3 # x = y; y = z; z = x
Now, standard out is going to standard error, and standard error is
going to standard out!
So next, redirect standard error (which contains the output) into the
file, using append >>
command 2>> logfile 3>&1 1>&2 2>&3
Finally, use tee to also append the standard output (which contains
errors) to the logfile and show it as well:
command 2>> logfile 3>&1 1>&2 2>&3 | tee -a logfile
(We could use 2>&3- to close 3 after moving it to 2, to be a little
less sloppy).
Ah well, doesn't work for me.
The shell's >> and tee -a are not actually using proper append mode on
the file descriptor, and so are racing with each other to overwrite the
output. Apparently, append means open the file normally, but seek to
the end before writing.
It was worth a shot.
It feels liberating to just give up.
By the way, which one of the two is the culprit? I bet you it's tee.
I'm using bash. The documentation says that >> opens in append mode. We
can freaking prove it. One would be to run strace on bash while it runs
a script that appends, see if the O_APPEND mode is used. But that would
be boring, so let's be empiricists:
append ()
{
local i=0;
exec 3>> logfile
while [ $i -lt $1 ]; do
echo $2_$i >3;
i=$((i + 1));
done
}
append 10000 a & append 10000 b
In my resulting logfile, not a damn thing is missing. There are 10000
messages from both a, and b, with no missing numbers, and the thing is
consequently 22000 lines long.
It's that tee not appending like it should, piece of shit! So we have
to drop it from consideration, and declare ourselves stuck.
No wait, can we /write/ our own tee using the shell? You bet. All it
has to do is read standard input, which is easy: while read line ; do
.... done. And each line has to go to a file, and to standard output.
That's easy, we just need logfile opened in append mode, and whee,
right?
command 2>>logfile 3>&1 1>&2 2>&3 | (exec 3>>logfile ; while read line
; do echo $line ; echo $line >3 ; done )
Hmm, this doesn't quite seem to work for me, but this does:
command 2>>logfile 3>&1 1>&2 2>&3 | while read line ; do echo $line ;
echo $line >logfile ; done
Anyway, one finishing touch: create the error log at the beginning, or
truncate existing one to zero:
> logfile ; command 2>> logfile ... etc
Not sure how portable that all is, but it seems to work with bash
3.00.16 on a Fedora Core 4 Linux box.
| |
| Stephane CHAZELAS 2006-11-30, 7:27 am |
| 2006-11-29, 18:48(-08), paintedjazz@gmail.com:
[...]]
>
> Hi Stephane,
>
> Many thanks but both of these commands don't quite do what I was hoping
> for
> since they both send stdout to the screen and I only want stdout to go
> to the log file.
> Stderr, on the other hand, should go to both or just the screen. Sorry
> if I failed
> to make that clear.
[...]
~$ { bash -xc 'echo foo' 2>&1 >&3 3>&- | tee /dev/fd/2 >&3 3>&-; } 3> some_log
+ echo foo
~$ cat some_log
foo
+ echo foo
Seems to do what you asked for. What did you try?
--
Stéphane
| |
| Stephane CHAZELAS 2006-11-30, 7:27 am |
| 2006-11-29, 22:42(-08), Kaz Kylheku:
> paintedjazz@gmail.com wrote:
[...]
> First, we need to swap standard input and output. This has to be done
> by a rotation through three descriptor positions:
ITYM *error* and output.
>
> command 3>&1 1>&2 2>&3 # x = y; y = z; z = x
>
> Now, standard out is going to standard error, and standard error is
> going to standard out!
>
> So next, redirect standard error (which contains the output) into the
> file, using append >>
>
> command 2>> logfile 3>&1 1>&2 2>&3
Ok, so you now have command's stdout going to logfile and stderr
going to whatever was stdout before. That's the same as
command 2>&1 >> logfile
> Finally, use tee to also append the standard output (which contains
> errors) to the logfile and show it as well:
>
> command 2>> logfile 3>&1 1>&2 2>&3 | tee -a logfile
Same as:
command 2>&1 >> logfile | tee -a logfile
> (We could use 2>&3- to close 3 after moving it to 2, to be a little
> less sloppy).
ITYM 3>&-
> Ah well, doesn't work for me.
Works for me.
> The shell's >> and tee -a are not actually using proper append mode on
> the file descriptor, and so are racing with each other to overwrite the
> output. Apparently, append means open the file normally, but seek to
> the end before writing.
They shouldn't. Only the Bourne shell >> is behaving as you
describe. But standard shs and tee should not.
--
Stéphane
| |
| Stephane CHAZELAS 2006-11-30, 7:27 am |
| 2006-11-29, 22:42(-08), Kaz Kylheku:
[...]
> command 2>>logfile 3>&1 1>&2 2>&3 | (exec 3>>logfile ; while read line
> ; do echo $line ; echo $line >3 ; done )
[...]
Should be:
{
while IFS= read -r line; do
printf '%s\n' "$line"
printf '%s\n' "$line" >&3
done
printf %s "$line"
printf %s "$line" >&3
} 3>> logfile
And it would still not be equivalent to "tee" (except if the
shell is zsh) as tee should be able to deal with non-text files
(the NUL character).
Another tee -a implementation:
awk '{print; print >> "logfile"}'
same problem with non-text files, except for some versions of
GNU awk where you could do:
awk '{printf "%s", $0 RT; printf "%s", $0 RT >> "logfile"}'
and awk may buffer its output which tee is not supposed to do.
--
Stéphane
|
|
|
|
|