Unix questions - help with bash script

Author

help with bash script

Martin Jørgensen

2005-10-03, 2:50 am

Hello,

I have 2 problems on suse 9.3 and I'm not experienced enough to be able
to solve the problems myself:

1)
I need a (bash preferably) script which can delete all sub-directories
except those named "0", "constant", "system" and "data". So I'll
probably need something with a pipe but what causes me problems is that
I don't know how to exclude these directories from removal.

I was thinking about something with "xargs rm -r" in combination with
"find" perhaps?

2)
I need to extract data from the file called result.dat.txt from within
each subdirectory and only one line (for instance line nr. 20 from the
top of the file) from within each result.dat.txt-file, should be put
into (and appended) to a file: "big_result.dat.txt". How would I do this?

The directories are called something like (0, 0.5, 1, 1.5, 2, 2.5 3...
or similar), i.e. starting from zero and with an interval the number of
the directory gets bigger. So I assume there's no need for a
sort-command, since I think the directories are already in the correct
order when they were written.

Thanks in advance for anyone who can help me or at least lead me closer
to a solution.

Med venlig hilsen / Best regards
Martin Jørgensen

--
---------------------------------------------------------------------------
Home of Martin Jørgensen - http://www.martinjoergensen.dk

Chris F.A. Johnson

2005-10-03, 2:50 am

On 2005-10-03, Martin Jørgensen wrote:
> Hello,
>
> I have 2 problems on suse 9.3 and I'm not experienced enough to be able
> to solve the problems myself:
>
> 1)
> I need a (bash preferably) script which can delete all sub-directories
> except those named "0", "constant", "system" and "data". So I'll
> probably need something with a pipe but what causes me problems is that
> I don't know how to exclude these directories from removal.
>
> I was thinking about something with "xargs rm -r" in combination with
> "find" perhaps?

Do you want to remove direcories through an entire hierarchy, or
just subdirectories directly below the current (or other)
directory?

If the latter:

for dir in */
do
case $dir in
0|constant|system|data) ;;
*) rm -rf "$dir" ;;
esac
done

If you want to remove them throughout a hierarchy, do you want to
remove only the end nodes of the tree, or the entire tree? Do you
want to keep higher level directories which contain one or more of
"0", "constant", "system" and "data"?

> 2)
> I need to extract data from the file called result.dat.txt from within
> each subdirectory and only one line (for instance line nr. 20 from the
> top of the file) from within each result.dat.txt-file, should be put
> into (and appended) to a file: "big_result.dat.txt". How would I do this?
>
> The directories are called something like (0, 0.5, 1, 1.5, 2, 2.5 3...
> or similar), i.e. starting from zero and with an interval the number of
> the directory gets bigger. So I assume there's no need for a
> sort-command, since I think the directories are already in the correct
> order when they were written.

If there are not too many files:

awk 'FNR == 20' $(find */ -name result.dat.txt -print) > big_result.dat.txt

If that gives you an "argument list too long" error:

find */ -name result.dat.txt -print) | xargs awk 'FNR == 20' > big_result.dat.txt

--
Chris F.A. Johnson <http://cfaj.freeshell.org>
========================================
==========================
Shell Scripting Recipes: A Problem-Solution Approach, 2005, Apress
<http://www.torfree.net/~chris/books/cfaj/ssr.html>

Martin Jørgensen

2005-10-03, 6:04 pm

Chris F.A. Johnson wrote:
> On 2005-10-03, Martin Jørgensen wrote:
-snip-

> Do you want to remove direcories through an entire hierarchy, or
> just subdirectories directly below the current (or other)
> directory?

Not sure if I understand the difference. All subdirectories except "0",
"constant", "system" and "data" should be removed in the current pwd.
Those 4 directories are only in the current directory (so I think it's
the latter).

i.e.

/home/martin/result/ - delete everything except those 4 subdirectories
should remain.

> If the latter:
>
> for dir in */
> do
> case $dir in
> 0|constant|system|data) ;;
> *) rm -rf "$dir" ;;
> esac
> done

Ok, so I just store these lines in a file called .script in
/home/martin/result and do a chmod a+x followed by a ./script which does
the trick on subdirectories below /home/martin/result?

> If you want to remove them throughout a hierarchy, do you want to
> remove only the end nodes of the tree, or the entire tree? Do you

Entire tree, I think (but the depth is only one level I think).

> want to keep higher level directories which contain one or more of
> "0", "constant", "system" and "data"?

Those 4 are only placed in /home/martin/result - not in the
subdirectories. They shouldn't be deleted.

>
>
> If there are not too many files:
>
> awk 'FNR == 20' $(find */ -name result.dat.txt -print) > big_result.dat.txt
>
> If that gives you an "argument list too long" error:
>
> find */ -name result.dat.txt -print) | xargs awk 'FNR == 20' > big_result.dat.txt

Okay, so that'll dig into each subdirectory and find the files and
extract the lines? I'll have a look and do some testing this weekend to
see if it works.

It seems like you're pretty experienced with making scripts :-)

Med venlig hilsen / Best regards
Martin Jørgensen

--
---------------------------------------------------------------------------
Home of Martin Jørgensen - http://www.martinjoergensen.dk

Bit Twister

2005-10-03, 6:04 pm

On Mon, 03 Oct 2005 21:58:29 +0200, Martin Jørgensen wrote:
>
> Ok, so I just store these lines in a file called .script in

Or just script instead of .script

> /home/martin/result and do a chmod a+x followed by a ./script which does
> the trick on subdirectories below /home/martin/result?

See, it would be ./.script instead of ./script

> Entire tree, I think (but the depth is only one level I think).
>
> Those 4 are only placed in /home/martin/result - not in the
> subdirectories. They shouldn't be deleted.

There shoul be no THINKING, you have to KNOW, when playing with
destructive commands. When in doubt, use ls instead of the actual
command or maybe a printf/echo command to see what will happen.

You can do a
mkdir -p ~-/tmp/dir1
mkdir -p ~-/tmp/dir2
mkdir -p ~-/tmp/dir3

touch ~-/tmp/dir1/fn1
touch ~-/tmp/dir1/fn2
..
..
..
touch ~-/tmp/dir3/fn3

to create test case directories and files.

Peter T. Breuer

2005-10-03, 6:04 pm

In comp.os.linux.misc Martin Joergensen <unoder.spam@spam.jay.net> wrote:
> Chris F.A. Johnson wrote:
[vbcol=seagreen]
> Not sure if I understand the difference.

That's a problem - what do you mean by "subdirectory"? Do you mean
those that are in the current directory, or those lower down? And what
about THEIR subdirectories?

> All subdirectories except "0",
> "constant", "system" and "data" should be removed in the current pwd.
> Those 4 directories are only in the current directory (so I think it's
> the latter).

Oh - then what's the problem ...

> /home/martin/result/ - delete everything except those 4 subdirectories
> should remain.
[vbcol=seagreen]

This is one of the most obvious and trivial ways. Clearly you can also
do it with find

find . -maxdepth 1 \
$ -name "0" \
-o -name "constant" \
-o -name "system" \
-o -name "data" \
$ -prune \
-o -type d -exec rm -fr {} \;

or something like that (run with echo, not rm, first).

Assuming that that is what you MEANT, of course.

Peter

Chris F.A. Johnson

2005-10-03, 6:04 pm

On 2005-10-03, Martin Jørgensen wrote:
> Chris F.A. Johnson wrote:
> -snip-
>
>
> Not sure if I understand the difference. All subdirectories except "0",
> "constant", "system" and "data" should be removed in the current pwd.
> Those 4 directories are only in the current directory (so I think it's
> the latter).
>
> i.e.
>
> /home/martin/result/ - delete everything except those 4 subdirectories
> should remain.
>
>
> Ok, so I just store these lines in a file called .script in
> /home/martin/result and do a chmod a+x followed by a ./script which does
> the trick on subdirectories below /home/martin/result?

I wouldn't; I'd put it in a directory in your PATH, such as
$HOME/bin, then you can call it from any directory you like.

>
> Entire tree, I think (but the depth is only one level I think).
>
>
> Those 4 are only placed in /home/martin/result - not in the
> subdirectories. They shouldn't be deleted.

Then the above script should suffice.

There's an extra parenthesis there; it should be:

find */ -name result.dat.txt -print | xargs awk 'FNR == 20' > big_result.dat.txt
[vbcol=seagreen]
> Okay, so that'll dig into each subdirectory and find the files and
> extract the lines? I'll have a look and do some testing this weekend to
> see if it works.
>
> It seems like you're pretty experienced with making scripts :-)

I've written a few. You can find a number of examples on my web
site, and even more in my book. See the URLs in my .sig.

--
Chris F.A. Johnson <http://cfaj.freeshell.org>
========================================
==========================
Shell Scripting Recipes: A Problem-Solution Approach, 2005, Apress
<http://www.torfree.net/~chris/books/cfaj/ssr.html>

Martin Jørgensen

2005-10-04, 5:56 pm

Peter T. Breuer wrote:
> In comp.os.linux.misc Martin Joergensen <unoder.spam@spam.jay.net> wrote:
-snip-

> Oh - then what's the problem ...

The problem was/is that I'm unexperienced with making bash scripts and
the like, but I think I got so much help that I should be able to put
something together...

>
>
>
>
>
> This is one of the most obvious and trivial ways. Clearly you can also
> do it with find
>
> find . -maxdepth 1 \
> $ -name "0" \
> -o -name "constant" \
> -o -name "system" \
> -o -name "data" \
> $ -prune \
> -o -type d -exec rm -fr {} \;
>
> or something like that (run with echo, not rm, first).
>
> Assuming that that is what you MEANT, of course.

That looks exactly like what I meant, but I'm pretty hooked up at the
moment. If it doesn't work correctly and if I can't make it work I'll
come back here. However I think I got so many good answers that there
should be a good chance of getting it to work.

Thanks a lot for the help - this weekend I'll look at it.

Med venlig hilsen / Best regards
Martin Jørgensen

--
---------------------------------------------------------------------------
Home of Martin Jørgensen - http://www.martinjoergensen.dk

Martin Jørgensen

2005-10-04, 5:56 pm

Chris F.A. Johnson wrote:
-snip-

>
>
> There's an extra parenthesis there; it should be:
>
> find */ -name result.dat.txt -print | xargs awk 'FNR == 20' > big_result.dat.txt

Thanks for the comment.

I didn't test it yet, though, since I suddenly got a lot of extra work
to do but I'll test it in a few days.

>
>
> I've written a few. You can find a number of examples on my web
> site, and even more in my book. See the URLs in my .sig.

I'll do that. I appreciate the help/comments.

Med venlig hilsen / Best regards
Martin Jørgensen

--
---------------------------------------------------------------------------
Home of Martin Jørgensen - http://www.martinjoergensen.dk

Martin Jørgensen

2005-10-04, 5:56 pm

Bit Twister wrote:
> On Mon, 03 Oct 2005 21:58:29 +0200, Martin Jørgensen wrote:
-snip-

> You can do a
> mkdir -p ~-/tmp/dir1
> mkdir -p ~-/tmp/dir2
> mkdir -p ~-/tmp/dir3
>
> touch ~-/tmp/dir1/fn1
> touch ~-/tmp/dir1/fn2
> .
> .
> .
> touch ~-/tmp/dir3/fn3
>
> to create test case directories and files.

Yep, rgr...

Med venlig hilsen / Best regards
Martin Jørgensen

--
---------------------------------------------------------------------------
Home of Martin Jørgensen - http://www.martinjoergensen.dk