|
Home > Archive > Unix Shell > October 2006 > piping multiple selections to mv or cp and exclusion
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
piping multiple selections to mv or cp and exclusion
|
|
|
| Hello,
Is there a common way to select a group of files and not others and
pipe them to mv or cp?
With gui, it would be selecting the needed files and dragging them to a
directory.
Does the shell offer something as quick?
The only ways I've found were loops or find -exec which I don't
remember off hand.
For a group of files with the same file format, say text, I tried
ls *txt | grep -v 'fileIDontWant'
but how do I pipe the stdout to mv or cp?
For a group of files with the different file formats, I'm at a lost as
to 'finding' the files I want as grep -v 'fileA' 'fileB' doesn't
exclude fileA and fileB
I'm slowly starting to learn unix, a sprinkle of sed, awk, grep,
regexp, etc so please bear with me.
Unfortunately we use windows at work so I'm using ports of unix tools:
unxutils and unixkit because cygwin is too big and I'd have to get
permission to install.
Any pointers are much obliged
thanks,
tony
| |
| Spiros Bousbouras 2006-10-21, 1:31 pm |
| tony wrote:
> Hello,
>
> Is there a common way to select a group of files and not others and
> pipe them to mv or cp?
You don't pipe stuff to mv or cp because mv and cp
do not read from stdin. What you want is to supply
certain files as *arguments* to mv or cp.
> With gui, it would be selecting the needed files and dragging them to a
> directory.
> Does the shell offer something as quick?
>
> The only ways I've found were loops or find -exec which I don't
> remember off hand.
Don't you have man pages ? Anyway it's
find . -type f -exec mv {} directory \;
This will move every regular file to the directory.
*** Do not use if the "directory" is under your current ***
*** directory. ***
>
> For a group of files with the same file format, say text, I tried
> ls *txt | grep -v 'fileIDontWant'
> but how do I pipe the stdout to mv or cp?
ls *txt gives you files whose name ends in txt ; it has nothing
to do with file format. Guessing from what you're saying above ,
you want to move (or copy) every such file to a directory , right ?
The way to do it is by
mv `ls *txt | grep -v 'fileIDontWant'` directory
>
> For a group of files with the different file formats, I'm at a lost as
> to 'finding' the files I want as grep -v 'fileA' 'fileB' doesn't
> exclude fileA and fileB
Which criterion are you using for picking the files you want
to move ?
> I'm slowly starting to learn unix, a sprinkle of sed, awk, grep,
> regexp, etc so please bear with me.
Sure. I'd leave sed and awk for later if I were you. Learn the simpler
stuff first.
> Unfortunately we use windows at work so I'm using ports of unix tools:
> unxutils and unixkit because cygwin is too big and I'd have to get
> permission to install.
All my comments above apply to Unix and similar. For all
I know on Windows file name and file format are not independent
from each other. I also have no idea if my suggestions will work
on Windows. Perhaps you should ask on a Windows group ?
| |
| Pierre Barbier de Reuille 2006-10-21, 1:31 pm |
| tony wrote:
> Hello,
>
> Is there a common way to select a group of files and not others and
> pipe them to mv or cp?
> With gui, it would be selecting the needed files and dragging them to a
> directory.
> Does the shell offer something as quick?
>
> The only ways I've found were loops or find -exec which I don't
> remember off hand.
>
> For a group of files with the same file format, say text, I tried
> ls *txt | grep -v 'fileIDontWant'
> but how do I pipe the stdout to mv or cp?
As a general tool, if you construct the argument list of a program in
stdout, you can use the programme "xargs" to convert it as arguments
like that :
((ls *txt | grep -v "fileIDontWant") && echo DestinationDir) | xargs mv
>
> For a group of files with the different file formats, I'm at a lost as
> to 'finding' the files I want as grep -v 'fileA' 'fileB' doesn't
> exclude fileA and fileB
For that, remember grep takes regular expressions. So you want:
grep -v '(fileA|fileB)'
>
> I'm slowly starting to learn unix, a sprinkle of sed, awk, grep,
> regexp, etc so please bear with me.
> Unfortunately we use windows at work so I'm using ports of unix tools:
> unxutils and unixkit because cygwin is too big and I'd have to get
> permission to install.
>
> Any pointers are much obliged
>
> thanks,
> tony
| |
| Spiros Bousbouras 2006-10-21, 1:31 pm |
| Pierre Barbier de Reuille wrote:
> tony wrote:
>
> For that, remember grep takes regular expressions. So you want:
>
> grep -v '(fileA|fileB)'
You might need to use some additional option to turn
on extended regular expressions or write 'egrep' instead
of 'grep'.
| |
| Janis Papanagnou 2006-10-21, 7:31 pm |
| Pierre Barbier de Reuille wrote:
> tony wrote:
>
>
> As a general tool, if you construct the argument list of a program in
> stdout, you can use the programme "xargs" to convert it as arguments
> like that :
>
> ((ls *txt | grep -v "fileIDontWant") && echo DestinationDir) | xargs mv
I don't think that works correct for _large sets_ of matching files.
Assume matching files A B ... Z don't fit into the exec buffer and is
split by xargs into two commands
mv A B ... K L
and
mv M N ... Z DestinationDir
where the first one will complain because L is no directory.
There's a similar problem with the solution posted elsethread,
mv `ls *txt | grep -v 'fileIDontWant'` directory
if the matching file set is too large.
Janis
| |
| Spiros Bousbouras 2006-10-21, 7:31 pm |
| Janis Papanagnou wrote:
> Pierre Barbier de Reuille wrote:
>
> I don't think that works correct for _large sets_ of matching files.
>
> Assume matching files A B ... Z don't fit into the exec buffer and is
> split by xargs into two commands
> mv A B ... K L
> and
> mv M N ... Z DestinationDir
> where the first one will complain because L is no directory.
What exec buffer ? And what makes you think that xargs
will split things into 2 commands ?
> There's a similar problem with the solution posted elsethread,
> mv `ls *txt | grep -v 'fileIDontWant'` directory
> if the matching file set is too large.
If the matching file set is too large I assume that an error
will be output rather than anything unsavory happen.
| |
| Janis Papanagnou 2006-10-21, 7:31 pm |
| Spiros Bousbouras wrote:
> Janis Papanagnou wrote:
>
>
>
>
> What exec buffer ?
The OS buffer where expanded arguments for external commands are built
to be passed to the external command (mv in this case).
> And what makes you think that xargs
> will split things into 2 commands ?
man xargs
Not 2 commands, that was an example, but as many parts as necessary to
fit each into the buffer. So in case where the data is small all file
names will be passed to a single instance of xargs command mv. If the
argument string is large it will be split, and more than instance will
be called. You can reduce the number or arguments to xargs by options
to xargs but the list cannot exceed beyond the buffer limit.
>
>
> If the matching file set is too large I assume that an error
> will be output rather than anything unsavory happen.
Yes, an error will be given in both cases.
Janis
| |
|
| Thank you for all your helpful replies and pointers to xargs, find exec
and the regexp.
Unfortunately the unix ports (http://jlb.twu.net/code/unixkit.php) that
allow me to use unix utilities on a corporate Windows machine doesn't
have man pages or xargs so I've been trying to add unxutils
(http://unxutils.sourceforge.net/) as needed as I learn.
I picked up basic unix books at the library, bought Chris Johnson's
Shell recipes book and I'll read up on the man pages on the net so that
will help too. Even though I'm training myself to use the command line
more, the crazy thing is that I'm using bloated programs like Excel or
features of a text editor to say parse the stdout of ls so I can paste
it back into the command line?! ie ls -1 > file then copy paste in text
editor and replace all line feeds with pipes | to get a string to paste
back into the grep statement.
Each little utility gem I learn offers a glimmer of hope out of this
madness but it is slow going without an overall picture of how to
string these programs together.
I guess the best way to learn is by trial and error, but I've been
wondering on a larger conceptual level how the shell handles file
operations (copy, move, rename) on multiple selections- basically how
do unix users organize large numbers of files which the gui handles
easily. The answer I'm learning is find a program that works, or learn
to write scripts and learn regular expressions. Anything else?
I'll post once I find a solution since I didn't know what to search for
in the group.
Thanks for your patience
Best,
tony
Janis Papanagnou wrote:
> Spiros Bousbouras wrote:
>
> The OS buffer where expanded arguments for external commands are built
> to be passed to the external command (mv in this case).
>
>
> man xargs
>
> Not 2 commands, that was an example, but as many parts as necessary to
> fit each into the buffer. So in case where the data is small all file
> names will be passed to a single instance of xargs command mv. If the
> argument string is large it will be split, and more than instance will
> be called. You can reduce the number or arguments to xargs by options
> to xargs but the list cannot exceed beyond the buffer limit.
>
>
> Yes, an error will be given in both cases.
>
> Janis
| |
| Spiros Bousbouras 2006-10-22, 1:16 pm |
| tony wrote:
> I guess the best way to learn is by trial and error, but I've been
> wondering on a larger conceptual level how the shell handles file
> operations (copy, move, rename) on multiple selections- basically how
> do unix users organize large numbers of files which the gui handles
> easily. The answer I'm learning is find a program that works, or learn
> to write scripts and learn regular expressions. Anything else?
You either create a loop as you said earlier in the thread or write
a programme using awk for example which writes a programme
which performs what you want. So assume for example that you
want to move a large number of files. Then you may write an awk
script whose output would be of the form
mv file1 dir
mv file2 dir
....
....
....
Then you can either pipe this output to a shell or put it in
a file and use source to execute the script.
| |
| Martijn Lievaart 2006-10-22, 7:19 pm |
| On Sat, 21 Oct 2006 19:43:26 -0700, tony wrote:
> Thank you for all your helpful replies and pointers to xargs, find exec
> and the regexp.
>
> Unfortunately the unix ports (http://jlb.twu.net/code/unixkit.php) that
> allow me to use unix utilities on a corporate Windows machine doesn't
> have man pages or xargs so I've been trying to add unxutils
> (http://unxutils.sourceforge.net/) as needed as I learn.
Try cigwin. Man pages, Xserver, the lot.
M4
--
Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering
their time inventing and implementing new, exciting ways for software
to suck. -- Toni Lassila in the Monastry
| |
| Martijn Lievaart 2006-10-22, 7:19 pm |
| On Sun, 22 Oct 2006 22:05:57 +0200, Martijn Lievaart wrote:
> Try cigwin. Man pages, Xserver, the lot.
Ugh! That's Cygwin.....
M4
--
Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering
their time inventing and implementing new, exciting ways for software
to suck. -- Toni Lassila in the Monastry
| |
| Spiros Bousbouras 2006-10-22, 7:19 pm |
| Martijn Lievaart wrote:
> On Sat, 21 Oct 2006 19:43:26 -0700, tony wrote:
>
>
> Try cigwin. Man pages, Xserver, the lot.
But in the very first post of the thread tony wrote:
> Unfortunately we use windows at work so I'm using ports of unix tools:
> unxutils and unixkit because cygwin is too big and I'd have to get
> permission to install.
| |
| Dean G. 2006-10-24, 7:17 pm |
|
tony wrote:
> Thank you for all your helpful replies and pointers to xargs, find exec
> and the regexp.
>
> Unfortunately the unix ports (http://jlb.twu.net/code/unixkit.php) that
> allow me to use unix utilities on a corporate Windows machine doesn't
> have man pages or xargs so I've been trying to add unxutils
> (http://unxutils.sourceforge.net/) as needed as I learn.
>
> I picked up basic unix books at the library, bought Chris Johnson's
> Shell recipes book and I'll read up on the man pages on the net so that
> will help too. Even though I'm training myself to use the command line
> more, the crazy thing is that I'm using bloated programs like Excel or
> features of a text editor to say parse the stdout of ls so I can paste
> it back into the command line?! ie ls -1 > file then copy paste in text
> editor and replace all line feeds with pipes | to get a string to paste
> back into the grep statement.
Are you grepping the same thing out of each file, or grepping the file
names from another location ? If the former, then
grep <regexp> `ls -1`
or (since you are just using ls -1 with no arguments)
grep <regexp> *
should work.
If the later, then consider using grep with the -f option for the
expresion file. It can save you the trouble of pasting the string
together, because the expression file is a list separated by line
feeds, which seems to be what you have.
grep -f <expfile> <filelist>
Dean G.
| |
|
| Hi,
I finally found something that worked for me in unixkit, if anyone has
the same issue.
Once a list of selected files is found from regex, find, egrep, comm or
other tools, redirect the list to a file, say, resultset.txt
Here's the clunky script to copy the filenames in resultset to the
directory named 'newdir':
sed 's/^/cp /' resultset.txt | sed 's/$/ newdir/' | bash
What it does:
Basically it brackets each filename with 'cp' and 'newdir' so there is
a command for each file:
1 prepend cp to all lines of resultset and pipe stdout to statement 2
2 append newdir or name of directory where you want the files to go to
each line
3 pipe to shell to execute each command statement
I couldn't figure out how to make it shorter, but at least it works.
Then today I found this post that uses loops to process multiple files:
http://www.basicallytech.com/blog/i...mmand-line.html
Unix is indeed lifelong learning :-)
Best,
tony
Dean G. wrote:
> tony wrote:
>
> Are you grepping the same thing out of each file, or grepping the file
> names from another location ? If the former, then
>
> grep <regexp> `ls -1`
> or (since you are just using ls -1 with no arguments)
> grep <regexp> *
>
> should work.
>
> If the later, then consider using grep with the -f option for the
> expresion file. It can save you the trouble of pasting the string
> together, because the expression file is a list separated by line
> feeds, which seems to be what you have.
>
> grep -f <expfile> <filelist>
>
> Dean G.
| |
| Dean G. 2006-10-27, 1:17 pm |
|
tony wrote:
> Hi,
>
> I finally found something that worked for me in unixkit, if anyone has
> the same issue.
>
> Once a list of selected files is found from regex, find, egrep, comm or
> other tools, redirect the list to a file, say, resultset.txt
>
> Here's the clunky script to copy the filenames in resultset to the
> directory named 'newdir':
>
> sed 's/^/cp /' resultset.txt | sed 's/$/ newdir/' | bash
>
> What it does:
> Basically it brackets each filename with 'cp' and 'newdir' so there is
> a command for each file:
>
> 1 prepend cp to all lines of resultset and pipe stdout to statement 2
> 2 append newdir or name of directory where you want the files to go to
> each line
> 3 pipe to shell to execute each command statement
>
> I couldn't figure out how to make it shorter, but at least it works.
If I understand correctly, resultset.txt has a list of files you want
to copy to a sinlge dir.
for i in `cat resultset.txt`
do
cp $i newdir
done
OR
cp `cat resultset.txt` newdir
The second one is shorter, and possibly quicker, but has a limit on the
number of files. This limit can be surmounted using xargs, but the
first method allows you to do more, since you can do other things with
the file inside the do loop in addition to the copying.
Dean G.
|
|
|
|
|