|
Home > Archive > Unix Shell > May 2004 > How can I copy Multipal files(10000) in to one single .txt file
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
How can I copy Multipal files(10000) in to one single .txt file
|
|
|
| "copy files using ksh shell script"
thanks you very much who as help me to solve my queary.
****************************************
***************************************
when I try to copy as follows
-------------------------------------------------------------------------------
for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6
do
c='/d/Build/AXX/Procedures11'
d='/d/Build/AXX/Script11'
cd "$c"/"$Folders"
cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
Echo "_____________Combine script $Folders 11 is done____________"
done
-------------------------------------------------------------------------------
error is as follows :
cat: /usr/bin/cat: cannot execute [Arg list too long]
each folder is having more than 10000 files , now I want to club all
the txt file and want to create only one file
will you help me ?
| |
| Alan Connor 2004-05-20, 5:37 pm |
| On 20 May 2004 04:52:35 -0700, Anand <versatile_anand@yahoo.com> wrote:
>
>
> "copy files using ksh shell script"
> thanks you very much who as help me to solve my queary.
> ****************************************
***************************************
> when I try to copy as follows
> -------------------------------------------------------------------------------
> for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6
> do
>
> c='/d/Build/AXX/Procedures11'
> d='/d/Build/AXX/Script11'
>
> cd "$c"/"$Folders"
> cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
> Echo "_____________Combine script $Folders 11 is done____________"
> done
> -------------------------------------------------------------------------------
> error is as follows :
> cat: /usr/bin/cat: cannot execute [Arg list too long]
>
> each folder is having more than 10000 files , now I want to club all
> the txt file and want to create only one file
>
> will you help me ?
Maybe this would work:
find /dir1 /dir2 /dir3 /dir4 /dir5 /dir6 -type f -name '*.txt' \
-exec cat {} >> destination_file \;
AC
| |
| Yaroslav Yakovets 2004-05-20, 5:37 pm |
| I think somthing like this will help you:
# *** start ***
resFile="<PATH_TO_FILE>/res.txt"
mask="*"
Path_to_folders="<SOME_FOLDER>"
Folders="FOLDER1 FOLDER2 FOLDER3"
for folder in $Folders
do
files=${Path_to_folders}/${folder}/${mask}
for file in $files
do
#append current file to res-file
cat $file >> $resFile
done
done
# *** finish ***
Maybe someone can explain the better way?
--
Best regards,
Yaroslav Yakovets
"Anand" <versatile_anand@yahoo.com> wrote in message
news:13d31bbd.0405200352.39fa5679@posting.google.com...
> "copy files using ksh shell script"
> thanks you very much who as help me to solve my queary.
>
****************************************
************************************
***
> when I try to copy as follows
> --------------------------------------------------------------------------
-----
> for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6
> do
>
> c='/d/Build/AXX/Procedures11'
> d='/d/Build/AXX/Script11'
>
> cd "$c"/"$Folders"
> cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
> Echo "_____________Combine script $Folders 11 is done____________"
> done
> --------------------------------------------------------------------------
-----
> error is as follows :
> cat: /usr/bin/cat: cannot execute [Arg list too long]
>
> each folder is having more than 10000 files , now I want to club all
> the txt file and want to create only one file
>
> will you help me ?
| |
| Kevin Collins 2004-05-20, 5:37 pm |
| In article <UK1rc.2453$Tn6.384@newsread1.news.pas.earthlink.net>, Alan Connor
wrote:
> On 20 May 2004 04:52:35 -0700, Anand <versatile_anand@yahoo.com> wrote:
>
> Maybe this would work:
>
>
> find /dir1 /dir2 /dir3 /dir4 /dir5 /dir6 -type f -name '*.txt' \
> -exec cat {} >> destination_file \;
>
Alan,
I seriously doubt forking 60000+ subprocesses (OP stated 6 dirs each with
10k+ files each) is a good solution here, especially when xargs could do it much
more efficiently...
Kevin
| |
| Kevin Collins 2004-05-20, 5:37 pm |
| In article <13d31bbd.0405200352.39fa5679@posting.google.com>, Anand wrote:
> "copy files using ksh shell script"
> thanks you very much who as help me to solve my queary.
> ****************************************
***************************************
> when I try to copy as follows
> -------------------------------------------------------------------------------
> for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6
> do
>
> c='/d/Build/AXX/Procedures11'
> d='/d/Build/AXX/Script11'
Since these 2 lines do no variable interpolation, they should be moved outside
the loop, otherwise their use is unneccearily repeated.
> cd "$c"/"$Folders"
^^^^^^^^^^^^^^^
This should be (and is also, not needed):
"$c/$Folders"
> cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
And this should be:
"$d/$Folders/${Folders}_P11.txt"
> Echo "_____________Combine script $Folders 11 is done____________"
> done
> -------------------------------------------------------------------------------
> error is as follows :
> cat: /usr/bin/cat: cannot execute [Arg list too long]
>
> each folder is having more than 10000 files , now I want to club all
> the txt file and want to create only one file
The answer is to use xargs, which can efficiently handle this:
cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
becomes:
ls | grep '\.txt$' | xargs -i cat {} > "$d/$Folders/${Folders}_P11.txt"
So, I would rewrite your script to this:
c='/d/Build/AXX/Procedures11'
d='/d/Build/AXX/Script11'
for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6
do
ls "$c/$Folders" | grep '\.txt$' | \
xargs -i cat {} > "$d/$Folders/${Folders}_P11.txt"
Echo "_____________Combine script $Folders 11 is done____________"
done
> will you help me ?
Hopefully, this does 
Kevin
| |
|
| On Thu, 20 May 2004 04:52:35 -0700, Anand wrote:
> "copy files using ksh shell script"
> thanks you very much who as help me to solve my queary.
> ****************************************
***************************************
> when I try to copy as follows
> -------------------------------------------------------------------------------
> for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6 do
>
> c='/d/Build/AXX/Procedures11'
> d='/d/Build/AXX/Script11'
>
> cd "$c"/"$Folders"
> cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
> Echo "_____________Combine script $Folders 11 is done____________"
> done
> -------------------------------------------------------------------------------
> error is as follows :
> cat: /usr/bin/cat: cannot execute [Arg list too long]
>
> each folder is having more than 10000 files , now I want to club all the
> txt file and want to create only one file
>
> will you help me ?
Take a look at the xargs command. Something like this:
( ls *.txt | xargs cat ) >> "$d"/"$Folders"/"$Folders""_P11.txt"
WARNING!!! UNTESTED COMMAND!!!
The idea is that xargs creates a command line based on what's fed to it in
stdin. But, it is clever enough to know the length limitation of the
command line and, if that would be exceeded, it (xargs) constructs
multiple command lines until all of stdin is processed.
It is because of the possibiliy of multiple command lines we need to use[vbcol=seagreen]
I used the parends because xargs likes to append data from stdin to the
end of the command line. There are ways around that but I think they may
defeat the purpose. Not sure.
| |
| rakesh sharma 2004-05-22, 10:28 pm |
| versatile_anand@yahoo.com (Anand) wrote in message news:
>
> for Folders in Folder1 Folder2 Folder3 Folder4 Folder5 Folder6
> do
>
> c='/d/Build/AXX/Procedures11'
> d='/d/Build/AXX/Script11'
>
> cd "$c"/"$Folders"
> cat *.txt > "$d"/"$Folders"/"$Folders""_P11.txt"
> Echo "_____________Combine script $Folders 11 is done____________"
> done
> -------------------------------------------------------------------------------
> error is as follows :
> cat: /usr/bin/cat: cannot execute [Arg list too long]
>
> each folder is having more than 10000 files , now I want to club all
> the txt file and want to create only one file
>
using GNU find and xargs we can do as follows:
c=/d/Build/AXX/Procedures11
d=/d/Build/AXX/Script11
Folders="${c}Folder1 ${c}Folder2 ${c}Folder3\
${c}Folder4 ${c}Folder5 ${c}Folder6"
for i in $Folders;do
{
find "$i" -maxdepth=1 -type f -print0 |\
xargs -0 cat
} > "$d/${i}/${i}_P11.txt"
done
|
|
|
|
|