Unix Programming - socket broken pipe error on sending huge data.

Author

socket broken pipe error on sending huge data.

rohit

2007-07-16, 1:20 pm

Dear All,
I have developed a simple client server application, the server needs
to send around 1.5 MB of data to the client for each transaction
continously. But the server is only able to send only 81660 bytes of
data and throws
"ERROR writing to socket: Broken pipe"

1) Cannot we send huge data through sockets and that too continously ?
2) What can be possible reason for this issue?
3) How to resolve the same?
4) Any code snippet will be of great use.
I am a NO expert in this area, so all comments are most welcome.
/////// SERVER CODE SNIPPET///
int serverConnection()
{
int sockfd, newsockfd, portno, clilen;
char buffer[256];

struct sockaddr_in serv_addr, cli_addr;

int n;

sockfd = socket(AF_INET, SOCK_STREAM, 0);

if (sockfd < 0)
error("ERROR opening socket");

bzero((char *) &serv_addr, sizeof(serv_addr));

portno = 5902;

serv_addr.sin_family = AF_INET;
serv_addr.sin_addr.s_addr = INADDR_ANY;
serv_addr.sin_port = htons(portno);

if (bind(sockfd, (struct sockaddr *) &serv_addr,
sizeof(serv_addr)) < 0)
error("ERROR on binding");

listen(sockfd,5);
clilen = sizeof(cli_addr);

newsockfd = accept(sockfd,
(struct sockaddr *) &cli_addr,
&clilen);

if (newsockfd < 0)
error("ERROR on accept");

bzero(buffer,256);

n = read(newsockfd,buffer,255);

if (n < 0) error("ERROR reading from socket");

printf("Here is the message: %s\n",buffer);

do{
fillBuffer
n = write(newsockfd, &HUGEDATA,sizeof(HUGEDATA));
if( n < 1 && errno != EINTR ) {
error("ERROR writing to socket");
return 0;
}

if( n == EINTR ) {
fprintf(stdout,"SOAP:\t:write interupted...tyring again");
continue;
}

}while(1);

if (n < 0) error("ERROR writing to socket");
return 0;

}
///////////
////////////////CLIENT SNIPPET///
int receivedData()
{
int sockfd, n;
struct sockaddr_in serv_addr;
struct hostent *server;

char buffer[256];

sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd < 0)
error("ERROR opening socket");
server = gethostbyname("localhost");

if (server == NULL) {
fprintf(stderr,"ERROR, no such host\n");
exit(0);
}

bzero((char *) &serv_addr, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
bcopy((char *)server->h_addr,
(char *)&serv_addr.sin_addr.s_addr,
server->h_length);
serv_addr.sin_port = htons(5902);
if (connect(sockfd,&serv_addr,sizeof(serv_addr)) < 0)
error("ERROR connecting");

printf("Please enter the message: ");
bzero(buffer,256);
strcpy(buffer,"rohit");
n = write(sockfd,buffer,strlen(buffer));
if (n < 0)
error("ERROR writing to socket");
while(1)
{
n = read(sockfd,&HUGEDATA,sizeof(HUGEDATA));
if (n < 0)
error("ERROR reading from socket");

}

return 0;
}

David Schwartz

2007-07-16, 7:20 pm

On Jul 16, 10:23 am, rohit <PowerPCDevelo...@gmail.com> wrote:
> Dear All,
> I have developed a simple client server application, the server needs
> to send around 1.5 MB of data to the client for each transaction
> continously. But the server is only able to send only 81660 bytes of
> data and throws

Short version -- the 'read' needs to be in a 'while' loop. You need to
keep reading until you get all the data. If you stop after you get one
bit of data, the server will soon be sending to nothing.

DS

Bill Marcum

2007-07-16, 7:20 pm

On Mon, 16 Jul 2007 17:23:20 -0000, rohit
<PowerPCDeveloper@gmail.com> wrote:
>
>
> Dear All,
> I have developed a simple client server application, the server needs
> to send around 1.5 MB of data to the client for each transaction
> continously. But the server is only able to send only 81660 bytes of
> data and throws
> "ERROR writing to socket: Broken pipe"
>
> 1) Cannot we send huge data through sockets and that too continously ?
> 2) What can be possible reason for this issue?
> 3) How to resolve the same?
> 4) Any code snippet will be of great use.
> I am a NO expert in this area, so all comments are most welcome.
> /////// SERVER CODE SNIPPET///

A broken pipe error means the receiving process died. Maybe you could
use strace or some other tool to find out why.

--
Using TSO is like kicking a dead whale down the beach.
-- S. C. Johnson

Igmar Palsenberg

2007-07-17, 7:21 am

rohit wrote:
> Dear All,
> I have developed a simple client server application, the server needs
> to send around 1.5 MB of data to the client for each transaction
> continously. But the server is only able to send only 81660 bytes of
> data and throws
> "ERROR writing to socket: Broken pipe"
>
> 1) Cannot we send huge data through sockets and that too continously ?
> 2) What can be possible reason for this issue?
> 3) How to resolve the same?
> 4) Any code snippet will be of great use.

In short : You can't assume that writing or reading (large) amounts of
data in one read() / write() call succeeds.

Igmar

rohit

2007-07-18, 1:23 am

Thanks, b/w can we use UDP i.e datagrams to send huge amounts (approx
2MB) of data per transaction continously?
Rohit
On Jul 17, 12:50 pm, Igmar Palsenberg <ig...@palsenberg.local> wrote:
> rohit wrote:
>
>
> In short : You can't assume that writing or reading (large) amounts of
> data in one read() / write() call succeeds.
>
> Igmar

Ian Collins

2007-07-18, 1:23 am

rohit wrote:

Please don't top-post.

> On Jul 17, 12:50 pm, Igmar Palsenberg <ig...@palsenberg.local> wrote:
> Thanks, b/w can we use UDP i.e datagrams to send huge amounts (approx
> 2MB) of data per transaction continously?
> Rohit

Not reliably unless you add your own higher layer protocol over UDP.

--
Ian Collins.

Barry Margolin

2007-07-18, 1:23 am

In article <1184730709.367083.129570@j4g2000prf.googlegroups.com>,
rohit <PowerPCDeveloper@gmail.com> wrote:

> Thanks, b/w can we use UDP i.e datagrams to send huge amounts (approx
> 2MB) of data per transaction continously?

The maximum size of a UDP datagram is 64K bytes, including the UDP
header.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

David Schwartz

2007-07-18, 7:23 am

On Jul 17, 12:50 am, Igmar Palsenberg <ig...@palsenberg.local> wrote:

> In short : You can't assume that writing or reading (large) amounts of
> data in one read() / write() call succeeds.

Although in this case, it does succeed. It just doesn't read as much
data as he might like.

DS

rohit

2007-07-19, 1:20 pm

On Jul 18, 12:33 pm, David Schwartz <dav...@webmaster.com> wrote:
> On Jul 17, 12:50 am, Igmar Palsenberg <ig...@palsenberg.local> wrote:
>
>
> Although in this case, it does succeed. It just doesn't read as much
> data as he might like.
>
> DS

Please clarify for following: I want to send 1.5 MB of data from
server to client continously
(a) So from the server part do i need to create a single buffer of
1.5 MB ? Or make small buffers from the server itself?
(b) Then at client end, do i need to allocate a small buffer area,
lets say 100 KB each, keep on getting the data and add in my 1.5MB
buffer ? Till i read all the data?

As posted in previous post, I am a novice in socket programming area,
so all comments, code snippet, knowledge sharing things are most
welcome.

rohit

2007-07-19, 1:20 pm

Pascal Bourguignon

2007-07-19, 1:20 pm

rohit <rohit.dhamija@gmail.com> writes:

> On Jul 18, 12:33 pm, David Schwartz <dav...@webmaster.com> wrote:
>
> Please clarify for following: I want to send 1.5 MB of data from
> server to client continously

Oh! What do you do when you've done sent it? If I were you, I would
send it very slowly. Since the expected age of the liveable universe
is about 15e9 years, I'd not send more than one bit every 1250 years.

> (a) So from the server part do i need to create a single buffer of
> 1.5 MB ? Or make small buffers from the server itself?

Depends where these bits come from. If you can generate them faster
than you can send them, there's no need for a buffer. Otherwise, you
may need to generate them beforehand, or you may send them slower. If
you generate them beforehand, it might be worthwhile to store them
outside of the main memory. With a good OS, this would be done
automatically,, but if you don't have a true OS, you may have to do it
yourself, loading small chunks from a mass data storage into RAM
buffers, etc.

> (b) Then at client end, do i need to allocate a small buffer area,
> lets say 100 KB each, keep on getting the data and add in my 1.5MB
> buffer ? Till i read all the data?

Depends what you need to do on the other end. You've only said you
needed to send the data continuously, not that it was to be subjected
to any processing on the other side. You could just as well throw it
away as soon as received. Or if you can process it faster than you
can receive it, then you don't need to buffer it. Or if you have
enough RAM you can just store the whole 1.5MB in RAM. Or you may also
need to store it on mass storage, then you may need buffers of a size
depending on the relative speed at which you can receive and you can
write to the mass storage. How can we guess? It's up to you to study
it!

In the mean time, you may be interested in knowing that the IP
protocol cannot send data in packages of more than 64KB (usually it
uses smaller packet size, to accomodate the smaller packet size of the
underlying network layers), therefore even if TCP provide a stream
abstraction, you won't receive more than 64KB (usually much less) "at
once".

Of course, allocating one big buffer, sending it in one write syscall,
and on the other end allocating one big buffer, and receiving it in
one read syscall MAY work and be simplier to write. But it's quite
probable that you'll have to implement at least one loop to do that,
because for example, the unix read/write syscall API allows for
partial transfers (see the EINTR error from man 2 read and man 2
write, and their return value).

--
__Pascal Bourguignon__ http://www.informatimago.com/
Wanna go outside.
Oh, no! Help! I got outside!
Let me back inside!

Ian Collins

2007-07-19, 7:19 pm

rohit wrote:
> On Jul 18, 12:33 pm, David Schwartz <dav...@webmaster.com> wrote:
>
> Also, how can I know in advance how much amount of data a socket can
> read or write ?
>
You shouldn't have to. Just keep calling write until all of the data is
transferred and read until there is nothing left to read.

--
Ian Collins.

David Schwartz

2007-07-21, 7:27 am

On Jul 19, 8:52 am, rohit <rohit.dham...@gmail.com> wrote:

> Also, how can I know in advance how much amount of data a socket can
> read or write ?

You can't because that would be asking the system to predict the
future, which it cannot know.

DS