Unix Programming - sending images over a socket

Author

sending images over a socket

Joaquin López

2004-01-23, 5:17 pm

Hello, I'm writing a program for sending images over a socket.

The size of the image is 640x480 pixels in pgm format.

The problem is that when the client sends the image to the server (307200
bytes) the server only receives 65533.

The thing is, is there any limits on the information sent from one point
(send) to another (recv)?.
In such case, how is the way I have to send one image of 640x480 pixels?
Perhaps in two times?

Thank you for your attention,
Joaquin.

Fletcher Glenn

2004-01-23, 5:17 pm

Joaquin López wrote:

quote:

> Hello, I'm writing a program for sending images over a socket.
>
> The size of the image is 640x480 pixels in pgm format.
>
> The problem is that when the client sends the image to the server (307200
> bytes) the server only receives 65533.
>
> The thing is, is there any limits on the information sent from one point
> (send) to another (recv)?.
> In such case, how is the way I have to send one image of 640x480 pixels?
> Perhaps in two times?
>
> Thank you for your attention,
> Joaquin.
>
>

Let me guess: you wrote the image to the socket with a single write
statement? And now you expect to read the image with a single read??

That's now how it is done. The transport (the stuff behind the socket)
cannot handle messages of that size, and it breaks up the message into
pieces which are then sent to the receiving socket. When you read, you
will only get part of the message, and so you must repeatedly read the
socket until you get all of the expected data. How, you ask, do you
know when you have all of the data?? You know because you sent the
length of the message before you sent the message.

--

Fletcher Glenn

Lew Pitcher

2004-01-23, 5:17 pm

On Mon, 8 Dec 2003 17:14:30 +0100, "Joaquin López" <lsanchez@lsi.uji.es> wrote:

quote:

>Hello, I'm writing a program for sending images over a socket.
>
>The size of the image is 640x480 pixels in pgm format.
>
>The problem is that when the client sends the image to the server (307200
>bytes) the server only receives 65533.
>
>The thing is, is there any limits on the information sent from one point
>(send) to another (recv)?.
>
>In such case, how is the way I have to send one image of 640x480 pixels?
>Perhaps in two times?

It is likely that you don't understand TCP communications and/or stream sockets.

A recv() call only returns the data received up to that point. There is no
synchronization between the data sent and the data received. That is to say that
although you may /send/ massive amounts of data in one call, the corresponding
recv() call does not have to (and usually won't) receive the data in one recv()
call. This is the nature of TCP/IP, and is not a flaw in the sockets
implementation, or anything else.

It appears that your /first/ recv() gets 64k of the data sent. You need to
continue issueing recv() calls (and appending the results together) until you
receive all the data you sent. How you determine that you have received /all/
the data depends on your sending program:
- It may close it's sending socket when it has sent all the data; the receiver
will detect this as an end-of-file on the receiving socket, or
- It may send the length of the data as a data item prior to the data itself;
the receiver must fetch this length first, then recv() until the length has
been exhausted, or
- It may send a sentinal in the data stream to mark the end of the data; the
receiver must be sensitive to this sentinal and trim it's recv()d data
according to the placement of the sentinal, or
- The sender and receiver always know exactly how much data is transferred as
the sender always sends a fixed amount of data, and the receiver knows what
that fixed amount is; both sender and receiver must agree on what this amount
is prior to the transmission of the data.

In all cases, both sender application and receiver application must agree upon
the strategy to be used to detect end of data, and both applications must be
coded to enact that strategy.

--
Lew Pitcher
IT Consultant, Enterprise Technology Solutions
Toronto Dominion Bank Financial Group

(Opinions expressed are my own, not my employers')

Barry Margolin

2004-01-23, 5:17 pm

In article <3fd4a376@news.vodafone.es>,
"Joaquin López" <lsanchez@lsi.uji.es> wrote:

quote:

> Hello, I'm writing a program for sending images over a socket.
>
> The size of the image is 640x480 pixels in pgm format.
>
> The problem is that when the client sends the image to the server (307200
> bytes) the server only receives 65533.
>
> The thing is, is there any limits on the information sent from one point
> (send) to another (recv)?.
> In such case, how is the way I have to send one image of 640x480 pixels?
> Perhaps in two times?

There's no limit to how much you can send over a TCP socket. Large web
pages and huge emails are sent routinely this way.

It sounds like you're not calling recv() enough. You have to call it in
a loop until you get everything. TCP is a byte stream transport, it
doesn't keep track of message boundaries. So there's no reason to
expect that everything sent in a single send() call will be returned by
a single recv() call.

--
Barry Margolin, barmar@alum.mit.edu
Woburn, MA

Mike Chirico

2004-01-23, 5:17 pm

"Joaquin López" <lsanchez@lsi.uji.es> wrote in message
news:3fd4a376@news.vodafone.es...

quote:

> Hello, I'm writing a program for sending images over a socket.
>
> The size of the image is 640x480 pixels in pgm format.
>
> The problem is that when the client sends the image to the server (307200
> bytes) the server only receives 65533.

Put the read in a loop. If you do a tcpdump

tcpdump ether src port <your port>

Look for the mss value "maximum segment size" in the 3 way hand, which
should be the number of bytes for each call to read.

/** server side **/
process_stuff(int sockfd)
{
ssize_t n;
char line[MAXLINE];

sockfd_to_family(sockfd);
while(1) {
if ( ( n = read( sockfd, line, MAXLINE)) == 0 )
{
fprintf(stderr,"client had nothing else to say \n");
return;
}
line[n]='\0';
fprintf(stderr,"client said :n=%d %s\n",n,line);
} /* end while */
}

When the client is done sending, it can exit and you can look for the
WNOHANG
/* server side */
sig_chld(int signo)
{
pid_t pid;
int stat;

while( (pid = waitpid ( -1, &stat, WNOHANG)) > 0 )
fprintf(stderr,"child %d terminated \n",pid); /* not recommended to
have a print here */
return;

}

/******* start: Complete toy server for text******************/
#include <stdlib.h>
#include <arpa/inet.h>
#include <assert.h>
#include <errno.h>
#include <netinet/in.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <unistd.h>

#define SA struct sockaddr /* keeps the lines shorter */
#define MAXLINE 4096
/* Following could be derived from SOMAXCONN in <sys/socket.h>, but many
kernels still #define it as 5, while actually supporting many more */
#define LISTENQ 1024 /* 2nd argument to listen() */
#define MAXSOCKADDR 128 /* max socket address structure size */

/* normally these would go in a header */
void sig_chld(int);
void process_stuff(int);
int sockfd_to_family(int );

/*
The follwing function prevents zombie or defunct processes:
which occurs with server2way when the client is killed:

$ps -U chirico -auf
chirico 30923 0.0 0.0 1352 252 pts/0 S 11:02 0:00 \_
../server2way
chirico 30924 0.0 0.0 0 0 pts/0 Z 11:02 0:00 \_
[server2way <defunct>]

So, the function below...prevents the problem.

*/

void
sig_chld(int signo)
{
pid_t pid;
int stat;

while( (pid = waitpid ( -1, &stat, WNOHANG)) > 0 )
fprintf(stderr,"child %d terminated \n",pid); /* not recommended to
have a print here */
return;

}

void
process_stuff(int sockfd)
{
ssize_t n;
char line[MAXLINE];

sockfd_to_family(sockfd);
while(1) {
if ( ( n = read( sockfd, line, MAXLINE)) == 0 )
{
fprintf(stderr,"return in process_stuff ... read == 0\n");
return;

}
line[n]='\0';
fprintf(stderr,"client:n=%d %s\n",n,line);
write(sockfd, line, n);
}
}

int
sockfd_to_family(int sockfd)
{

struct sockaddr_in sa;
socklen_t len;

len = sizeof(sa);
if (getpeername(sockfd, (struct sockaddr*) &sa, &len) < 0 )
return -1;

fprintf(stderr,"client: %s: %d\n",inet_ntoa(sa.sin_addr),(int)
ntohs(sa.sin_port));

return(sa.sin_port);
}

/*

These are the major 4 steps:
socket
bind
listen
-- signal handle (optional but good if client it killed)
accept

*/

int
main(int argc, char **argv)
{
int listenfd, connfd;
pid_t childpid;
socklen_t clilen;
struct sockaddr_in cliaddr, servaddr;

listenfd = socket(AF_INET, SOCK_STREAM, 0);

bzero(&servaddr, sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
servaddr.sin_port = htons(10000); /* change this to your favorate port
*/

bind(listenfd, (SA *) &servaddr, sizeof(servaddr));
listen(listenfd, LISTENQ);

signal(SIGCHLD, sig_chld);

while(1)
{
clilen = sizeof(cliaddr);
connfd = accept(listenfd, (SA *) &cliaddr, &clilen);

if ( (childpid = fork()) == 0 ) {
close(listenfd);
process_stuff(connfd);
exit(0);
}
close(connfd);
}

}

/******* end: Complete toy server ******************/

Sorry about all the code; but, it's hard to understand with just pieces.

Excellent reference: "Unix Network Programming.." W. Richard Stevens.

Regards,

Mike Chirico

Joseph Dionne

2004-01-23, 5:17 pm

however, since TCP is a STREAM of bytes, it is advised you use an NBO=20
long, htol() of the number of bytes to follow so your receiver knows how =

many bytes to expect.

Barry Margolin wrote:

quote:

> In article <3fd4a376@news.vodafone.es>,
> "Joaquin L=F3pez" <lsanchez@lsi.uji.es> wrote:
>=20
>=20
00[QUOTE][color=darkred]
t[QUOTE][color=darkred]
?[QUOTE][color=darkred]
>=20
>=20
> There's no limit to how much you can send over a TCP socket. Large web=

=20

quote:

> pages and huge emails are sent routinely this way.
>=20
> It sounds like you're not calling recv() enough. You have to call it i=

n=20

quote:

> a loop until you get everything. TCP is a byte stream transport, it=20
> doesn't keep track of message boundaries. So there's no reason to=20
> expect that everything sent in a single send() call will be returned by=

=20

quote:

> a single recv() call.
>=20

Lew Pitcher

2004-01-23, 5:17 pm

On Mon, 08 Dec 2003 17:48:46 GMT, Joseph Dionne <jdionne@sdcnov1.sdcweb.net>
wrote:

quote:

>however, since TCP is a STREAM of bytes, it is advised

by whom?

quote:

> you use an NBO=20
>long, htol() of the number of bytes to follow so your receiver knows how =
>
>many bytes to expect.

Or any one of the alternate strategies, like
- having the source send a sentinal value at the end of the data, or
- having the source close it's outbound data stream after it has written
the last byte of data, or
- having the source and receiver programs agree that the transferred data will
always be a specific, fixed number of octets. The source sends exactly that
number of octets, and the receiver reads exactly that number of octets
preprogrammed.

[snip]

--
Lew Pitcher
IT Consultant, Enterprise Technology Solutions
Toronto Dominion Bank Financial Group

(Opinions expressed are my own, not my employers')

Joseph Dionne

2004-01-23, 5:17 pm

I stand corrected, Lew, any of the alternatives you have suggested will
work just fine, except perhaps disconnect which might be delivered OOB,
causing a less than full read.

However, a "common" practice for binary data transmission is the use a
length bytes, two or four bytes (I recommend NBO -- network byte order)
to eliminate the need for concern over Endian differences between client
and server.

HTML, one of the most popular STREAMs over sockets is token, and "line",
delimited, but a "sentinel" in a binary stream would require one to use
transparency to allow the "end sentinel" appearing in the image being sent.

Just a thought. I find lengths bytes serve two purposes; 1) the
receiver knows how much data to perhaps malloc() a buffer, 2) restores
message structure to the TCP STREAM if one plans to use the connection
multiple times.

Just a thought.

Lew Pitcher wrote:

quote:

> On Mon, 08 Dec 2003 17:48:46 GMT, Joseph Dionne <jdionne@sdcnov1.sdcweb.net>
> wrote:
>
>
>
>
> by whom?
>
>
>
>
> Or any one of the alternate strategies, like
> - having the source send a sentinal value at the end of the data, or
> - having the source close it's outbound data stream after it has written
> the last byte of data, or
> - having the source and receiver programs agree that the transferred data will
> always be a specific, fixed number of octets. The source sends exactly that
> number of octets, and the receiver reads exactly that number of octets
> preprogrammed.
>
>
>
> [snip]
>

Barry Margolin

2004-01-23, 5:17 pm

In article <PK3Bb.31593$b01.723209@twister.tampabay.rr.com>,
Joseph Dionne <jdionne@sdcnov1.sdcweb.net> wrote:

quote:

> I stand corrected, Lew, any of the alternatives you have suggested will
> work just fine, except perhaps disconnect which might be delivered OOB,
> causing a less than full read.

It shouldn't be delivered OOB. Closing the connection results in an EOF
being read.

--
Barry Margolin, barmar@alum.mit.edu
Woburn, MA

Joaquin =?ISO-8859-15?Q?L=F3pez?=

2004-01-23, 5:17 pm

Lew Pitcher wrote:

quote:

> On Mon, 8 Dec 2003 17:14:30 +0100, "Joaquin López" <lsanchez@lsi.uji.es>
> wrote:
>
>
> It is likely that you don't understand TCP communications and/or stream
> sockets.
>
> A recv() call only returns the data received up to that point. There is no
> synchronization between the data sent and the data received. That is to
> say that although you may /send/ massive amounts of data in one call, the
> corresponding recv() call does not have to (and usually won't) receive the
> data in one recv() call. This is the nature of TCP/IP, and is not a flaw
> in the sockets implementation, or anything else.
>
> It appears that your /first/ recv() gets 64k of the data sent. You need to
> continue issueing recv() calls (and appending the results together) until
> you receive all the data you sent. How you determine that you have
> received /all/ the data depends on your sending program:
> - It may close it's sending socket when it has sent all the data; the
> receiver
> will detect this as an end-of-file on the receiving socket, or
> - It may send the length of the data as a data item prior to the data
> itself;
> the receiver must fetch this length first, then recv() until the length
> has been exhausted, or
> - It may send a sentinal in the data stream to mark the end of the data;
> the
> receiver must be sensitive to this sentinal and trim it's recv()d data
> according to the placement of the sentinal, or
> - The sender and receiver always know exactly how much data is transferred
> as
> the sender always sends a fixed amount of data, and the receiver knows
> what that fixed amount is; both sender and receiver must agree on what
> this amount is prior to the transmission of the data.
>
> In all cases, both sender application and receiver application must agree
> upon the strategy to be used to detect end of data, and both applications
> must be coded to enact that strategy.
>
>

Thank you to all of us for helping me. The problem was exactly this, I
expected recv() will get all the data in only one call. It was my fault.

Lew Pitcher

2004-01-23, 5:18 pm

Barry Margolin wrote:

quote:

> In article <PK3Bb.31593$b01.723209@twister.tampabay.rr.com>,
> Joseph Dionne <jdionne@sdcnov1.sdcweb.net> wrote:
>
>
>
>
> It shouldn't be delivered OOB. Closing the connection results in an EOF
> being read.

More specifically, a close (or half close on the sending connection) will
cause the sending stack to set the FIN flag in the last sent segment. The
receiving stack will ACKnowledge the segment, and pass the appropriate EOF
signal to the receiving application at the appropriate time in it's processing.

If I read the doc correctly, the FIN will either accompany the last data
segment (in the case of a partially full segment buffer) or as a seperate
datagram following the last data segment (in the case of an empty segment
buffer). In other words, the FIN never precedes any pending data.

Likewise, the receiver should not deliver the resulting EOF out-of-band. If
the reader application has not yet exhausted the receive buffer, it should
read data from that buffer until the data has been gathered. The FIN tells the
stack that, once the buffer is emptied, further reads cannot be satisfied, and
an EOF condition should be asserted.

--
Lew Pitcher

Master Codewright and JOAT-in-training
Registered Linux User #112576 (http://counter.li.org/)
Slackware - Because I know what I'm doing.

James Antill

2004-01-23, 5:18 pm

On Mon, 08 Dec 2003 18:51:27 +0000, Joseph Dionne wrote:

quote:

> I stand corrected, Lew, any of the alternatives you have suggested will
> work just fine, except perhaps disconnect which might be delivered OOB,
> causing a less than full read.

I assume you are thinking of people calling poll() and getting a
POLLUP before all the data is read? Well HHTTP 1.0 closes the
connection after each request and HTTP 1.1 can do it ... so it
can and does work fine in practice.

quote:

> However, a "common" practice for binary data transmission is the use a
> length bytes, two or four bytes (I recommend NBO -- network byte order)
> to eliminate the need for concern over Endian differences between client
> and server.

I would argue that it is _not_ common, at all. If you want to pass
lengths then passing them as a sequence of ASCII is _much_ more common
than a NBO binary.
Also with SMTP, IMAP, POP3 and HTTP all using the sentinel (admittedly
HTTP only for the headers) I'd argue that using a sentinel is more common
and using lengths.

quote:

> HTML, one of the most popular STREAMs over sockets is token, and "line",
> delimited, but a "sentinel" in a binary stream would require one to use
> transparency to allow the "end sentinel" appearing in the image being sent.

1. HTML is not a protocol, HTTP is ... maybe you meant that.
2. HTTP works happily with binary data.
3. After using sentinels for the headers, HTTP uses a Content-Length
header with an ASCII length following it (and/or the disconnection method).

quote:

> Just a thought. I find lengths bytes serve two purposes; 1) the
> receiver knows how much data to perhaps malloc() a buffer, 2) restores
> message structure to the TCP STREAM if one plans to use the connection
> multiple times.

Using ASCII lengths is easier to debug and generally easier to write. And
if you want to be really special you can use the format at...

http://cr.yp.to/proto/netstrings.txt

....which is implemented a few places, and so even easier use.

--
James Antill -- james@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/

Joseph Dionne

2004-01-23, 5:18 pm

I can agree with all you say, James, however I believe the image being
sent is in binary form, thus sentinel characters will appear in the
binary image data. HTML, SMTP, POP3, are all clear text transfers,
using tokens and/or newlines as data sentinels. Transferring pure
binary data needs extra consideration.

Now, perhaps wrapping the image in a MIMEd STREAM is a better idea all
around.

While I agree debugging with ASCII digits lengths, that are converted
from/to integers, does help development of software, a two byte NBO
short adds little to the packet overhead, and is "almost" as easy to
read if you have the skills to read HEX values.

But, there are perhaps as many solutions as programmers in the world. I
work on many UNIX OSes, and find TCP stack behaviors vary quite a lot,
and most times on undocumented ways. IP routines tend to be write/debug
once use many times, so the advantage of ease of debugging at a packet
level dimities. I have use the same IP routines for almost twenty
years. I hardly ever look at the code these days, and only edit when I
need to port to a new compiler/OS.

Joseph

James Antill wrote:

quote:

> On Mon, 08 Dec 2003 18:51:27 +0000, Joseph Dionne wrote:
>
>
>
>
> I assume you are thinking of people calling poll() and getting a
> POLLUP before all the data is read? Well HHTTP 1.0 closes the
> connection after each request and HTTP 1.1 can do it ... so it
> can and does work fine in practice.
>
>
>
>
> I would argue that it is _not_ common, at all. If you want to pass
> lengths then passing them as a sequence of ASCII is _much_ more common
> than a NBO binary.
> Also with SMTP, IMAP, POP3 and HTTP all using the sentinel (admittedly
> HTTP only for the headers) I'd argue that using a sentinel is more common
> and using lengths.
>
>
>
>
> 1. HTML is not a protocol, HTTP is ... maybe you meant that.
> 2. HTTP works happily with binary data.
> 3. After using sentinels for the headers, HTTP uses a Content-Length
> header with an ASCII length following it (and/or the disconnection method).
>
>
>
>
> Using ASCII lengths is easier to debug and generally easier to write. And
> if you want to be really special you can use the format at...
>
> http://cr.yp.to/proto/netstrings.txt
>
> ...which is implemented a few places, and so even easier use.
>