pnet benchmarking

Hello everyone,

I am trying to benchmark pnet toolbox for sending and receiving both tcp/udp messages between the machines. My goal is to measure the throughput and latency for different pay-load sizes.

I wanted some advice on getting the basic loop of the script running. I wrote the following code for the listener:

sPort = '1234';
if ((sPort > 1024))
    tcpCon = pnet('tcpsocket', sPort); % only create this socket once 
else
    tcpCon = -1;
end

serv = pnet(tcpCon,'tcplisten');

try
    pnet(tcpCon, 'close');
catch exception
    retCode = 1;
end

But I get the error:

??? Error using ==> pnet
Invalid socket for LISTEN, Already open, or UDP?...

Error in ==> test_pnet at 24
serv = pnet(tcpCon,'tcplisten');

Can someone please guide me to what could be the problem. If anyone has a simple working sender/receiver script that I could develop on that would be great!

Answer 1

[Ian]

Hi, perhaps you have another application that has opened that port? Try a higher port number (9876 for example), or check if that port is open (sudo lsof -i TCP -P in os x/linux, use sysinternals tcpview in windows).

I use pnet for both UDP and TCP communication, wrapping them in an oop object. My ghetto code needs refactoring, but here it is warts and all: https://gist.github.com/899981

To open a server:

sconn=dataConnection(struct('lPort', 8765, 'protocol', 'tcp', 'autoOpen', 1, 'type', 'server'));

To open a client:

cconn=dataConnection(struct('rPort', 8765, 'rAddress', '127.0.0.1', 'protocol', 'tcp', 'autoOpen', 1));

Check connection:

sconn.checkClient; %server needs to check if a client has connected
sconn.write('Ping'); %write data to the connection
cconn.read % (ans='Ping')

See alse http://stackoverflow.com/questions/5104396/controling-a-matlab-script-pause-reset/5148504#5148504 for an example using UDP.

EDIT: as Walter points out, the port number needs to be a number for pnet not a string; the address should be a string.

Answer 2

[Walter Roberson]

Your code

sPort = '1234';

sets sPort to be a string. Then in the next line, you try to compare the string to an integer. The result is to compare each character of the string to the integer, and to find the "if" to be true only if all of the comparisons are true. None of the comparisons will be true, though, as '1' is only about value 48...

Answer 3

[Priya Bhat]

Hi Guys!

Thanks for your responses. I have created a preliminary version of the benchmarking code. I have uploaded it here https://www.acis.ufl.edu/~pbhat/pnet_benchmarking.v4.tar in case any one would like to have a look. Any feedback would be greatly appreciated.

The basic idea of the code is to have two sessions (one server and one receiver) running and transmit packets of different sizes for several iterations and measure the time elapsed.

I compared the results of throughput with those obtained with netperf on the same set of machines and it seems (from the surface) that pnet is very slow:

Results from netperf:

      pbhat@BMIUbuntu64:~$ netperf -H 192.168.140.22TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.140.22 (192.168.140.22) port 0 AF_INET : demo
    Recv   Send    Send                          
    Socket Socket  Message  Elapsed              
    Size   Size    Size     Time     Throughput  
    bytes  bytes   bytes    secs.    10^6bits/sec  
     87380  16384  16384    10.00    4302.63

Output of pnet_benchmarking is:

                 Send     Throughput    MeanLatency 
               Size           10^6        10^(-3) 
              bytes       bits/sec            sec 
                100          4.435          0.181
                500         21.845          0.185
               1000         44.114          0.183
               1500         65.640          0.184
               2000         86.150          0.187
               5000        193.530          0.208
              16384        460.342          0.285

In this case the better term for Throughput would probably be "Average Bandwidth". As you can see there is order of magnitude difference between the two results. I am wondering if this is something I am doing or if pnet is little inefficient.

Also copypasting my code here for easier refernce:

Server Code

      function server(iterNumber, repeat)
    % This function tests how to send and receive packets using pnet over the
    % network.
    if (nargin < 1)          iterNumber     = 1000;                 end
    if (nargin < 2)          repeat         = 1000;                 end
    %Set path
    if isunix
            pnetPath ='/home/pbhat/ModifiedBmiModels/TestCwFolder/pnetLib';
        else
            pnetPath = 'C:\Users\Administrator\Documents\ModifiedBmiModels\TestCwFolder\pnetLib';
    end
    addpath(pnetPath);
    %Set up Connection
    sPort = '1677';
    sockcon = pnet('tcpsocket', sPort);
    tcpCon=pnet(sockcon,'tcplisten');
    %Initilize
    ByteSize = [100, 500, 1000, 1500, 2000,5000, 16384];
    byte_sizes = length(ByteSize);
    data_collected = 0;
    %Receive and send data
    for i = 1: byte_sizes
            byteSize                = ByteSize(i);
            for j = 1: iterNumber
                    for r = 1: repeat
                            element = pnet(tcpCon,'read',byteSize,'uint8');
    %                       pnet(tcpCon, 'write' , element, 'uint8');
                    end
                    pnet(tcpCon, 'write' , element, 'uint8');
                    data_collected  = data_collected + length(element);
           end
    end
    %Store the final result
    str =  ['Test completed. Number of uint8 elements received: ' num2str(data_collected)];
    disp(str);
    try
        pnet(tcpCon, 'close');
    catch exception
        retCode = 1;
    end

Client File 2: client_test.m

function TimeElapsed =  client_test(con, byteSize, iterNumber, repeat)

if (nargin < 1)         connectionType = 'tcpconnect';          end
if (nargin < 2)         byteSize = 100;                         end
if (nargin < 3)         iterNumber = 1000;                      end
if (nargin < 4)         repeat = 1000;                          end

%Initialize
payload                 = uint8(rand(byteSize,1));
TimeElapsed             = zeros(iterNumber,1);

%Carry out experiment
for i   = 1:iterNumber
        tStart  = tic;
        for r   = 1: repeat
                %send
                pnet(con,'write',payload, 'uint8');
        end
        %receive
        element = pnet (con, 'read', byteSize, 'uint8');
        TimeElapsed(i) = (toc(tStart)) / repeat;
end

Client function 1: benchmarking_pnet code

      function [Results_thoughput, Results_latency] = benchmarking_pnet(host, remote, iterNumber, repeat, draw_plot)
    %How to run
    % Start this function after startigng the server function in another matlab session. 'host and remote values have been associated with hardcoded addresses for this test bench
    if (nargin < 1)         host            = 'local';      end
    if (nargin < 2)         remote          = 'local';      end
    if (nargin < 3)         iterNumber      = 1000;         end
    if (nargin < 4)         repeat          = 1000;         end
    if (nargin < 5)         draw_ plot      = 1;            end
    %Add pnet to path
    if isunix
            pnetPath ='../pnet_benchmarking/pnetLib';
        else
            pnetPath = '..\pnet_benchmarking\pnetLib';
        end
    addpath(pnetPath);
    %Hardcoded IP addresses of machines on which test are performed
    if strcmp(remote,'Win32')
        remoteHost = '192.168.140.24';
    elseif strcmp(remote,'Win64')
        remoteHost = '192.168.141.253';
    elseif strcmp(remote,'Lin64')
        remoteHost = '192.168.140.22';
    elseif strcmp(remote,'Lin32')
        remoteHost = '192.168.140.23';
    else
            remoteHost = '127.0.0.1';
    end
    disp(remoteHost);
    %Initialize
    ByteSize                = [100, 500, 1000, 1500, 2000, 5000, 16384];
    byte_sizes              = length(ByteSize);
    TimeElapsed             = zeros(iterNumber, byte_sizes);
    Results                 = zeros(byte_sizes,7);  
      %Set up connection
    connectionType          = 'tcpconnect';
    con                     = pnet(connectionType, remoteHost, 1677);
    if con == -1
            Error = ['connection could not be established, Error:' num2str(con)];
            disp(Error);
            return;
    end
    %Begin Testing
    for i = 1: byte_sizes
            byteSize = ByteSize(i);
            TimeElapsed(:, i) = client_test(con, byteSize, iterNumber, repeat);
            %Compute the Latency. 
    %       if (i > 1)
    %               TimeElapsed(:,i) = TimeElapsed(:, i) - byteSize .* (TimeElapsed(:,1);
    %       end
            [muhat,sigmahat,muci,sigmaci] = normfit(TimeElapsed(:, i));
            Results(i, 1)   = 8* mean( byteSize ./ TimeElapsed(:, i) );
            Results(i, 2)   = muhat;
            Results(i, 3)   = sigmahat;
            Results(i, 4)   = muci(1);
            Results(i, 5)   = muci(2);
            Results(i, 6)   = sigmaci(1);
            Results(i, 7)   = sigmaci(2);
    end
    %Store Results and Display plots
    str = ['Total uint8 items sent from the client program:' num2str(sum(ByteSize) * iterNumber)];
    disp(str);
    filename = ['Host-' host 'Remote-' remote];
    result_filename = ['Results/' filename 'Results.txt'];
    dlmwrite(result_filename, Results, 'delimiter', '\t', 'precision', 6);
    %print output on console
    fprintf('%15s%15s%15s \n', 'Send','Throughput','MeanLatency');
    fprintf('%15s%15s%15s \n', 'Size', '10^6','10^(-3)');
    fprintf('%15s%15s%15s \n\n', 'bytes', 'bits/sec','sec');
    for i = 1: byte_sizes
            fprintf('%15d%15.3f%15.3f\n',ByteSize(i),Results(i,1)/(10^6), Results(i,2)*(10^3));
    end
% plotting the Elapsed Time graph
if (draw_plot == 1)
        iteration               = 1: iterNumber;
        figure_filename         = ['Figures/' filename 'Plot.png'];
        handle                  = figure('Visible', 'off');
        plot(iteration, TimeElapsed, '--s', 'LineWidth',1,'MarkerSize',5);
        xlabel('Iteration Number');
        ylabel('Latency (seconds)');
        legend('100', '500', '1000', '1500', '2000', '5000');
        title(['Latency Plot for' filename]);
        print(handle, '-dpng' , figure_filename);
end
%End of figurediaplay

pnet('closeall');
clear con;

[Walter Roberson]

Could you post the corresponding netperf results over the same machines?

I might have missed it, but at first glance I don't see where you have set buffer sizes? Buffer sizes can make large differences for TCP connections.

[Priya Bhat]

I am sorry, I don't completely understand. My same machine do you mean that I do a "netperf -H 127.0.0.1"? These results are actually on the same machime (server and client for pnet_benchmarking are running on the one machine, which has the IP address 192.168.140.22).

So the results shown above are: Netperf shows a throughput of 4302.63 Mb/s and pnet_benchmarking shows a throughput of 460.342 Mb/s when the packet size is 16384 bytes.

Please send me a quick line if you meant something else and I would be happy to post the results

[Walter Roberson]

It appears the pnet you are using is what I know as tcpudpip, http://www.mathworks.com/matlabcentral/fileexchange/345-tcpudpip-toolbox-2-0-6

I haven't done any benchmarking of that.

[Priya Bhat]

Well thanks anyway. And do let me know if you have any thoughts about this.

Answer 4

[Priya Bhat]

Does anyone have an idea about how much longer it takes to send a very small (20 bytes) packet using pnet as compared to ping.

ping <machine_address> -s 50

I am getting an order of magnitude difference and I want to know if someone can corroborate that. On my machine the ping time is 0.021 ms while the pnet round trip time is ~.50 ms

Answer 5

[Erik Flister]

you need to set your snd/rcv buff sizes, they are not tuned for fast small messages by default.

http://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-recvbuf

also turn off nagle's algorithm. http://blogs.msdn.com/b/windowsazurestorage/archive/2010/06/25/nagle-s-algorithm-is-not-friendly-towards-small-requests.aspx

specific edits given here: http://www.mathworks.com/matlabcentral/fileexchange/authors/88910