Taxi datset - lat lon colocation

Hi everyone,

I’m practicing data science with NY Taxi data (https://www.kaggle.com/competitions/nyc-taxi-trip-duration/data). I want to check if certain taxis were collocated within a short distance from each other.

The task I’m trying to accomplish is:

• Measure the distance for each taxi (against all other taxis)
• If the taxis were within 20 m of each other – just an example
• Check the date column associate with those lat lon matches
• If the date match, meaning the two taxis visited the general area on the same date
• Return the two unique IDs of those taxis, the date, the distance, and the lat lon of both of them

Here is my progress and I would really appreciate your help.

taxi = readtable('yellow_tripdata_2015-01.csv') % reading the data 
taxi.uniqueID = randi([1000000,9999999],[height(taxi),1]) % Creating random hopefully unique ID :)
% measuring the distance from pick up to drop off
taxi.distance_travelled_in_M = deg2km(distance(taxi.pickup_latitude, taxi.pickup_longitude, taxi.dropoff_latitude, taxi.dropoff_longitude))

% spliting date and time in two diffrent columns 
taxi.date = datestr(datetime(taxi.tpep_pickup_datetime),'dd/mm/yyyy') 
taxi.time = datestr(datetime(taxi.tpep_pickup_datetime),'HH:MM:SS')

% In this loop I took each lat lon point and ran it agaist the rest of the
% data- probably not an efficient way - open to suggestions 
% then converting the distance to meters 
% checking if the distance within for example 20 meters 
% and the date is the same - this step is not implemented yet , need help
% return the unique ids of both taxis, date of that event, the location
% (lat,lon), the distacne how far from each other 

for idx = 1:length(taxi.pickup_latitude)
    my_dist = distance(taxi.pickup_latitude(idx), taxi.pickup_longitude(idx), taxi.pickup_latitude, taxi.pickup_longitude); % distqances in degrees 
    my_dist = deg2km(my_dist); % distance in meters 
    if my_dist < 2
        pos_results = [pos_results;my_dist;taxi(idx:idx, ["uniqueID", "date", "pickup_latitude", "pickup_longitude"])]
    end 


end

[Walter Roberson]

    my_dist = distance(taxi.pickup_latitude(idx), taxi.pickup_longitude(idx), taxi.pickup_latitude, taxi.pickup_longitude); % distqances in degrees 

Not really degrees. You are doing Euclidean distance calculations on a non-linear surface. The result is only meaningful if the data is recorded near the equator. New York City is 40.7128N and cosd() of that is about 0.76 so degrees longitude there are only roughly 3/4 of the distance of a degree latitude.

Have you considered calculating Great Circle Distance ?

[LeoAiE]

Hi @Walter Roberson

Thank you for your comments! I have not. I will research how to calculate great circle disctance!

Answer 1

[Walter Roberson]

Instead, do a rangesearch which is a knnsearch by distance.

Either do a custom distance calculation of Great Circle Distance, or else divide the longitudes by cosd() of the latitudes in order to adjust the two of them to be on the same scale, and then use Euclidean.

[LeoAiE]

@Walter Roberson still trying to process your method in my head lol very high level, but I found a function that possibly takes care of the distance. I'm still not sure how to implement the rest of my steps

Steve Ratts (2022). Compute the Great Circle Distance between two points (https://www.mathworks.com/matlabcentral/fileexchange/23026-compute-the-great-circle-distance-between-two-points), MATLAB Central File Exchange. Retrieved November 12, 2022.

function d = greatCircleDistance(phi_s, lambda_s, phi_f, lambda_f, r)
% compute the great circle distance given lat and long for two points
% optionally, a fifth parameter (r) can be specified. If this paramter
% isn't specified it's assumed to be the mean radius of the earth. The
% calculation is done using the Vincenty formula.
%
% INPUTS:
% phi_s    = latitude of the standpoint (base) [rad]
% lambda_s = longitude of the standpoint (base) [rad]
% phi_f    = latitude of the forepoint (destination) [rad]
% lambda_f = longitude of the forepoint (destination) [rad]
% r        = radius of the sphere [units determine units of d]
%
% OUTPUT:
% d        = great circle distance from standpoint to forepoint
%
% See http://en.wikipedia.org/wiki/Great-circle_distance
% If no arguments, bail out
if nargin < 4
    fprintf('Usage: greatCircleDistance(phi_s, lambda_s, phi_f, lambda_f, r)\n')
    return
end
% If no radius supplied, assume the mean radius of the earth in km
if nargin < 5
    r = 6371.01; % km
end
% convert from degrees minutes seconds to radians as needed
if isstruct(phi_s) || (length(phi_s) > 1 && ~isstruct(phi_s))
    phi_s = dms2r(phi_s);
end
if isstruct(lambda_s) || (length(lambda_s) > 1 && ~isstruct(lambda_s))
    lambda_s = dms2r(lambda_s);
end
if isstruct(phi_f) || (length(phi_f) > 1 && ~isstruct(phi_f))
    phi_f = dms2r(phi_f);
end
if isstruct(lambda_f) || (length(lambda_f) > 1 && ~isstruct(lambda_f))
    lambda_f = dms2r(lambda_f);
end
% Compute Delta lambda (delta longitude)
Delta_lambda = lambda_f - lambda_s;
% Compute Delta sigma (central angle)
Delta_sigma = atan2(sqrt((cos(phi_f)*sin(Delta_lambda))^2 + (cos(phi_s)*sin(phi_f) - sin(phi_s)*cos(phi_f)*cos(Delta_lambda))^2), ...
    sin(phi_s)*sin(phi_f) + cos(phi_s)*cos(phi_f)*cos(Delta_lambda));
d = r*Delta_sigma;

function r = dms2r(dms)
if isstruct(dms)
    r = sign(dms.deg)*(abs(dms.deg) + (dms.min + dms.sec/60)/60)*pi/180;
elseif length(dms) == 3
    r = sign(dms(1))*(abs(dms(1)) + (dms(2) + dms(3)/60)/60)*pi/180;
elseif length(dms) == 2
    r = sign(dms(1))*(abs(dms(1)) + dms(2)/60)*pi/180;
else
    r = nan;
end

[Walter Roberson]

https://www.mathworks.com/help/map/ref/distance.html if you have the mapping toolbox

[LeoAiE]

Yes I do have it and thats why I use the distance function in my original post but I guess I have to specify somehting like this

s = referenceSphere('Earth')
distance(taxi.pickup_latitude(1), taxi.pickup_longitude(1), taxi.pickup_latitude(2), taxi.pickup_longitude(2),s)

The issue now how to check the date column associate with those lat lon matches and If the date match, meaning the two taxis visited the general area on the same date then return the two unique IDs of those taxis, the date, the distance, and the lat lon of both of them