How to only follow certain links with matlab spider
조회 수: 2 (최근 30일)
이전 댓글 표시
Hi I am struggling with only allowing certain urls to be followed when using a spider to build a web graph. Basically I only want the spider to follow links that point to the uni server( shef.ac.uk), any other urls need to be discarded, opposed to the current state were all links are followed. Probably quiet a simple fix.
U = cell(n,1);
hash = zeros(n,1);
L = logical(sparse(n,n));
m = 1;
U{m} = root;
hash(m) = hashfun(root);
for j = 1:n
try
disp(['open ' num2str(j) ' ' U{j}])
page = urlread(U{j});
catch
disp(['fail ' num2str(j) ' ' U{j}])
continue
end
for f = findstr('http:',page);
e = min(findstr('"',page(f:end)));
if isempty(e), continue, end
url = deblank(page(f:f+e-2));
url(url<' ') = '!';
if url(end) == '/', url(end) = []; end
skips = {'.gif','.jpg','.ico'};
skip = any(url=='!') | any(url=='?');
k=0;
while ~skip && (k < length(skips))
k = k+1;
skip = ~isempty(findstr(url,skips{k}));
end
if skip
if isempty(findstr(url,'.gif')) & isempty(findstr(url,'.jpg'))
disp([' skip' url])
end
continue
end
i=0;
for k = find(hash(1:m) == hashfun(url))';
if isequal(U{k},url)
i = k;
break
end
end
if (i == 0) & (m < n)
m = m+1;
U{m} = url;
hash(m) = hashfun(url);
i=m;
end
if i > 0
disp([' link ' int2str(i) ' ' url])
L(i,j) = 1;
end
end
end
댓글 수: 0
답변 (0개)
참고 항목
카테고리
Help Center 및 File Exchange에서 Antennas and Electromagnetic Propagation에 대해 자세히 알아보기
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!