필터 지우기
필터 지우기

Feeding in large amounts of data from an online database

조회 수: 1 (최근 30일)
Muneer
Muneer 2014년 2월 6일
댓글: Muneer 2014년 2월 11일
Would appreciate someone pointing me in the right direction:
Currently I am using urlread to feed in data from a password protected database. I read in all the data at once as a block and then start breaking it up with textscan. Currently, I place a limit on the amount of data I'm pulling in. You can see it below (password removed):
url = 'http://assa104/eurofot/csv';
block = urlread(url, 'Authentication', 'Basic', 'Post', {'login', '***', 'passwd', '***', 'trips', '1332244409', 'signals', '1.AVL_BRTORQ_SUM.AVL_BRTORQ_SUM_DVCH', 'limit', '300000','content_type', 'text'});
but when I remove this limit, I get the following errors:
Error using urlreadwrite (line 97)
Error downloading URL. Your network connection may be down
or your proxy settings improperly configured.
Error in urlread (line 36)
[s,status] =
urlreadwrite(mfilename,catchErrors,url,varargin{:});
My question is two-fold: 1. Any idea why I am returning such an error? 2. This method takes a while (almost a minute with my limit at 300k) and I want to read in even more data in the future. Is there an alternative to my current method? Maybe one that doesn't save the whole set in a block at once but one that reads in rows at a time?
I'm not very experienced with urlread, so any assistance would be greatly appreciated.
Thanks in advance.
  댓글 수: 1
Muneer
Muneer 2014년 2월 11일
Here is the message for the exception. Hopefully this sheds some light on the problem.
Java exception occurred:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.grow(Unknown Source)
at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:99)
at com.mathworks.mlwidgets.io.InterruptibleStreamCopier.copyStream(InterruptibleStreamCopier.java:76)

댓글을 달려면 로그인하십시오.

답변 (0개)

카테고리

Help CenterFile Exchange에서 Downloads에 대해 자세히 알아보기

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by