Circshift function on very large array

My question is I am performing BWT transform whose first step is to perform circular shift on input text which can be from any txt file and save each shift to form NxN matrix, where N is length of input text from file. Now, if I input small text like name or some sentence of 30 character everything works very well. However,it sticks and gives error not enough memory if I input 1mb or more than 1 mb file. Please anyone here suggest me the solution?

[James Tursa]

You are trying to create and manipulate a 1MB x 1MB matrix?

[Muhammad Waqas]

Yes, size can carry depending upon size of input file

[Muhammad Waqas]

I input 1mb txt as a vector of length N and then try to create it's matrix of dimension NxN by applying circshift function

Answer 1

[John D'Errico]

Yes, but you need to understand that just because your mind can conceive of an idea, does not mean your computer can solve the task.

The array that you want to create, for a vector with one million elements, for example, will create an array of size 1e6x1e6.

Assume that you are creating character arrays, so every element will require 2 bytes of storage. The result will be a total of 2 terabytes of memory required. Does your computer have more than 2 terabytes of contiguous RAM to store this array, and problem 2 or 3 times that, since you may need to make copies of it? I usually suggest 3 times the total memory is a good safe amount. These days, most computers seem to have between 4 and 64 gigabytes of memory. You would need roughly 1000 times as much memory to store this array.

So do you have terabyte sized RAM chips in your computer? Even with moderately inexpensive chips these days, terabyte RAM chips are still expensive. This is more memory than most people probably have in a disk drive.

The point is, your computer is not infinitely large in its capacity. If it tells you it does not have enough memory, then it says you are trying to do something impossible.

Computing is sometimes be viewed as an art. The art lies in learning to find better ways to solve a problem than just using brute force algorithms to solve a problem, that would be impossible to solve otherwise. We have no idea why you are wanting to do something impossibly large. But there are often better ways to crack a walnut than by dropping a Mack truck on top of it.