Board index » Visual Studio » dynamic array allocation

Visual Studio326
Hello,



In an application that I am developing I need a 4 dimensions

array (such a[5][400][200][200]). My PC has 2G Ram and running windows XP

professional.







Since this arrays should be a dynamic one ( the size of each dimension could

be changed in run time), I tried to write a class to wrap it. In my class I

tried to allocate a one dimension array and then intelligently selecting the

best one. But I was unsuccessful. A search shows me that I can't do this

as:







1- I can't allocate more than 32 MB at each time. ( is it a real

restriction?)



2- I am getting out of memory when I am trying to allocate memory.







To solve the problem I am looking for help:



1- Is there any tutorial or document on how to solve the problem?



2- Is there any a library or a class for multidimensional array

allocation.



3- Can I use a memory map file to solve the problem? Is it a good

idea?



4- Any other help or suggestions are very welcomed.











Best regards


-

ma wrote:



The quantity of RAM is (mostly) irrelevant to your problem....



memory...





easiest solution at hand, though it is a bit clumsy and is probably not

optimum in memory usage.

boost::multi_array may also be an option. But since we don't know

anything about your problem (what is the use of this memory?), we can't

really help.



Except if you have a quite big application, this is not a problem per

se.



Arnaud

MVP -VC



-

Thanks for your reply and sorry for lack of information:







The code that I have is something such as follow:



class myarray



{



private:



double * m_data;







public:



void init(int dim1,int dim2,int dim3,int dim4)



{



m_data= new double[dim1 * dim2 * dim3 * dim4]; // this is the place

that I get out of memory error



}



}







the call to function is as follow:



...



myarray anarray;



anarray.init(50,400,200,200);



...







I like to know how I can allocate a multidimensional array dynamically.



I need the array to do the following:



1- Read and write to array and pass it to different function. (One instance

of the array)



2- Writing to a file (another instance of the array)



3- Reading from a file (another instance of the file)











In some run of the application, the size of this array may increase to 1 or

2GByte. What is the real restriction on memory allocation?











Best regards





<adebaene@club-internet.fr>wrote in message



-

//what i understood from ur code is that u trying to allocate memory

dynamically for 4 dimensional array.But this will allocate a 1 dimension

array of size dim1 * dim2 * dim3 * dim4





-

ma wrote:

Your program has a limit of 2 GB of address space available. The

program code (both yours and DLLs) will occupy some fraction. So you

probably have 1.x GB of addresses available.



However, you are trying to allocate contiguous memory. The available

addresses are probably fragmented into multiple ranges, so allocating a

contiguous range>1 GB is probably impractical.



A memory-mapped file does not change this problem: The addresses

available remain the same. However, you can create a large memory

mapped file (>2 GB ) and map a smaller section of it into your address

space. This is a solution, but awkward to program and slow in performance.



Another solution is to move your application to a 64-bit machine with a

64-bit program. The memory space limits effectively disappear with a

64-bit address space.



--

Scott McPhillips [VC++ MVP]



-

ma wrote:

First of all, you should probably drop the init function and do the

initialization in the constructor instead.



It is generally hard to allocate a single array of more than about 512MB

on a 32-bit machine, though you can get lucky, and you can generally

allocate more if you allocate the memory up front, when your program

starts. You can increase this a bit using this:

http://msdn.microsoft.com/library/default.asp?url" rel="nofollow" target="_blank">msdn.microsoft.com/library/default.asp=/library/en-us/memory/base/4gt_ram_tuning.asp



Finally, you would be better off allocating your memory in smaller

chunks, by splitting off one of the dimensions. e.g.



double** m_array;



m_array = new double*[dim1];

for (int i = 0; i != dim1; ++i)

{

m_array[i] = new double[dim2 * dim3 * dim4];

}



or similar (preferably using std::vector or a similar class).



Tom

-

ma wrote:

Ok, so this is double data, and with the "dimensions" you try in your

code, this would allocate : sizeof(double)*50*400*200*200=6.4 GB !!!



Are you sure you need *that much* memory? Can't you arrange your

algorithm to be less greedy? If you describe what you intend to do,

someone may help to come with a less expensive solution...



As you have been explained, under Win32, you've got 2GB of user virtual

memory space to hold *all* your data : code, DLLs, stacks, variables,

etc... What is more, this memory space may be fragmented so that it is

impossible to allocate all the reee memory in one contigous chunk, as

you are trying to do.





it's roughly 1.3 GB.



Either switch to a 64 bits platform with plenty of RAM, either use a

memory mapped file by mapping into memory only small "chunks" of the

data ones at a time (but be aware that the file itself would be 6.4

GB!). The simpliest solution however is almost certainly to modify the

algorithm to require less memory.



Arnaud

MVP - VC



-

This is 8*50*400*200*200 = 6,4000,000,000 as I do the computation. You just asked for the

impossible: 6.4GB of data allocated in a 32-bit address space with an absolute maximum of

4.2GB.



Even if you use the value 5 instead of 50, that's 640MB. That's asking a lot. The issue

is not the total space available, but the total amount of contiguous space available. This

is a function of how your memory is fragmented by executable code, data segments, etc.

You should essentially not expect to get this much memory in a single chunk. Arrays of

arrays are a better approach (the example of nested std::vector, for example). because

each fragment might fit, where the entire array will not. NOTHING is going to allow you

to allocate this much memory on 32-bit windows. You can use memory-mapped files to window

into pieces of something this large, or you can use Win64 where you can have 8TB of

virtual memory. Note that you had better have about 10GB of swap file allocated, and I

suspect that the performance of working on an array this size will be dominated by page

fault time, and therefore execution will be unrealistically long if all you do is iterate

over the array.



joe





On Fri, 14 Oct 2005 12:42:39 +0100, "ma" <ma@nowhere.com>wrote:



email: newcomer@flounder.com

Web: www.flounder.com">www.flounder.com

MVP Tips: www.flounder.com/mvp_tips.htm">www.flounder.com/mvp_tips.htm

-