[Ilugc] Can we access the SWAP SPACE in Python
rajasuperman at gmail.com
Fri Oct 7 18:09:32 IST 2011
On Fri, Oct 7, 2011 at 5:16 PM, Arulalan T <arulalant at gmail.com> wrote:
> Is there any way to store the data (temporarily) in the SWAP SPACE instead
> of RAM ?
No, it's best to let the kernel handle memory management. But read below.
> Is there any way to store some data in SWAP (but not in RAM) and get the
> data from the SWAP partition whenever we need it in Python or C language ?
> yup. It may be slower to access the swap area rather than RAM, but it is
> much faster than access the data of files which are stored in the hard disk.
Use mmap to map a file directly into your processes address space. When
you reference the memory address, the kernel will read/write data from the
file using the same mechanisms it uses to swap. You will bypass all file
IO buffering and have direct access to the files. This is how all DB engines
handle IO. mmap() vs read() will give you a 3 x speedup for IO bound jobs.
Advanced Programming in the Unix Environment by Stevens has a good
example of file copy implemented using mmap.
> It takes 2 days to complete this job and eats 3GB RAM memory.
Mechanisms to speed up scientific code:
1. Make fewer passes over your data. Ideally you want to read the
data once, complete all processing and never have to read it again.
Do not make several passes over data doing a little processing each
2. Use efficient IO like mmap.
3. Use the correct algorithms suitable for your data size and work.
4. Profile your code to understand which step takes the most time.
Then start fine tuning.
5. Run a 64bit OS, you'll sometimes need to use more RAM if you
want faster performance. It's a trade-off.
6. Code the hot spots in C if you need more speed.
> One of the reasons is ' 2 x 360 x 720 ( = 518400 ) times needs to access
> python file object to retrieve the data of corresponding latitude,
OS, python libs will cache IO. So you will not hit disk each time you
read the data.
More information about the ilugc