Temporary Data Storage
Home directories in TCM are too small and slow to be appropriate for temporary storage of large (or even moderate) amounts of data. The usual compromise between speed, capacity, price, reliability and backups was weighted against speed and capacity. But there are two other areas for data storage.
Firstly, /rscratch/{uid}. Like your home directory, this is on a central fileserver and is accessible equally from all PCs, so it is equally convenient. Quotas are generally at least ten times bigger, and currently the link to the fileserver is ten times faster than that to your home directories. Unlike your home directories, it is not backed up. However, it is on a RAID system, so it should survive a single disk failure.
Secondly, /scratch/{uid}. This is merely the local disk of each machine. Access to it is not via any network, so it is much faster, and aggressive local caching can occur. Compilations here are likely to be noticeably faster than on remote directories. However, there is neither backup nor protection against disk failure.
Quotas etc.
Quotas exist (in part) to prevent people filling disks, and thus spoiling things for everyone else. The default quota on /rscratch is merely 2GB because many people choose not to use it, and in order to ensure that the disk cannot become full, I keep the sum of all quotas less than the total capacity of the disk. If you are interested in using it more, a doubling or quadrupling of your quota on this disk would readily be granted. (On home directories requests for quota increases are also often granted, but the bottle-neck of backups means that large factors are likely to be resisted.)
Quotas do not exist for /scratch, so people can fill this disk. To find out how much space is available:
df -h /scratch(or df -k /scratch). To find out who is using what, type repscratch.
To find out which PCs have lots of space on their /scratch disks,
rscratch -freeand to find out where you have left your data
rscratch
Please delete data you no longer need. These disks are all finite, and once a PCs /scratch disk is full, it becomes useless for running most codes, and Firefox starts misbehaving too.
Other storage
Two other solutions exist for data storage. Either the purchase of some form of USB disk, which you can plug in, mount (mount /usb), use, unmount (umount /usb), and remove. (If using two simultaneously, the second will be /usb2). Or the purchase of an extra SATA disk to put inside a computer.
USB disks are conveniently removeable and portable. They are slightly slow - the theoretical bit rate for USB2 is 480MBit/s (60MB/s), but the protocol has quite large overheads, so one would be very lucky to see more than 50MB/s, and more likely 30-40MB/s. This is a problem for large disks, for 40MB/s is about 140GB/hr, so a 1.5TB disk takes over ten hours to fill.
USB disks also give rise to filesystem problems. Our PCs will accept FAT32, ext2 and ext3. Windows accepts FAT32 and NTFS, MacOS accepts FAT32 and HFS+. The common demoninator is clearly FAT32, but FAT32 has serious performance problems for volumes bigger than 500GB (although in theory it can be used up to 2TB). It also has a 4GB limit on the maximum size of a single file.
Internal SATA disks are likely to end up ext4 formatted, and may manage transfer rates of around 100MB/s.
There are two issues to consider with both of these routes. Firstly, backups for the data. A single disk is not very secure against failure, and any scheme involving copying 1TB of data across a 100MBit/s network is unlikely to be popular - it will take 100% of the network bandwidth for 24hr. Secondly, on which PC should the data be placed? If one ever wishes to transfer it to another machine, the best solution for a USB drive is to unplug it and physically move it. Not a solution for internal drives, or for target machines to which one lacks physical access. Although a 1GBit/s network would reduce this problem, even then one needs about 3hr to move 1TB, and 3TB drives are cheap...
Finally, there are always writable CDs and DVDs.