Computer Laboratory

Filespace

The department is fairly well-endowed with disc space, and has a sophisticated backup system. Our file server is known as elmer or the filer. The filer is a Redundant Array of Independent Discs, also known as a RAID array. We use double RAID, which means the filer can survive a double disc failure. The lab also has a second filer in a separate building which partially mirrors the main filer, so your files are well-protected. The filer also maintains data about your old files, so you can retrieve any files you might have deleted by accident.

Lab members typically have access to several types of filer space, such as the home directory, group filespace, global temporary space and global scratch space. You can access the filer from either Windows, Unix, or Mac machines. If you would like to make any changes to the documents stored on the filer (especially any changes in permissions using chmod in Unix or under "Properties|Security" in Windows), make sure to only do so from one system for each given set of files, as changing the same permissions from more than one system will likely result in conflict.

Please note that this page uses Unix filepaths when describing locations on the filer (e.g. /auto/userfiles/gsm10. For Windows filepaths, reverse the direction of the slashes, make the first slash a double, and replace the first part of the Unix name (eg "auto", "anfs", "usr") with "filer".

Make sure you only store important files in either your home directory or group space, as temporary space and scratch space do not get backed up.

The home directory

elmer

When your CL account is created, an allocation is made for you on the file server. We create a filespace known as your superhome, which holds both your Unix home directory (known as your unix_home) and your Windows home directory (known as your windows_home), as well as your Windows roaming profile. Everybody gets both a Unix and a Windows home, even if they have stated that they are going to be Unix- or Windows-only users. The preferred path for viewing your superhome is via /auto/userfiles/ under Unix or \\filer\userfiles under Windows.

Your filespace is subject to a certain quota: the default quota is (as of 2016) 50 GB, which covers both your Unix usage and your Windows usage. You can examine your current usage and quota using the CL website (requires Raven authentication); alternatively, there is a command cl-rquota on most Linux systems which will give the same information.

The filer is best at keeping large amounts of static data, and does not perform as well with small files which change frequently. Because of this, we would prefer users to have a large enough quota to hold their working set of files, rather than continuously copying files on and off their filer space. If you find yourself continuously copying files to other places to avoid going over quota, then please

We expect you to be realistic and honest in assessing your needs, bearing in mind that maintaining a high level of availability and reliability of filespace is fairly expensive.

Everything in your home directory (and superhome) is intended for your personal use. Please do not create directories and invite other people to create files in them, or accept invitations from others to write into their space; any collaborative work should be done in a group filespace.

Unix

Your Unix home directory is located in /home/CRSid/. When you try to access your home directory for the first time, make sure you navigate to it directly, as it is auto-mounted and as such will not initially appear in /home.

Some commands such as df will reveal the "real" file structure that underlies this auto-mounting:

$ df /home/gsm10
Filesystem           1K-blocks      Used Available Use% Mounted on
elmer:/vol/vol1/homes-1
                      62914560  43748280  19166280  70% /Nfs/Mounts/homes-1

This reveals that your home directory is in fact housed on /Nfs/Mounts/homes-1/CRSid, your superhome. Despite that, you should always refer to your home filespace as /home/CRSid/, because system administrators can move filespaces around to correct faults or balance allocations, and the /home/CRSid/ route is the only one guaranteed to stay in place.

Aside from the space on filer, Linux users can store their files in two local filespaces:

Local temporary space

For very fast access, you can use your temporary disc space located under /var/tmp. This space tends to be used by applications (from web browsers to compilers), and because it is fairly small, you can prevent some applications from working by filling it up. Additionally, Linux reboot sequences are likely to delete things in /tmp, and system management scripts may do the same from time to time; as this space doesn't get backed up, you should only use it for small, temporary items. For anything bigger or more important, consider using local scratch space.

Local scratch space

"Scratch" space is space that is not backed up; if you delete a file in scratch space, it is permanently lost. Any data in scratch space is stored locally, so access to it is very fast. Scratch space is thus ideally suited to bulky data that can be regenerated relatively easily.

To start with, you won't have access to anything in a scratch directory: you need to create a directory to use. Under Unix the command

$ sudo cl-mkscratchdir

will make you a directory on /local/scratch; the directory's name is your user identifier, so if user gsm10 were to issue the command, he would get a directory /local/scratch/gsm10. If given the extra argument "1" it will create /local/scratch-1/gsm10 etc.

cl-mkscratchdir has a number of obscure failure modes, and just one that is relatively common:

  error: /local/scratch is on the / partition.
  Please mail sys-admin for help.

This error happens on a system that hasn't been set up to allow local scratch in a separate partition (it is unwise to use the root partition for scratch as unpleasant things can happen if it gets full up). As the message says, you need assistance from sys-admin (probably, users of FC6 machines may be able to do this repartitioning for themselves).

Windows

Your Windows home directory will be mapped to a drive letter (typically Z:) when you log on to a lab-managed Windows machines. On other Windows machines, you can map your home directory to a drive or add it as a shortcut using the path \\filer\userfiles\CRSid\windows_home (or unix_home).

Windows roaming profiles are copied to the local mach ine from the file server whenever you log on, and copied back to the file server when you log off. For this reason, you should only keep the smallest possible amount of data in your roaming profile, as it does not get backed up until you log out; additionally, large amounts of data will slow down your login/logout process considerably.

Users of workstation do not get a roaming profile by default, you should ensure that your redirect your My Documents folder to the Z: drive. Every terminal server user does get a roaming profile so that you can login to any of the terminal servers and see the same desktop. However, that profile is capped at 30M. If you exceed the limit you will be nagged until you tidy up and the profile will not be saved.

Group filespace

Shared space may be allocated for use by a research or administrative group of people. Group file space is stored on the file server, and located at:

/usr/groups/group-name

in Linux; in Windows, you can find it under

\\filer\groups\group-name

Just as with home directories, the group space directories are automatically created by the auto-mounter the first time you navigate to them. All group space data is backed up regularly.

Group space allocation is subject to quota, and is negotiated by the researcher (typically a project principal investigator) who arranges for the space in the first place. By contrast to home directory quotas, group space quotas apply to the container, rather than to the user who has access to the container.

Global temporary space

For temporary items that don't need to be in your home directory, you can use the global temporary space located at /anfs/bigtmp. This is a single resource for the whole department that can be accessed from any lab-managed machine and is cleared regularly (i.e. does not get backed-up). It's much larger than the typical /tmp partition under Linux, but will be slower, since it's on the other side of the network.

Global scratch space

The department's central file server maintains a system-wide scratch space, known as bigdisc. Data access to bigdisc is not as fast as access to local scratch on Linux, but you can access it from any lab-managed computer, provided you have been allocated space on bigdisc. It is thus useful for common-access items such as local caches of CVS repositories, etc. As with global temporary space, your bigdisc space is not backed-up.

You may ask for an allocation of bigdisc space by emailing sys-admin; when the request is granted, a directory (/anfs/bigdisc/CRSid)is created for you, and a quota allocation made.