Table of Contents
- What is a scratch folder?
- How can I access the scratch folder on Schooner?
- What is the Schooner scratch autodelete policy?
- Why do we have an autodelete policy?
- Where can I put my files?
Your /scratch folder is meant to be a space for temporary files, even large ones, and is ideal for jobs that do a lot of file accesses (reading, writing, etc.) and/or consume a lot of disk space. You can modify your programs so that the first step is to copy the files to /scratch before processing them. Once all the processing is done, they can be copied to elsewhere. If you're really done with them, you should consider deleting them from /scratch as well, so that others can use the space. If you're not done, you can leave them there to be used again. We strongly recommend users to treat scratch as a temporary storage. We do not back up the scratch folder. While OSCER currently does not enforce quotas on /scratch, we do have a two week deletion policy. Please see the Schooner scratch autodelete policy for more information.
After logging into Schooner, you can go to your scratch folder by using the following command: cd /scratch/username (replace username with the username you used to log into Schooner).
- When a file is created in your /scratch directory, its clock starts ticking.
- Every file in your /scratch directory will get at least two warning e-mails before being effectively deleted but STILL RECOVERABLE.
- Those warning e-mails will be at least 5 calendar days apart.
- No file will be effectively deleted (but still recoverable) until the later of:
- at least 7 calendar days after the second such warning e-mail
- at least 14 calendar days after file creation,
whichever comes later.
- Once a file is effectively deleted (but still recoverable), you will receive at least one more warning e-mail about it, at least 5 calendar days after the most recent warning e-mail. That'll be a warning about:
- that file being temporarily eligible to be restored (upon request);
- the file becoming eligible to be physically deleted soon.
- No file will be physically deleted until the later of:
- at least 21 calendar days after it was initially created in your /scratch directory
- after you have received a total of at least 3 e-mail warnings about it,
whichever comes later.
- Once a file is physically deleted, it is GONE FOREVER AND NEVER RECOVERABLE.
- NO OTHER Schooner filesystem is subject to automatic file deletion -- only /scratch.
Before we had an automatic deletion policy in place, our /scratch filesystems were typically between 95% and 100% full. That made it extremely difficult for anyone to get any work done -- either they couldn't write files at all, or writing files was very slow.
Our experience has shown that a predictable and recurring scheduled autodelete policy on scratch allows our users to better prepare for their file deletions. This is far better than ad-hoc and irregularly scheduled deletions that are often performed with little notice because the scratch storage has reached critical levels.
Everyone has files that they need to retain for longer than 2 weeks. Where to put them?
- Oklahoma PetaStore
You can archive your files on the Oklahoma PetaStore and download them to /scratch when you need them. Here's what you'll need to do:
- Request a PetaStore account (as an extension of your current OSCER account).
- Fill out and submit a PetaStore Use Agreement form.
- Do an hour of training.
- Buy tape cartridges, which are incredibly cheap - https://www.cdwg.com/shop/products/IBM-TotalStorage-LTO-Ultrium-x-1-2.5-TB/2890034.aspx. For OU users we can probably beat that price by around 5-10%, using OU's internal "Crimson Corner" purchasing system. (Look for item CWD#2890034.)
(We recommend buying enough tape cartridges for dual copies of all your files, in case a tape cartridge breaks -- it doesn't happen often, but it definitely does happen.)
NOTE: Access to the PetaStore is available to researchers and educators at US institutions only.
- /work Folder
If you have a compelling need for some of your files to be accessible for longer than 2 weeks, please contact us at firstname.lastname@example.org to discuss having a /work directory for your files that you expect to need to access for longer than 2 weeks.
We can make a /work directory available on request. But, we'll need to understand why it's needed. Then we'll revisit with you each month to see whether you continue to need it. Please contact us if you want us to set up a /work directory for you.
- /data Folder
A /data directory is very much like a /work directory, except: A /data directory contains files that are needed by more than one research group.
We can make a /data directory available on request. But, we'll need to understand why it's needed. Then we'll revisit with you each semester to see whether those multiple research groups continue to need it.
Please contact us at email@example.com if you want us to set up a /data directory for you.