Using Bioarchive

Bioarchive

Bioarchive is set up for long-term data storage at a low upfront cost, which allows grant PIs to preserve data well beyond the funding period. Storage space can be purchased before the end of the grant, and data will be safely kept for 10 years from purchase. If a longer term of storage is needed, PIs may contact Grants and Contracts for approval. Large datasets that are not actively used should also be moved to Bioarchive to save for the higher storage cost on Biocluster or other storage services.Any type of data, except for the HIPAA data, is allowed on Bioarchive.​

How Bioarchive Works

Bioarchive is a disk-to-tape system that utilizes a Spectralogic Black Perl disk system and a Spectralogic T950 tape library with LTO8 tape drives. This setup allows users to transfer data to the archive quickly because data is written to spinning disks, and then the software will move the data to the tape. Unless specified otherwise, all data on Bioarchive is written onto two tapes. One of them stays in the library in case the data is needed, while the other tape is taken to a secure offsite storage facility.​

How Bioarchive Billing Works

Each private storage unit is called a Bucket (like a folder), and each bucket needs to be associated with a CFOP number. A user is granted 50GB of storage space before any charge. This is to ensure that the service meets their needs before commitment. Once a bucket passes this initial 50GB threshold, we bill $200 for each TB of data in the next monthly billing period. Ten years after a Bucket is billed, the PI will be offered the choice of removing the data or paying for an additional 10 years.To check your Bioarchive billing, go to https://www-app.igb.illinois.edu/bioarchive. if you need to update the CFOP associated with your account, emailhelp@igb.illinois.edu.​

How do I start using Bioarchive

Please fill out the Bioarchive Bucket Request Form:https://www.igb.illinois.edu/webform/archive_bucket_request_form

You will receive an email from CNRG once your account is created.The email includes a temporary password. Change your passwordIMMEDIATELY. To do so, go tohttps://bioarchive-login.igb.illinois.eduand log in using your NetID and the temporary password.

• Click on your NetID at the top-right corner and then choose "User Profile".
• Click on "Action" at the top-left corner and then choose "Edit".
• Enter your temporary password in the Current Password field, and then create a new password and confirm it. Be sure to click "Save" when you are done.

​

Get S3 Keys

The credentials to access your data are the S3 keys that are assigned to your account. To obtain the keys:

• Log into<a class="external free" rel="nofollow">https://bioarchive-login.igb.illinois.eduwith</a> the username and password created in the previous step, or continue from there if you just changed your password.
• Click on "Action" at the top-left corner and then choose "Show S3 Credentials".
• A pop-up box will appear with your S3 Access ID and S3 Secret Key. Copy them to a safe place as you will need them to access Bioarchive.

​

How to transfer data to Bioarchive

We support data transfer from your local computer system and from Biocluster.To prepare for data transfer, you are recommended to put all data for one project in a directory and compress it into a file. Add a readme file to explain the contents inside. Put both the compressed and readme files in a folder.​

Data transfer from a computer

The easiest way is to use Eon Browser. Download and install the Eon Browser GUI on your computer.Windows -BlackPearlEonBrowserSetup-5.0.12.msiMacOS -BlackPearlEonBrowser-5.0.12.dmgNote for OSX systemsIf the system says that the file is corrupted, open a terminal session after you copy the program to your applications. Type the following to the terminal.​

Option B - Install Eon Browser

You can download and install the Eon Browser GUI for the archive on your local system.

Windows - BlackPearlEonBrowserSetup-5.0.12.msi

MacOS - BlackPearlEonBrowser-5.0.12.dmg

This is useful if you are archiving data from your local computer or from systems other than Biocluster.

Note for OSX systems

If the system says that the file is corrupted, doing the following in a terminal session after you copy the program to your applications should fix it.

cd /Applications
sudo xattr -cr BlackPearlEonBrowser.app

==

===Using Eon Browser

=

The Spectralogic User Guide for the Eon Browser is athttps://developer.spectralogic.com/wp-content/uploads/2018/11/90990126_C_BlackPearlEonBrowser-UserGuide.pdf.

Use this information when opening a new session on the archive:

• Name: Bioarchive
• Data Path Address: bioarchive.igb.illinois.edu
• Port: 443
• Check the box for SSL
• Access ID: Your S3 Access ID
• Secret Key: Your S3 Secret Key
• Check the box for Default Session if you want it to automatically connect to Bioarhive when you open the program.
• Click Save to save your session settings so that future connections will use the same info.
• Click Open to open the archive session.

You should see your local computer directories on the lefthand side, and your available buckets on the right. To transfer data, just drag from your local computer directories and drop it in the appropriate bucket. You will see the progress in the bottom of the program window.When transferring a file, it will first go into the cache and then get written to tape. You can see this happening in the Eon Browser by looking at the storage locations column. A purple disk icon shows the data is in cache, and the green tape icon shows that the data is on tape.While transferring a file, the session will not show a complete transfer until all data is written to the tape. This can take some time. Please check the sleep settings on your computer in advance of the transfer, which can get interrupted if your computer goes to sleep. Using an ethernet instead of a wireless connection would also facilitate data transfer.Another way to transfer data from your local system to Bioarchive is to use a command line tool. To do so, download and install the command line tool on your local system: https://developer.spectralogic.com/clients/. The Spectralogic user guide for the command line tool is athttps://developer.spectralogic.com/java-command-line-interface-cli-reference-and-examples/. You may need to reconcile your local computer environment with the tool for this to work. Thus, if you aren’t sure about the difference between the two environments, use Eon Browser. The CNRG would not provide further support.​

Data transfer from Biocluster

The tool to access Bioarchive is already installed on Biocluster. You will need to load the module.

module load ds3_java_cli

This will add the utilities to your environment. Then, load the program we created to set up the environment variables. Enter your secret key and access key when prompted. All the bold textin commands below denotes items that need to be replaced with information pertinent to you.[]$ archive_environment.py

Directory exists, but file does not Enter S3 Access ID:S3_Access_ID Enter S3 Secret Key:S3_Secret_Key Archive variables already loaded in .bashrc Environment updated for next login. To use new values in this session, please type 'source .archive/credentials'

Be sure to type:

source .archive/credentials

​In all of the following commands below, we are assuming you are using biocluster to do the transfer and have run this program. If this is not the case then you need to include: <brclass="single_linebreak">

-a S3_Access_ID -k S3_Secret_Key -e bioarchive.igb.illinois.edu<brclass="single_linebreak"> On every line immediately after the ds3_java_cli command and before the rest of the commands as shown below.

Limits

There are not many, but as we find them, they will be placed here

• You can only put 500,000 objects at a time on the system. If you need up put more than 500,000 files and folders, you will need to split that up between two or more submissions. However this is assuredly a sign that your data needs to be organized better. Please contact us if you run into this issue.

• You can only put 500,000 objects at a time on the system. If you need up put more than 500,000 files and folders, you will need to split that up between two or more submissions. However this is assuredly a sign that your data needs to be organized better. Please contact us if you run into this issue.

Upload Items in A Directory

ds3_java_cli -b Bucket_Name -c put_bulk -d Directory_Name

• put_bulk tells the program to upload everything inside the directory given
• Directory_Name is the directory to upload, note this will upload everything in this directory, not the directory itself

Upload One File

ds3_java_cli -b Bucket_Name -c put_object -o File_Name

• uploads a single file to the archive
• uploading a file with -o SomeDirectory/somefile will put the uploaded file in the directory SomeDirectory
• to put all files in a certain subdirectory, use the prefix option -p path/to/another/directory/

See Files in Archive

ds3_java_cli -b Bucket_Name -c get_bucket

• This will list all of the files in the archive and the total size of data in the bucket

Download a File From the Archive

ds3_java_cli -b Bucket_Name -c get_object -o File_Name

• This will download the object named File_Name
• Note if there are other read or write jobs running on the archive and the file is not in cache, this will appear to hang while it waits for the drives to become available.

Download a Directory From the Archive

ds3_java_cli -b Bucket_Name -c get_bulk

• This will download all objects in a bucket. Use carefully.
• You can add -p prefix to restrict which files get restored

List Available Buckets

ds3_java_cli -c get_service

• Returns a list of buckets that the user has permission to access.

Best Practices for Uploading Data

Create the Environment

First make sure you set up your command line environment properly via the archive_accounting.py script outline above. You can check if this is working by doing a:<brclass="single_linebreak">

ds3_java_cli -c get_service<brclass="single_linebreak"> and making sure that your buckets show.

Upload With a Separate Folder

To upload data, it is best that you have all data in one directory named something like "to_archive" then you can put all of the data you want to upload to the archive in that folder, exactly like you want it to be on the archive. Make sure these files do not already exist in those locations in the archive with either the Eon Browser or the command line tools. When you are ready, then run this command:<brclass="single_linebreak">

ds3_java_cli -b Bucket_Name -c put_bulk -d to_archive<brclass="single_linebreak"> Note that the to_archive folder does not exist in the bucket now. This command moves everything inside of the to_archive folder into the archive, but not the to_archive folder itself.

References

• https://developer.spectralogic.com/java-command-line-interface-cli-reference-and-examples/