preservazione a lungo termine

As explained in the introductory page, IA2 provides storage space for"Data Sharing"and for"Long-Term Preservation." The latter refers to saving data to a). Quest’ultima si riferisce al salvataggio dei dati su libreria a nastro (works).

How the Tape Libraryworks)

• Once written to the Tape Library, data can no longer be deleted or modified.
• Access to data stored on the Tape Library occurs asynchronously via VOSpace.

How to Request an Account

• Accounts are personal and are not issued to groups. If group data needs to be saved, it must be done through an account linked to a designated contact person.
• Accounts must be requested directly at ia2@inaf.it and will use IDEM credentials.
• The request must specify the size of the data to be saved and the planned frequency of access to that data.

How to Write a Data Management Plan (DMP)

The Data Management Plan must provide all relevant information for proper data preservation in accordance with FAIR principles.

You can follow these guidelines to write your DMP:

How to PerformData Ingestion)

1. Once the account is created, the user will be assigned a scratch area where they can transfer their data;
2. Transfers can be performed via scp or rsync. Any additional software must be agreed upon directly with the IA2 team;
3. Once the user completes the transfer of a data block (a directory with subdirectories), it will be taken over (upon user notification) and frozen;
4. We kindly ask you to prepare and structure data in directories containing no more than 2,000 units, or keep in mind that leaf directories with more than 2,000 files will be converted into .tar files during the archiving procedure. A Bash script can be used to run a check on the files prior to uploading. For each directory, please generate a file listing the computed checksums for its files to verify file integrity. Please leave each checksum file inside its corresponding directory.

Note: Having too many files in a single directory (more than 2,000 files per directory) can negatively impact both file system efficiency and data transfer performance. It is strongly recommended to create a .tar archive before initiating the transfer.

Executing the Checksum Script

Before starting this process, verify that files or directories do not contain special characters such as:

[ ] < > ? \ / " : | ' *`

Assuming your data is located in /home//:

Place the script in /home/ and execute it after modifying execution permissions:

Bash

cd /home/
chmod +x checksum
bash checksum my_data &

This script will run in the background and generate a log file. To check the execution status of the script, use:

Bash

pgrep -f checksum

Once the data is saved to tape, it can be managed by the user in read-only mode.

DataRetrieval)

Data stored on tape can be retrieved via VOSpace. From the VOSpace interface, users can trigger an asynchronous recall (async-recall) for both individual files and folders. Once recalled, download links will appear on the VOSpace interface.

• For small files, users can download files directly from the VOSpace interface.
• For large files, large batches of files, or full folders: We recommend downloading via scp or rsync.

In all cases, the user must initiate an asynchronous recall (async-recall) of the data to download it. Requested data will be placed in the user's directory while preserving the original path structure.

Note: Users do not have direct write access to the Tape Library.

FAQ

#!/bin/bash

# Exit on errors
set -e

# Set main dir (can be absolute or not, it doesn't matter)
ROOT_DIR=$(echo "$1" | rev | cut -d '/' -f 1 | rev)

# Check for a valid input dir name
if [ -z "$ROOT_DIR" ]; then
  echo "Invalid directory name."
  exit 1
fi

# Print all the columns after first one (path dirs)
SUBDIRS_LIST=$(du "$ROOT_DIR" | awk '{$1=""; print substr($0,2)}')
CWD="$(pwd)"

# Use a log file
now="$(date +'%d-%m-%Y_%H-%M-%S')"
LOG="${CWD}"/checksums_"${now}".log

# Iterate over all the dirs (handles dir names with spaces)
echo "$SUBDIRS_LIST" | while read i
  do
    MD5_FILE=$(echo "$i" | rev | cut -d '/' -f 1 | rev)-md5sum.txt
    cd "$i"
    FILES=$(find -maxdepth 1 -type f \( ! -iname "*-md5sum.txt" \))
    if [[ ! -e "../${MD5_FILE}" && ! -z "$FILES" ]]; then
      now="$(date +'[ %d/%m/%Y %H:%M:%S ] ')"
      echo "${now} Generating \"${MD5_FILE}\" for \"${i}\"..." >> "$LOG"
      find -maxdepth 1 -type f \( ! -iname "*-md5sum.txt" \) -exec md5sum '{}' \; > ../"${MD5_FILE}" && \
      now="$(date +'[ %d/%m/%Y %H:%M:%S ] ')" && \
      echo "${now} Done." >> "$LOG"
    fi
    cd "$CWD"
  done