F Bartusch: Created page with "= Data Organization on BinAC 2 project directories = This guide explains how to organize and manage your data in the project directory at `/pfs/10/project` BinAC 2. Each project has its own dedicated directory with flexible permission systems to help you control access to your files and collaborate effectively with team members. '''Important''' Use workspaces for actual computations. Data in workspaces are stored on '''much''' faster storage! == Project Di..."

2025-09-17T07:16:39Z

Created page with "= Data Organization on BinAC 2 project directories = This guide explains how to organize and manage your data in the project directory at <code>/pfs/10/project</code> BinAC 2. Each project has its own dedicated directory with flexible permission systems to help you control access to your files and collaborate effectively with team members. '''Important''' Use workspaces for actual computations. Data in workspaces are stored on '''much''' faster storage! == Project Di..."

New page

= Data Organization on BinAC 2 project directories =

This guide explains how to organize and manage your data in the project directory at <code>/pfs/10/project</code> BinAC 2.
Each project has its own dedicated directory with flexible permission systems to help you control access to your files and collaborate effectively with team members.

'''Important''' Use workspaces for actual computations. Data in workspaces are stored on '''much''' faster storage!

== Project Directory Structure ==

Every project gets a dedicated directory located at:
<pre>/pfs/10/project/<project_id>/</pre>

For example, if your project ID is <code>bw16f003</code>, your project directory would be:
<pre>/pfs/10/project/bw16f003/</pre>

=== Understanding Your Project Directory ===

When you list your project directory, you'll see something like this:
<pre>$ ls -ld /pfs/10/project/bw16f003/
drwxrwx---. 5 root bw16f003 33280 Jul 25 14:13 /pfs/10/project/bw16f003/</pre>

Let's break down what this means:
* '''<code>d</code>''': This is a directory (not a file)
* '''<code>rwxrwx---</code>''': The permissions (explained in detail below)
* '''<code>root</code>''': The owner of the directory
* '''<code>bw16f003</code>''': The group that owns the directory (your project group)
* '''<code>33280</code>''': The size in bytes
* '''<code>Jul 25 14:13</code>''': Last modification date
* '''<code>/pfs/10/project/bw16f003/</code>''': The full path

== Understanding Unix Permissions ==

Unix permissions control who can read, write, or execute files and directories. They are displayed as a 10-character string like <code>drwxrwx---</code>.

=== Permission Characters ===

* '''<code>r</code>''' (read): Can view file contents or list directory contents
* '''<code>w</code>''' (write): Can modify files or create/delete files in directories
* '''<code>x</code>''' (execute): Can run files as programs or enter directories

=== Permission Groups ===

Permissions are shown for three groups of users:

# '''Owner''' (positions 2-4): The user who owns the file/directory
# '''Group''' (positions 5-7): Users who belong to the file's group
# '''Others''' (positions 8-10): Everyone else

=== Example Breakdown ===

For <code>drwxrwx---</code>:
* '''<code>d</code>''': Directory
* '''<code>rwx</code>''' (owner): Owner can read, write, and execute
* '''<code>rwx</code>''' (group): Group members can read, write, and execute
* '''<code>---</code>''' (others): Others have no permissions

=== Numeric Permission Values ===

Permissions on files and directories are changed with the tool <code>chmod</code>, which uses numerical values for describing permissions.

{| class="wikitable"
|-
! Number !! Binary !! Permissions !! Description
|-
| 0 || 000 || --- || No permissions
|-
| 1 || 001 || --x || Execute only
|-
| 2 || 010 || -w- || Write only
|-
| 3 || 011 || -wx || Write and execute
|-
| 4 || 100 || r-- || Read only
|-
| 5 || 101 || r-x || Read and execute
|-
| 6 || 110 || rw- || Read and write
|-
| 7 || 111 || rwx || Read, write, and execute
|}

=== Common Permission Patterns ===

{| class="wikitable"
|-
! Pattern !! Use Case !! Description
|-
| <code>700</code> || Private files/directories || Only owner can access
|-
| <code>750</code> || Shared read-only || Owner full access, group read-only
|-
| <code>770</code> || Shared read-write || Owner and group full access
|-
| <code>644</code> || Public readable files || Owner can edit, others can read
|-
| <code>755</code> || Public executable directories || Owner can edit, others can read/execute
|}

== Data Organization Strategies ==

=== Private Directories ===

Create directories that '''only you''' can access:

<pre># Create a private directory
mkdir /pfs/10/project/<project_id>/$USER

# Set permissions so only you can access it
chmod 700 /pfs/10/project/<project_id>/$USER</pre>

'''Permission breakdown for <code>700</code>:'''
* '''Owner''': <code>rwx</code> (7 = 4+2+1) - full access
* '''Group''': <code>---</code> (0) - no access
* '''Others''': <code>---</code> (0) - no access

=== Shared Project Directories ===

Create directories that '''all''' project members can access:

<pre># Create a shared directory
mkdir /pfs/10/project/<project_id>/<your directory name>

# Set permissions for group access
chmod 770 /pfs/10/project/<project_id>/<your directory name></pre>

'''Permission breakdown for <code>770</code>:'''
* '''Owner''': <code>rwx</code> (7) - full access
* '''Group''': <code>rwx</code> (7) - full access for project members
* '''Others''': <code>---</code> (0) - no access

=== Read-Only Shared Directories ===

Create directories where project members can read but not modify:

<pre># Create a read-only shared directory
mkdir /pfs/10/project/<project_id>/<your directory name>

# Set read-only permissions for group
chmod 750 /pfs/10/project/<project_id>/<your directory name></pre>

'''Permission breakdown for <code>750</code>:'''
* '''Owner''': <code>rwx</code> (7) - full access
* '''Group''': <code>r-x</code> (5 = 4+1) - can read and enter, but not write
* '''Others''': <code>---</code> (0) - no access

== Advanced Access Control with ACLs ==

For more fine-grained control beyond basic Unix permissions, you can use Access Control Lists (ACLs). ACLs allow you to set permissions for specific users.

=== Checking Current ACLs ===

<pre># Check if a directory has ACLs
getfacl /pfs/10/project/<project_id>/<your directory name></pre>

=== Setting User-Specific Permissions ===

<pre># Give user 'alice' read and execute permissions
setfacl -m u:alice:rx /pfs/10/project/<project_id>/<your directory name>

# Give user 'bob' full permissions
setfacl -m u:bob:rwx /pfs/10/project/<project_id>/<your directory name></pre>

=== Removing ACLs ===

<pre># Remove ACL for specific user
setfacl -x u:alice /pfs/10/project/<project_id>/<your directory name>

# Remove all ACLs and revert to basic permissions
setfacl -b /pfs/10/project/<project_id>/<your directory name></pre>

=== Default ACLs for New Files ===

Set default ACLs that will be applied to new files created in a directory:

<pre># Set default ACL for new files in directory
setfacl -d -m u:alice:rw /pfs/10/project/<project_id>/<your directory name></pre>

== Practical Examples ==

=== Example 1: Personal Workspace with Shared Results ===

<pre># Create your personal workspace
mkdir /pfs/10/project/<project_id>/user_alice
chmod 700 /pfs/10/project/<project_id>/user_alice

# Create a results directory that others can read
mkdir /pfs/10/project/<project_id>/user_alice/results
chmod 755 /pfs/10/project/<project_id>/user_alice/results</pre>

=== Example 2: Collaborative Analysis Directory ===

<pre># Create collaborative space
mkdir /pfs/10/project/<project_id>/collaborative_analysis
chmod 770 /pfs/10/project/<project_id>/collaborative_analysis

# Create subdirectories for different types of work
mkdir /pfs/10/project/<project_id>/collaborative_analysis/{scripts,data,results}
chmod 770 /pfs/10/project/<project_id>/collaborative_analysis/*</pre>

=== Example 3: Mixed Access with ACLs ===

<pre># Create a directory with complex permissions
mkdir /pfs/10/project/<project_id>/complex_project
chmod 750 /pfs/10/project/<project_id>/complex_project

# Give specific users different levels of access
setfacl -m u:alice:rwx /pfs/10/project/<project_id>/complex_project
setfacl -m u:bob:rx /pfs/10/project/<project_id>/complex_project
setfacl -m g:external_collaborators:r /pfs/10/project/<project_id>/complex_project</pre>

== Best Practices ==

=== Plan Your Directory Structure ===

Before creating directories, plan your organization:
* '''Personal directories''': For work-in-progress and private files
* '''Shared directories''': For collaboration and shared resources

=== Use Descriptive Names ===

Choose clear, descriptive directory names:
* ✗ <code>protein_folding_analysis</code>
* ✗ <code>shared_reference_genomes</code>
* ✗ <code>alice_optimization_scripts</code>
* ✗ <code>stuff</code>
* ✗ <code>temp</code>
* ✗ <code>dir1</code>

=== Set Permissions Appropriately ===

* Start with restrictive permissions and open up as needed
* Use <code>750</code> for directories you want to share read-only
* Use <code>770</code> for full collaboration
* Use ACLs for complex permission requirements

=== Regular Backups ===

'''Important''' The cluster does not provide automatic backups for project directories. You are responsible for backing up your important data.

'''Why Backups Matter:'''

Hardware failures can result in permanent data loss
Human errors (accidental deletion, wrong commands) happen
Corrupted files from failed jobs or system issues
No recovery possible once data is lost from the cluster

=== Document Your Organization ===

Create a README file in your project directory explaining the structure:

<pre># Create documentation
cat > /pfs/10/project/<project_id>/README.md << 'EOF'
# Project <project_id> Directory Structure

## Overview
This project focuses on protein folding simulation and analysis.

## Directory Structure
- `shared_data/`: Reference datasets (read-only for group)
- `collaborative_analysis/`: Shared analysis scripts and results
- `user_alice/`: Alice's personal workspace
- `user_bob/`: Bob's personal workspace
- `archive/`: Completed analysis and backups

## Permissions
- Group members have read access to shared_data/
- All group members can contribute to collaborative_analysis/
- Personal directories are private to each user
EOF</pre>

== Troubleshooting ==

=== Permission Denied Errors ===

If you get "Permission denied" errors:

# Check the permissions:
<pre>ls -la /pfs/10/project/<project_id>/problematic_directory</pre>

# Check if you're in the correct group:
<pre>groups</pre>

# Check ACLs if basic permissions look correct:
<pre>getfacl /pfs/10/project/<project_id>/problematic_directory</pre>

=== Cannot Create Files in Directory ===

This usually means the directory lacks write permissions for your user/group:

<pre># Check directory permissions
ls -ld /pfs/10/project/<project_id>/target_directory

# Fix if you own the directory
chmod g+w /pfs/10/project/<project_id>/target_directory</pre>

=== Accidentally Locked Yourself Out ===

If you accidentally removed your own permissions:

<pre># As the owner, you can always restore permissions
chmod 755 /pfs/10/project/<project_id>/locked_directory</pre>

BinAC2/Project Data Organization - Revision history