Accessing Datasets on Fermi Data Platform#

FDP supports data access utilizing the following protocols:

HTTP(s)/WebDAV
XRootD
Globus

The data is organized in hierarchical directory tree structure. Public data (accessible without authentication) is located in /amsc/public/<project name>. Data requiring authentication for read access is located in /amsc/<project name>. All write access requires authentication.

Authentication and Authorization#

To allow authorized access to the data FDP utilizes OpenID Connect (OIDC) authentication protocol within overall authorization flow implemented using OAuth2 authorization framework.

Currently only Fermilab OIDC authentication is supported, although it is planned to extend this to other identity providers.

See Getting Started for instructions on getting access to FDP.

Globus#

File access in a browser via the globus.org website#

https://www.globus.org/

Log in with Fermilab credentials, or with any Globus enabled identity provider. The latter case will only allow read access to public data.

Search for “Fermi Data Platform” in the collection search bar. There are several collections associated with FDP. The “Fermi Data Platform” collection allows read access to all publicly available data. The other collections allow read and write access to the corresponding group, given the user has existing access to that group.

From the command line:#

globus transfer [source-endpoint-ID]:[/path/to/source] [destination-ID]:[/path/to/destination] 

The Fermi Data Platform (read-only) endpoint ID is 73edb319-d876-4143-b5fc-5a47c61b0c02 and the path is /amsc/public

The IDs of the other collections can be found either from globus.org (click the … to view collection details), or from the following globus command

globus endpoint search "fermi data platform"

Instructions for using the AmSC data movement API to transfer files via Globus can be found here

Note: for uploads, the underlying dCache storage will not allow overwriting existing files, and trying to do so will fail with an “Operation not permitted” error. Either delete the existing copy beforehand, or submit the transfer with sync level 0 which will skip trying to transfer any file which already exists on the destination.

Https/WebDAV#

Download public data using browser#

https://amsc.fnal.gov:2880/amsc/public

Allows to browse and download publicly available data.

Download public data to local host using curl#

curl -sSfL -O https://amsc.fnal.gov:2880/[source_path]

The source_path is the path to the data on /amsc/public/.... If you want to write the output somewhere other than the current directory, replace -O with -o /path/to/output/file (which must end with a file name, not a directory).

Download public data to local host using gfal#

Using gfal, which requires the gfal2-* packages:

gfal-ls https://amsc.fnal.gov:2880/amsc 

gfal-copy https://amsc.fnal.gov:2880/amsc/[source_path] [destination_path]

Download project specific data requiring authorization using curl#

Below is an example of using an OIDC token authorized to access the dune project directories for reading data. The access token is obtained from the htgettoken utility (this is preinstalled on many fnal.gov systems).

htgettoken -a htvaultprod.fnal.gov -i amsc -r duneread 
export BEARER_TOKEN=$(< $XDG_RUNTIME_DIR/bt_u$(id -u))
curl -f -L -s -H "Authorization: Bearer ${BEARER_TOKEN}" https://amsc.fnal.gov:2880/[source_path] -o [destination_path]

Download project specific data requiring authorization using gfal#

Below is an example of using an OIDC token authorized to access dune project directories for read:

htgettoken -a htvaultprod.fnal.gov -i amsc -r duneread 
export BEARER_TOKEN=$(< $XDG_RUNTIME_DIR/bt_u$(id -u))
gfal-copy https://amsc.fnal.gov:2880/[source_path] [destination_path]

Upload data from local host using curl#

Below is an example of using an OIDC token to upload data to FDP using dunewrite role:

htgettoken -a htvaultprod.fnal.gov -i amsc -r dunewrite
export BEARER_TOKEN=$(< $XDG_RUNTIME_DIR/bt_u$(id -u))
curl -f -L -s -H "Authorization: Bearer ${BEARER_TOKEN}"  -T[source_path]  https://amsc.fnal.gov:2880/[destination_path]

Upload data from local host using gfal#

Below is an example of using OIDC token to upload data to FDP using dunewrite role:

htgettoken -a htvaultprod.fnal.gov -i amsc -r dunewrite
export BEARER_TOKEN=$(< $XDG_RUNTIME_DIR/bt_u$(id -u))
gfal-copy [source_path]  https://amsc.fnal.gov:2880/[destination_path]

Copy data from public dCache to FDP#

htgettoken -a htvaultprod.fnal.gov -i dune
export TOKEN_SRC=$(< $XDG_RUNTIME_DIR/bt_u$(id -u))

htgettoken -a htvaultprod.fnal.gov -i amsc -r dunewrite
export TOKEN_DST=$(< $XDG_RUNTIME_DIR/bt_u$(id -u))

PULL mode copy:

curl -s --capath /etc/grid-security/certificates -L -X COPY \
      -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credential: none' \
      -H "Authorization: Bearer $TOKEN_DST" -H "TransferHeaderAuthorization: Bearer $TOKEN_SRC" \
      -H "Source: https://fndcadoor.fnal.gov:2880/[source_path]" https://amsc.fnal.gov:2880/[destination_path]

PUSH mode copy:

curl -s --capath /etc/grid-security/certificates -L -X COPY \
      -H 'Secure-Redirection: 1' -H 'X-No-Delegate: 1' -H 'Credential: none' \ 
      -H "Authorization: Bearer $TOKEN_SRC" -H "TransferHeaderAuthorization: Bearer $TOKEN_DST" \
      -H "Destination: https://amsc.fnal.gov:2880/[destinaion_path]"  https://fndcadoor.fnal.gov:2880/[source_path] 

XRootD#

These commands require the xrootd-client package.

From the command line:

Copy a file:

xrdcp root://amsc.fnal.gov/[source_path] [destination_path]

List a directory:

xrdfs root://amsc.fnal.gov ls -l /amsc/public

Using gfal, which requires the gfal2-* packages:

gfal-ls root://amsc.fnal.gov/amsc/public 

gfal-copy root://amsc.fnal.gov/amsc/[source_path] [destintation_path] 

Using the `fsspec` python library#

For applications that use the Python fsspec library, there are plugins for both xrootd and WebDAV that allow streaming of data from FDP.

For example

import fsspec
import h5py

with fsspec.open('root://amsc.fnal.gov/amsc/public/treasure/aoj/data/RunG_batch0.h5', 'rb') as filestream:
    with h5py.File(filestream, 'r') as f:
        print('Top level keys:')
        for k in f.keys():
            print(f'  {k}')

will output the following (without having to read every part of the file).

Top level keys:
  PFCands
  event_info
  jet_kinematics
  jet_tagging

Data formats that use fsspec include Apache Arrow and Parquet, among others

Accessing Datasets on Fermi Data Platform

Contents