Difference between revisions of "BwUniCluster2.0/Software/R/terra"

From bwHPC Wiki
Jump to: navigation, search
(Created page with " = General information = sf and terra allow to use the following tools for handling spatial structures in R * the 'Geospatial' Data Abstraction Library [https://gdal.org/...")
 
 
(9 intermediate revisions by the same user not shown)
Line 14: Line 14:
   
 
Please enter the following code, presented in the boxes below, directly into your shell/command line on bwUniCluster.
 
Please enter the following code, presented in the boxes below, directly into your shell/command line on bwUniCluster.
  +
  +
The whole process will take approximately 45 minutes.
  +
  +
  +
== Preparations ==
  +
Prepare .R directory (if it does not already exists). This is then filled with information how R should compile the packages (so-called 'compilation flags'). These are written (and can be reviewed) into the (text) file Makevars.
  +
  +
If you already have a .R/Makevars file, check whether these flags are already set.
  +
  +
<pre>
  +
cat ~/.R/Makevars
  +
</pre>
  +
  +
In this case, skip the following block of five commands. If not, please enter the following commands in your shell (command line):
  +
<pre>
  +
mkdir -p ~/.R
  +
  +
echo "CXX14=icpc" > ~/.R/Makevars
  +
echo "CXX17=icpc" >> ~/.R/Makevars
  +
echo "CXX14FLAGS=-O3 -fPIC -std=c++14 -wd308 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp" >> ~/.R/Makevars
  +
echo "CXX17FLAGS=-O3 -fPIC -std=c++17 -wd308 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp" >> ~/.R/Makevars
  +
echo "CXXFLAGS += -wd308" >> ~/.R/Makevars
  +
echo "PKG_CXXFLAGS += -std=c++14 -wd308" >> ~/.R/Makevars
  +
</pre>
   
 
== Install external programs ==
 
== Install external programs ==
Line 24: Line 48:
   
 
<pre>
 
<pre>
salloc -n 4 -t 30 -p dev_single
+
salloc -n 4 -t 60 -p single
 
</pre>
 
</pre>
   
Line 36: Line 60:
 
Then, download and install PROJ
 
Then, download and install PROJ
 
<pre>
 
<pre>
PROJ_VER=6.3.2
+
PROJ_VER=9.3.1
 
wget http://download.osgeo.org/proj/proj-$PROJ_VER.tar.gz
 
wget http://download.osgeo.org/proj/proj-$PROJ_VER.tar.gz
 
tar xf proj-$PROJ_VER.tar.gz
 
tar xf proj-$PROJ_VER.tar.gz
 
cd proj-$PROJ_VER
 
cd proj-$PROJ_VER
  +
./configure --prefix=$HOME/sw/R
 
  +
mkdir build
make -j 8
 
  +
cd build
make install
 
  +
cd ..
 
  +
cmake -DCMAKE_INSTALL_PREFIX=$HOME/sw/R ..
  +
cmake --build . -j 8
  +
cmake --build . --target install
 
</pre>
 
</pre>
   
Then, install gdal
+
Then, install GDAL
   
 
<pre>
 
<pre>
  +
cd $HOME/src
GDAL_VER=3.4.1
 
  +
GDAL_VER=3.8.3
 
wget http://download.osgeo.org/gdal/$GDAL_VER/gdal-$GDAL_VER.tar.gz
 
wget http://download.osgeo.org/gdal/$GDAL_VER/gdal-$GDAL_VER.tar.gz
 
tar xf gdal-$GDAL_VER.tar.gz
 
tar xf gdal-$GDAL_VER.tar.gz
 
cd gdal-$GDAL_VER
 
cd gdal-$GDAL_VER
  +
./configure --prefix=$HOME/sw/R --with-proj=$HOME/sw/R
 
  +
mkdir build
make -j 8
 
  +
cd build
make install
 
  +
cd ..
 
  +
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/sw/R ..
  +
cmake --build . -j 8
  +
cmake --build . --target install
  +
 
</pre>
 
</pre>
   
 
Finally, install GEOS
 
Finally, install GEOS
 
<pre>
 
<pre>
  +
cd $HOME/src
GEOS_VER=3.9.2
 
  +
GEOS_VER=3.12.1
 
wget http://download.osgeo.org/geos/geos-$GEOS_VER.tar.bz2
 
wget http://download.osgeo.org/geos/geos-$GEOS_VER.tar.bz2
 
tar xf geos-$GEOS_VER.tar.bz2
 
tar xf geos-$GEOS_VER.tar.bz2
 
cd geos-$GEOS_VER
 
cd geos-$GEOS_VER
  +
./configure --prefix=$HOME/sw/R
 
  +
mkdir _build
make -j 8
 
  +
cd _build
  +
  +
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/sw/R ..
  +
make -j 8
  +
ctest -j 8
 
make install
 
make install
cd ..
 
 
</pre>
 
</pre>
   
Line 76: Line 113:
   
 
<pre>
 
<pre>
  +
export LD_LIBRARY_PATH=$HOME/sw/R/lib:$LD_LIBRARY_PATH
 
  +
export LD_LIBRARY_PATH=$HOME/sw/R/lib64:$LD_LIBRARY_PATH
 
export PATH=$PATH:$HOME/sw/R/bin
 
export PATH=$PATH:$HOME/sw/R/bin
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$HOME/sw/R/lib/pkgconfig
+
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$HOME/sw/R/lib64/pkgconfig
 
export GDAL_DATA=$HOME/sw/R/share/gdal
 
export GDAL_DATA=$HOME/sw/R/share/gdal
 
</pre>
 
</pre>
Line 92: Line 130:
   
   
  +
Now, we install terra and sf from within R.
Now, we install rgdal and rgeos from within R. Note that, since we install a package from a local repository the order of the package installation is relevant. Installing rgdal will fail if sp is not available.
 
   
 
<pre>
 
<pre>
 
module load math/R/4.1.2
 
module load math/R/4.1.2
   
cd
 
wget https://cran.r-project.org/src/contrib/Archive/rgdal/rgdal_1.6-4.tar.gz
 
wget https://cran.r-project.org/src/contrib/Archive/rgeos/rgeos_0.6-1.tar.gz
 
   
 
R -q
 
R -q
> install.packages("sp", repos="https://ftp.gwdg.de/pub/misc/cran/")
+
R> install.packages("terra")
  +
R> library(terra)
> install.packages("~/rgdal_1.6-4.tar.gz", repos=NULL, type="source")
 
  +
terra 1.7.71
> install.packages("~/rgeos_0.6-1.tar.gz", repos=NULL, type="source")
 
  +
  +
  +
R> install.packages("sf")
  +
R> library(sf)
  +
Linking to GEOS 3.12.1, GDAL 3.8.3, PROJ 9.3.1; sf_use_s2() is TRUE
   
> library("rgdal")
 
> library("rgeos")
 
 
</pre>
 
</pre>
   
== Preparations to use the rgdal/rgeos packages ==
+
== Preparations to use the terra and sf packages ==
Since rgdal and rgeos depend on the external programs we installed, several environment variables have to be set before using the packages to allow R to address these programs.
+
Since terra and sf depend on the external programs we installed, several environment variables have to be set before using the packages to allow R to address these programs.
   
 
We recommend to add the export commands
 
We recommend to add the export commands
   
 
<pre>
 
<pre>
export LD_LIBRARY_PATH=$HOME/sw/R/lib:$LD_LIBRARY_PATH
+
export LD_LIBRARY_PATH=$HOME/sw/R/lib64:$LD_LIBRARY_PATH
 
export PATH=$PATH:$HOME/sw/R/bin
 
export PATH=$PATH:$HOME/sw/R/bin
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$HOME/sw/R/lib/pkgconfig
+
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$HOME/sw/R/lib64/pkgconfig
 
export GDAL_DATA=$HOME/sw/R/share/gdal
 
export GDAL_DATA=$HOME/sw/R/share/gdal
 
</pre>
 
</pre>
   
to your [[BwUniCluster_2.0_Slurm_common_Features#sbatch_Examples | batch job scripts]] that use rgdal and rgeos or to run them directly in the command line if you use an [[BwUniCluster_2.0_Batch_Queues | interactive session]].
+
to your [[BwUniCluster_2.0_Slurm_common_Features#sbatch_Examples | batch job scripts]] that use terra and sf or to run them directly in the command line if you use an [[BwUniCluster_2.0_Batch_Queues | interactive session]].

Latest revision as of 15:30, 15 February 2024

1 General information

sf and terra allow to use the following tools for handling spatial structures in R

  • the 'Geospatial' Data Abstraction Library GDAL
  • Projection/transformation operations from the PROJ library
  • Interface to the open source Geometry Engine GEOS


2 Installation

Please enter the following code, presented in the boxes below, directly into your shell/command line on bwUniCluster.

The whole process will take approximately 45 minutes.


2.1 Preparations

Prepare .R directory (if it does not already exists). This is then filled with information how R should compile the packages (so-called 'compilation flags'). These are written (and can be reviewed) into the (text) file Makevars.

If you already have a .R/Makevars file, check whether these flags are already set.

cat ~/.R/Makevars

In this case, skip the following block of five commands. If not, please enter the following commands in your shell (command line):

mkdir -p ~/.R

echo "CXX14=icpc" > ~/.R/Makevars
echo "CXX17=icpc" >> ~/.R/Makevars
echo "CXX14FLAGS=-O3 -fPIC -std=c++14 -wd308 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp" >> ~/.R/Makevars
echo "CXX17FLAGS=-O3 -fPIC -std=c++17 -wd308 -axCORE-AVX512,CORE-AVX2,AVX -xSSE4.2 -fp-model strict -qopenmp" >> ~/.R/Makevars
echo "CXXFLAGS += -wd308" >> ~/.R/Makevars
echo "PKG_CXXFLAGS += -std=c++14 -wd308" >> ~/.R/Makevars

2.2 Install external programs

First, we download the sources of GDAL, PROJ, GEOS and install the three programs.

We will gather them in a folder src, unpack there and then compile.

We strongly recommend to use a interactive session with multiple cores.

salloc -n 4 -t 60 -p single


First, provide the source directory (if not yet existing)

mkdir -p ~/src
cd ~/src

Then, download and install PROJ

PROJ_VER=9.3.1
wget http://download.osgeo.org/proj/proj-$PROJ_VER.tar.gz
tar xf proj-$PROJ_VER.tar.gz
cd proj-$PROJ_VER

mkdir build
cd build

cmake -DCMAKE_INSTALL_PREFIX=$HOME/sw/R ..
cmake --build . -j 8
cmake --build . --target install

Then, install GDAL

cd $HOME/src
GDAL_VER=3.8.3
wget http://download.osgeo.org/gdal/$GDAL_VER/gdal-$GDAL_VER.tar.gz
tar xf gdal-$GDAL_VER.tar.gz
cd gdal-$GDAL_VER

mkdir build
cd build

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/sw/R ..
cmake --build . -j 8
cmake --build . --target install

Finally, install GEOS

cd $HOME/src
GEOS_VER=3.12.1
wget http://download.osgeo.org/geos/geos-$GEOS_VER.tar.bz2
tar xf geos-$GEOS_VER.tar.bz2
cd geos-$GEOS_VER

mkdir _build
cd _build

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/sw/R ..
make -j 8
ctest -j 8
make install

2.3 Installing the R packages

In order to install the two R packages, we need R to understand where we installed the 3 underlying programs, so we export the necessary paths.


export LD_LIBRARY_PATH=$HOME/sw/R/lib64:$LD_LIBRARY_PATH
export PATH=$PATH:$HOME/sw/R/bin
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$HOME/sw/R/lib64/pkgconfig
export GDAL_DATA=$HOME/sw/R/share/gdal


Additionally, the R package installation features compilation of built-in C++ code, for which we specify compilation options ('compiler flags')

export CFLAGS=-I$HOME/sw/R/include
export CXX="icpc -std=c++11"
export CXX17=icpc


Now, we install terra and sf from within R.

module load math/R/4.1.2


R -q
R> install.packages("terra")
R> library(terra)
terra 1.7.71


R> install.packages("sf")
R> library(sf)
Linking to GEOS 3.12.1, GDAL 3.8.3, PROJ 9.3.1; sf_use_s2() is TRUE

2.4 Preparations to use the terra and sf packages

Since terra and sf depend on the external programs we installed, several environment variables have to be set before using the packages to allow R to address these programs.

We recommend to add the export commands

export LD_LIBRARY_PATH=$HOME/sw/R/lib64:$LD_LIBRARY_PATH
export PATH=$PATH:$HOME/sw/R/bin
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$HOME/sw/R/lib64/pkgconfig
export GDAL_DATA=$HOME/sw/R/share/gdal

to your batch job scripts that use terra and sf or to run them directly in the command line if you use an interactive session.