Development/Python - Revision history

H Winkhardt: typo

2025-10-29T14:11:48Z

typo

← Older revision		Revision as of 16:11, 29 October 2025
Line 343:		Line 343:
	*: Good for big data files. Reduces storage need.		*: Good for big data files. Reduces storage need.
	*: Used via libraries like PyArrow and FastParquet. The latter is a version with less features but therefore better performance when used with small to medium sized datasets. [See the comparison [https://dev.to/alexmercedcoder/all-about-parquet-part-08-reading-and-writing-parquet-files-in-python-338d here]]		*: Used via libraries like PyArrow and FastParquet. The latter is a version with less features but therefore better performance when used with small to medium sized datasets. [See the comparison [https://dev.to/alexmercedcoder/all-about-parquet-part-08-reading-and-writing-parquet-files-in-python-338d here]]
	*: ~~Sinificantly~~ slower when storing a mixed dataframe in contrast to a numerical dataframe.		*: Significantly slower when storing a mixed dataframe in contrast to a numerical dataframe.
	* '''hdf5'''		* '''hdf5'''
	* '''Database'''		* '''Database'''

H Schumacher: Added Snakemake config recommendations and File Formats section

2025-10-29T11:46:54Z

Added Snakemake config recommendations and File Formats section

@@ Line 12: / Line 12: @@
   module avail devel/python
   </syntaxhighlight>
 * '''Python distributions and virtual environments:''' By using python distributions such as Anaconda, you can easily install the needed python version into a virtual environment. For the use of conda on bwHPC clusters, please refer to [[Development/Conda|Conda]]. Alternatively, you can use more python specific tools for installing python. Some options are listed in [[#Virtual Environments and Package Management | Virtual Environments and Package Management]]
 * '''[[Development/Containers | Container]]:''' Containers can contain their own python installation. Keep this in mind when you are working with containers provided by others.
@@ Line 259: / Line 260: @@
 When you have an existing project, you can create a virtual environment at .venv with a specific python version as follows:
 <syntaxhighlight lang="bash">
-uv venv --python 3.11
+uv python list # get available versions
 </syntaxhighlight>
 A specific name or path can be given instead of "venv" as well. If the python version is not available on the system, uv downloads it.
@@ Line 317: / Line 319: @@
 Leaving the python world, a common choice for bioinformaticians would be Nextflow as there is a strong community that focuses on standardisation. Those more standardised pipelines can be found at the [https://nf-co.re/pipelines/ nf-core website].
 In other research fields, other workflow management systems might be more common.
 == Best Practices ==

H Schumacher: Fix in uv section

2025-10-15T11:13:46Z

Fix in uv section

← Older revision		Revision as of 13:13, 15 October 2025
Line 271:		Line 271:
	If you activated the virtual environment, you can install packagages as usual. Otherwise, you need to add "uv" before the command to run it within the virtual environment:		If you activated the virtual environment, you can install packagages as usual. Otherwise, you need to add "uv" before the command to run it within the virtual environment:
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	uv ~~pip install~~ <package_name>		uv add <package_name>
	</syntaxhighlight>		</syntaxhighlight>
			Note: Additionally, there is a pip interface for uv but it should only be used in fitting edge cases.


	To execute a python file within the virtual environment:		To execute a python file within the virtual environment:
Line 282:		Line 284:

	→ '''[https://docs.astral.sh/uv/concepts/projects/workspaces/ workspaces]'''		→ '''[https://docs.astral.sh/uv/concepts/projects/workspaces/ workspaces]'''


	=== Pixi ===		=== Pixi ===

P Hematty: /* Virtual Environments and Package Management */ UV ersetzt Pixi nicht, weil Pixi auch Conda Pakete installieren kann. Daher habe ich das aus dem Beschreibungstext rausgenommen. Außerdem gibts bei pipx soweit ich weiß keine Möglichkeit was bei PyPI zu publishen, was wahrscheinlich aber auch eher ein Fehler beim Übertragen war oder so.

2025-10-06T11:36:43Z

Virtual Environments and Package Management: UV ersetzt Pixi nicht, weil Pixi auch Conda Pakete installieren kann. Daher habe ich das aus dem Beschreibungstext rausgenommen. Außerdem gibts bei pipx soweit ich weiß keine Möglichkeit was bei PyPI zu publishen, was wahrscheinlich aber auch eher ein Fehler beim Übertragen war oder so.

← Older revision		Revision as of 13:36, 6 October 2025
Line 108:		Line 108:
	\| yes		\| yes
	\| yes (only for single applications)		\| yes (only for single applications)
	\| ~~yes~~		\| no
	\|-		\|-
	\| uv		\| uv
	\| Replaces ~~pixi,~~ poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)		\| Replaces poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)
	\| yes		\| yes
	\| yes		\| yes
Line 121:		Line 121:
	\|-		\|-
	\| pixi		\| pixi
	\| For installing and managing python as well as conda packages. Uses uv.		\| For installing and managing python as well as conda packages. Uses uv in the background
	\| yes		\| yes
	\| yes		\| yes

H Schumacher: Added section for Pixi and workflow management systems.

2025-10-05T22:53:38Z

Added section for Pixi and workflow management systems.

← Older revision		Revision as of 00:53, 6 October 2025
Line 283:		Line 283:
	→ '''[https://docs.astral.sh/uv/concepts/projects/workspaces/ workspaces]'''		→ '''[https://docs.astral.sh/uv/concepts/projects/workspaces/ workspaces]'''


			=== Pixi ===

			==== Creation ====

			With uv you can create two types of new projects. Either an application (default) or a library.

			→ '''[https://pixi.sh/latest/python/tutorial/ Python Tutorial]'''
			<syntaxhighlight lang="bash">
			pixi init <project_name> --format pyproject
			</syntaxhighlight>

			==== Usage ====
			Add a new package to the environment:
			<syntaxhighlight lang="bash">
			# Get it from conda:
			pixi add <package_name>
			# Get it from PyPi
			pixi add <package_name> --pypi
			</syntaxhighlight>

			To execute a command within the virtual environment:
			<syntaxhighlight lang="bash">
			pixi run <command>
			</syntaxhighlight>

			== Workflow Management Systems ==

			There are a lot of different workflow management systems. One example is [https://snakemake.readthedocs.io/en/stable/index.html Snakemake] which is often used in the field of biology/medicine. Two use cases for snakemake workflows would be:
			* Use an existing workflow to analyze data.
			* Create your own workflow to make your research more reproducible.
			Leaving the python world, a common choice for bioinformaticians would be Nextflow as there is a strong community that focuses on standardisation. Those more standardised pipelines can be found at the [https://nf-co.re/pipelines/ nf-core website].
			In other research fields, other workflow management systems might be more common.

	== Best Practices ==		== Best Practices ==

H Schumacher: Added uv section

2025-10-05T22:35:37Z

Added uv section

@@ Line 209: / Line 209: @@
 When you want to create a virtual environment for an already existing project, you can go to the top directory and run
 <syntaxhighlight lang="bash">
-poetry init      # Create virtual enviroment
+poetry init      # Create virtual environment
 </syntaxhighlight>
 Otherwise start with the demo project:
@@ Line 243: / Line 243: @@
 poetry env remove # delete virtual environment
 </syntaxhighlight>
 == Best Practices ==

H Schumacher: Changed text in the beginning, removed setuptools from table

2025-10-05T18:17:35Z

Changed text in the beginning, removed setuptools from table

@@ Line 33: / Line 33: @@
 == Virtual Environments and Package Management ==
 === Overview ===
-The following table provides an overview of both and explains the differences between the various options. After deciding on a specific tool, it can be installed by following the given link.
+The following table provides an overview of common tools in the field of virtual environments and package management. The main differences between the various options are highlighted. After deciding on a specific tool, it can be installed by following the given link.
 To make it short, if you plan to use...
 {| class="wikitable"
@@ Line 83: / Line 89: @@
 | yes
 | no
 |-
 | poetry

K Siegmund: /* Best Practices */

2025-09-09T08:36:06Z

Best Practices

← Older revision		Revision as of 10:36, 9 September 2025
Line 253:		Line 253:
	# Version the Python source files and the <code>requirements.txt</code> file with a version control system, e.g. git. Exclude unnecessary folders and files like for example <code>venv</code> via an entry in the ignore file, for example <code>.gitignore</code>.		# Version the Python source files and the <code>requirements.txt</code> file with a version control system, e.g. git. Exclude unnecessary folders and files like for example <code>venv</code> via an entry in the ignore file, for example <code>.gitignore</code>.
	# Create and activate a virtual environment.		# Create and activate a virtual environment.
			# Use specialized number crunching python modules (e.g. numpy and scipy), don't use plain python for serious caluclations
			## check if optimized compiled modules are available on the cluster (numpy, scipy)
	# Update pip <code>pip install --upgrade pip</code>.		# Update pip <code>pip install --upgrade pip</code>.
	# Install all required packages via the <code>requirements.txt</code> file in the case of venv. Or by using the corresponding command of your chosen tool.		# Install all required packages via the <code>requirements.txt</code> file in the case of venv. Or by using the corresponding command of your chosen tool.

H Winkhardt: typo

2025-09-09T08:30:42Z

typo

← Older revision		Revision as of 10:30, 9 September 2025
Line 115:		Line 115:
	\|-		\|-
	\| uv		\| uv
	\| Replaces pixi, poetry, pyenv, pip etc. ~~dand~~ is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)		\| Replaces pixi, poetry, pyenv, pip etc. and is very fast (https://www.loopwerk.io/articles/2025/uv-keeps-getting-better/)
	\| yes		\| yes
	\| yes		\| yes
Line 134:		Line 134:
	\| yes		\| yes
	\|}		\|}


	=== Pip ===		=== Pip ===

H Winkhardt: typo

2025-09-09T08:29:07Z

typo

← Older revision		Revision as of 10:29, 9 September 2025
Line 214:		Line 214:
	When you want to create a virtual environment for an already existing project, you can go to the top directory and run		When you want to create a virtual environment for an already existing project, you can go to the top directory and run
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	~~poery~~ init # Create virtual enviroment		poetry init # Create virtual enviroment
	</syntaxhighlight>		</syntaxhighlight>
	Otherwise start with the demo project:		Otherwise start with the demo project: