Skip to content

PyPI Mirror

Posit Package Manager supports creating a Python repository with a source that mirrors PyPI.

Adding a PyPI repository to a Package Manager installation will:

  • Provide a full mirror of all packages available on PyPI.
  • Enable fully reproducible dependency management through historic PyPI snapshots.
  • Locally cache all downloaded Python packages for quicker installs.

System Requirements#

In addition to the recommended system requirements, supporting Python packages requires additional disk storage depending on the number of packages being used.

Important

The entirety of PyPI currently requires more than 20 TB of storage. Your actual storage needs depend on your usage. Deep learning packages, such as Tensorflow and PyTorch, are notoriously large, with hundreds of gigabytes needed for each project's collection of files. If you do not anticipate using deep learning packages, a starting storage size of 50 GB is likely adequate. If you do intend to use deep learning packages, you should plan for 500 GB or more.

Quickstart#

The quickest way to make PyPI packages available for your Package Manager installation is by running these commands:

Terminal
$ rspm create repo --name=pypi --type=python --description='Access PyPI packages'
$ rspm subscribe --repo=pypi --source=pypi

For more information about these commands, refer to the Python PyPI Repository section below.

Note

Package Manager serves Python packages through PyPI-compliant HTTP endpoints. These packages are routed via relative URLs, but in cases where subpaths like /ppm/ or domain names are used, the configuration option Server.Address needs to be set.

See the Server.Address configuration documentation for more information.

User Configuration#

Once a Python repository has been successfully created, users need to configure their local system and pip to install from Package Manager.

To find instructions specific to your Package Manager installation:

  1. Follow the Quickstart or Creating a Python PyPI Repository instructions.
  2. Navigate to the Package Manager homepage.
  3. Select the relevant Python repository from the sidebar.
  4. Click the Setup button at the top of the page.

In general, users can either install from Package Manager in a one-off basis:

Terminal
$ pip install --index-url http(s)://[HOST:PORT]/latest/simple PACKAGE-TO-INSTALL

or configure pip to use Package Manager in a persistent manner:

Terminal
$ pip config set global.index-url http(s)://[HOST:PORT]/latest/simple

HTTP Configuration

If you use HTTP, pip will ignore your repository by default. Using only the configuration above, pip displays a warning message such as:

WARNING: The repository located at [HOST] is not a trusted or secure host and is being ignored. If this repository is available via HTTPS we recommend you use HTTPS instead, otherwise you may silence this warning and allow it anyway with '--trusted-host [HOST]'.

To configure pip to use the unencrypted HTTP Package Manager server, you must use the --trusted-host flag or configuration option:

Terminal
$ pip install --trusted-host [HOST] --index-url http://[HOST:PORT]/latest/simple PACKAGE-TO-INSTALL

or configure pip to use Package Manager in a persistent manner:

Terminal
$ pip config set global.index-url http://[HOST:PORT]/latest/simple
$ pip config set global.trusted-host [HOST]

HTTPS Configuration

If you use HTTPS but do not provide your Package Manager installation with a valid SSL certificate, pip returns a SSL: CERTIFICATE_VERIFY_FAILED errors when installing packages, because it attempts to verify proper HTTPS configuration by default. To configure pip to ignore these errors, you need to use the --trusted-host flag or configuration option:

Terminal
$ pip install --trusted-host [HOST] --index-url https://[HOST:PORT]/latest/simple PACKAGE-TO-INSTALL

or configure pip to use Package Manager in a persistent manner:

Terminal
$ pip config set global.index-url https://[HOST:PORT]/latest/simple
$ pip config set global.trusted-host [HOST]

Creating a Python PyPI Repository#

The Quickstart section above provides commands for quickly making PyPI packages available. This section further describes these commands and operations.

  • Create a Python repository with a description:
Terminal
$ rspm create repo --name=pypi --type=python --description='Access PyPI packages'
<< Repository: pypi - Python
  • Subscribe the repository to the preconfigured PyPI source:
Terminal
$ rspm subscribe --repo=pypi --source=pypi
<< Repository: pypi
<< Sources:
<< --pypi (Python)

Note

If you try subscribing a non-Python type repository to a Python source, you'll get the error source type must be compatible with repository type.

PyPI and the Posit Package Service#

The PyPI source retrieves its metadata from the Posit Package Service automatically. Package Manager checks for PyPI updates from the Posit Package Service every 10 minutes. If you need to force an immediate update, you can synchronize manually:

Terminal
$ rspm sync --type=pypi

Note

Although Package Manager checks for updates regularly, the Posit Package Service may not update packages every day.