Home:ALL Converter>How to use Kedro with Pipenv?

How to use Kedro with Pipenv?

Ask Time:2019-11-26T01:28:55         Author:jayBana

Json Formatter

I am currently using kedro, version 0.15.4 with pipenv, version 2018.11.26.

At the moment, I have to do the following if I want to use Pipenv (For this example, I want this project to reside in the kedro-pipenv directory):

  1. mkdir kedro-pipenv && cd kedro-pipenv
    • This is needed so that the virtualenv created is "tied" to the project directory which only really means that it has a name that is based on the directory name from which the pipenv install kedro or pipenv shell commands are executed.
    • Unfortunately, at the moment Pipenv doesn't have the functionality to support custom virtualenv names.
  2. pipenv install kedro
    • In my example, this generates a virtualenv with name kedro-pipenv-AB9IGRnB which resides in the following location ~/.local/share/virtualenvs/kedro-pipenv-AB9IGRnB/
  3. pipenv shell
  4. kedro info
    • all is looking good, kedro was successfully installed in the virualenv handled by Pipenv
  5. cd .. & kedro new
    • I navigate one directory up and provide kedro-pipenv as the directory name for the project. Given that the directory was already created before in step 1, this fails which is expected and I get the following message:
cookiecutter.exceptions.OutputDirExistsException:
Error: "/Users/xyz/projects/kedro-pipenv" directory already exists
Run with --verbose to see the full exception
Error: Failed to generate project.

In order to "work around" this, I do the following whilst still in the same virtualenv as before:

  • mv kedro-pipenv kedro-pipenv_tmp
    • rename the existing directory
  • kedro new
    • I provide kedro-pipenv as the directory name for the project.
  • mv kedro-pipenv_tmp/Pipfile* kedro-pipenv && rm -rf kedro-pipenv_tmp
    • This step is needed so that the Python version and the single kedro dependency is maintained.
  • cd kedro-pipenv
  • kedro install
  • Using kedro build-reqs for managing project requirements.

I am familiar with conda as well and it seems that it is a much cleaner way of handling environments for Kedro at the moment.

However, for most of my other projects I have been using pyenv in conjunction with Pipenv for environment and dependency management. This allows me to have environment information tied to specific project spaces by having a Pipfile in each of my projects' root directory.

Has anyone got any suggestions on how to improve the above workflow?

Author:jayBana,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/59037182/how-to-use-kedro-with-pipenv
yy