---
title: "Calling Python from R"
output: 
  rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Calling Python from R}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---


```{r setup, include=FALSE}
knitr::opts_chunk$set(eval = FALSE)
```

<link rel="stylesheet" type="text/css" href="extra.css"/>


## Overview

The **reticulate** package provides an R interface to Python modules, classes, and functions. For example, this code imports the Python `os` module and calls some functions within it:

```{r, eval=FALSE}
library(reticulate)
os <- import("os")
os$listdir(".")
```
```
 [1] ".git"             ".gitignore"       ".Rbuildignore"    ".RData"          
 [5] ".Rhistory"        ".Rproj.user"      ".travis.yml"      "appveyor.yml"    
 [9] "DESCRIPTION"      "docs"             "external"         "index.html"      
[13] "index.Rmd"        "inst"             "issues"           "LICENSE"         
[17] "man"              "NAMESPACE"        "NEWS.md"          "pkgdown"         
[21] "R"                "README.md"        "reticulate.Rproj" "src"             
[25] "tests"            "vignettes"      
```

Functions and other data within Python modules and classes can be accessed via the `$` operator (analogous to the way you would interact with an R list, environment, or reference class).

The **reticulate** package is compatible with all versions of Python >= 2.7. Integration with NumPy is optional and requires NumPy >= 1.6.

## Python Version

By default, reticulate uses the version of Python found on your `PATH` (i.e. `Sys.which("python")`). The `use_python()` function enables you to specify an alternate version, for example:

```r
library(reticulate)
use_python("/usr/local/bin/python")
```

The `use_virtualenv()` and `use_condaenv()` functions enable you to specify versions of Python in virtual or conda environments, for example:

```r
library(reticulate)
use_virtualenv("myenv")
```

See the article on [Python Version Configuration](versions.html) for additional details.

## Python Packages

You can install any required Python packages using standard shell tools like `pip` and `conda`. Alternately, reticulate includes a set of functions for managing and installing packages within virtualenvs and Conda environments. See the article on [Installing Python Packages](https://rstudio.github.io/reticulate/articles/python_packages.html) for additional details.

## Type Conversions

When calling into Python, R data types are automatically converted to their equivalent Python types. When values are returned from Python to R they are converted back to R types. Types are converted as follows:

| R  | Python | Examples |
|---|---|---|
| Single-element vector   | Scalar |  `1`, `1L`, `TRUE`, `"foo"` |
| Multi-element vector | List  |  `c(1.0, 2.0, 3.0)`, `c(1L, 2L, 3L)` |
| List of multiple types  | Tuple  |  `list(1L, TRUE, "foo")`
| Named list | Dict  |  `list(a = 1L, b = 2.0)`, `dict(x = x_data)`
| Matrix/Array | NumPy ndarray  | `matrix(c(1,2,3,4), nrow = 2, ncol = 2)`
| Data Frame | Pandas DataFrame | ` data.frame(x = c(1,2,3), y = c("a", "b", "c"))`  |
| Function | Python function | `function(x) x + 1`
| NULL, TRUE, FALSE  | None, True, False  |  `NULL`, `TRUE`, `FALSE`

If a Python object of a custom class is returned then an R reference to that object is returned. You can call methods and access properties of the object just as if it was an instance of an R reference class.

## Importing Modules

The `import()` function can be used to import any Python module. For example:

```{r}
difflib <- import("difflib")
difflib$ndiff(foo, bar)

filecmp <- import("filecmp")
filecmp$cmp(dir1, dir2)
```

The `import_main()` and `import_builtins()` functions give you access to the main module where code is executed by default and the collection of built in Python functions. For example:

```{r}
main <- import_main()

builtins <- import_builtins()
builtins$print('foo')
```

The main module is generally useful if you have executed Python code from a file or string and want to get access to it's results (see the section below for more details).

## Sourcing Scripts

The `source_python()` function will source a Python script and make the objects it creates available within an R environment (by default the calling environment). For example, consider the following Python script:

```python
def add(x, y):
  return x + y
```

We source it using the `source_python()` function and then can call the `add()` function directly from R:

```{r}
source_python('add.py')
add(5, 10)
```
```
[1] 15
```

## Executing Code

You can execute Python code within the main module using the `py_run_file` and `py_run_string` functions. You can then access any objects created using the `py` object exported by reticulate:

```{r}
library(reticulate)

py_run_file("script.py")

py_run_string("x = 10")

# access the python main module via the 'py' object
py$x
```

## Object Conversion

By default when Python objects are returned to R they are converted to their equivalent R types. However, if you'd rather make conversion from Python to R explicit and deal in native Python objects by default you can pass `convert = FALSE` to the `import` function. In this case Python to R conversion will be disabled for the module returned from `import`. For example:

```{r}
# import numpy and specify no automatic Python to R conversion
np <- import("numpy", convert = FALSE)

# do some array manipulations with NumPy
a <- np$array(c(1:4))
sum <- a$cumsum()

# convert to R explicitly at the end
py_to_r(sum)
```

As illustrated above, if you need access to an R object at end of your computations you can call the `py_to_r()` function explicitly.


## Getting Help

You can print documentation on any Python object using the `py_help()` function. For example:

```r
os <- import("os")
py_help(os$chdir)
```



## Lists, Tuples, and Dictionaries

The automatic conversion of R types to Python types works well in most cases, but occasionally you will need to be more explicit on the R side to provide Python the type it expects.

For example, if a Python API requires a list and you pass a single element R vector it will be converted to a Python scalar. To overcome this simply use the R `list` function explicitly:

```r
foo$bar(indexes = list(42L))
```

Similarly, a Python API might require a `tuple` rather than a list. In that case you can use the `tuple()` function:

```r
tuple("a", "b", "c")
```

R named lists are converted to Python dictionaries however you can also explicitly create a Python dictionary using the `dict()` function:

```r
dict(foo = "bar", index = 42L)
```

This might be useful if you need to pass a dictionary that uses a more complex object (as opposed to a string) as it's key.

## Numeric Types and Indexes

R and Python have different default numeric types. If you write `42` in R it is considered a floating point number whereas `42` in Python is considered an integer.

This means that when a Python API expects an integer, you need to be sure to use the `L` suffix within R. For example, if the `foo` function requires an integer as it's `index` argument you would do this:

```r
foo$bar(index = 42L)
```

Python collections are addressed using 0-based indices rather than the 1-based indices you might be familiar with from R. So to address the first item of an array in R you would write:

```r
items[[1]]
```

Whereas if you are calling a method in Python via reticulate that takes an index you would write this to address the first item:

```r
items$get(0L)
```

Note the use of the 0-based index as well as the `L` to indicate t that the value is an integer.

## Arrays

R matrices and arrays are converted automatically to and from [NumPy](http://www.numpy.org/) arrays. 

When converting from R to NumPy, the NumPy array is mapped directly to the underlying memory of the R array (no copy is made). In this case, the NumPy array uses a column-based in memory layout that is compatible with R (i.e. Fortran style rather than C style). When converting from NumPy to R, R receives a column-ordered copy of the NumPy array. 

You can also manually convert R arrays to NumPy using the `np_array()` function. For example, you might do this if you needed to create a NumPy array with C rather than Fortran style in-memory layout (for higher performance in row-oriented computations) or if you wanted to control the data type of the NumPy array more explicitly. Here are some example uses of `np_array()`:

```r
a <- np_array(c(1:8), dtype = "float16")
a <- np_array(c(1:8), order = "C")
```

Reasoning about arrays which use distinct in-memory orders can be tricky. The [Arrays in R and Python](arrays.html) article provides additional details.

Also, always remember that when calling NumPy methods array indices are 0 rather than 1 based and require the `L` suffix to indicate they are integers.

## Data Frames

R data frames can be automatically converted to and from [Pandas](https://pandas.pydata.org/) DataFrames. By default, columns are converted using the same rules governing R array <-> NumPy array conversion, but a couple extensions are provided:

| R      | Python                                    |
| ------ | ----------------------------------------- |
| Factor | Categorical Variable                      |
| POSIXt | NumPy array with dtype = `datetime64[ns]` |

If the R data frame has row names, the generated Pandas DataFrame will be re-indexed using those row names (and vice versa). Special handling is also available for a `DatetimeIndex` associated with a Pandas DataFrame; however, because R only supports character vectors for row names they are converted to character first.

## With Contexts

The R `with` generic function can be used to interact with Python context manager objects (in Python you use the `with` keyword to do the same). For example:

```{r}
py <- import_builtins()
with(py$open("output.txt", "w") %as% file, {
  file$write("Hello, there!")
})
```

This example opens a file and ensures that it is automatically closed at the end of the with block. Note the use of the `%as%` operator to alias the object created by the context manager.

## Iterators

If a Python API returns an [iterator or generator](http://anandology.com/python-practice-book/iterators.html) you can interact with it using the `iterate()` function. The `iterate()` function can be used to apply an R function to each item yielded by the iterator:

```{r}
iterate(iter, print)
```

If you don't pass a function to `iterate` the results will be collected into an R vector:

```{r}
results <- iterate(iter)
```

Note that the `Iterators` will be drained of their values by `iterate()`:

```{r}
a <- iterate(iter) # results are not empty
b <- iterate(iter) # results are empty since items have already been drained
```

### Element Level Iteration

You can also iterate on an element-by-element basis using the `iter_next()` function. For example:

```{r}
while (TRUE) {
  item <- iter_next(iter)
  if (is.null(item))
    break
}
```

By default `iter_next()` will return `NULL` when the iteration is complete but you can provide a custom `completed` value it will be returned instead. For example:

```{r}
while (TRUE) {
  item <- iter_next(iter, completed = NA)
  if (is.na(item))
    break
}
```

Note that some iterators/generators in Python are infinite. In that case the caller will need custom logic to determine when to terminate the loop.


### Generators

Python [generators](https://wiki.python.org/moin/Generators) are functions that implement the Python iterator protocol. Similarly, the reticulate `generator()` function enables you to create a Python iterator from an R function.

In Python, generators produce values using the `yield` keyword. In R, values are simply returned from the function. One benefit of the `yield` keyword is that it enables successive iterations to use the state of previous iterations. In R, this can be done by returning a function that mutates it's enclosing environment via the <<- operator. For example:

```{r}
# define a generator function
sequence_generator <-function(start) {
  value <- start
  function() {
    value <<- value + 1
    value
  }
}

# convert the function to a python iterator
iter <- py_iterator(sequence_generator(10))
```

If you want to indicate the end of the iteration, return `NULL` from the function:

```{r}
sequence_generator <-function(start) {
  value <- start
  function() {
    value <<- value + 1
    if (value < 100)
      value
    else
      NULL
  }
}
```

Note that you can change the value that indicates the end of the iteration using the `completed` parameter (e.g. `py_iterator(func, completed = NA)`).

## Advanced Functions

There are several more advanced functions available that are useful principally when creating high level R interfaces for Python libraries.

### Python Objects

Typically interacting with Python objects from R involves using the `$` operator to access whatever properties for functions of the object you need. When using the `$`, Python objects are automatically converted to their R equivalents when possible. The following functions enable you to interact with Python objects at a lower level (e.g. no conversion to R is done unless you explicitly call the `py_to_r` function):

| Function  | Description | 
|----------------|------------------------------------------------------------------|
| `py_has_attr()` | Check if an object has a specified attribute.  | 
| `py_get_attr()` | Get an attribute of a Python object.  | 
| `py_set_attr()` | Set an attribute of a Python object.  | 
| `py_list_attributes()` | List all attributes of a Python object. |
| `py_len()` | Length of Python object. |
| `py_call()` | Call a Python callable object with the specified arguments.  | 
| `py_to_r()` | Convert a Python object to it's R equivalent |
| `r_to_py()` | Convert an R object to it's Python equivalent |

### Pickle

You can save and load Python objects (via pickle) using the `py_save_object` and `py_load_object` functions:

| Function  | Description | 
|----------------|------------------------------------------------------------------|
| `py_save_object()` | Save a Python object to a file with pickle. |
| `py_load_object()` | Load a previously saved Python object from a file. |

### Configuration

The following functions enable you to query for information about the Python configuration available on the current system.

| Function  | Description | 
|----------------|------------------------------------------------------------------|
| `py_available()` | Check whether a Python interface is available on this system. |
| `py_numpy_available()` | Check whether the R interface to NumPy is available (requires NumPy >= 1.6) |
| `py_module_available()` | Check whether a Python module is available on this system. |
| `py_config()` | Get information on the location and version of Python in use.  | 


### Output Control

These functions enable you to capture or suppress output from Python:

| Function  | Description | 
|----------------|------------------------------------------------------------------|
| `py_capture_output()` | Capture Python output for the specified expression and return it as an R character vector. | 
| `py_suppress_warnings()` | Execute the specified expression, suppressing the display Python warnings.   | 


### Miscellaneous

The functions provide miscellaneous other lower-level capabilities:

| Function  | Description | 
|----------------|------------------------------------------------------------------|
| `py_set_seed()` | Set Python and NumPy random seeds. |
| `py_unicode()` | Convert a string to a Python unicode object. |
| `py_str()` | Get the string representation of Python object. |
| `py_id()`  | Get a unique identifier for a Python object |
| `py_is_null_xptr()` | Check whether a Python object is a null externalptr. |
| `py_validate_xptr()` | Check whether a Python object is a null externalptr and throw an error if it is. |


## Learning More

The following articles cover additional aspects of using **reticulate**:

- [R Markdown Python Engine](r_markdown.html) 

- [Python Version Configuration](versions.html)

- [Installing Python Packages](python_packages.html) 

- [Using reticulate in an R Package](package.html)

- [Arrays in R and Python](arrays.html)


