For better or worse, we live in a world that is constantly changing. focus on Better,One striking example is the abundance and rapid ,development of software that helps us achieve our goals. But that blessing comes with challenges. We should actually be able to use Install new features, install new libraries, and integrate new technologies into our packages.
with torch
, there is so much that can be achieved as it stands, but only a small part of it has been hinted at in this blog. But one thing we can be sure of is that there will never be a shortage of demand to do more. Here are three scenarios that come to mind:
-
Load pre-trained models defined in Python (no need to manually port all code)
-
Modify the neural network module to incorporate some new algorithmic improvements (without incurring the performance cost of running custom code in R).
-
Take advantage of one of the many extension libraries available in the PyTorch ecosystem (with as little coding as possible).
This post describes each of these use cases in order. From a practical standpoint, this means gradually moving from a user perspective to a developer perspective. But behind it all, there are the same components that support it all.
aid: torchexport
And torch script
R package torchexport
And (on the PyTorch side) TorchScript operates at a very different scale and plays a very different role. Nonetheless, both are important in this context, even for “small” actors (torchexport
) is a really essential component from an R user's perspective. In part, this is because all three scenarios are true, whereas TorchScript is only concerned with the first scenario.
torchexport: Manages the “type stack” and handles errors.
In R torch
, the depth of the “type stack” is dizzying. The user-facing code is written in R. Low-level functionality is packaged as follows: libtorch
C++ shared libraries that depend on torch
The same goes for PyTorch. As is often the case, the arbiter is Rcpp. But that's not where the story ends. Due to OS-specific compiler incompatibilities, an additional intermediate bi-directional operation layer (Rcpp or libtorch
, resp.), leaving only the raw memory pointer and adding it back to another pointer. The end result is a fairly complex call stack. As you can imagine, this requires carefully placed and level-headed error handling to ensure that the user is presented with usable information at the end.
What now holds? torch
Applies to all R-side extensions that add custom code or call external C++ libraries. This place torchexport
As an extension author, all you have to do is write a very small portion of the code that is needed overall. The rest are generated by: torchexport
. We will return to this in Scenarios 2 and 3.
TorchScript: Enables “on-the-fly” code generation.
We've already encountered TorchScript in previous posts, but from a different angle and emphasized a different set of terms. In that post we will learn how to train a model in R and track The result is an optimized intermediate representation that can be saved and loaded into other (perhaps non-R) environments. The conceptual focus here was on the agent that makes this workflow possible: the PyTorch Just-in-time Compiler (JIT), which generates a representation of the problem. We quickly mentioned that there is another way to invoke JIT on the Python side. That is, you are calling the JIT rather than an instantiated “living” model. Script model definition code. The second method is named accordingly. ScriptingThis is relevant to the current situation.
Even if scripting is not available in R (the scripted code is not written in Python), you can still benefit from its presence. If the Python-side extension library uses TorchScript (instead of plain C++ code), There is no need to add bindings to each function on the R (C++) side.. Instead, PyTorch handles everything.
This is completely transparent to the user, but makes scenario 1 possible. (Python) The pretrained models provided by TorchVision often use (model-dependent) special operators. Because the script is written, there is no need to add bindings for each operator, let alone reimplement it on the R side.
Now that we've outlined some of the basic features, we'll introduce the scenario itself.
Scenario 1: Load a TorchVision pre-trained model
You've probably already used one of the pre-trained models provided by TorchVision. Some of these were ported manually. torchvision
, R package. But there's more to it. many more. Many people use special operators. This is rarely needed outside the context of some algorithms. There seems to be little use for creating R wrappers for those operators. Of course, continued porting efforts will be required on our part if new models continue to emerge.
Fortunately, there is an elegant and effective solution. All required infrastructure is organized into a concise, dedicated package. torchvisionlib
. (It can be lean as it makes liberal use of TorchScript on the Python side, as explained in the previous section. But in this scenario, these details are not important to the user in my view.)
After installing and loading torchvisionlib
, you can choose from numerous image recognition related models. The process then consists of two parts:
-
Instantiate a model in Python. script Save it and save it.
-
Load and use the model in R.
The first steps are: How to create a model before writing a script eval
The mode ensures that all layers exhibit inference time behavior.
import torch
import torchvision
model = torchvision.models.segmentation.fcn_resnet50(pretrained = True)
model.eval()
scripted_model = torch.jit.script(model)
torch.jit.save(scripted_model, "fcn_resnet50.pt")
The second step is much shorter. Loading the model into R requires one line:
library(torchvisionlib)
model <- torch::jit_load("fcn_resnet50.pt")
At this point, you can also use the model to obtain predictions or integrate them as components of a larger architecture.
Scenario 2: Implementing a custom module
Every new and well-received algorithm, every promising new variant of a layer type, or even better, an algorithm we have in mind to reveal to the world in our next paper has already been discovered. torch
?
Well, maybe. But maybe not. A much more sustainable solution would be to make it reasonably easy to scale. torch
Each is a small, dedicated package that serves a clear purpose and is quick to install. Detailed, practical walkthroughs of the process are provided in the package. lltm
. This package has a recursive touch. At the same time, it is also an instance of C++. torch
enlargement, and It serves as a tutorial showing how to create these extensions.
The README itself explains how the code should be structured and why. If you're curious about how torch
It's designed in-house, so it's easy to understand if you read this, whether you're planning on writing an extension or not. In addition to this behind-the-scenes information, the README has step-by-step instructions on how to actually proceed. The source code is also richly documented to match the purpose of the package.
As already hinted at in the “Enabling Factors” section, the reason I dare write “to make it reasonably easy” is to torch
extension) is torchexport
, a package that automatically generates conversion-related and error-handling C++ code for multiple layers of the “type stack”. Typically, the amount of auto-generated code is much greater than the amount of code you write yourself.
Scenario 3: Interface to PyTorch extension embedded in C++ code
It's highly unlikely that one day you'll find a PyTorch extension that you'll want to use in R. If your extension is written (exclusively) in Python, it must be translated “directly” into R. Utilize all applicable features torch
provide. However, sometimes those extensions contain a mix of Python and C++ code. You then need to bind to a low-level C++ function in a way similar to the following: torch
bind to libtorch
– All input requirements described above now apply to extensions in the same way.
Also, it is torchexport
That comes to the rescue. And here too, lltm
README still applies. You just add bindings to externally provided C++ functions instead of writing custom code. You are done. torchexport
Generates all necessary infrastructure code.
A template of sorts can be found here: torchsparse
Package (currently in development). All functions in csrc/src/torchsparse.cpp call PyTorch Sparse, along with function declarations in that project's csrc/sparse.h.
Integrating with external C++ code in this way may raise additional questions. Take an example: torchsparse
. In the header file you can see the return type: std::tuple<torch::Tensor, torch::Tensor>
, <torch::Tensor, torch::Tensor, <torch::optional<torch::Tensor>>, torch::Tensor>>
… And more. In R torch
(C++ layer) we have torch::Tensor
and we torch::optional<torch::Tensor>
, also. However, there is not a custom type for every possible item. std::tuple
You can build it. As is the base torch
Providing all kinds of specialized, domain-specific functionality is not sustainable. Therefore, it makes little sense to predict all the types of things that will be in demand.
Therefore, you must define the type in a package that requires that type. How to do this torchexport
Custom type vignette. If you use these custom types: torchexport
We need to know how to name the types created at the various levels. Therefore, in such cases, instead of a concise explanation, //[[torch::export]]
You will see a line like /. [[torch::export(register_types=c("tensor_pair", "TensorPair", "void*", "torchsparse::tensor_pair"))]]
. The vignette explains this in more detail.
what future plans
“What’s next?” is a common way to conclude a post, replacing “conclusion” or “conclusion.” But here it must be taken literally. We look forward to doing our best to leverage, interface, and extend it. torch
As easily as possible. So, please let us know about any difficulties or problems you are currently facing. Create an issue in torchexport, lltm, torch, or any applicable repository.
As always, thanks for reading!
Photo: Antonino Visalli, Unsplash