When we're still out of memory

We were able to shrink the size to just under the memory limit. However, it is not always possible or desirable to do this. If our dependencies are complex and our files are large, we have two more options:

  • First, the lambda can be just an entry point—it could invoke a Docker image (an isolated virtual environment running somewhere else in the cloud), pass the parameters there, and communicate the results back.
  • Alternatively—and this is a bit of a hack—we could download all the dependencies in the same way that we downloaded our model: during runtime. This data will be lost once the server is down, which will happen if a serverless function is not triggered by anything. Redeployment can take a significant amount of time, so it might make sense to try keeping the server running. All this is an additional hassle, but luckily, it can be taken care of by another package—Zappa (http://github.com/Miserlou/Zappa). 

We won't do any of that in this book, but both of those options are available.

Let's now take a stab at using serverless for scheduled data pipelines.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.15.141.206