fix typos in resource monitoring Python notebook

753d808c · Andrei Plamada · 6d1cfe16 · 753d808c
Verified Commit 753d808c authored 5 months ago by Andrei Plamada
--- a/Resource_Monitoring_and_Benchmarking/Python.ipynb
+++ b/Resource_Monitoring_and_Benchmarking/Python.ipynb
 {
 "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2c609917-7b4c-460e-b1f5-28f0a9558b1b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!which python"
+   ]
+  },
  {
   "cell_type": "markdown",
   "id": "720e584b-b00d-4ec8-b3a6-6a5e0b67a52c",
@@ -111,8 +121,7 @@
   "source": [
    "## Profiling with `%prun`, `%lprun`\n",
    "- `%prun` is the python code profiler see https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-prun\n",
-    "- `%lprun` is a line profiler available in the [`line_profiler`](https://github.com/pyutils/line_profiler) package (more at https://github.com/pyutils/line_profiler)\n",
-    "- "
+    "- `%lprun` is a line profiler available in the [`line_profiler`](https://github.com/pyutils/line_profiler) package (more at https://github.com/pyutils/line_profiler)"
   ]
  },
  {
@@ -293,7 +302,6 @@
    "\n",
    "\n",
    "**What is faster:**\n",
-    "\n",
    "```{python}\n",
    "n = 10\n",
    "sum(range(n))\n",
@@ -445,6 +453,14 @@
    "%%memray_flamegraph\n",
    "np.arange(n).sum()"
   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a3f3426-255b-4ac7-8933-d1655346ec92",
+   "metadata": {},
+   "outputs": [],
+   "source": []
  }
 ],
 "metadata": {
@@ -463,7 +479,7 @@
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
-   "version": "3.11.8"
+   "version": "3.11.10"
  }
 },
 "nbformat": 4,

+%% Cell type:code id:2c609917-7b4c-460e-b1f5-28f0a9558b1b tags:
+
+``` python
+!which python
+```
+
 %% Cell type:markdown id:720e584b-b00d-4ec8-b3a6-6a5e0b67a52c tags:


 # Resource monitoring

 [`jupyter-resource-usage`](https://github.com/jupyter-server/jupyter-resource-usage) displays:
 - usage information in the status bar (refreshed every 5s)
 - more information in the sidebar
 - see https://github.com/jupyter-server/jupyter-resource-usage

 # Benchmarking and Profiling Python code

 We start with the previous python example

 %% Cell type:code id:a5da3e84-47df-40e4-8181-72abc3655b0f tags:

 ``` python
 %pycat example_2.py
 ```

 %% Cell type:code id:00f85e8f-86df-4f04-901a-d3251dff8b17 tags:

 ``` python
 import os
 from array import array
 ```

 %% Cell type:code id:3546c39e-248f-48c9-abf8-d67bc0a7aa49 tags:

 ``` python
 def sum_pow(n, exponent: int = 88, modulo: int = 100) -> int:
    # memory allocation
    array("l", range(10**8))

    # some computation
    return sum(pow(i, exponent, modulo) for i in range(n))
 ```

 %% Cell type:markdown id:b1bd0410-06ac-4e17-899e-95c2528018fe tags:

 ## Line and Cell magics
 - Line magics (line starting with `%`)
 - Cell magics (cell starting with `%%`)

 %% Cell type:markdown id:9a042650-abbb-4d87-b515-55543e78eaf2 tags:

 ## `%%time`
 - similar to the `time` from bash
 - more at https://ipython.readthedocs.io/en/stable/interactive/magics.html#line-magics

 %% Cell type:code id:8f11bab5-9a71-473c-812a-19bf329ca75c tags:

 ``` python
 %%time
 sum_pow(0)
 ```

 %% Cell type:code id:3ee78b06-ffd5-42f4-a21b-735eda8ec4d1 tags:

 ``` python
 %%time
 sum_pow(19)
 ```

 %% Cell type:markdown id:e5626c16-a373-4968-920b-1d4a1d4597f0 tags:

 Pay attention at the Kernel usage

 %% Cell type:markdown id:2724ad5b-21ff-4d5a-bc86-542208cb6fca tags:

 ## Profiling with `%prun`, `%lprun`
 - `%prun` is the python code profiler see https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-prun
 - `%lprun` is a line profiler available in the [`line_profiler`](https://github.com/pyutils/line_profiler) package (more at https://github.com/pyutils/line_profiler)
-

 %% Cell type:code id:c04ba7ae-45a6-48d4-bd0d-c550676e286f tags:

 ``` python
 %prun sum_pow(0)
 ```

 %% Cell type:code id:7e0c8fb5-9d38-4347-9ced-a497ecf6d370 tags:

 ``` python
 # needed for %lprun
 %load_ext line_profiler
 ```

 %% Cell type:code id:a3e4bfa8-2fff-4b1d-a573-ff5e84ba33db tags:

 ``` python
 %lprun -f sum_pow sum_pow(0)
 ```

 %% Cell type:markdown id:df42a017-a583-4cf9-bcf6-efc4ea6d5ce4 tags:

 ## Memory profiling with `%%memray_flamegraph`
 - allows to understand the memoroy usage
 - more at https://bloomberg.github.io/memray/overview.html

 %% Cell type:code id:0192a451-821d-4035-9aa1-46122bbc3cff tags:

 ``` python
 # needed for %%memray_flamegraph
 %load_ext memray
 ```

 %% Cell type:code id:91c6335f-26d7-4bd4-befc-5ebacc8e19ee tags:

 ``` python
 %%memray_flamegraph
 sum_pow(0)
 ```

 %% Cell type:markdown id:b69a8600-7b5f-4ee1-86f5-c60393fb043f tags:

 ## Measuring the memory of Python objects with `pympler`
 - see https://pympler.readthedocs.io/en/latest/
 - pay attention that `sys.getsizeof()` is not what you might expect (more at https://docs.python.org/3/library/sys.html#sys.getsizeof)
  > Only the memory consumption directly attributed to the object is accounted for, not the memory consumption of objects it refers to.

 %% Cell type:code id:0fddaae0-8757-44c1-96e2-2a8068816855 tags:

 ``` python
 # source https://stackoverflow.com/questions/5194057/better-way-to-convert-file-sizes-in-python
 import math

 def convert_size(size_bytes):
   if size_bytes == 0:
       return "0B"
   size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
   i = int(math.floor(math.log(size_bytes, 1024)))
   p = math.pow(1024, i)
   s = round(size_bytes / p, 2)
   return "%s %s" % (s, size_name[i])
 ```

 %% Cell type:code id:e5f4a317-5663-4d34-8dc8-d7c86f946aa3 tags:

 ``` python
 from pympler import asizeof
 import sys
 ```

 %% Cell type:code id:db2b6d49-44a0-4c4c-9156-6e9561adffb7 tags:

 ``` python
 obj = array("l", range(10**8))
 ```

 %% Cell type:code id:5b96c471-f6d2-4284-b9e7-515deabcd0b4 tags:

 ``` python
 convert_size(asizeof.asizeof(obj))
 ```

 %% Cell type:code id:6eb9ca79-2cc0-4a4e-bc9b-179fcb1d600e tags:

 ``` python
 convert_size(sys.getsizeof(obj))
 ```

 %% Cell type:code id:c31e99f7-b4fc-4cc3-8ebe-b9e4795bd25c tags:

 ``` python
 obj_in_a_list=[obj]
 ```

 %% Cell type:code id:0ef58021-bfee-4d2e-8c6f-2a642e960d1a tags:

 ``` python
 convert_size(sys.getsizeof(obj_in_a_list))
 ```

 %% Cell type:code id:b0783ace-db47-4baf-b8a1-2d2160118c3d tags:

 ``` python
 convert_size(asizeof.asizeof(obj_in_a_list))
 ```

 %% Cell type:markdown id:c8dd4f40-da22-442e-b4b6-bc7a6ee0ac33 tags:

 ## Microbenchmarking with `%timeit`

 - useful for accurate measurements of fast snippets of code. The code is executed multiple times in a loop.
 - it reports the statistics of 7 repeats of the loop
 - see more at https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-timeit


 **What is faster:**
-
 ```{python}
 n = 10
 sum(range(n))
 np.arange(n).sum()
 ```

 %% Cell type:code id:22e14962-3757-4841-9a61-60ad33e8bb41 tags:

 ``` python
 n = 10
 ```

 %% Cell type:code id:c004f03a-5d2f-4e41-9f7d-892f4e485daa tags:

 ``` python
 sum(range(n))
 ```

 %% Cell type:code id:7761b208-a469-4c81-bba0-7501d6212ee5 tags:

 ``` python
 import numpy as np
 np.arange(n).sum()
 ```

 %% Cell type:code id:b445c4c9-ec2d-4e3a-b18c-11b220e15a71 tags:

 ``` python
 %%time
 sum(range(n))
 ```

 %% Cell type:code id:86530b5f-7fbe-4003-a568-6b8b6650bbbe tags:

 ``` python
 %%time
 np.arange(n).sum()
 ```

 %% Cell type:code id:4301440b-2742-4883-bd9b-979cb9568eb9 tags:

 ``` python
 %%timeit
 sum(range(n))
 ```

 %% Cell type:code id:41d1fb63-8d19-4612-885c-e406febc166f tags:

 ``` python
 %%timeit
 np.arange(n).sum()
 ```

 %% Cell type:markdown id:adae33c2-f15f-4489-93bc-ecdfb00f4186 tags:

 What about `n = 10_000_000`

 %% Cell type:code id:e73102f6-8fea-4907-b1f7-f00079b3ccba tags:

 ``` python
 n = 10_000_000
 ```

 %% Cell type:code id:fe9cd648-294c-4960-a7d8-ea7d5a9682d0 tags:

 ``` python
 %%timeit
 sum(range(n))
 ```

 %% Cell type:code id:3481b3a9-c05d-455f-8220-e5bc8a148a64 tags:

 ``` python
 %%timeit
 np.arange(n).sum()
 ```

 %% Cell type:markdown id:b0b19b62-e33e-4856-a82a-e7161ad035f3 tags:

 But NumPy should be faster!

 %% Cell type:code id:682e52c8-d4b5-47ff-af7a-c4f543744fc9 tags:

 ``` python
 %%memray_flamegraph
 sum(range(n))
 ```

 %% Cell type:code id:ce666348-abc7-45c2-b9e7-8a81e4fdb938 tags:

 ``` python
 %%memray_flamegraph
 np.arange(n).sum()
 ```
+
+%% Cell type:code id:1a3f3426-255b-4ac7-8933-d1655346ec92 tags:
+
+``` python
+```