# Python
## build
### github-actions
- <https://calmcode.io/github-actions/introduction.html>
### poetry
**Using Poetry**
- 1. [Installation](https://python-poetry.org/docs/#windows-powershell-install-instructions)
- 1.1 Configuration: `poetry completions bash > /etc/bash_completion.d/poetry.bash-completion`
- 2. Usage `poetry new poetry-demo` to make a new directory project or `poetry init` in current project
- 3. Activate venv in the project `poetry shell`
- 4. Add dependencies `poetry add pandas` add developer dependencies with `poetry add --dev pytest`
- 4.1 get latest versions of all dependencies `poetry update` or list which to update `poetry update pandas`
- 5. Build the source and wheel archives `poetry build`
- 6. deactivate venv `exit`
- 7. [publish to PyPi](https://python-poetry.org/docs/libraries/#publishing-to-pypi) `poetry publish`
- Opening a poetry project and installing them `poetry install`
- update poetry `poetry self update`
- poetry help pages easy to access `poetry help <command>`
## conventions
- Classes: Proper Snake Case
- Constants: Upper Snake Case
- Functions: Lower Snake Case
- Variables: Lower Snake Case
- Spaces > Tabs
## data Structures
### Dictionary
```python
# key value pairs, basically an object in javascript
my_dictionary = {"key":"my_key", "value":666}
my_dictionary.keys()
my_dictionary.values()
# to add more key value pairs after initialization to the dictionary use object name and brackets around the key and set equal to the value:
my_dictionary["name"] = "Bryan Jenks"
# with bracket method the key needs quotes around it but in the constructor function you DONT need quotes on the key names:
dict(message="test", language="english")
for key, value in my_dictionary.items():
print(key, "=", value)
```
### List
```python
# basically an array as you understand them
a = [1, 2, 3, 4, 5]
a.append(17) # adds a new item to the end of the list
a.append(19) # adds a new item to the end of the list
a = [1, 2, 3, 4, 5, 17, 19]
a[-1] # shows the item at the end of the list by wrapping around
# you can only wrap around completely, once, otherwise you will get an error.
a[2:5] # this slices out a smaller list. the starting index number element is included and the ending index number is not included so indexes returned will be 2, 3, 4 but not 5
b = ['a', 'b', 'c']
a + b # = [1, 2, 3, 4, 5, 17, 19, 'a', 'b', 'c']
# adding lists together causes concatenation
```
### Data Structures are passed by reference not value
```python
# Data structures are passed by reference, not by value.
# So the dictionary can be passed into a class instance, operated upon, and then when the class is over the data itself is mutated
# making it so classes dont need to play hot potato with data. and just just be called and manipulate data at the global state level.
class Mutator:
def __init__(self, data) -> None:
self.data = data # Grab a reference to the data for the class to use internally
def mutate(self):
self.data["fname"] = "John" # Manipulate the data in the class
print(F"After function call within class instance:\n\t{data}") # Show result of manipulation
if __name__ == "__main__":
data = {
"fname": "Bryan",
"lname": "Jenks",
"age": 29,
}
print(F"Before function call and at global scope:\n\t{data}") # show original data in before manipulation
m = Mutator(data) # Pass the data into the class
m.mutate() # Manipulate the data
print(F"After class instance, and at global scope:\n\t{data}") # Show result of manipulation at global scope level (not in class)
```
### Data Structures do not even need to be passed to class \_\_init\_\_ functions
```python
# Data structures like lists and dictionaries dont even need to be passed into the classes init function.
# The data lives at global scope, is mutated by reference not value.
class Mutator:
def mutate(self):
data.append(3) # >>> [0, 1, 2, 3]
def add_four(data):
data.append(4)
add_four(data) # >>> [0, 1, 2, 3, 4]
if __name__ == "__main__":
data = [0, 1, 2]
print(data) # >>> [0, 1, 2]
m = Mutator()
m.mutate()
print(data) # >>> [0, 1, 2, 3, 4]
```
### Set
```python
# a set is like a hashtable, the order is not important and they are not ordered like an array
# sets cannot contain duplicate values
a = set([1,2,3,4])
a = (1, 2, 3, 4)
# methods
a.add() # adds an item to the set
a.remove() # removes an existing item from the set
a.discard() # acts like remove but if item is not a part of the set, do nothing and throw no errors (remove quietly)
a.clear() # removes all items from the set and it becomes empty
len(a) # lets you see how many items are in the set
# faster way of making a set is to pass an `[]` array to the set() function
a = set([1,3,5,7,9])
b = set([2,4,6,8,10])
c = set([2, 3, 5 ,7])
a.union(b) # = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
a.intersection(c) # = 3, 5, 7
2 in b # result is True
9 not in a # Result is False 9 IS in the set of odd numbers
```
#### Frozen Set
```python
# less methods than sets
# immutable
```
### Tuple
```python
# can be created just by assigning values encapsulated in parens
# or for a single value leaving a trailing comma
a = (1, 2, 3, 4)
# or
b = ('a',)
# or
c = 1, 2, 3, 4,
# tuples are immutable, they are smaller in memory than lists, have less methods available to them, and are faster that lists.
survey = (27, "vietnam", True)
age, country, knows_python = survey # this works
print(age)
print(country)
print(knows_python)
```
## data Types
### booleans
```python
True & False # these ARE the operators
true & false # These are NOT the operators
TRUE & FALSE # These are NOT the operators
bool(" ") # True
bool("") # False
bool(1) # True
bool(0) # False
```
### generators
```python
# Instead of holding values in memory:
x = range(1_000_000)
# Use a generator that only spits out values as needed
x = (i for i in range(1_000_000))
# ================================================ #
# Using functions as a generator
def gen(n):
for i in range(n):
yield i
for i in gen(5):
print(i)
# V.S.
x = range(5)
for i in x:
print(i)
```
### integers
```python
# Integer whole numbers
a = 496
print(type(a)) # int for integer
# Float for floating point number
a = 496
print(type(a)) # float
# Complex Numbers (in math i is used for imaginary numbers, in programming and python it is 'j')
a = 496 - 6.1j #the j is the complex part
print(type(a)) # complex
print(a.real) # view the real part of the number
print(a.imag) # view the imaginary part of the number
```
### Strings
```python
# This is an ordinary string
print("This is a string")
# This is a function Doc string
def function(x):
"""This is a doc string explaining
the inner workings of this function
"""
y = x * 7
reutrn(y)
x = 17
printer = "hewlet packard"
# This is a template literal "F string"
print(f"I just printed {x} pages to the printer {printer}")
# or for less elegence
print("I just printed" + x + "pages to the printer" + printer)
```
#### fstrings
- [Python 3's f-Strings: An Improved String Formatting Syntax (Guide)](https://realpython.com/python-f-strings/)
- F String format specifications
!!! **DO NOT USE F-STRINGS FOR SQL STATEMENTS** use the `.format` syntax as this santizes a lot of malicious input
##### Date and number formatting
```python
import datetime
from datetime import datetime
today = datetime.today()
print(f'{datetime.datetime.utcnow():%Y-%m-%d}')
print(f"Today is {today}")
# Today is 2021-07-31 18:20:48.956829
print(f"Today is {today:%B %d, %Y}")
# Today is July 31, 2021
print(f"Today is {today:%m-%d-%Y}")
# Today is 07-31-2021
print(f"Today is {datetime.today()}")
# Today is 2021-07-31 18:20:48.956829
```
##### nested-formatting-conditions
```python
num_value = 123.456
nested_format = ".2f"
print(f'{num_value:{nested_format}}')
# >>> 123.46
```
##### Number Formatting
###### Floats
```python
pi = 3.1415926
print(f'Pi is approximately equal to {pi:.2f}')
# Pi is approximately equal to 3.14
```
###### Integer
```python
id = 1 # need to print a 3-digit number
print(f"The id is {id:03d}")
# The id is 001
```
###### Number separator
```python
N = 1000000000 # need to add separator
print(f'His networth is ${N:,d}')
# His networth is $1,000,000,000
```
##### Printing Ascii
- to print the repr of the value end the fstring with a `!r`:
```python
x = "hello ✅"
print(f'{x!r}')
# >>> "hello ✅"
```
- to print only ascii values end the fstring with a `!a`:
```python
x = "hello ✅"
print(f'{x!a}')
# >>> "hello \u2705"
```
- to print string values without quotes end the fstring with a `!s`:
```python
x = "hello ✅"
print(f'{x!a}')
# >>> hello ✅
```
##### Reuse variable name
- f strings that have a trailing ` = ` will print a statement like this:
```python
x=7
print(f'{x=}')
# >>> x=7
print(f'{x =}')
# >>> x =7
print(f'{x = }')
# >>> x = 7
```
### type hints
#### Any Type
Any Type is the same as not adding an annotation but more explicit
```python
from typing import Any
def foo(output: Any):
pass
```
#### Callable Type:
```python
from typing import callable, Optional
def foo(func: Callable[[int, int, Optional[int]], int]) -> None:
func(1, 2)
def add(x: int, y: int) -> int:
return x + y
foo(add)
#=================================================================#
def foo() -> Callable[[int, int, Optional[int]], int]):
def add(x: int, y: int) -> int:
return x + y
return add
foo()
#=================================================================#
def foo() -> Callable[[int, int], int]):
func: Callable[[int, int], int]) = Lambda x, y: x + y
return func
foo()
```
#### custom
- Using the `typing` library you can import types for data structures and custom types more than just the standard primitive types
##### Custom Typing
```python
from typing import List
Vector = List[float]
def foo(v: Vector) -> Vector:
print(v)
```
![[Pasted_image_20211215085306.png]]
##### Can also use our own custom types like this:
```python
from typing import List
Vector = List[float]
Vectors = List[Vector]
def foo(v: Vectors) -> Vectors:
print(v)
```
#### Generics:
```python
from typing import TypeVar, List
T = TypeVar('T')
def get_item(lst: List[T], index: int) -> T:
return lst[index]
```
#### Optional typing
```python
from typing import Optional
def foo(output: Optional[bool]=False):
pass
foo()
```
#### Sequence Type
```python
from typing import Sequence
def foo(seq: Sequence[str]):
pass
foo("Hello") # This is fine because a string is a sequence of characters
foo(("a", "b", "c")) # a Tuple is an ordered and immutable indexed Object
foo(["a", "b", "c"]) # A list is an ordered and indexed object
foo({1, 2, 3}) # A set is hashed and not indexed or ordered so it cannot be a sequence
foo(1)
#>>> Last one throws an error because static analysis determines that it is an incompabile type
```
#### Tuple Type:
```python
from typing import Tuple
# This is an error because the tuple can contain items of differing types
# so you need to specify the type of each item within it
x: Tuple[int] = (1, 2, 3)
x: Tuple[int, int, int] = (1, 2, 3)
```
#### union
[[union-operator-type-hint]]
## files
```python
with open("employees.txt","r") as employee_file: # Filename, Mode (r:read a:append w:write)
print(employee_file.readable()) # returns T/F if we can read the file
print(employee_file.readline()) # reads the first line of the file
print(employee_file.readline()) # reads the next line (second) in the file
print(employee_file.readline()[2]) # reads the (third) in the file
# always close open files
employee_file.close() # not necessary if using a context manager `with`
# Python program to illustrate
# Append vs write mode
with open("myfile.txt", "w") as file1:
L = ["This is Delhi \n", "This is Paris \n", "This is London"]
file1.writelines(L)
# Append-adds at last
with open("myfile.txt", "a") as file1: # append mode
file1.write("Today \n")
with open("myfile.txt", "r") as file1:
print("Output of Readlines after appending")
print(file1.read())
print()
# Write-Overwrites
with open("myfile.txt", "w") as file1: # write mode
file1.write("Tomorrow \n")
with open("myfile.txt", "r") as file1:
print("Output of Readlines after writing")
print(file1.read())
print()
# Output:
#> Output of Readlines after appending
#> This is Delhi
#> This is Paris
#> This is LondonToday
#>
#>
#> Output of Readlines after writing
#> Tomorrow
```
## flow
### Case Statement
- Python 3.10 added Match Case statements
```python
x = "hello"
match x:
case "hello":
print("hello")
case "hi":
print("hi")
case _:
print("default case")
```
### Exception Handling
```python
try:
number = int(input("enter a number: "))
print(number/0) #divide by zero
except ZeroDivisionError as err:
print(err)
print("Divided By Zero")
except ValueError:
print("invalid input")
# OUTPUT:
## ./test.py
## enter a number: 1
## integer division or modulo by zero
## Divided By Zero
```
### loops
##### For Loop
```python
for letter in "giraffe academy":
print(letter)
friends = ["Jim", "Suzy", "Kevin"]
for name in friends:
print(name)
for index in range(10):
print(index) # prints 1-9 not including 10 so always increment upwards by 1
```
##### While Loop
```python
i = 1
while i <= 10:
print(i)
i+=1
```
### Ternary Operator
```python
#You'll typically see:
reputation = 30
if reputation > 25:
name = "admin"
else:
name = "visitor"
print(name)
reputation = 20
name = "admin" if reputation > 25 else "visitor"
print(name)
#>>> admin
#>>> visitor
```
## Functions
```python
def pow(p):
"""Calculate the power"""
return p*p
```
### The \_\_main\_\_ Function
```python
import time
def useful_function():
for i in range(5):
print("I sweat I'm useful: {}".format(i))
time.sleep(1.0)
# This function call if un commented would make it so if the script was run
# on the CLI then it would execute the function code. However, if we want to
# Re-use the code as a module import and not have it run immediately on import
# Then you need the following logical construct
# useful_function()
# This code checks if the entry point to the current session is this file.
# i.e. did we run this file like a script? or are we importing from another
# location that is actually "__main__" and we're a sub process declaring
# variables, functions, and classes? Anything under this construct is ran and
# defined only if the script is executed as a stand alone script and not imported
if __name__ == "__main__":
useful_function()
```
### decorators
```python
def hello_decorator(func):
def wrapper(*args, **kwargs):
result = func(*args, **kwargs)
return result
return wrapper
@hello_decorator
def add(a, b):
return a + b
if __name__ == '__main__':
output = add(2, 2)
print(output)
```
![[Pasted image 20210916111645.png]]
- [Python Decorators in 15 Minutes](https://youtu.be/r7Dtus7N4pI) \| [Calm Code Decorators](https://calmcode.io/decorators/introduction.html) #💻️/⭐
- [Decorators 101](https://sureshdsk.dev/python-decorators-101)
- [Decorators 201](https://sureshdsk.dev/python-decorators-201)
- To get the correct doc strings and indicate that wrapping has occurred on the given function we can use the [[s.l.python.libs.functools]] module with the `wraps` function
- [Decorators with params](https://sureshdsk.dev/python-decorators-with-parameters)
- Useful with using Classes so the extra param passing boiler plate can be avoided
- [3 Essential Decorators in Python You Need To Know](https://betterprogramming.pub/3-essential-decorators-in-python-you-need-to-know-654650bd5c36)
- Can use logging decorators with [[s.l.python.libs.functools]] to log data changes:
```python
from functools import wraps
import datetime as dt
def log_step(func):
@wraps(func)
def wrapper(*args, **kwargs):
tic = dt.datetime.now()
result = func(*args, **kwargs)
time_taken = str(dt.datetime.now() - tic)
print(f"just ran step {func.__name__} shape={result.shape} took {time_taken}s")
return result
return wrapper
```
### Map Filter Reduce
```python
import math
def area(r):
"""Area of a circle with radius 'r'."""
return math.pi * (r2)
radii = [2, 5, 7.1, 0.3, 10]
# Method 1: Direct Method
areas = []
for r in radii:
a = area(r)
areas.append(a)
print(areas)
radii = [2, 5, 7.1, 0.3, 10]
# map(<function>, <list, tuple, or other iterable object>)
list(map(area, radii))
print(list(map(area, radii)))
# Celcius to Farenheit with map and lambda
temps = [("Berlin",29), ("Cairo", 36), ("Buenos Aires", 19), ("Los Angelas", 26), ("Tokyo", 27), ("New York", 28), ("London", 22), ("Beijing", 32)]
c_to_f = lambda data: (data[0], (9/5)*data[1] + 32)
print(list(map(c_to_f, temps)))
import statistics
data = [1.3, 2.7, 0.8, 4.1, 4.3, -0.1]
avg = statistics.mean(data)
print(avg)
print(list(filter(lambda x: x > avg, data)))
# Remove Missing Data
countries = ["", "Argentina", "", "Brazil", "Chile", "", "Columbia", "", "Ecuador", "", "", "Venezuela"]
print(list(filter(None, countries)))
# dont use reduce, just use an explicit for loop
```
### Functions Can Be Nested
```python
def func1(a, b):
def inner_func(x):
return x*x*x
return inner_func(a) + inner_func(b)
```
## machine learning
- [perceptilabs](https://www.perceptilabs.com/papers)
- Model Building Steps:
1. _Define_: What type of model will it be? A decision tree? Some other type of model? Some other parameters of the model type are specified too.
2. _Fit_: Capture patterns from provided data. This is the heart of modeling.
3. _Predict_: Just what it sounds like
4. _Evaluate_: Determine how accurate the model's predictions are.
## oop
- Reference:
- [Operator Overloading](https://www.programiz.com/python-programming/operator-overloading)
- [[Python Data Classes|s.l.python.oop.data-classes]]
### abstract base classes
> you should use abstract classes to specify which methods must be implemented by the child classes. The abc module "make sure" that you implement all those methods before being able to instantiate the objects.
```python
def animate(animal):
print(animal.get_sound())
for leg in animal.legs_number:
print("Moving leg", leg)
print("Moved!")
```
```python
from abc import ABC, abstractmethod
class Animal(ABC):
@abstractmethod
def get_sound(self) -> str:
pass
@property # you need to add the @propertydecorator both in the abstract class and in every sub-class.
@abstractmethod
def legs_number(self) -> int:
pass
# >>>a = Animal()
# Traceback (most recent call last):
# File "c:\abstract_class.py", line 9, in <module>
# a = Animal()
# TypeError: Can't instantiate abstract class Animal with abstract method get_sound
class Cat(Animal):
def get_sound(self) -> str:
return "Meow"
@property # you need to add the @propertydecorator both in the abstract class and in every sub-class.
def legs_number(self) -> int:
return 4
# >>> a = Cat()
# >>> a.get_sound()
# Meow
```
---
- Reference:
- [[r.2021.11.10.level-up-your-python-code-with-abstract-classes]]
### Benefits
> **The benefits of using it and when to use it.**
> The most important to take from this decorator is the purpose of using it.
> You should consider this decorator when writing any function that you believe is connected to the “subject” or the class that you are using but do not necessarily need that “subject” or class where you created the function.
> By doing this you can decouple the function itself, but still, keep it organized and with a clear meaning for which purpose can be used.
> From now on, any piece of code that you want to be reusable, but, at the same time you want to make it clear the “meaning” of when or how it could be used you should consider using this decorator.
### data classes
- [x] [If you're not using Python DATA CLASSES yet, you should 🚀](https://youtu.be/vRVVyl9uaZc)
- <https://youtu.be/CvQ7e6yUtnw>
```python
# Regular classes
class Person:
name: str
job: str
age: int
def __init__(self, name, job, age)
self.name = name
self.job = job
self.age = age
person1 = Person("Geralt", "Witcher", 30)
person2 = Person("Yennefer", "Sorceress", 25)
person3 = Person("Yennefer", "Sorceress", 25)
print(id(person2))
print(id(person3))
print(person1)
print(person3 == person2)
# Data Class
from dataclasses import dataclass, field
@dataclass(order=True,frozen=False)
class Person:
sort_index: int = field(init=False, repr=False)
name: str
job: str
age: int
strength: int = 100
def __post_init__(self):
object.__setattr__(self, 'sort_index', self.strength)
def __str__(self):
return f'{self.name}, {self.job} ({self.age})'
person1 = Person("Geralt", "Witcher", 30, 99)
person2 = Person("Yennefer", "Sorceress", 25)
person3 = Person("Yennefer", "Sorceress", 25)
print(person1)
print(id(person2))
print(id(person3))
print(person3 == person2)
print(person1 > person2)
```
```python
from dataclasses import dataclass, field
@dataclass
class Person:
name: str
address: str
active: bool = True
email_addresses: list[str] = field(default_factory=list)
id: str = field (init=False, default_factory=generate_id)
_search_string: str = field(init=False)
def __post_init__(self) -> None:
self._search_string = f" {self.name} {self.address}"
```
#### Pydantic for Data Class Type Hints
> [Do we still need dataclasses? // PYDANTIC tutorial](https://youtu.be/Vj-iU-8_xLs)
> Source: <https://pydantic-docs.helpmanual.io/>
```python
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel
class User(BaseModel):
id: int
name = 'John Doe'
signup_ts: Optional[datetime] = None
friends: List[int] = []
external_data = {
'id': '123',
'signup_ts': '2019-06-01 12:22',
'friends': [1, 2, '3'],
}
user = User(**external_data)
print(user.id)
#> 123
print(repr(user.signup_ts))
#> datetime.datetime(2019, 6, 1, 12, 22)
print(user.friends)
#> [1, 2, 3]
print(user.dict())
"""
{
'id': 123,
'signup_ts': datetime.datetime(2019, 6, 1, 12, 22),
'friends': [1, 2, 3],
'name': 'John Doe',
}
"""
```
### Deleting A Class Instance
```python
class User
def __init__(self):
print("I Live!")
def say_hi(self):
print("Hi!")
My_User = User()
#>>> "I Live!"
My_User.say_hi()
#>>> "Hi!"
MY_User
#>>> <__main__.User object at 0x000001BF37B38D30>
del My_User
My_User
#>>> Traceback (most recent call last):
#>>> File "<stdin>", line 1, in <module>
#>>> NameError: name 'My_User' is not defined
```
### Initializing A Class
```python
class User
def __init__(self):
print("I Live!")
My_User = User()
#>>> "I Live!"
```
### making-a-class
#### Making A Class
```python
class User
def say_hi(self):
print("Hi!")
My_User = User()
My_User.say_hi()
#>>> "Hi!"
```
#### Class Properties
```python
class Circle:
def __init__(self, radius):
self._radius = radius
@property
def radius(self):
"""The radius property"""
print("Get Radius")
return self._radius
@radius.setter
def radius(self, value):
print("Set Radius")
self._radius = value
@radius.deleter
def radius(self):
print("Delete Radius")
del self._radius
circle = Circle(42.0)
circle.radius
circle.radius = 100
circle.radius
del circle.radius
circle.radius
```
### Sample Implementation
> from: <https://stackoverflow.com/a/50255847/12339658>
```python
from typing import Protocol
class SupportsClose(Protocol):
def close(self) -> None:
# ...
class Resource:
# ...
def close(self) -> None:
self.file.close()
self.lock.release()
def close_all(things: Iterable[SupportsClose]) -> None:
for thing in things:
thing.close()
file = open('foo.txt')
resource = Resource()
close_all([file, resource]) # OK!
close_all([1]) # Error: 'int' has no 'close' method
```
---
- Related:
- [[s.l.python.libs.typing]]
- [[s.l.python.oop.abstract-base-classes]]
### static methods
#### Before
```python
class Calendar:
def __init__(self, date):
self.year = date.year
self.month = date.month
self.day = date.day
self.weekday = date.weekday()
def is_leap_year(self):
return self.year % 400 == 0 or
(self.year % 4 == 0 and self.year % 100 != 0)
def is_weekend(self):
return self.weekday > 4
```
![[Pasted_image_20211213151355.png]]
We want to decouple the `is_weekend` and `is_leap_year` from the calendar class instance because they relate to any date not just that class instance.
#### After
```python
class Calendar:
def __init__(self, date):
self.year = date.year
self.month = date.month
self.day = date.day
self.weekday = date.weekday()
@staticmethod
def is_leap_year(date):
return date.year % 400 == 0 or
(date.year % 4 == 0 and date.year % 100 != 0)
@staticmethod
def is_weekend(date):
return date.weekday() > 4
```
Ok, the changes are:
1. Add the decorator `@staticmethod`;
2. Rename the expected parameter to be a date object;
3. Adjust the is_weekend function to get weekday as function and not variable.
Usage:
![[Pasted_image_20211213151513.png]]
### super
[[+ 2022-03-16 Super Pythons Most Misunderstood Feature]]
## peps
> PEP stands for Python Enhancement Proposal
| PEP # | Subject |
| :--------------------------------------------------- | :--------------------- |
| [PEP 8](https://www.python.org/dev/peps/pep-0008/) | Code Style Conventions |
| [PEP 282](https://www.python.org/dev/peps/pep-0282/) | Logging System |
| [PEP 484](https://www.python.org/dev/peps/pep-0484/) | Type Hints |
| [PEP 572](https://www.python.org/dev/peps/pep-0572/) | Walrus Operator `:=` |
## pkg
- Workflow
```bash
python -m pip install jupyterlab
python -m jupyter lab
```
```
%load_ext autoreload
%autoreload 2
```
- Upload package to PyPi (Python Package Index)
```bash
python -m pip install twine wheel
```
- Documentation
- Create Documentation automatically. [Example config file](https://github.com/koaning/clumper/blob/main/mkdocs.yml) \| [Explanation](https://calmcode.io/docs/mkdocs.yml.html)
- also [[DevLog/Z/python libraries/sphinx]] and [[Restructured Text|restructured-text]]
### documentation
- Create Documentation automatically. [Example config file](https://github.com/koaning/clumper/blob/main/mkdocs.yml) \| [Explanation](https://calmcode.io/docs/mkdocs.yml.html)
- also [[s.l.python.libs.sphinx]] and [[s.m.restructured-text]]
### resources
- [Calm Code Setup](https://calmcode.io/setup/introduction.html)
### workflow
- Setup Git Repo and clone
- Make directory and `*.py` file for your project name
- In that directory add an `__init__.py` file and in that file add code to import your desired objects from your package
- like "Install module but also add to working session these objects"
- in the root set up a `setup.py` file
- This file lets the local files be installed by pip as a proper package
- Live iteration through [[s.l.python.tools.jupyter]] lab with magics:
```bash
python -m pip install jupyterlab
python -m jupyter lab
```
```
%load_ext autoreload
%autoreload 2
```
- Upload package to PyPi (Python Package Index)
```bash
python -m pip install twine wheel
```
## proj
- [[p.backlog.ftp-file-transfer-application]]
- [[p.backlog.make-a-discord-bot]]
- [Turn your Python Script into a 'Real' Program with Docker](https://python.plainenglish.io/turn-your-python-script-into-a-real-program-with-docker-c200e15d5265)
- [Project Repo](https://github.com/adamcyber1/mypythondocker)
## quirks
### dunder methods
#### Resources
- [What's the meaning of underscores in Python variable names](https://youtu.be/ALZmCy2u0jQ)
- [Special Attributes](https://docs.python.org/3/reference/datamodel.html#:~:text=formal%20parameter%20list.-,Special%20attributes%3A,-Attribute)
- [Dunder/Magic Methods in Python](https://www.section.io/engineering-education/dunder-methods-python/)
#### in code
##### In General Code
- `__annotations__` gets a returned dictionary of type annotations for the object
- `class.__bases__` The tuple of base classes of a class object.
- `__builtins__` <https://newbedev.com/python-what-s-the-difference-between-builtin-and-builtins>
- `__cached__` is the path to any compiled version of the code
- `instance.__class__` The class to which a class instance belongs.
- _R_ `__closure__` `None` or a tuple of cells that contain bindings for the function’s free variables.
- `__code__` The code object representing the compiled function body.
- `__defaults__` A tuple containing default args values for those args that have defaults, or `None`
- `object.__dict__` The namespace supporting arbitrary function attributes.
- `__doc__` The function’s documentation string, or None if unavailable; not inherited by subclasses.
- `__file__` The absolute path to the file that the code currently executing is within
- _R_ `__globals__` A reference to the dict that holds the function’s global vars
- the global namespace of the module in which the function was defined.
- `__import__` will return the top level module of a package, unless you pass a nonempty `fromlist` arg
- `__kwdefaults__` A dict containing defaults for keyword-only parameters.
- `__loader__` <https://stackoverflow.com/questions/22185888/pythons-loader-what-is-it>
- `__main__` The Entry point to a program, used in `if __name__ == "__main__":`
- `__module__` The name of the module the function was defined in, or None if unavailable.
- `__mro__` [https://docs.python.org/3/library/stdtypes.html#class.\_\_mro\_\_](https://docs.python.org/3/library/stdtypes.html#class.__mro__)
- `__name__` The function’s name. same as `__qualname__` but not showing the parents
- `__package__` <https://stackoverflow.com/a/21233334/12339658>
- `__path__` <https://stackoverflow.com/a/2700924/12339658>
- `__qualname__` A dotted name showing the “path” from the global scope to an item in that module
- `__spec__` `None` in `__main__` or interactive mode otherwise, information about the module
- `class.__subclasses__()` Returns a list of weak references to its immediate alive subclasses.
##### Variable Names
- Name a private identifier with a leading underscore ( `_username`)
- These items are not meant to be part of the public interface
- They may just be managing some part of internal state
- **CAN** be accessed via `obj._method_` but items flow to the bottom of the suggestions
- Name a strongly private identifier with two leading underscores (`__password`)
- when viewing objects `dir(obj)` these methods are not depicted as
- `__method`
- But rather as `_obj__method`
- This prevents collisions when in a subclass from `obj` wanting to re-use the `__method`name
- **CANNOT** be accessed via `obj.__method` an AttributeError will be thrown and cant be found
- Only way to find it is with the mangled name: `instance._obj__method` but not recommended
- Special identifiers in Python end with two leading underscores.
- _A.K.A. Dunder methods (double under-score)_ `__main__`
#### in files
##### \_\_init\_\_
- `__init__.py`
- Determines what happens when the directory it's in is imported as a package
##### \_\_main\_\_
- `__main__.py`
- When there is a file with this name in a module folder it makes it so that you dont have to explicitly point to a script to run the module.
- instead of: `python repo/script.py`
- it can just be `python repo`
#### In OOP Code
> items can have their operations, `__add__` == `( + )` have their operations of arithmatic assignments `__iadd__` == `( += )` and If one of those methods does not support the operation with the supplied arguments, it should return `NotImplemented` == `__radd__`.
- `__add__` == `( + )` [[s.l.python.quirks.dunder-methods.in-oop.__add__]]
- `__sub__` == `( - )`
- `__mul__` == `( * )`
- `__matmul__` == `( @ )`
- `__truediv__` == `( \ )`
- `__floordiv__` == `( \\ )`
- `__mod__` == `( % )`
- `__divmod__` == `divmod()`
- `__pow__` == `( ** )`
- `__lshift__` == `( << )`
- `__rshift__` == `( >> )`
- `__and__` == `( & )`
- `__xor__` == `( ^ )`
- `__or__` == `( | )`
- `__lt__` == `( < )`
- `__le__` == `( <= )`
- `__eq__` == `( == )`
- `__ne__` == `( != )`
- `__gt__` == `( > )`
- `__ge__` == `( >= )`
- `__neg__` == `( - )`
- `__pos__` == `( + )`
- `__abs__` == `abs()`
- `__invert__` == `( ~ )`
- `__complex__` == `complex()`
- `__int__` == `int()`
- `__float__` == `float()`
- `__index__` [https://docs.python.org/3/reference/datamodel.html#object.\_\_index\_\_](https://docs.python.org/3/reference/datamodel.html#object.__index__)
- `__round__` == `round()
- `__trunc__` == `math.trunc()`
- `__floor__` == `math.floor()`
- `__ceil__` == `math.ceil()`
- [[s.l.python.quirks.dunder-methods.in-oop.dictionary-dunders]]
- `__setitem__` When assigning values in a dictionary.q
- `__getitem__` Executed when we run the `dict['key']` type of operation
- `__delitem__` ran when deleting a dictionary value `del dict['key']`
- `__contains__` Executed when we run the `in` operator `if item in class_instance:`
- `__len__` Is called if we ran a `len()` operation on our class instance
- `__call__` if `instance = Class()` then `__call__` is ran when we do: `instance()` right after
- `__init__` init function is run as soon as a class instance is generated. It's the constructor.
- `__post_init__` used in [[s.l.python.oop.data-classes]] for after `__init__` as thats overwritten by data class
- `__getattr__` Called when the default attribute access fails with an AttributeError
- `__getattribute__` Called unconditionally to implement attribute accesses for instances of the class.
- If the class also defines `__getattr__()`, the latter will not be called unless
- `__getattribute__()` either calls it explicitly or raises an AttributeError.
- `__setattr__` Called when an attribute assignment is attempted.
- This is called instead of the normal mechanism (i.e. store the value in the instance dictionary).
- `__delattr__` Like `__setattr__()` but for attribute deletion instead of assignment.
- This should only be implemented if `del obj.name` is meaningful for the object.
- `__dir__` Called when `dir()` is called on the object. A sequence must be returned.
- dir() converts the returned sequence to a list and sorts it.
- `__format__` when the class is being formatted in an fstring for with `.format()`
- `__bool__` [https://docs.python.org/3/reference/datamodel.html#object.\_\_bool\_\_](https://docs.python.org/3/reference/datamodel.html#object.__bool__)
- `__del__` The destructor of the class and what runs when attempting to `del instance`
- `__bytes__` Computes a byte-string representation of an object. This should return a bytes object.
- `__format__` produces a “formatted” string representation of an object.
- `__hash__` [https://docs.python.org/3/reference/datamodel.html#object.\_\_hash\_\_](https://docs.python.org/3/reference/datamodel.html#object.__hash__)
- `__repr__` Returns a representation of an object instance as a string [[s.l.python.quirks.dunder-methods.in-oop.__repr__]]
- `__str__` when not defined it calls `__repr__` by default to represent the instance as a string [[s.l.python.quirks.dunder-methods.in-oop.__str__]]
- `__unicode__` New preferred method compared to `__str__` in python 2
- Python 3 uses unicode and therefore defaults to the `__str__` method
- `__enter__` Enter the runtime context related to this object using the `with` statement
- `__exit__` Exit the runtime context related to this object
- `__reversed__` not used with slice `[::-1]` but rather with `reversed()`
- `__slots__` [[s.l.python.quirks.dunder-methods.in-oop.__slots__]]
##### \_\_add\_\_
```python
class point:
x = None
y = None
def __init__(self, x , y):
self.x = x
self.y = y
def __str__(self):
s = f'({self.x},{self.y})'
return s
def __add__(self, p2):
x = self.x + p2.x
y = self.y + p2.y
return point(x,y)
def __iadd__(self, p2):
self.x += p2.x
self.y += p2.y
return self
p1 = point(5,4)
p2 = point(2,3)
p3 = p1 + p2 # this runs the __add__ method
# >>> p3 = (7,7)
p1 += p2
# >>> p1 = (7,7)
```
##### \_\_new\_\_
```python
class A:
def __new__(cls, *args, **kwargs):
print ('new', cls, args, kwargs)
return super().__new__(cls)
def __init__(self, *args, **kwargs):
print ('init', self, args, kwargs)
x = A()
```
`__new__` allows you to modify immutable types before the object instance is created. `__new__` occurs before `__init__` so things like this are possible:
```python
class Uppercase Tuple(tuple):
def __new__(cls, iterable):
upper_iterable = (s.upper() for s in iterable)
return super().__new__(cls, upper_iterable)
```
Whereas this was not possible because the class object already exists even if the instance doesnt yet exist. the OBJECT is what we're trying to beat to the punch
```python
class Uppercase Tuple(tuple):
def __init__(self, iterable):
print (f'init {iterable}')
for i, arg in enumerate (iterable):
self[i] = arg. upper()
```
##### \_\_repr\_\_
```python
# A simple Person class
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
def __repr__(self):
rep = 'Person(' + self.name + ',' + str(self.age) + ')'
return rep
# Let's make a Person object and print the results of repr()
person = Person("John", 20)
print(repr(person))
# >>> Person(John,20)
```
##### \_\_slots\_\_
- <https://medium.com/@stephenjayakar/a-quick-dive-into-pythons-slots-72cdc2d334e>
- <https://youtu.be/Fot3_9eDmOs>
###### 20% speed boost
just by using slots and making the attributes static
```python
class Person:
def __init__(self, name: str, address: str, email: str):
self.name = name
self.address = address
self.email = email
class PersonSlots:
__slots__ = "name", "address", "email"
def __init__(self, name: str, address: str, email: str):
self.name = name
self.address = address
self.email = email
```
###### Combined with dataclasses to reduce code duplication
```python
@dataclass(slots=False)
class Person:
name: str
address: str
email: str
@dataclass(slots=True)
class PersonSlots:
name: str
address: str
email: str
```
###### Issues
- Multiple inheritance is not possible because it wont know what slots to use so a class inheriting from 2 other classes will not work
- a good things because multiple inheritance can cause a TON of problems and Issues
- slots prevent adding new dynamic attributes to a class
- also a good thing because a class should be well thought out and planned versus just having new stuff added on the fly
##### \_\_str\_\_
```python
class MyClass:
x = 0
y = ""
def __init__(self, anyNumber, anyString):
self.x = anyNumber
self.y = anyString
myObject = MyClass(12345, "Hello")
print(myObject.__str__())
print(myObject.__repr__())
print(myObject)
# >>> <__main__.MyClass object at 0x7f75d994f1f0>
# >>> <__main__.MyClass object at 0x7f75d994f1f0>
# >>> <__main__.MyClass object at 0x7f75d994f1f0>
#==========================================================================#
class MyClass:
x = 0
y = ""
def __init__(self, anyNumber, anyString):
self.x = anyNumber
self.y = anyString
def __str__ (self):
return 'MyClass(x=' + str(self.x) + ' ,y=' + self.y + ')'
myObject = MyClass(12345, "Hello")
print(myObject.__str__())
print(myObject)
print(str(myObject))
print(myObject.__repr__())
# >>> MyClass(x=12345 ,y=Hello)
# >>> MyClass(x=12345 ,y=Hello)
# >>> MyClass(x=12345 ,y=Hello)
# >>> <__main__.MyClass object at 0x7f8f023ef1f0>
#==========================================================================#
class MyClass:
x = 0
y = ""
def __init__(self, anyNumber, anyString):
self.x = anyNumber
self.y = anyString
def __repr__ (self):
return 'MyClass(x=' + str(self.x) + ' ,y=' + self.y + ')'
myObject = MyClass(12345, "Hello")
print(myObject.__str__())
print(myObject)
print(str(myObject))
print(myObject.__repr__())
# >>> MyClass(x=12345 ,y=Hello)
# >>> MyClass(x=12345 ,y=Hello)
# >>> MyClass(x=12345 ,y=Hello)
# >>> MyClass(x=12345 ,y=Hello)
```
##### dictionary dunders
```python
class softwares:
names = []
versions = {}
def __init__(self, names):
print("running init method")
if names:
self.names = names.copy()
for name in names:
self.versions[name] = 1
else:
raise Exception("Please Enter the names")
def __setitem__(self, name, version):
print("running set item")
if name in self.versions:
self.versions[name] = version
else:
raise Exception("Software Name doesn't exist")
def __getitem__(self,name):
print("running get item")
if name in self.versions:
return self.versions[name]
else:
raise Exception("Software Name doesn't exist")
def __delitem__(self,name):
print("running del item")
if name in self.versions:
del self.versions[name]
self.names.remove(name)
else:
raise Exception("Software Name doesn't exist")
def __len__(self):
return len(self.names)
def __contains__(self,name):
if name in self.versions:
return True
else:
return False
#==========================================================================#
p = softwares(['S1','S2','S3'])
p['S1'] = 2 # assigning values to a dictionary is when the __setitem__ dunder is invoked
p['2'] = 2 # assigning values to a dictionary is when the __setitem__ dunder is invoked
print(p['S1']) # using the brackets to access a dict value runes the __getitem__ method
print(p['2']) # using the brackets to access a dict value runes the __getitem__ method
# print(p['1']) # throws exception
del p['S1'] # deleting a dictionary key/val pair runs the __delitem__ method
len(p) # runs the class's __len__ method
if 'S2' in p: # this usage of the `in` operator uses the __contains__ function
print('found!')
```
### whitespace and line terminations
- Python uses whitespace indentation to denote scope without braces.
- It is bad practice to:
- Use tabs instead of spaces 😭
- mix whitespace types (tabs with spaces)
- line termination in python files is denoted with the standard `\n`
- however early on in python to aid in transitions from [[s.l.clang]] languages there is an optional usage of `;` especially to end a line and have multiple statements on a single line like this:
- `import datetime; print(f'{datetime.datetime.utcnow():%Y-%m-%d}')`
---
- Related:
- [[r.2021.12.13.the-secret-world-of-newline-characters]]
## releases
### 3.10
#### Parenthesized Context Managers
```python
with (open('file1.txt', 'r') as fin, open('file2.txt', 'w') as fout):
fout.write(fin.read())
# In essence: cat file1.txt > file2.txt
# if file2 were in append mode 'a' then it would be like:
# cat file1.txt >> file2.txt
```
#### Structural Pattern Matching
##### From
```python
name = input()
match name:
case "Misha":
return "Hello Misha"
case "John":
return "Hello John"
case _:
return "Go away"
```
##### To
```python
match name:
case "Misha" | "John":
return f"Hello {name}"
case "Michelle":
return "Long time no see, Michelle"
case _:
return "Go away"
```
- Add Additional Conditions on matches with `if`
```python
def get_car_price(make, is_turbocharged):
match make:
case "Subaru" if is_turbocharged:
return 10000
case "Toyota" if not is_turbocharged:
return 7500
case _:
return 2300
```
- [Video on Pattern Matching](https://youtu.be/-79HGfWmH_w)
#### Union Operator for Type Hints
```python
def add(x: int | float, y: int | float):
return x + y
```
## resources
- [[2_page_python_cheatsheet.pdf]]
- [Comprehensive Cheatsheet](https://github.com/gto76/python-cheatsheet)
- [[s.l.python.peps]] - [PEP's](https://www.python.org/dev/peps/)
- [Dev Docs](https://devdocs.io/python~3.9/)
- [Google's Dependency Management service](https://deps.dev/) **COMING SOON**
### python and docker
- [scaffoldy](https://scaffoldy.io/) service to make the generation of [[s.containers.docker]] template files easier
## run
### global interpreter lock
- Reference:
- [[Python behind the scenes 13 the GIL and its effects on Python multithreading|r.python-behind-the-scenes-13-the-gil-and-its-effects-on-python-multithreading]]
## Security
- [10 common security gotchas in Python and how to avoid them](https://hackernoon.com/10-common-security-gotchas-in-python-and-how-to-avoid-them-e19fbe265e03)
- [SQL injection Cheat Sheet](https://www.netsparker.com/blog/web-security/sql-injection-cheat-sheet/)
## tips tricks
### default parameters
```python
# In Python 3 you can use a bare "*" asterisk
# in function parameter lists to force the
# caller to use keyword arguments for certain
# parameters:
def f(a, b, *, c='x', d='y', e='z'):
return 'Hello'
# To pass the value for c, d, and e you
# will need to explicitly pass it as
# "key=value" named arguments:
f(1, 2, 'p', 'q', 'v')
# TypeError: "f() takes 2 positional arguments but 5 were given"
f(1, 2, c='p', d='q',e='v')
'Hello'
```
### dictionary access with get
#### Stop Using Square Brackets To Get Dictionary Items — Use .get()
##### Before
```python
nik = {
'age':32,
'gender':'male',
'employed':True,
}
print(nik['location'])
# Returns:
# KeyError: 'location'
```
##### After
```python
nik = {
'age':32,
'gender':'male',
'employed':True,
}
print(nik.get('location'))
# Returns:
# None
```
### ellipses
#### For ... Else
```python
#case 1
for letter in 'foo':
print(letter)
else:
print("All letters have been printed")
#case 2
for letter in 'foo':
print(letter)
if letter == 'o':
break
else:
print("Letters have been printed")
# Output:
# case 1
# f
# o
# o
# All letters have been printed
# case 2
# f
# o
```
### enumerate
#### Replace len() with enumerate()
##### Before
```python
# Define a collection, such as list:
names = ['Nik', 'Jane', 'Katie', 'Jim', 'Luke']
# Using the range(len(collection)) method, you'd write:
for i in range(len(names)):
print(i, names[i])
# Using enumerate, you can define this by writing:
for idx, name in enumerate(names):
print(idx, name)
# Both ways of doing this return:
# 0 Nik
# 1 Jane
# 2 Katie
# 3 Jim
# 4 Luke
```
##### After
```python
# Define a collection, such as list:
names = ['Nik', 'Jane', 'Katie', 'Jim', 'Luke']
# Using enumerate, you can define this by writing:
for idx, name in enumerate(names, start=1):
print(idx, name)
# This returns:
# 1 Nik
# 2 Jane
# 3 Katie
# 4 Jim
# 5 Luke
```
### enums
```python
from enum import Enum
Season = Enum('Season', 'winter summer spring autumn')
print(Season.summer.name)
print(Season.summer.value)
#using class
class Season(Enum):
winter = 1
summer = 2
spring = 3
autumn = 4
print(Season.winter.name)
print(Season.winter.value)
# Output:
# summer
# 2
# winter
# 1
```
### itertools permutatio generator
```python
import itertools
def main() -> None:
items = ["a", "b", "c"]
perms = itertools.permutations(items)
for perm in perms:
print(perm)
```
### kwargs
```python
# Python 3.5+ allows passing multiple sets
# of keyword arguments ("kwargs") to a
# function within a single call, using
# the "" syntax:
def process_data(a, b, c, d):
print(a, b, c, d)
x = {'a': 1, 'b': 2}
y = {'c': 3, 'd': 4}
process_data(**x, **y)
# >>> 1 2 3 4
process_data(**x, c=23, d=42)
# >>> 1 2 23 42
```
### multi list iteration with zip
#### Simplify Iterating Over Multiple Lists With Zip()
##### Before
```python
names = ['Nik', 'Jane', 'Melissa', 'Doug']
ages = [32, 28, 37, 53]
gender = ['Male', 'Female', 'Female', 'Male']
# Old boring way:
for_looped = []
for i in range(len(names)):
for_looped.append((names[i], ages[i], gender[i]))
print(for_looped)
# Returns:
# [('Nik', 32, 'Male'), ('Jane', 28, 'Female'), ('Melissa', 37, 'Female'), ('Doug', 53, 'Male')]
```
##### After
```python
names = ['Nik', 'Jane', 'Melissa', 'Doug']
ages = [32, 28, 37, 53]
gender = ['Male', 'Female', 'Female', 'Male']
# Zipping through lists with zip()
zipped = zip(names, ages, gender)
zipped_list = list(zipped)
print(zipped_list)
# Returns:
# [('Nik', 32, 'Male'), ('Jane', 28, 'Female'), ('Melissa', 37, 'Female'), ('Doug', 53, 'Male')]
```
### multi variable assignment
```python
x = 0
y = 0
# can be:
x, y = 0, 0
```
### named tuples
```python
from collections import namedtuple
Coordinate = namedtuple("Coordinate", "longitude latitude")
location = Coordinate(90, 37.5)
print("location:", location)
# accessing attributes with dot notation
print(location.longitude, location.latitude)
# Output:
# location: Coordinate(longitude=90, latitude=37.5)
# (90, 37.5)
```
### underscore variable
#### The \_ in Python
- The `_` in python can hold the last value in an interactive shell session but can be used like the unnamed register in [[cli.cmd.vim]] and you can use it to avoid issues when unpacking tuples or just throwing something away:
```python
my_tuple = (1,2,3)
x, _, z = my_tuple # (1,2,3)
#> x = 1
#> _ = 2
#> z = 3
```
### variable swap
#### Swap Variable Values
```python
x, y = 1, 0
```
## tools
- jupyter
- vscode Extensions
- `ms-python.python` Python language support
- `ms-python.vscode-pylance` Python language support
- `brainfit.vscode-importmagic` Fix missing module imports
- `formulahendry.code-runner` Code runner to run the language for output and not just python code
- AI Code Completion
- [Comparisons here](https://medium.com/swlh/kite-vs-tabnine-which-ai-code-autocomplete-should-you-choose-eb6eba85c3a6)
- ❌️ `kiteco.kite` Kite
- kite
- ✅️ `tabnine.tabnine-vscode` TabNine
- `almenon.arepl` Coding REPL
- `njpwerner.autodocstring` Auto Generatre Doc Strings
- `kevinrose.vsc-python-indent` Correct Indentation
- `dongli.python-preview` Preview execution stack
- `littlefoxteam.vscode-python-test-adapter` Python Test explorer
- `njqdev.vscode-python-typehint` Data type hint
- `ms-pyright.pyright` for static type checking
- ❌️ `nikolapaunovic.tkinter-snippets` [[tkinter]] snippets
- `njqdev.vscode-python-typehint` Data type hint
- `jithurjacob.nbpreviewer` Jupyter Notebook Support and Viewing
- `ms-toolsai.jupyter` Jupyter Notebook Support and Viewing
### Build Tools
- poetry build tool
- [Pants](https://www.pantsbuild.org/docs)
### Linting
- black Code formatter
### Visualization Tools
- code2flow
- pydeps
### Project Oversight
- deep source
### jupyter
- **Themes**
- To get a custom theme (_gruvboxdark_) onto my jupyter notebook i used the tool [jupyter-themes](https://github.com/dunovank/jupyter-themes)
- Then the command i ran was `jt -t gruvboxd -f fira -fs 115`
- [SO answer about theming notebooks](https://stackoverflow.com/questions/46510192/change-the-theme-in-jupyter-notebook#46561480)
- **Line Magics**
- Running this in a cell `%lsmagic` will give you a list of magic commands you can run they look like the bash built-ins
- You can access them like `%ls` `%cp` `%rm` etc. if `Automagic` is turned on then you can run cells with straight up bash in them too. Command flags and everything.
- **Inline Charts**
- To display an inline chart you need line magic . The one in question is `%matplotlib inline` that will allow you to display charts inline in the notebook
- ==Jupyter Lab==
- **Installation**
- pip install jupyterlab
- **Extensions**
- pip install jupyterlab-git
- pip install jupyterlab-pullrequests
- pip install jupyterlab_github
- pip install jupyterlab_vim
- pip install jupyter_contrib_nbextensions
### kite
- <https://www.kite.com/>
## Workflow
- Make a new Git Repo
- vscode gitignore extension to add Python gitignore
- add gitattributes file
- Add config files for terminal/vscode/Extensions
- Setup Linting CI/CD with the `github super linter`
- Super linter file
```yaml
# https://aka.ms/yaml
# Initial code src: https://www.meziantou.net/running-github-super-linter-in-azure-pipelines.htm
# referenced docker container script: https://github.com/github/super-linter
# Official MS Documentation: https://docs.microsoft.com/en-us/azure/devops/pipelines/?view=azure-devops
# YAML Schema Info: https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
#============================================================================================================#
name: 'Super Linter -- $(Date:yyyyMMdd)$(Rev:.r)' #............................# Name of the Job Instance When Ran
trigger: #.....................................................................# What triggers the job to run?
- '*' #........................................................................# Everything triggers it
variables: #...................................................................# Variables for any parameter testing or changes
default_branch: 'prod' #.....................................................# What Branch is the default branch to run the job on
log_level: 'WARN' #..........................................................# What level and above of notices do you want to receive? WARN & ERROR
excluded_paths: '.*(\.github|\.vscode).*' #..................................# What Locations are ignored by this job?
jobs: #........................................................................# No stages so top level list of jobs to run
- job: 'super_linter' #......................................................# Name of the job
#pool: 'default' #.........................................................# The Agent Pool if we wanted to use our self hosted option
pool: #....................................................................# Which Agent Pool runs the job?
vmImage: 'ubuntu-latest' #...............................................# Microsoft Hosted Image to Run Job on
steps: #...................................................................# The list of steps that make up the job
- script: 'docker pull github/super-linter:latest' #.......................# Commands to Run in the default shell of the host machine See VmImage
displayName: 'Pull GitHub Super-Linter image' #..........................# Display name of this step when it executed in the pipeline
- script: >-
docker run \
-e IGNORE_GITIGNORED_FILES=true \
-e MARKDOWN_CONFIG_FILE=.markdownlint.jsonc \
-e JAVASCRIPT_ES_CONFIG_FILE=.eslintrc.json \
-e PYTHON_BLACK_CONFIG_FILE=.python-black \
-e PYTHON_FLAKE8_CONFIG_FILE=.flake8 \
-e PYTHON_ISORT_CONFIG_FILE=.isort.cfg \
-e DEFAULT_BRANCH=$(default_branch) \
-e LOG_LEVEL=$(log_level) \
-e FILTER_REGEX_EXCLUDE='$(excluded_paths)' \
-e RUN_LOCAL=true \
-v $(System.DefaultWorkingDirectory):/tmp/lint github/super-linter
# - script: >- #...............................................................# The Below is the multi-line inline script ran in the default shell formatted for legibility
# docker run \ #...........................................................# Running the docker image with the following commands `-e` is enviornmental variable `-v` is the volume to attach to (a file path)
# -e IGNORE_GITIGNORED_FILES=true \ #....................................# BOOL Variable asking if the job should lint .gitignored files
# -e MARKDOWN_CONFIG_FILE=.markdownlint.jsonc \ #........................# STRING Variable of Markdown config file
# -e JAVASCRIPT_ES_CONFIG_FILE=.eslintrc.json \ #........................# STRING Variable of JSON config file
# -e PYTHON_BLACK_CONFIG_FILE=.python-black \ #..........................# STRING Variable of Python Black config file
# -e PYTHON_FLAKE8_CONFIG_FILE=.flake8 \ #...............................# STRING Variable of Python flake8 config file
# -e PYTHON_ISORT_CONFIG_FILE=.isort.cfg \ #.............................# STRING Variable of Python isort config file
# -e DEFAULT_BRANCH=$(default_branch) \ #................................# STRING Variable of default branch to run job on when not targeted manually
# -e LOG_LEVEL=$(log_level) \ #..........................................# STRING Variable of desired log output level
# -e FILTER_REGEX_EXCLUDE=$(excluded_paths) \ #..........................# STRING Variable of regex path list to ignore for the job
# -e RUN_LOCAL=true \ #..................................................# BOOL Variable to Run the container locally (On VmImage)
# -v $(System.DefaultWorkingDirectory):/tmp/lint github/super-linter #...# Volume to attach to
displayName: 'Run GitHub Super-Linter' #.................................# Name of this step in the job
```
- [[black]] with a `pyproject.toml` addition
```toml
[tool.black]
line-length = 88
include = '\.pyi?
exclude = '''
/(
\.git
| \.hg
| \.mypy_cache
| \.tox
| \.venv
| _build
| buck-out
| build
| dist
| docs
| notes
)/
'''
```
- [[s.l.python.libs.flake8]] with a `.flake8` file
```
[flake8]
max-line-length = 88
max-complexity = 18
exclude =
.git,
__pycache__,
build,
dist
select = B,C,E,F,W,T4,B9
ignore =
E203, # E203: whitespace before ‘:’
E266, # E266: too many leading ‘#’ for block comment
E501, # E501: line too long (82 > 79 characters)
W503, # W503: line break before binary operator
F401, # F401: module imported but unused
F403 # F403: ‘from module import *’ used; unable to detect undefined names
```
- [[s.l.python.libs.isort]] with a `pyproject.toml` addition
```toml
[tool.isort]
line_length = 88
multi_line_output = 3
include_trailing_comma = true
skip_glob = []
known_third_party = []
```
- [[s.l.python.libs.venv.vulture]] with a `pyproject.toml` addition
```toml
[tool.vulture]
exclude = []
ignore_decorators = ["@app.route", "@require_*"]
ignore_names = []
make_whitelist = true
min_confidence = 80
paths = ["src/"]
sort_by_size = true
verbose = false
```
- setup repo branch policies and settings
- [[s.l.python.build.poetry]] setup
- `poetry new <PROJECT>` or `poetry init`
- Spin up virtual environment `poetry shell`
- `poetry add --dev pre-commit pytest pytest-cov`
- setup pre-commit hooks for formatting
- make `.pre-commit-config.yaml` file
- `poetry run pre-commit install`
- `poetry run pre-commit autoupdate`
- `poetry run pre-commit run` (to test that it works)
```yaml
repos:
- repo: https://github.com/asottile/seed-isort-config
rev: v2.2.0
hooks:
- id: seed-isort-config
- repo: https://github.com/pre-commit/mirrors-isort
rev: v5.8.0
hooks:
- id: isort
- repo: https://github.com/psf/black
rev: 21.6b0
hooks:
- id: black
language_version: python3
description: "Black: The uncompromising Python code formatter"
- repo: https://gitlab.com/pycqa/flake8
rev: 3.9.2
hooks:
- id: flake8
- repo: https://github.com/jendrikseipp/vulture
rev: 'v2.3'
hooks:
- id: vulture
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.0.1
hooks:
- id: trailing-whitespace
log_file: src/logs/hooks.log
- id: end-of-file-fixer
log_file: src/logs/hooks.log
- id: check-docstring-first
log_file: src/logs/hooks.log
- id: requirements-txt-fixer
log_file: src/logs/hooks.log
- id: check-added-large-files
log_file: src/logs/hooks.log
- id: check-json
log_file: src/logs/hooks.log
exclude: .vscode/settings.json
- id: check-yaml
log_file: src/logs/hooks.log
- id: check-toml
log_file: src/logs/hooks.log
- id: check-xml
log_file: src/logs/hooks.log
- id: debug-statements
log_file: src/logs/hooks.log
- id: name-tests-test
log_file: src/logs/hooks.log
- id: check-shebang-scripts-are-executable
log_file: src/logs/hooks.log
- id: check-merge-conflict
log_file: src/logs/hooks.log
- id: check-symlinks
log_file: src/logs/hooks.log
- id: destroyed-symlinks
log_file: src/logs/hooks.log
- id: debug-statements
log_file: src/logs/hooks.log
- id: detect-private-key
log_file: src/logs/hooks.log
- id: no-commit-to-branch
log_file: src/logs/hooks.log
args: [--branch, prod]
```
- Set up logging and this directory structure to start:
```
.
├───.github
│ ├───ISSUE_TEMPLATE
│ ├───linters
│ └───workflows
├───doc
└───src
├───logger
│ ├───__init__.py
│ └───logger.py
├───logs
├───mypackage
│ ├───__init__.py
│ └───mypackage.py
└───app.py
```
- Set up unit testing framework with [[s.l.python.libs.pytest]]
- `pytest-cov --cov <SRC DIR> --cov-report html`
- This tests the coverage of the entire source code directory and writes an intricate [[s.m.html]] report for viewing test results
- Set up auto documentation with [[s.l.python.libs.sphinx]] & [[s.m.restructured-text]]
- **More on Packaging**
- build your distribution: `poetry build`
- publish your distribution: `poetry publish`
- **Opening someone elses poetry project**
- install dependencies `poetry install`
## Libraries
- [Alive-Progress](https://github.com/rsalmei/alive-progress)
- [PyPi - Blessed](https://pypi.org/project/blessed/)
- [Creating PDF Invoices in Python with borb](https://stackabuse.com/creating-pdf-invoices-in-python-with-borb/)
- [Click](https://click.palletsprojects.com/en/8.0.x/)
- [email](https://realpython.com/python-send-email)
- [holoviz](https://holoviews.org/)
- [panel](https://panel.holoviz.org/)
- [hvplot](https://hvplot.holoviz.org/)
- [pydeps](https://github.com/thebjorn/pydeps)
- [pysimplegui](https://github.com/PySimpleGUI/PySimpleGUI)
- [ray](https://calmcode.io/ray/introduction.html)
- [Scrapy](https://github.com/scrapy/scrapy)
- [yarl](https://calmcode.io/shorts/yarl.py.html)