Python etc

@pythonetc Нравится 1
Это ваш канал? Подтвердите владение для дополнительных возможностей

Regular tips about Python and programming in general — @pushtaevhttps://ko-fi.com/pythonetc — © CC BY-SA 4.0
Гео и язык канала
Россия, Английский
Категория
Технологии


Гео канала
Россия
Язык канала
Английский
Категория
Технологии
Добавлен в индекс
28.12.2018 17:35
реклама
Попробуйте рекламу на Altt.me
Активная аудитория и отслеживаемая эффективность.
Реклама в каналах? Ну а может, лучше
Реклама на Altt.me – прозрачная эффективность и бонусы
NewsOne - новости 24/7
Рекламу увидят более 100 000 вовлеченных пользователей
4 165
подписчиков
~2.8k
охват 1 публикации
~1.7k
дневной охват
~4
постов / нед.
67.2%
ERR %
2.56
индекс цитирования
Репосты и упоминания канала
104 упоминаний канала
2 упоминаний публикаций
1 репостов
Мишка на Севере
PythonDigest
habr.com
ХабраХабр Pub.
Habr
Habr
TM FEED
PythonDigest
habr.com
Habr
Habr
TM FEED
PythonDigest
Мишка на Севере
PythonDigest
ХабраХабр Pub.
TM FEED
Мишка на Севере
PythonDigest
ХабраХабр Pub.
Habr
Habr
TM FEED
Мишка на Севере
PythonDigest
habr.com
ХабраХабр Pub.
Habr
TM FEED
Habr
ITGram
Information X
Habr
PythonDigest
Мишка на Севере
Sysadmin Tools
habr.com
ХабраХабр Pub.
Habr
TM FEED
TheFrontEnd🔥
Канал айтишника
Каналы, которые цитирует @pythonetc
MRG Champs
Python Books
TheFrontEnd🔥
TheFrontEnd🔥
ITGram
Последние публикации
Удалённые
С упоминаниями
Репосты
Python etc 19 Aug, 13:00
In Python, None is equal to None so it looks like you can check for None with ==:

ES_TAILS = ('s', 'x', 'z', 'ch', 'sh')


def make_plural(word, exceptions=None):
if exceptions == None: # ← ← ←
exceptions = {}

if word in exceptions:
return exceptions[word]
elif any(word.endswith(t) for t in ES_TAILS):
return word + 'es'
elif word.endswith('y'):
return word[0:-1] + 'ies'
else:
return word + 's'

exceptions = dict(
mouse='mice',
)

print(make_plural('python'))
print(make_plural('bash'))
print(make_plural('ruby'))
print(make_plural('mouse', exceptions=exceptions))

This is a wrong thing to do though. None is indeed is equal to None, but it’s not the only thing that is. Custom objects may be equal to None too:

>>> class A:
... def __eq__(self, other):
... return True
...
>>> A() == None
True
>>> A() is None
False

The only proper way to compare with None is to use is None.
👍 72
👎
Читать полностью
Python etc 15 Aug, 14:01
Different data structures are merged by different means in Python.

Lists use +:

>>> [1, 2] + [2, 3]
[1, 2, 2, 3]

Tuples and strings use + as well:

>>> (1, 2) + (2, 3)
(1, 2, 2, 3)
>>> "12" + "23"
'1223'

The same is true for deque:

>>> deque([1,2]) + deque([2,3])
deque([1, 2, 2, 3])
Sets, on the other hand, use |:

>>> {1, 2} | {2, 3}
{1, 2, 3}

Dicts really stand apart here. Merging dicts is not that simple: the order is important if both operands contain the same key:

>>> {**dict(a=1, b=2), **dict(b=3, c=4)}
{'a': 1, 'b': 3, 'c': 4}
>>> {**dict(b=3, c=4), **dict(a=1, b=2)}
{'b': 2, 'c': 4, 'a': 1}

Counters can be merged with +, the values are summed in the case:

>>> Counter(dict(a=1, b=2)) + Counter(dict(b=3, c=4))
Counter({'b': 5, 'c': 4, 'a': 1})
👍 100
👎 1
Читать полностью
Python etc 14 Aug, 14:00
Python has a very short list of built-in constants. One of them is Ellipsis which is also can be written as .... This constant has no special meaning for the interpreter but is used in places where such syntax looks appropriate.

numpy supports Ellipsis as a __getitem__ argument, e. g. x[...] returns all elements of x.

PEP 484 defines additional meaning: Callable[..., type] is a way to define a type of callables with no argument types specified.

Finally, you can use ... to indicate that function is not yet implemented. This is a completely valid Python code:

def x():
...
👍 102
👎
Читать полностью
Python etc 11 Aug, 13:08
Functions declared in a class body can’t see the class scope. It makes sense since the class scope only exists during class creation.

>>> class A:
... x = 2
... def f():
... print(x)
... f()
...
[...]
NameError: name 'x' is not defined
That is usually not a problem: methods are declared inside a class only to become methods and be called later:

>>> class A:
... x = 2
... def f(self):
... print(self.x)
...
>>>
>>>
>>> A().f()
2
Somewhat surprisingly, the same is true for comprehensions. They have their own scopes and can’t access the class scope as well. That really make sense for generator comprehensions: they evaluate expressions after the class creation is already finished.

>>> class A:
... x = 2
... y = [x for _ in range(5)]
...
[...]
NameError: name 'x' is not defined
Comprehensions, however, have no access to self. The only way to make it work is to add one more scope (yep, that’s ugly):

>>> class A:
... x = 2
... y = (lambda x=x: [x for _ in range(5)])()
...
>>> A.y
[2, 2, 2, 2, 2]
👍 71
👎 5
Читать полностью
Python etc 10 Aug, 14:24
If an instance of a class doesn’t have an attribute with the given name, it tires to access the class attribute with the same name.

>>> class A:
... x = 2
...
>>> A.x
2
>>> A().x
2

It’s fairly simple for an instance to have attribute that a class doesn’t or have the attribute with the different value:

>>> class A:
... x = 2
... def __init__(self):
... self.x = 3
... self.y = 4
...
>>> A().x
3
>>> A.x
2
>>> A().y
4
>>> A.y
AttributeError: type object 'A' has no attribute 'y'

If it’s not that simple, however, if you want an instance behave like it doesn’t have an attribute despite the class having it. To make it happen you have to create custom descriptor that doesn’t allow access from the instance:

class ClassOnlyDescriptor:
def __init__(self, value):
self._value = value
self._name = None # see __set_name__

def __get__(self, instance, owner):
if instance is not None:
raise AttributeError(
f'{instance} has no attribute {self._name}'
)

return self._value

def __set_name__(self, owner, name):
self._name = name


class_only = ClassOnlyDescriptor


class A:
x = class_only(2)


print(A.x) # 2
A().x # raises AttributeError

See also how the Django classonlymethod decorator works: https://github.com/django/django/blob/b709d701303b3877387020c1558a590713b09853/django/utils/decorators.py#L6
👍 48
👎 1
Читать полностью
Python etc 8 Aug, 12:40
You can customize index completions in Jupiter notebook by providing the _ipython_key_completions_ method. This way you can control what is displayed when you press tab after something like d["x:
Python etc 8 Aug, 12:40
Python etc 8 Aug, 12:40
Note that the method doesn't get the looked up string as an argument.
👍 57
👎 8
Python etc 5 Aug, 12:04
Today's post is written by @MrMrRobat
Python etc 5 Aug, 12:04
__slots__ can be used to speed up attributes access and reduce size of an object.

However, if you want to be able to assign new attributes, you need to use mutable container inside the __slots__ to store values in.

The problem is that using such container might be a way slower than using class with __dict__:

class A:
__slots__ = ('__items_type__', '__values__')

def __init__(self, items_type, **values):
object.__setattr__(self, '__values__', values)
object.__setattr__(self, '__items_type__', items_type)

def __getattr__(self, item):
try:
return self.__values__[item]
except KeyError:
raise AttributeError(f'{self} does not have attribute {item}')

def __setattr__(self, key, value):
if type(value) is not self.__items_type__:
raise TypeError(f'Value has wrong type {type(value)}, {self.__items_type__} expected')
self.__values__[key] = value

class B:
def __init__(self, items_type, **values):
object.__setattr__(self, '__dict__', values)
object.__setattr__(self, '__items_type__', items_type)

def __setattr__(self, key, value):
if type(value) is not self.__items_type__:
raise TypeError(f'Value has wrong type {type(value)}, {self.__items_type__} expected')
self.__dict__[key] = value

a = A(int, first=1)
b = B(int, first=1)

print(timeit('a.first; a.second=2', globals={'a': a}, number=10000000))
print(timeit('b.first; b.second=2', globals={'b': b}, number=10000000))

# 4.367306873999951
# 1.7298872390001634

To solve this problem without sacrificing faster access to attributes that stored in __slots__ you can mention __dict__ inside, so new attributes will be stored in there:

class C:
__slots__ = ('__items_type__', '__dict__')

def __init__(self, items_type, **values):
object.__setattr__(self, '__dict__', values)
object.__setattr__(self, '__items_type__', items_type)

def __setattr__(self, key, value):
if type(value) is not self.__items_type__:
raise TypeError(f'Value has wrong type {type(value)}, {self.__items_type__} expected')
self.__dict__[key] = value

c = C(int, first=1)

print(timeit('c.first; c.second=2', globals={'c': c}, number=10000000))
# 1.5695846090002306
Here's an example of boost attrs access speed up to 14 times, just by renaming one attribute to __dict__:
https://github.com/samuelcolvin/pydantic/issues/711
👍 28
👎 9
Читать полностью
Python etc 3 Aug, 11:18
Dictionaries that are used for storing object attributes are not the same that you create with dict though they look exactly the same:

>>> from sys import getsizeof
>>> class A:
... pass
...
>>> a = dict()
>>> b = A().__dict__
>>> type(a)

>>> type(b)

>>> a
{}
>>> b
{}
>>> getsizeof(a)
240
>>> getsizeof(b)
112

For reduction in memory used, dictionaries for __dict__ are implemented differently. They are sharing keys across all instances of A. Mind, however, that b is not actually smaller than a, it’s just how getsizeof works.

Read all the details here: https://www.python.org/dev/peps/pep-0412/
👍 74
👎 2
Читать полностью
Python etc 1 Aug, 12:00
The attributes of classes are stored in dictionaries, and that could be a problem since they don't preserve order in Python 3.5 and older:


$ cat test.py
class M:
def __new__(meta, cls, bases, ns):
print(ns)

class A(metaclass=M):
a = 1
b = 2
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'a': 1, 'b': 2, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'__qualname__': 'A', 'a': 1, '__module__': '__main__', 'b': 2}
$ python3.4 test.py
{'b': 2, 'a': 1, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'b': 2, '__qualname__': 'A', '__module__': '__main__', 'a': 1}
However, you can replace the attribute container by using the __prepare__ metaclass method:

$ cat test.py
from collections import OrderedDict


class M:
def __new__(meta, cls, bases, ns):
print(ns)

@classmethod
def __prepare__(metacls, cls, bases):
return OrderedDict()

class A(metaclass=M):
a = 1
b = 2
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
There is no need to do such thing in Python 3.6+, since dictionaries now preserve order:

$ cat test.py
class M:
def __new__(meta, cls, bases, ns):
print(ns)

class A(metaclass=M):
a = 1
b = 2
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
👍 31
👎 14
Читать полностью
Python etc 29 Jul, 16:37
You can slice dicts using dict comprehensions:

In [1]: d = dict(
...: a=1,
...: b=2,
...: c=3,
...: )

In [2]: {k: d.get(k) for k in ['a', 'c', 'e']}
Out[2]: {'a': 1, 'c': 3, 'e': None}

In [3]: {k: d.get(k) for k in ['a', 'c', 'e'] if k in d}
Out[3]: {'a': 1, 'c': 3}

In [4]: {k: d.get(k) for k in d.keys() if k in {'a', 'c', 'e'}}
Out[4]: {'a': 1, 'c': 3}
👍 60
👎 21
Python etc 27 Jul, 14:03
One of the techniques of metaprogramming is to use code generation. That means that you write some program that produces another program as an output. Even if your goal is to create Python code, the program that generates code can be written in any language.

Here is the Perl program that generates a line of Python source code, that is executed by Python interpreter afterward:

$ perl -e 'print "print(100)"' | python
100
It's worth noting that not every program that is syntactically correct may be executed, it's possible that expression is simply too long. That usually doesn't happen when you write source code manually, but it's entirely possible to have such problem while generating code:

$ python -c 'print("not " * 1000000 + "False")' | python
s_push: parser stack overflow
MemoryError
👍 31
👎 13
Читать полностью
Python etc 24 Jul, 14:30
If you want to pass some information down the call chain, you usually use the most straightforward way possible: you pass it as functions arguments.

However, in some cases, it may be highly inconvenient to modify all functions in the chain to propagate some new piece of data. Instead, you may want to set up some kind of context to be used by all functions down the chain. How can this context be technically done?

The simplest solution is a global variable. In Python, you also may use modules and classes as context holders since they are, strictly speaking, global variables too. You probably do it on a daily basis for things like loggers.

If your application is multi-threaded, a bare global variable won't work for you since they are not thread-safe. You may have more than one call chain running at the same time, and each of them needs its own context. The threading module gets you covered, it provides the threading.local() object that is thread-safe. Store there any data by simply accessing attributes: threading.local().symbol = '@'.

Still, both of that approaches are concurrency-unsafe meaning they won't work for coroutine call-chain where functions are not only called but can be awaited too. Once a coroutine does await, an event loop may run a completely different coroutine from a completely different chain. That won't work:

import asyncio
import sys

global_symbol = '.'

async def indication(timeout):
while True:
print(global_symbol, end='')
sys.stdout.flush()
await asyncio.sleep(timeout)

async def sleep(t, indication_t, symbol='.'):
loop = asyncio.get_event_loop()

global global_symbol
global_symbol = symbol
task = loop.create_task(
indication(indication_t)
)
await asyncio.sleep(t)
task.cancel()

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(
sleep(1, 0.1, '0'),
sleep(1, 0.1, 'a'),
sleep(1, 0.1, 'b'),
sleep(1, 0.1, 'c'),
))

You can fix that by having the loop set and restore the context every time it switches between coroutines. You can do it with the contextvars module since Python 3.7.

import asyncio
import sys
import contextvars

global_symbol = contextvars.ContextVar('symbol')

async def indication(timeout):
while True:
print(global_symbol.get(), end='')
sys.stdout.flush()
await asyncio.sleep(timeout)

async def sleep(t, indication_t, symbol='.'):
loop = asyncio.get_event_loop()

global_symbol.set(symbol)
task = loop.create_task(indication(indication_t))
await asyncio.sleep(t)
task.cancel()

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(
sleep(1, 0.1, '0'),
sleep(1, 0.1, 'a'),
sleep(1, 0.1, 'b'),
sleep(1, 0.1, 'c'),
))
👍 47
👎 2
Читать полностью
Python etc 22 Jul, 12:01
The problem with calling repr of other objects in your own __repr__ method is that you can't guarantee none of the other objects is not equal to self and the call isn't recursive:

In : p = Pair(1, 2)
In : p
Out: Pair(1, 2)
In : p.right = p
In : p
Out: [...]
RecursionError: maximum recursion depth exceeded while calling a Python object

To easily solve this problem you can use the reprlib.recursive_repr decorator:

@reprlib.recursive_repr()
def __repr__(self):
class_name = type(self).__name__
return f'{class_name}({self.left!r}, {self.right!r})'

Now it works:

In : p = Pair(1, 2)
In : p.right = p
In : p
Out: Pair(1, ...)
👍 68
👎 2
Читать полностью
Python etc 19 Jul, 14:00
Nested context managers normally don’t know that they are nested. You can make them know by spawning inner context managers by the outer one:

from contextlib import AbstractContextManager
import time


class TimeItContextManager(AbstractContextManager):
def __init__(self, name, parent=None):
super().__init__()

self._name = name
self._parent = parent
self._start = None
self._substracted = 0

def __enter__(self):
self._start = time.monotonic()
return self

def __exit__(self, exc_type, exc_value, traceback):
delta = time.monotonic() - self._start
if self._parent is not None:
self._parent.substract(delta)

print(self._name, 'total', delta)
print(self._name, 'outer', delta - self._substracted)

return False

def child(self, name):
return type(self)(name, parent=self)

def substract(self, n):
self._substracted += n


timeit = TimeItContextManager


def main():
with timeit('large') as large_t:
with large_t.child('medium') as medium_t:
with medium_t.child('small-1'):
time.sleep(1)
with medium_t.child('small-2'):
time.sleep(1)
time.sleep(1)
time.sleep(1)


main()
👍 34
👎 3
Читать полностью
Python etc 18 Jul, 12:00
If you want to measure time between two events you should use time.monotonic() instead of time.time(). time.monotonic() never goes backwards even if system clock is updated:

from contextlib import contextmanager
import time


@contextmanager
def timeit():
start = time.monotonic()
yield
print(time.monotonic() - start)

def main():
with timeit():
time.sleep(2)

main()
👍 135
👎 3
Python etc 17 Jul, 13:00
Feel like you versed in Data Science but can't organize your knowledges properly and don't have enough practice?
Online-education center SkillFactory launches Data Science course https://clc.to/I9sEKQ This course will allow you not only to increase your existing skills but learn somethig new. All our teachers are industry professionals, who worked for Yandex and NVIDIA. They will share with you industry insides and intricacies of the job, which are not written in any books.

Our course program includes a Python learning block (including Pandas for Data Analysis), mathematics, statistics probability theory with solving problems in NumPy and landing on Data Science, introduction to Machine Learning, course of Data Engineering, Neural Networks and AI and management for a data scientist: the skill of implementing data analysis systems, machine learning and neural networks.

Want to become a one of a kind specialist and take part in a variety of competitions on Kaggle (with solutions analyses and various models of Machine Learning and Neural Networks training) - our course starts 11.06, click to register now https://clc.to/I9sEKQ
Читать полностью
Python etc 17 Jul, 13:00
Ads time :).