Advanced Scientific Programming in Python

advanced_python.handout.pdf

Video of this Lecture

We are writing a git replacement in Python, and we need to store long sequence of commits representing changes. The commits are identified by their numbers, and sometimes we need to remove “changes” from the list.

Execute the two following implementations of rm_change. Which one is faster? Why is there a difference?

from time import time
 
def rm_change(change):
    if change in COMMITS:
        COMMITS.remove(change)
 
 
COMMITS = range(10**7)
t = time()
rm_change(10**7); rm_change(10**7-1); rm_change(10**7-2)
print(time()-t)
 
 
def rm_change(change):
    try:
        COMMITS.remove(change)
    except ValueError:
        pass
 
 
 
COMMITS = range(10**7)
t = time()
rm_change(10**7); rm_change(10**7-1); rm_change(10**7-2)
print(time()-t)

Write a decorator which wraps functions to log function arguments and the return value on each call. Provide support for both positional and named arguments (your wrapper function should take both *args and **kwargs and print them both):

>>> @logged
... def func(*args, **kwargs):
...     return len(args) + len(kwargs) 
>>> func()
you called func()
it returned 0
0
>>> func(4, 4, 4)
you called func(4, 4, 4)
it returned 3
3
>>> func(x=1, y=2)
you called func(x=2, y=2)
it returned 2
2

Note: getting the output details perfectly is fun, but not essential. If you have the basic wrapping working, consider jumping to the next exercise.

class logged(object):
    def __init__(self, func):
        self.func = func
 
    def __call__(self, *args, **kwargs):
        print('you called {.__name__}({}{}{})'.format(
             func,
             str(list(args))[1:-1], # cast to list is because tuple
                                    # of length one has an extra comma
             ', ' if kwargs else '',
             ', '.join('{}={}'.format(*pair) for pair in kwargs.items()),
             ))
        val = func(*args, **kwargs)
 
        print('it returned', val)
        return val

def logged(func):
    """Print out the arguments before function call and
    after the call print out the returned value
    """
 
    def wrapper(*args, **kwargs):
        print('you called {.__name__}({}{}{})'.format(
             func,
             str(list(args))[1:-1], # cast to list is because tuple
                                    # of length one has an extra comma
             ', ' if kwargs else '',
             ', '.join('{}={}'.format(*pair) for pair in kwargs.items()),
             ))
        val = func(*args, **kwargs)
        print('it returned', val)
        return val
    return wrapper

Write a context manager which temporarily changes to the current working directory of the program to the specified path, and returns to the original directory afterwards.

(In order words, write a context manager which does what was open-coded on slide 32 in the lecture…)

>>> import os
>>> print(os.getcwd())
/home/zbyszek
>>> with Chdir('/tmp'):
...   print(os.getcwd())
/tmp
>>> print(os.getcwd())
/home/zbyszek

@contextlib.contextmanager
def Chdir(dir):
  old = os.getcwd()
  try:
     os.chdir(dir)
     yield
  finally:
     os.chdir(old)

Write a context manager similar to assertRaises, which checks if the execution took at most the specified amount of time, and prints an error if too much time was taken. (This is not very useful for unit testing, we would expect and exception here, but should work nicely for doctests and casual testing).

>>> with time_limit(10):
...       short_computation()
...
42
>>> with time_limit(10):
...       loooong_computation()
...
⚡ function took 13s to execute — too long

import time
import functools
def time_limit(limit):
    def decorator(func):
        def wraper(*args, **kwargs):
            t = time.time()
            ans = func(*args, **kwargs)
            actual = t - time.time()
            if actual > limit:
                 print('⚡ function took %fs to execute — too long'%actual)
                 return None
            return ans
        return functools.update_wrapper(wraper, func)
    return decorator

Memoization is the operation of caching computation results. When a specific combination of arguments is used for the first time, the original function is executed normally, but the result is stored. In subsequent invocations with the same arguments, the answer is retrieved from the cache and the function is not called. This makes sense for functions which take long to execute, but have arguments and results which are compact enough to store.

Write a decorator to memoize functions with an arbitrary set of arguments. Memoization is only possible if the arguments are hashable. If the wrapper is called with arguments which are not hashable, then the wrapped function should just be called without caching.

Note: To use args and kwargs as dictionary keys, they must be hashable, which basically means that they must be immutable. Variable args is already a tuple, which is fine, but kwargs have to be converted. One way is invoke tuple(sorted(kwargs.items())).

>>> @memoize
... def f(*args, **kwargs):
...     ans = len(args) + len(kwargs)
...     print(args, kwargs, '->', ans)
...     return ans
>>> f(3)
(3,) {} -> 1
1
>>> f(3)
1
>>> f(*[3])
1
>>> f(a=1, b=2)
() {'a': 1, 'b': 2} -> 2
2
>>> f(b=2, a=1)
2
>>> f([1,2,3])
([1, 2, 3],) {} -> 1
1
>>> f([1,2,3])
([1, 2, 3],) {} -> 1
1

import functools
 
def memoize(func):
    """
    >>> @memoize
    ... def f(*args, **kwargs):
    ...     ans = len(args) + len(kwargs)
    ...     print(args, kwargs, '->', ans)
    ...     return ans
    >>> f(3)
    (3,) {} -> 1
    1
    >>> f(3)
    1
    >>> f(*[3])
    1
    >>> f(a=1, b=2)
    () {'a': 1, 'b': 2} -> 2
    2
    >>> f(b=2, a=1)
    2
    >>> f([1,2,3])
    ([1, 2, 3],) {} -> 1
    1
    >>> f([1,2,3])
    ([1, 2, 3],) {} -> 1
    1
    """
    func.cache = {}
    def wrapper(*args, **kwargs):
        key = (args, tuple(sorted(kwargs.items())))
        try:
            ans = func.cache[key]
        except TypeError:
            # key is unhashable
            return func(*args, **kwargs)
        except KeyError:
            # value is not present in cache
            ans = func.cache[key] = func(*args, **kwargs)
        return ans
    return functools.update_wrapper(wrapper, func)