-
ROS index logo ROS Index
  1. Home
  2. Repos
  3. tblib
 

tblib repository

Repository Summary

Checkout URI https://github.com/ionelmc/python-tblib.git
VCS Type git
VCS Version master
Last Updated 2024-05-02
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Packages

Name Version
tblib 1.2.0

README

Overview

+———————————–+———————————–+ | docs | Documentation Status | +———————————–+———————————–+ | tests | | ![GitHub Actions Build Status] | | | (https://github.com/ionelmc/pytho | | | n-tblib/actions/workflows/github- | | | actions.yml/badge.svg) | | | | Coverage Status | +———————————–+———————————–+ | package | | ![PyPI Package latest release] | | | (https://img.shields.io/pypi/v/tb | | | lib.svg) | | | PyPI Wheel | | | Supported versions | | | Supported implementations | | | | Commits since latest release | | | | +———————————–+———————————–+

Serialization library for Exceptions and Tracebacks.

  • Free software: BSD license

It allows you to:

  • Pickle tracebacks and raise exceptions with pickled tracebacks in different processes. This allows better error handling when running code over multiple processes (imagine multiprocessing, billiard, futures, celery etc).
  • Create traceback objects from strings (the from_string method). No pickling is used.
  • Serialize tracebacks to/from plain dicts (the from_dict and to_dict methods). No pickling is used.
  • Raise the tracebacks created from the aforementioned sources.
  • Pickle an Exception together with its traceback and exception chain (raise ... from ...) (Python 3 only)

Again, note that using the pickle support is completely optional. You are solely responsible for security problems should you decide to use the pickle support.

Installation

pip install tblib

Documentation

::: {.contents local=””} :::

Pickling tracebacks

Note: The traceback objects that come out are stripped of some attributes (like variables). But you'll be able to raise exceptions with those tracebacks or print them - that should cover 99% of the usecases.

>>> from tblib import pickling_support
>>> pickling_support.install()
>>> import pickle, sys
>>> def inner_0():
...     raise Exception('fail')
...
>>> def inner_1():
...     inner_0()
...
>>> def inner_2():
...     inner_1()
...
>>> try:
...     inner_2()
... except:
...     s1 = pickle.dumps(sys.exc_info())
...
>>> len(s1) > 1
True
>>> try:
...     inner_2()
... except:
...     s2 = pickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s2) > 1
True

>>> try:
...     import cPickle
... except ImportError:
...     import pickle as cPickle
>>> try:
...     inner_2()
... except:
...     s3 = cPickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s3) > 1
True

Unpickling tracebacks

>>> pickle.loads(s1)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s2)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s3)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

Raising

>>> from six import reraise
>>> reraise(*pickle.loads(s1))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s2))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s3))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

Pickling Exceptions together with their traceback and chain (Python 3 only)

>>> try:  # doctest: +SKIP
...     try:
...         1 / 0
...     except Exception as e:
...         raise Exception("foo") from e
... except Exception as e:
...     s = pickle.dumps(e)
>>> raise pickle.loads(s)  # doctest: +SKIP
Traceback (most recent call last):
  File "<doctest README.rst[16]>", line 3, in <module>
    1 / 0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<doctest README.rst[17]>", line 1, in <module>
    raise pickle.loads(s)
  File "<doctest README.rst[16]>", line 5, in <module>
    raise Exception("foo") from e
Exception: foo

BaseException subclasses defined after calling pickling_support.install() will not retain their traceback and exception chain pickling. To cover custom Exceptions, there are three options:

  1. Use @pickling_support.install as a decorator for each custom Exception

    ``` {.sourceCode .python}

    from tblib import pickling_support

    Declare all imports of your package’s dependencies

    import numpy # doctest: +SKIP

    pickling_support.install() # install for all modules imported so far

    @pickling_support.install … class CustomError(Exception): … pass

    ```

    Eventual subclasses of CustomError will need to be decorated again.

  2. Invoke pickling_support.install() after all modules have been imported and all Exception subclasses have been declared

    ``` {.sourceCode .python}

    Declare all imports of your package’s dependencies

    import numpy # doctest: +SKIP from tblib import pickling_support

    Declare your own custom Exceptions

    class CustomError(Exception): … pass

    Finally, install tblib

    pickling_support.install()

    ```

  3. Selectively install tblib for Exception instances just before they are pickled

    ``` {.sourceCode .python} pickling_support.install(, [Exception instance], ...)

    ```

    The above will install tblib pickling for all listed exceptions as well as any other exceptions in their exception chains.

    For example, one could write a wrapper to be used with ProcessPoolExecutor, Dask.distributed, or similar libraries:

>>> from tblib import pickling_support
>>> def wrapper(func, *args, **kwargs):
...     try:
...         return func(*args, **kwargs)
...     except Exception as e:
...         pickling_support.install(e)
...         raise

What if we have a local stack, does it show correctly ?

Yes it does:

>>> exc_info = pickle.loads(s3)
>>> def local_0():
...     reraise(*exc_info)
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> local_2()
Traceback (most recent call last):
  File "...doctest.py", line ..., in __run
    compileflags, 1) in test.globs
  File "<doctest README.rst[24]>", line 1, in <module>
    local_2()
  File "<doctest README.rst[23]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[22]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[21]>", line 2, in local_0
    reraise(*exc_info)
  File "<doctest README.rst[11]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

It also supports more contrived scenarios

Like tracebacks with syntax errors:

>>> from tblib import Traceback
>>> from examples import bad_syntax
>>> try:
...     bad_syntax()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[58]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[57]>", line 2, in <module>
    bad_syntax()
  File "...tests...examples.py", line 18, in bad_syntax
    import badsyntax
  File "...tests...badsyntax.py", line 5
    is very bad
     ^
SyntaxError: invalid syntax

Or other import failures:

>>> from examples import bad_module
>>> try:
...     bad_module()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[61]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[60]>", line 2, in <module>
    bad_module()
  File "...tests...examples.py", line 23, in bad_module
    import badmodule
  File "...tests...badmodule.py", line 3, in <module>
    raise Exception("boom!")
Exception: boom!

Or a traceback that's caused by exceeding the recursion limit (here we're forcing the type and value to have consistency across platforms):

>>> def f(): f()
>>> try:
...    f()
... except RuntimeError:
...    et, ev, tb = sys.exc_info()
...    tb = Traceback(tb)
...
>>> reraise(RuntimeError, RuntimeError("maximum recursion depth exceeded"), tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  ...
RuntimeError: maximum recursion depth exceeded

Reference

tblib.Traceback

It is used by the pickling_support. You can use it too if you want more flexibility:

>>> from tblib import Traceback
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
tblib.Traceback.to_dict

You can use the to_dict method and the from_dict classmethod to convert a Traceback into and from a dictionary serializable by the stdlib json.JSONDecoder:

>>> import json
>>> from pprint import pprint
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...     tb_dict = tb.to_dict()
...     pprint(tb_dict)
{'tb_frame': {'f_code': {'co_filename': '<doctest README.rst[...]>',
                         'co_name': '<module>'},
              'f_globals': {'__name__': '__main__'},
              'f_lineno': 5},
 'tb_lineno': 2,
 'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                     'co_name': 'inner_2'},
                          'f_globals': {'__name__': '__main__'},
                          'f_lineno': 2},
             'tb_lineno': 2,
             'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                 'co_name': 'inner_1'},
                                      'f_globals': {'__name__': '__main__'},
                                      'f_lineno': 2},
                         'tb_lineno': 2,
                         'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                             'co_name': 'inner_0'},
                                                  'f_globals': {'__name__': '__main__'},
                                                  'f_lineno': 2},
                                     'tb_lineno': 2,
                                     'tb_next': None}}}}
tblib.Traceback.from_dict

Building on the previous example:

>>> tb_json = json.dumps(tb_dict)
>>> tb = Traceback.from_dict(json.loads(tb_json))
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
tblib.Traceback.from_string
>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: fail

If you use the strict=False option then parsing is a bit more lax:

>>> tb = Traceback.from_string("""
... File "bogus.py", line 123, in bogus
... Traceback (most recent call last):
...  File "tests/examples.py", line 2, in func_a
...   func_b()
...    File "tests/examples.py", line 6, in func_b
...     func_c()
...    File "tests/examples.py", line 10, in func_c
...   func_d()
...  File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """, strict=False)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "bogus.py", line 123, in bogus
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: fail

tblib.decorators.return_error

>>> from tblib.decorators import return_error
>>> inner_2r = return_error(inner_2)
>>> e = inner_2r()
>>> e
<tblib.decorators.Error object at ...>
>>> e.reraise()
Traceback (most recent call last):
  ...
  File "<doctest README.rst[26]>", line 1, in <module>
    e.reraise()
  File "...tblib...decorators.py", line 19, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 25, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

How's this useful? Imagine you're using multiprocessing like this:

# Note that Python 3.4 and later will show the remote traceback (but as a string sadly) so we skip testing this.
>>> import traceback
>>> from multiprocessing import Pool
>>> from examples import func_a
>>> pool = Pool()  # doctest: +SKIP
>>> try:  # doctest: +SKIP
...     for i in pool.map(func_a, range(5)):
...         print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    for i in pool.map(func_a, range(5)):
  File "...multiprocessing...pool.py", line ..., in map
    ...
  File "...multiprocessing...pool.py", line ..., in get
    ...
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()  # doctest: +SKIP

Not very useful is it? Let's sort this out:

>>> from tblib.decorators import apply_with_return_error, Error
>>> from itertools import repeat
>>> pool = Pool()
>>> try:
...     for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...         if isinstance(i, Error):
...             i.reraise()
...         else:
...             print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 4, in <module>
    i.reraise()
  File "...tblib...decorators.py", line ..., in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line ..., in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line ..., in apply_with_return_error
    return args[0](*args[1:])
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()

Much better !

What if we have a local call stack ?
>>> def local_0():
...     pool = Pool()
...     try:
...         for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...             if isinstance(i, Error):
...                 i.reraise()
...             else:
...                 print(i)
...     finally:
...         pool.close()
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> try:
...     local_2()
... except:
...     print(traceback.format_exc())
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    local_2()
  File "<doctest README.rst[...]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[...]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[...]>", line 6, in local_0
    i.reraise()
  File "...tblib...decorators.py", line 20, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 27, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line 47, in apply_with_return_error
    return args[0](*args[1:])
  File "...tests...examples.py", line 2, in func_a
    func_b()
  File "...tests...examples.py", line 6, in func_b
    func_c()
  File "...tests...examples.py", line 10, in func_c
    func_d()
  File "...tests...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
Other weird stuff

Clearing traceback works (Python 3.4 and up):

>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> import traceback, sys
>>> if sys.version_info > (3, 4):
...     traceback.clear_frames(tb)

Credits

CONTRIBUTING

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

Bug reports

When reporting a bug please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • Detailed steps to reproduce the bug.

Documentation improvements

tblib could always use more documentation, whether as part of the official tblib docs, in docstrings, or even on the web in blog posts, articles, and such.

Feature requests and feedback

The best way to send feedback is to file an issue at https://github.com/ionelmc/python-tblib/issues.

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that code contributions are welcome :)

Development

To set up [python-tblib]{.title-ref} for local development:

  1. Fork python-tblib (look for the "Fork" button).

  2. Clone your fork locally:

    git clone git@github.com:YOURGITHUBNAME/python-tblib.git

  3. Create a branch for local development:

    git checkout -b name-of-your-bugfix-or-feature

    Now you can make your changes locally.

  4. When you're done making changes run all the checks and docs builder with one command:

    tox

  5. Commit your changes and push your branch to GitHub:

    git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature
  6. Submit a pull request through the GitHub website.

Pull Request Guidelines

If you need some code review or feedback while you're developing the code just make the pull request.

For merging, you should:

  1. Include passing tests (run tox).
  2. Update documentation when there's new API, functionality etc.
  3. Add a note to CHANGELOG.rst about the changes.
  4. Add yourself to AUTHORS.rst.

Tips

To run a subset of tests:

tox -e envname -- pytest -k test_myfeature

To run all the test environments in parallel:

tox -p auto

Repository Summary

Checkout URI https://github.com/ionelmc/python-tblib.git
VCS Type git
VCS Version master
Last Updated 2024-05-02
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Packages

Name Version
tblib 1.2.0

README

Overview

+———————————–+———————————–+ | docs | Documentation Status | +———————————–+———————————–+ | tests | | ![GitHub Actions Build Status] | | | (https://github.com/ionelmc/pytho | | | n-tblib/actions/workflows/github- | | | actions.yml/badge.svg) | | | | Coverage Status | +———————————–+———————————–+ | package | | ![PyPI Package latest release] | | | (https://img.shields.io/pypi/v/tb | | | lib.svg) | | | PyPI Wheel | | | Supported versions | | | Supported implementations | | | | Commits since latest release | | | | +———————————–+———————————–+

Serialization library for Exceptions and Tracebacks.

  • Free software: BSD license

It allows you to:

  • Pickle tracebacks and raise exceptions with pickled tracebacks in different processes. This allows better error handling when running code over multiple processes (imagine multiprocessing, billiard, futures, celery etc).
  • Create traceback objects from strings (the from_string method). No pickling is used.
  • Serialize tracebacks to/from plain dicts (the from_dict and to_dict methods). No pickling is used.
  • Raise the tracebacks created from the aforementioned sources.
  • Pickle an Exception together with its traceback and exception chain (raise ... from ...) (Python 3 only)

Again, note that using the pickle support is completely optional. You are solely responsible for security problems should you decide to use the pickle support.

Installation

pip install tblib

Documentation

::: {.contents local=””} :::

Pickling tracebacks

Note: The traceback objects that come out are stripped of some attributes (like variables). But you'll be able to raise exceptions with those tracebacks or print them - that should cover 99% of the usecases.

>>> from tblib import pickling_support
>>> pickling_support.install()
>>> import pickle, sys
>>> def inner_0():
...     raise Exception('fail')
...
>>> def inner_1():
...     inner_0()
...
>>> def inner_2():
...     inner_1()
...
>>> try:
...     inner_2()
... except:
...     s1 = pickle.dumps(sys.exc_info())
...
>>> len(s1) > 1
True
>>> try:
...     inner_2()
... except:
...     s2 = pickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s2) > 1
True

>>> try:
...     import cPickle
... except ImportError:
...     import pickle as cPickle
>>> try:
...     inner_2()
... except:
...     s3 = cPickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s3) > 1
True

Unpickling tracebacks

>>> pickle.loads(s1)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s2)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s3)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

Raising

>>> from six import reraise
>>> reraise(*pickle.loads(s1))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s2))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s3))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

Pickling Exceptions together with their traceback and chain (Python 3 only)

>>> try:  # doctest: +SKIP
...     try:
...         1 / 0
...     except Exception as e:
...         raise Exception("foo") from e
... except Exception as e:
...     s = pickle.dumps(e)
>>> raise pickle.loads(s)  # doctest: +SKIP
Traceback (most recent call last):
  File "<doctest README.rst[16]>", line 3, in <module>
    1 / 0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<doctest README.rst[17]>", line 1, in <module>
    raise pickle.loads(s)
  File "<doctest README.rst[16]>", line 5, in <module>
    raise Exception("foo") from e
Exception: foo

BaseException subclasses defined after calling pickling_support.install() will not retain their traceback and exception chain pickling. To cover custom Exceptions, there are three options:

  1. Use @pickling_support.install as a decorator for each custom Exception

    ``` {.sourceCode .python}

    from tblib import pickling_support

    Declare all imports of your package’s dependencies

    import numpy # doctest: +SKIP

    pickling_support.install() # install for all modules imported so far

    @pickling_support.install … class CustomError(Exception): … pass

    ```

    Eventual subclasses of CustomError will need to be decorated again.

  2. Invoke pickling_support.install() after all modules have been imported and all Exception subclasses have been declared

    ``` {.sourceCode .python}

    Declare all imports of your package’s dependencies

    import numpy # doctest: +SKIP from tblib import pickling_support

    Declare your own custom Exceptions

    class CustomError(Exception): … pass

    Finally, install tblib

    pickling_support.install()

    ```

  3. Selectively install tblib for Exception instances just before they are pickled

    ``` {.sourceCode .python} pickling_support.install(, [Exception instance], ...)

    ```

    The above will install tblib pickling for all listed exceptions as well as any other exceptions in their exception chains.

    For example, one could write a wrapper to be used with ProcessPoolExecutor, Dask.distributed, or similar libraries:

>>> from tblib import pickling_support
>>> def wrapper(func, *args, **kwargs):
...     try:
...         return func(*args, **kwargs)
...     except Exception as e:
...         pickling_support.install(e)
...         raise

What if we have a local stack, does it show correctly ?

Yes it does:

>>> exc_info = pickle.loads(s3)
>>> def local_0():
...     reraise(*exc_info)
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> local_2()
Traceback (most recent call last):
  File "...doctest.py", line ..., in __run
    compileflags, 1) in test.globs
  File "<doctest README.rst[24]>", line 1, in <module>
    local_2()
  File "<doctest README.rst[23]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[22]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[21]>", line 2, in local_0
    reraise(*exc_info)
  File "<doctest README.rst[11]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

It also supports more contrived scenarios

Like tracebacks with syntax errors:

>>> from tblib import Traceback
>>> from examples import bad_syntax
>>> try:
...     bad_syntax()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[58]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[57]>", line 2, in <module>
    bad_syntax()
  File "...tests...examples.py", line 18, in bad_syntax
    import badsyntax
  File "...tests...badsyntax.py", line 5
    is very bad
     ^
SyntaxError: invalid syntax

Or other import failures:

>>> from examples import bad_module
>>> try:
...     bad_module()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[61]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[60]>", line 2, in <module>
    bad_module()
  File "...tests...examples.py", line 23, in bad_module
    import badmodule
  File "...tests...badmodule.py", line 3, in <module>
    raise Exception("boom!")
Exception: boom!

Or a traceback that's caused by exceeding the recursion limit (here we're forcing the type and value to have consistency across platforms):

>>> def f(): f()
>>> try:
...    f()
... except RuntimeError:
...    et, ev, tb = sys.exc_info()
...    tb = Traceback(tb)
...
>>> reraise(RuntimeError, RuntimeError("maximum recursion depth exceeded"), tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  ...
RuntimeError: maximum recursion depth exceeded

Reference

tblib.Traceback

It is used by the pickling_support. You can use it too if you want more flexibility:

>>> from tblib import Traceback
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
tblib.Traceback.to_dict

You can use the to_dict method and the from_dict classmethod to convert a Traceback into and from a dictionary serializable by the stdlib json.JSONDecoder:

>>> import json
>>> from pprint import pprint
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...     tb_dict = tb.to_dict()
...     pprint(tb_dict)
{'tb_frame': {'f_code': {'co_filename': '<doctest README.rst[...]>',
                         'co_name': '<module>'},
              'f_globals': {'__name__': '__main__'},
              'f_lineno': 5},
 'tb_lineno': 2,
 'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                     'co_name': 'inner_2'},
                          'f_globals': {'__name__': '__main__'},
                          'f_lineno': 2},
             'tb_lineno': 2,
             'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                 'co_name': 'inner_1'},
                                      'f_globals': {'__name__': '__main__'},
                                      'f_lineno': 2},
                         'tb_lineno': 2,
                         'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                             'co_name': 'inner_0'},
                                                  'f_globals': {'__name__': '__main__'},
                                                  'f_lineno': 2},
                                     'tb_lineno': 2,
                                     'tb_next': None}}}}
tblib.Traceback.from_dict

Building on the previous example:

>>> tb_json = json.dumps(tb_dict)
>>> tb = Traceback.from_dict(json.loads(tb_json))
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
tblib.Traceback.from_string
>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: fail

If you use the strict=False option then parsing is a bit more lax:

>>> tb = Traceback.from_string("""
... File "bogus.py", line 123, in bogus
... Traceback (most recent call last):
...  File "tests/examples.py", line 2, in func_a
...   func_b()
...    File "tests/examples.py", line 6, in func_b
...     func_c()
...    File "tests/examples.py", line 10, in func_c
...   func_d()
...  File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """, strict=False)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "bogus.py", line 123, in bogus
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: fail

tblib.decorators.return_error

>>> from tblib.decorators import return_error
>>> inner_2r = return_error(inner_2)
>>> e = inner_2r()
>>> e
<tblib.decorators.Error object at ...>
>>> e.reraise()
Traceback (most recent call last):
  ...
  File "<doctest README.rst[26]>", line 1, in <module>
    e.reraise()
  File "...tblib...decorators.py", line 19, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 25, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

How's this useful? Imagine you're using multiprocessing like this:

# Note that Python 3.4 and later will show the remote traceback (but as a string sadly) so we skip testing this.
>>> import traceback
>>> from multiprocessing import Pool
>>> from examples import func_a
>>> pool = Pool()  # doctest: +SKIP
>>> try:  # doctest: +SKIP
...     for i in pool.map(func_a, range(5)):
...         print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    for i in pool.map(func_a, range(5)):
  File "...multiprocessing...pool.py", line ..., in map
    ...
  File "...multiprocessing...pool.py", line ..., in get
    ...
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()  # doctest: +SKIP

Not very useful is it? Let's sort this out:

>>> from tblib.decorators import apply_with_return_error, Error
>>> from itertools import repeat
>>> pool = Pool()
>>> try:
...     for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...         if isinstance(i, Error):
...             i.reraise()
...         else:
...             print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 4, in <module>
    i.reraise()
  File "...tblib...decorators.py", line ..., in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line ..., in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line ..., in apply_with_return_error
    return args[0](*args[1:])
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()

Much better !

What if we have a local call stack ?
>>> def local_0():
...     pool = Pool()
...     try:
...         for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...             if isinstance(i, Error):
...                 i.reraise()
...             else:
...                 print(i)
...     finally:
...         pool.close()
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> try:
...     local_2()
... except:
...     print(traceback.format_exc())
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    local_2()
  File "<doctest README.rst[...]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[...]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[...]>", line 6, in local_0
    i.reraise()
  File "...tblib...decorators.py", line 20, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 27, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line 47, in apply_with_return_error
    return args[0](*args[1:])
  File "...tests...examples.py", line 2, in func_a
    func_b()
  File "...tests...examples.py", line 6, in func_b
    func_c()
  File "...tests...examples.py", line 10, in func_c
    func_d()
  File "...tests...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
Other weird stuff

Clearing traceback works (Python 3.4 and up):

>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> import traceback, sys
>>> if sys.version_info > (3, 4):
...     traceback.clear_frames(tb)

Credits

CONTRIBUTING

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

Bug reports

When reporting a bug please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • Detailed steps to reproduce the bug.

Documentation improvements

tblib could always use more documentation, whether as part of the official tblib docs, in docstrings, or even on the web in blog posts, articles, and such.

Feature requests and feedback

The best way to send feedback is to file an issue at https://github.com/ionelmc/python-tblib/issues.

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that code contributions are welcome :)

Development

To set up [python-tblib]{.title-ref} for local development:

  1. Fork python-tblib (look for the "Fork" button).

  2. Clone your fork locally:

    git clone git@github.com:YOURGITHUBNAME/python-tblib.git

  3. Create a branch for local development:

    git checkout -b name-of-your-bugfix-or-feature

    Now you can make your changes locally.

  4. When you're done making changes run all the checks and docs builder with one command:

    tox

  5. Commit your changes and push your branch to GitHub:

    git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature
  6. Submit a pull request through the GitHub website.

Pull Request Guidelines

If you need some code review or feedback while you're developing the code just make the pull request.

For merging, you should:

  1. Include passing tests (run tox).
  2. Update documentation when there's new API, functionality etc.
  3. Add a note to CHANGELOG.rst about the changes.
  4. Add yourself to AUTHORS.rst.

Tips

To run a subset of tests:

tox -e envname -- pytest -k test_myfeature

To run all the test environments in parallel:

tox -p auto

Repository Summary

Checkout URI https://github.com/ionelmc/python-tblib.git
VCS Type git
VCS Version master
Last Updated 2024-05-02
Dev Status MAINTAINED
CI status No Continuous Integration
Released RELEASED
Tags No category tags.
Contributing Help Wanted (0)
Good First Issues (0)
Pull Requests to Review (0)

Packages

Name Version
tblib 1.2.0

README

Overview

+———————————–+———————————–+ | docs | Documentation Status | +———————————–+———————————–+ | tests | | ![GitHub Actions Build Status] | | | (https://github.com/ionelmc/pytho | | | n-tblib/actions/workflows/github- | | | actions.yml/badge.svg) | | | | Coverage Status | +———————————–+———————————–+ | package | | ![PyPI Package latest release] | | | (https://img.shields.io/pypi/v/tb | | | lib.svg) | | | PyPI Wheel | | | Supported versions | | | Supported implementations | | | | Commits since latest release | | | | +———————————–+———————————–+

Serialization library for Exceptions and Tracebacks.

  • Free software: BSD license

It allows you to:

  • Pickle tracebacks and raise exceptions with pickled tracebacks in different processes. This allows better error handling when running code over multiple processes (imagine multiprocessing, billiard, futures, celery etc).
  • Create traceback objects from strings (the from_string method). No pickling is used.
  • Serialize tracebacks to/from plain dicts (the from_dict and to_dict methods). No pickling is used.
  • Raise the tracebacks created from the aforementioned sources.
  • Pickle an Exception together with its traceback and exception chain (raise ... from ...) (Python 3 only)

Again, note that using the pickle support is completely optional. You are solely responsible for security problems should you decide to use the pickle support.

Installation

pip install tblib

Documentation

::: {.contents local=””} :::

Pickling tracebacks

Note: The traceback objects that come out are stripped of some attributes (like variables). But you'll be able to raise exceptions with those tracebacks or print them - that should cover 99% of the usecases.

>>> from tblib import pickling_support
>>> pickling_support.install()
>>> import pickle, sys
>>> def inner_0():
...     raise Exception('fail')
...
>>> def inner_1():
...     inner_0()
...
>>> def inner_2():
...     inner_1()
...
>>> try:
...     inner_2()
... except:
...     s1 = pickle.dumps(sys.exc_info())
...
>>> len(s1) > 1
True
>>> try:
...     inner_2()
... except:
...     s2 = pickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s2) > 1
True

>>> try:
...     import cPickle
... except ImportError:
...     import pickle as cPickle
>>> try:
...     inner_2()
... except:
...     s3 = cPickle.dumps(sys.exc_info(), protocol=pickle.HIGHEST_PROTOCOL)
...
>>> len(s3) > 1
True

Unpickling tracebacks

>>> pickle.loads(s1)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s2)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

>>> pickle.loads(s3)
(<...Exception'>, Exception('fail'...), <traceback object at ...>)

Raising

>>> from six import reraise
>>> reraise(*pickle.loads(s1))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s2))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
>>> reraise(*pickle.loads(s3))
Traceback (most recent call last):
  ...
  File "<doctest README.rst[14]>", line 1, in <module>
    reraise(*pickle.loads(s2))
  File "<doctest README.rst[8]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

Pickling Exceptions together with their traceback and chain (Python 3 only)

>>> try:  # doctest: +SKIP
...     try:
...         1 / 0
...     except Exception as e:
...         raise Exception("foo") from e
... except Exception as e:
...     s = pickle.dumps(e)
>>> raise pickle.loads(s)  # doctest: +SKIP
Traceback (most recent call last):
  File "<doctest README.rst[16]>", line 3, in <module>
    1 / 0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<doctest README.rst[17]>", line 1, in <module>
    raise pickle.loads(s)
  File "<doctest README.rst[16]>", line 5, in <module>
    raise Exception("foo") from e
Exception: foo

BaseException subclasses defined after calling pickling_support.install() will not retain their traceback and exception chain pickling. To cover custom Exceptions, there are three options:

  1. Use @pickling_support.install as a decorator for each custom Exception

    ``` {.sourceCode .python}

    from tblib import pickling_support

    Declare all imports of your package’s dependencies

    import numpy # doctest: +SKIP

    pickling_support.install() # install for all modules imported so far

    @pickling_support.install … class CustomError(Exception): … pass

    ```

    Eventual subclasses of CustomError will need to be decorated again.

  2. Invoke pickling_support.install() after all modules have been imported and all Exception subclasses have been declared

    ``` {.sourceCode .python}

    Declare all imports of your package’s dependencies

    import numpy # doctest: +SKIP from tblib import pickling_support

    Declare your own custom Exceptions

    class CustomError(Exception): … pass

    Finally, install tblib

    pickling_support.install()

    ```

  3. Selectively install tblib for Exception instances just before they are pickled

    ``` {.sourceCode .python} pickling_support.install(, [Exception instance], ...)

    ```

    The above will install tblib pickling for all listed exceptions as well as any other exceptions in their exception chains.

    For example, one could write a wrapper to be used with ProcessPoolExecutor, Dask.distributed, or similar libraries:

>>> from tblib import pickling_support
>>> def wrapper(func, *args, **kwargs):
...     try:
...         return func(*args, **kwargs)
...     except Exception as e:
...         pickling_support.install(e)
...         raise

What if we have a local stack, does it show correctly ?

Yes it does:

>>> exc_info = pickle.loads(s3)
>>> def local_0():
...     reraise(*exc_info)
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> local_2()
Traceback (most recent call last):
  File "...doctest.py", line ..., in __run
    compileflags, 1) in test.globs
  File "<doctest README.rst[24]>", line 1, in <module>
    local_2()
  File "<doctest README.rst[23]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[22]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[21]>", line 2, in local_0
    reraise(*exc_info)
  File "<doctest README.rst[11]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

It also supports more contrived scenarios

Like tracebacks with syntax errors:

>>> from tblib import Traceback
>>> from examples import bad_syntax
>>> try:
...     bad_syntax()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[58]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[57]>", line 2, in <module>
    bad_syntax()
  File "...tests...examples.py", line 18, in bad_syntax
    import badsyntax
  File "...tests...badsyntax.py", line 5
    is very bad
     ^
SyntaxError: invalid syntax

Or other import failures:

>>> from examples import bad_module
>>> try:
...     bad_module()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[61]>", line 1, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[60]>", line 2, in <module>
    bad_module()
  File "...tests...examples.py", line 23, in bad_module
    import badmodule
  File "...tests...badmodule.py", line 3, in <module>
    raise Exception("boom!")
Exception: boom!

Or a traceback that's caused by exceeding the recursion limit (here we're forcing the type and value to have consistency across platforms):

>>> def f(): f()
>>> try:
...    f()
... except RuntimeError:
...    et, ev, tb = sys.exc_info()
...    tb = Traceback(tb)
...
>>> reraise(RuntimeError, RuntimeError("maximum recursion depth exceeded"), tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  File "<doctest README.rst[32]>", line 1, in f
    def f(): f()
  ...
RuntimeError: maximum recursion depth exceeded

Reference

tblib.Traceback

It is used by the pickling_support. You can use it too if you want more flexibility:

>>> from tblib import Traceback
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
tblib.Traceback.to_dict

You can use the to_dict method and the from_dict classmethod to convert a Traceback into and from a dictionary serializable by the stdlib json.JSONDecoder:

>>> import json
>>> from pprint import pprint
>>> try:
...     inner_2()
... except:
...     et, ev, tb = sys.exc_info()
...     tb = Traceback(tb)
...     tb_dict = tb.to_dict()
...     pprint(tb_dict)
{'tb_frame': {'f_code': {'co_filename': '<doctest README.rst[...]>',
                         'co_name': '<module>'},
              'f_globals': {'__name__': '__main__'},
              'f_lineno': 5},
 'tb_lineno': 2,
 'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                     'co_name': 'inner_2'},
                          'f_globals': {'__name__': '__main__'},
                          'f_lineno': 2},
             'tb_lineno': 2,
             'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                 'co_name': 'inner_1'},
                                      'f_globals': {'__name__': '__main__'},
                                      'f_lineno': 2},
                         'tb_lineno': 2,
                         'tb_next': {'tb_frame': {'f_code': {'co_filename': ...,
                                                             'co_name': 'inner_0'},
                                                  'f_globals': {'__name__': '__main__'},
                                                  'f_lineno': 2},
                                     'tb_lineno': 2,
                                     'tb_next': None}}}}
tblib.Traceback.from_dict

Building on the previous example:

>>> tb_json = json.dumps(tb_dict)
>>> tb = Traceback.from_dict(json.loads(tb_json))
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[21]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "<doctest README.rst[21]>", line 2, in <module>
    inner_2()
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail
tblib.Traceback.from_string
>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: fail

If you use the strict=False option then parsing is a bit more lax:

>>> tb = Traceback.from_string("""
... File "bogus.py", line 123, in bogus
... Traceback (most recent call last):
...  File "tests/examples.py", line 2, in func_a
...   func_b()
...    File "tests/examples.py", line 6, in func_b
...     func_c()
...    File "tests/examples.py", line 10, in func_c
...   func_d()
...  File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """, strict=False)
>>> reraise(et, ev, tb.as_traceback())
Traceback (most recent call last):
  ...
  File "<doctest README.rst[42]>", line 6, in <module>
    reraise(et, ev, tb.as_traceback())
  File "bogus.py", line 123, in bogus
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: fail

tblib.decorators.return_error

>>> from tblib.decorators import return_error
>>> inner_2r = return_error(inner_2)
>>> e = inner_2r()
>>> e
<tblib.decorators.Error object at ...>
>>> e.reraise()
Traceback (most recent call last):
  ...
  File "<doctest README.rst[26]>", line 1, in <module>
    e.reraise()
  File "...tblib...decorators.py", line 19, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 25, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "<doctest README.rst[5]>", line 2, in inner_2
    inner_1()
  File "<doctest README.rst[4]>", line 2, in inner_1
    inner_0()
  File "<doctest README.rst[3]>", line 2, in inner_0
    raise Exception('fail')
Exception: fail

How's this useful? Imagine you're using multiprocessing like this:

# Note that Python 3.4 and later will show the remote traceback (but as a string sadly) so we skip testing this.
>>> import traceback
>>> from multiprocessing import Pool
>>> from examples import func_a
>>> pool = Pool()  # doctest: +SKIP
>>> try:  # doctest: +SKIP
...     for i in pool.map(func_a, range(5)):
...         print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    for i in pool.map(func_a, range(5)):
  File "...multiprocessing...pool.py", line ..., in map
    ...
  File "...multiprocessing...pool.py", line ..., in get
    ...
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()  # doctest: +SKIP

Not very useful is it? Let's sort this out:

>>> from tblib.decorators import apply_with_return_error, Error
>>> from itertools import repeat
>>> pool = Pool()
>>> try:
...     for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...         if isinstance(i, Error):
...             i.reraise()
...         else:
...             print(i)
... except:
...     print(traceback.format_exc())
...
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 4, in <module>
    i.reraise()
  File "...tblib...decorators.py", line ..., in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line ..., in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line ..., in apply_with_return_error
    return args[0](*args[1:])
  File "...examples.py", line 2, in func_a
    func_b()
  File "...examples.py", line 6, in func_b
    func_c()
  File "...examples.py", line 10, in func_c
    func_d()
  File "...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
>>> pool.terminate()

Much better !

What if we have a local call stack ?
>>> def local_0():
...     pool = Pool()
...     try:
...         for i in pool.map(apply_with_return_error, zip(repeat(func_a), range(5))):
...             if isinstance(i, Error):
...                 i.reraise()
...             else:
...                 print(i)
...     finally:
...         pool.close()
...
>>> def local_1():
...     local_0()
...
>>> def local_2():
...     local_1()
...
>>> try:
...     local_2()
... except:
...     print(traceback.format_exc())
Traceback (most recent call last):
  File "<doctest README.rst[...]>", line 2, in <module>
    local_2()
  File "<doctest README.rst[...]>", line 2, in local_2
    local_1()
  File "<doctest README.rst[...]>", line 2, in local_1
    local_0()
  File "<doctest README.rst[...]>", line 6, in local_0
    i.reraise()
  File "...tblib...decorators.py", line 20, in reraise
    reraise(self.exc_type, self.exc_value, self.traceback)
  File "...tblib...decorators.py", line 27, in return_exceptions_wrapper
    return func(*args, **kwargs)
  File "...tblib...decorators.py", line 47, in apply_with_return_error
    return args[0](*args[1:])
  File "...tests...examples.py", line 2, in func_a
    func_b()
  File "...tests...examples.py", line 6, in func_b
    func_c()
  File "...tests...examples.py", line 10, in func_c
    func_d()
  File "...tests...examples.py", line 14, in func_d
    raise Exception("Guessing time !")
Exception: Guessing time !
<BLANKLINE>
Other weird stuff

Clearing traceback works (Python 3.4 and up):

>>> tb = Traceback.from_string("""
... File "skipped.py", line 123, in func_123
... Traceback (most recent call last):
...   File "tests/examples.py", line 2, in func_a
...     func_b()
...   File "tests/examples.py", line 6, in func_b
...     func_c()
...   File "tests/examples.py", line 10, in func_c
...     func_d()
...   File "tests/examples.py", line 14, in func_d
... Doesn't: matter
... """)
>>> import traceback, sys
>>> if sys.version_info > (3, 4):
...     traceback.clear_frames(tb)

Credits

CONTRIBUTING

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

Bug reports

When reporting a bug please include:

  • Your operating system name and version.
  • Any details about your local setup that might be helpful in troubleshooting.
  • Detailed steps to reproduce the bug.

Documentation improvements

tblib could always use more documentation, whether as part of the official tblib docs, in docstrings, or even on the web in blog posts, articles, and such.

Feature requests and feedback

The best way to send feedback is to file an issue at https://github.com/ionelmc/python-tblib/issues.

If you are proposing a feature:

  • Explain in detail how it would work.
  • Keep the scope as narrow as possible, to make it easier to implement.
  • Remember that this is a volunteer-driven project, and that code contributions are welcome :)

Development

To set up [python-tblib]{.title-ref} for local development:

  1. Fork python-tblib (look for the "Fork" button).

  2. Clone your fork locally:

    git clone git@github.com:YOURGITHUBNAME/python-tblib.git

  3. Create a branch for local development:

    git checkout -b name-of-your-bugfix-or-feature

    Now you can make your changes locally.

  4. When you're done making changes run all the checks and docs builder with one command:

    tox

  5. Commit your changes and push your branch to GitHub:

    git add .
git commit -m "Your detailed description of your changes."
git push origin name-of-your-bugfix-or-feature
  6. Submit a pull request through the GitHub website.

Pull Request Guidelines

If you need some code review or feedback while you're developing the code just make the pull request.

For merging, you should:

  1. Include passing tests (run tox).
  2. Update documentation when there's new API, functionality etc.
  3. Add a note to CHANGELOG.rst about the changes.
  4. Add yourself to AUTHORS.rst.

Tips

To run a subset of tests:

tox -e envname -- pytest -k test_myfeature

To run all the test environments in parallel:

tox -p auto