alex.utils package¶
Submodules¶
alex.utils.analytics module¶
alex.utils.audio module¶
alex.utils.audio_play module¶
alex.utils.autopath module¶
self cloning, automatic path configuration
copy this into any subdirectory of pypy from which scripts need to be run, typically all of the test subdirs. The idea is that any such script simply issues
import autopath
and this will make sure that the parent directory containing “pypy” is in sys.path.
If you modify the master “autopath.py” version (in pypy/tool/autopath.py) you can directly run it which will copy itself on all autopath.py files it finds under the pypy root directory.
This module always provides these attributes:
pypydir pypy root directory path this_dir directory where this autopath.py resides
alex.utils.cache module¶
-
alex.utils.cache.
lfu_cache
(maxsize=100)[source]¶ Least-frequently-used cache decorator.
Arguments to the cached function must be hashable. Cache performance statistics stored in f.hits and f.misses. Clear the cache with f.clear(). http://en.wikipedia.org/wiki/Least_Frequently_Used
-
alex.utils.cache.
lru_cache
(maxsize=100)[source]¶ Least-recently-used cache decorator.
Arguments to the cached function must be hashable. Cache performance statistics stored in f.hits and f.misses. Clear the cache with f.clear(). http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used
alex.utils.caminfodb module¶
alex.utils.config module¶
-
class
alex.utils.config.
Config
(file_name=None, project_root=False, config=None)[source]¶ Bases:
object
Config handles configuration data necessary for all the components in Alex. It is implemented using a dictionary so that any component can use arbitrarily structured configuration data.
Before the configuration file is loaded, it is transformed as follows:
- ‘{cfg_abs_path}’ as a string anywhere in the file is replaced by an
absolute path of the configuration files. This can be used to make the configuration file independent of the location of programs that use it.
-
DEFAULT_CFG_PPATH
= u'resources/default.cfg'¶
-
config_replace
(p, s, d=None)[source]¶ Replace a pattern p with string s in the whole config (recursively) or in a part of the config given in d.
-
classmethod
load_configs
(config_flist=[], use_default=True, log=True, *init_args, **init_kwargs)[source]¶ Loads and merges configs from paths listed in `config_flist’. Use this method instead of direct loading configs, as it takes care of not only merging them but also processing some options in a special way.
- Arguments:
- config_flist – list of paths to config files to load and merge;
- order matters (default: [])
- use_default – whether to insert the default config
- ($ALEX/resources/default.cfg) at the beginning of `config_flist’ (default: True)
- log – whether to log the resulting config using the system logger
- (default: True)
- init_args – additional positional arguments will be passed to
- constructors for each config
- init_kwargs – additional keyword arguments will be passed to
- constructors for each config
-
merge
(other)[source]¶ Merges self’s config with other’s config and saves it as a new self’s config.
- Keyword arguments:
- other: a Config object whose configuration dictionary to merge
into self’s one
-
unfold_lists
(pattern, unfold_id_key=None, part=[])[source]¶ Unfold lists under keys matching the given pattern into several config objects, each containing one item. If pattern is None, all lists are expanded.
Stores a string representation of the individual unfolded values under the unfold_id_key if this parameter is set.
Only expands a part of the whole config hash (given by list of keys forming a path to this part) if the path parameter is set.
-
alex.utils.config.
callback_download_progress
(blocks, block_size, total_size)[source]¶ callback function for urlretrieve that is called when connection is created and when once for each block
Parameters: - blocks – number of blocks transferred so far
- block_size – in bytes
- total_size – in bytes, can be -1 if server doesn’t return it
-
alex.utils.config.
load_as_module
(path, force=False, encoding=u'UTF-8', text_transforms=[])[source]¶ Loads a file pointed to by `path’ as a Python module with minimal impact on the global program environment. The file name should end in ‘.py’.
- Arguments:
path – path towards the file force – whether to load the file even if its name does not end in
‘.py’encoding – character encoding of the file text_transforms – collection of functions to be run on the original
file text
Returns the loaded module object.
-
alex.utils.config.
online_update
(file_name)[source]¶ This function can download file from a default server if it is not available locally. The default server location can be changed in the config file.
The original file name is transformed into absolute name using as_project_path function.
Parameters: file_name – the file name which should be downloaded from the server Returns: a file name of the local copy of the file downloaded from the server
-
alex.utils.config.
set_online_update_server
(server_name)[source]¶ Set the name of the online update server. This function can be used to change the server name from inside a config file.
Parameters: server_name – the HTTP(s) path to the server and a location where the desired data reside. Returns: None
alex.utils.cuda module¶
alex.utils.czech_stemmer module¶
Czech stemmer Copyright © 2010 Luís Gomes <luismsgomes@gmail.com>.
- Ported from the Java implementation available at:
- http://members.unine.ch/jacques.savoy/clef/index.html
alex.utils.enums module¶
alex.utils.env module¶
alex.utils.excepthook module¶
Depending on the hook_type, ExceptionHook class adds various hooks how to catch exceptions.
-
class
alex.utils.excepthook.
ExceptionHook
(hook_type, logger=None)[source]¶ Bases:
object
Singleton objects for registering various hooks for sys.exepthook. For registering a hook, use set_hook.
-
apply
()[source]¶ The object can be used to store settings for excepthook. a = ExceptionHook(‘log’) # now it logs b = ExceptionHook(‘ipdb’) # now it uses ipdb a.apply() # now it logs again
-
logger
= None¶
-
alex.utils.exceptions module¶
-
exception
alex.utils.exceptions.
ConfigException
[source]¶ Bases:
alex.AlexException
-
exception
alex.utils.exceptions.
SessionClosedException
[source]¶ Bases:
alex.AlexException
-
exception
alex.utils.exceptions.
SessionLoggerException
[source]¶ Bases:
alex.AlexException
alex.utils.filelock module¶
Context manager for locking on a file. Obtained from
http://www.evanfosmark.com/2009/01 /cross-platform-file-locking-support-in-python/,
licensed under BSD.
This is thought to work safely on NFS too, in contrast to fcntl.flock(). This is also thought to work safely over SMB and else, in contrast to fcntl.lockf(). For both issues, consult http://oilq.org/fr/node/13344.
Use as simply as
- with FileLock(filename):
- <critical section for working with the file at `filename’>
-
class
alex.utils.filelock.
FileLock
(file_name, timeout=10, delay=0.05)[source]¶ Bases:
object
A file locking mechanism that has context-manager support so you can use it in a with statement. This should be relatively portable as it doesn’t rely on msvcrt or fcntl for the locking.
alex.utils.fs module¶
Filesystem utility functions.
-
class
alex.utils.fs.
GrepFilter
(stdin, stdout, breakchar=u'n')[source]¶ Bases:
multiprocessing.process.Process
-
add_listener
(regex, callback)[source]¶ Adds a listener to the output strings.
- Arguments:
- regex – the compiled regular expression to look for
- (`regex.search’) in any piece of output
- callback – a callable that is invoked for output where `regex’ was
found. This will be called like this:
outputting &= callback(output_unicode_str)That means, callback should take the unicode string argument containing what would have been output and return a boolean value which is True iff outputting should stop.
Returns the index of the listener for later reference.
-
-
alex.utils.fs.
find
(dir_, glob_, mindepth=2, maxdepth=6, ignore_globs=[], ignore_paths=None, follow_symlinks=True, prune=False, rx=None, notrx=None)[source]¶ A simplified version of the GNU `find’ utility. Lists files with basename matching `glob_‘ found in `dir_‘ in depth between `mindepth’ and `maxdepth’.
The `ignore_globs’ argument specifies a glob for basenames of files to be ignored. The `ignore_paths’ argument specifies a collection of real absolute pathnames that are pruned from the search. For efficiency reasons, it should be a set.
In the current implementation, the traversal resolves symlinks before the file name is checked. However, taking symlinks into account can be forbidden altogether by specifying `follow_symlinks=False’. Cycles during the traversal are avoided.
The returned set of files consists of real absolute pathnames of those files.
alex.utils.htk module¶
alex.utils.interface module¶
alex.utils.lattice module¶
alex.utils.mfcc module¶
alex.utils.mproc module¶
Implements useful classes for handling multiprocessing implementation of the Alex system.
-
class
alex.utils.mproc.
InstanceID
[source]¶ Bases:
object
This class provides unique ids to all instances of objects inheriting from this class.
-
instance_id
= <Synchronized wrapper for c_int(0)>¶
-
lock
= <Lock(owner=None)>¶
-
-
class
alex.utils.mproc.
SystemLogger
(output_dir, stdout_log_level='DEBUG', stdout=True, file_log_level='DEBUG')[source]¶ Bases:
object
This is a multiprocessing-safe logger. It should be used by all components in Alex.
-
get_session_dir_name
(*args, **kw)[source]¶ Return directory where all the call related files should be stored.
-
get_time_str
()[source]¶ Return current time in dashed ISO-like format.
It is useful in constructing file and directory names.
-
levels
= {'INFO': 20, 'CRITICAL': 40, 'EXCEPTION': 50, 'SYSTEM-LOG': 0, 'WARNING': 30, 'ERROR': 60, 'DEBUG': 10}¶
-
lock
= <RLock(None, 0)>¶
-
log
(*args, **kw)[source]¶ Logs the message based on its level and the logging setting. Before writing into a logging file, it locks the file.
-
session_end
(*args, **kw)[source]¶ WARNING: Deprecated Disables logging into the session-specific directory.
We better do not end a session because very often after the session_end() method is called there are still incoming messages. Therefore, it is better to wait for the session_start() method to set a new destination for the session log.
-
session_start
(*args, **kw)[source]¶ Create a specific directory for logging a specific call.
NOTE: This is not completely safe. It can be called from several processes.
-
-
alex.utils.mproc.
async
(func)[source]¶ A function decorator intended to make “func” run in a separate thread (asynchronously). Returns the created Thread object
E.g.: @async def task1():
do_something@async def task2():
do_something_toot1 = task1() t2 = task2() ... t1.join() t2.join()
-
alex.utils.mproc.
etime
(name='Time', min_t=0.3)[source]¶ This decorator measures the execution time of the decorated function.
-
alex.utils.mproc.
file_lock
(file_name)[source]¶ Multiprocessing lock using files. Lock on a specific file.
-
alex.utils.mproc.
file_unlock
(lock_file)[source]¶ Multiprocessing lock using files. Unlock on a specific file.
alex.utils.nose_plugins module¶
alex.utils.parsers module¶
-
class
alex.utils.parsers.
CamTxtParser
(lower=False)[source]¶ Bases:
object
Parser of files of the following format: <<BOF>> [record]
[record]
... <<EOF>>
where [record] has the following format:
<<[record]>> [property name]([property value]) <</[record]>>
[property name] and [property value] are arbitrary strings
Any ” or ‘ characters are stripped from the beginning and end of each [property value].
-
line_expr
= <_sre.SRE_Pattern object>¶
-
alex.utils.procname module¶
alex.utils.rdb module¶
alex.utils.sessionlogger module¶
-
class
alex.utils.sessionlogger.
SessionLogger
[source]¶ Bases:
multiprocessing.process.Process
This is a multiprocessing-safe logger. It should be used by Alex to log information according the SDC 2010 XML format.
Date and times should also include time zone.
Times should be in seconds from the beginning of the dialogue.
alex.utils.test_analytics module¶
alex.utils.test_fs module¶
Unit tests for alex.util.fs.
-
class
alex.utils.test_fs.
TestFind
(methodName='runTest')[source]¶ Bases:
unittest.case.TestCase
alex.utils.test_sessionlogger module¶
alex.utils.test_text module¶
alex.utils.text module¶
-
class
alex.utils.text.
Escaper
(chars=u''"', escaper=u'\', re_flags=0)[source]¶ Bases:
object
Creates a customised escaper for strings. The characters that need escaping, as well as the one used for escaping can be specified.
-
ESCAPED
= 1¶
-
ESCAPER
= 0¶
-
NORMAL
= 2¶
-
annotate
(esced)[source]¶ Annotates each character of a text that has been escaped whether:
Escaper.ESCAPER - it is the escape character Escaper.ESCAPED - it is a character that was escaped Escaper.NORMAL - otherwise.It is expected that only parts of the text may have actually been escaped.
Returns a list with the annotation values, co-indexed with characters of the input text.
-
static
re_literal
(char)[source]¶ Escapes the character so that when it is used in a regexp, it matches itself.
-
-
alex.utils.text.
escape_special_characters_shell
(text, characters=u'\'"')[source]¶ Simple function that tries to escape quotes. Not guaranteed to produce the correct result!! If that is needed, use the new `Escaper’ class.
-
alex.utils.text.
min_edit_dist
(target, source)[source]¶ Computes the min edit distance from target to source.
-
alex.utils.text.
min_edit_ops
(target, source, cost=<function <lambda>>)[source]¶ Computes the min edit operations from target to source.
Parameters: - target – a target sequence
- source – a source sequence
- cost – an expression for computing cost of the edit operations
Returns: a tuple of (insertions, deletions, substitutions)
-
alex.utils.text.
parse_command
(command)[source]¶ Parse the command name(var1=”val1”,...) into a dictionary structure:
E.g. call(destination=”1245”,opt=”X”) will be parsed into:
- { “__name__”: “call”,
- “destination”: “1245”, “opt”: “X”}
Return the parsed command in a dictionary.
-
alex.utils.text.
split_by
(text, splitter, opening_parentheses=u'', closing_parentheses=u'', quotes=u'\'"')[source]¶ Splits the input text at each occurrence of the splitter only if it is not enclosed in parentheses.
text - the input text string splitter - multi-character string which is used to determine the position
of splitting of the text- opening_parentheses - an iterable of opening parentheses that has to be
- respected when splitting, e.g. “{(” (default: ‘’)
- closing_parentheses - an iterable of closing parentheses that has to be
- respected when splitting, e.g. “})” (default: ‘’)
quotes - an iterable of quotes that have to come in pairs, e.g. ‘”’
alex.utils.ui module¶
alex.utils.various module¶
-
alex.utils.various.
flatten
(list_, ltypes=(<type 'list'>, <type 'tuple'>))[source]¶ Flatten nested list into a simple list.
-
alex.utils.various.
get_text_from_xml_node
(node)[source]¶ Get text from all child nodes and concatenate it.
-
alex.utils.various.
group_by
(objects, attrs)[source]¶ Groups `objects’ by the values of their attributes `attrs’.
Returns a dictionary mapping from a tuple of attribute values to a list of objects with those attribute values.