| Name | cs.lex JSON |
| Version |
20240316
JSON |
| download |
| home_page | |
| Summary | Lexical analysis functions, tokenisers, transcribers: an arbitrary assortment of lexical and tokenisation functions useful for writing recursive descent parsers, of which I have several. There are also some transcription functions for producing text from various objects, such as `hexify` and `unctrl`. |
| upload_time | 2024-03-16 06:59:28 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | |
| license | GNU General Public License v3 or later (GPLv3+) |
| keywords |
python2
python3
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Lexical analysis functions, tokenisers, transcribers:
an arbitrary assortment of lexical and tokenisation functions useful
for writing recursive descent parsers, of which I have several.
There are also some transcription functions for producing text
from various objects, such as `hexify` and `unctrl`.
*Latest release 20240316*:
Fixed release upload artifacts.
Generally the get_* functions accept a source string and an offset
(usually optional, default `0`) and return a token and the new offset,
raising `ValueError` on failed tokenisation.
## Function `as_lines(chunks, partials=None)`
Generator yielding complete lines from arbitrary pieces of text from
the iterable of `str` `chunks`.
After completion, any remaining newline-free chunks remain
in the partials list; they will be unavailable to the caller
unless the list is presupplied.
## Function `camelcase(snakecased, first_letter_only=False)`
Convert a snake cased string `snakecased` into camel case.
Parameters:
* `snakecased`: the snake case string to convert
* `first_letter_only`: optional flag (default `False`);
if true then just ensure that the first character of a word
is uppercased, otherwise use `str.title`
Example:
>>> camelcase('abc_def')
'abcDef'
>>> camelcase('ABc_def')
'abcDef'
>>> camelcase('abc_dEf')
'abcDef'
>>> camelcase('abc_dEf', first_letter_only=True)
'abcDEf'
## Function `common_prefix(*strs)`
Return the common prefix of the strings `strs`.
Examples:
>>> common_prefix('abc', 'def')
''
>>> common_prefix('abc', 'abd')
'ab'
>>> common_prefix('abc', 'abcdef')
'abc'
>>> common_prefix('abc', 'abcdef', 'abz')
'ab'
>>> # contrast with cs.fileutils.common_path_prefix
>>> common_prefix('abc/def', 'abc/def1', 'abc/def2')
'abc/def'
## Function `common_suffix(*strs)`
Return the common suffix of the strings `strs`.
## Function `cropped(s: str, max_length: int = 32, roffset: int = 1, ellipsis: str = '...')`
If the length of `s` exceeds `max_length` (default `32`),
replace enough of the tail with `ellipsis`
and the last `roffset` (default `1`) characters of `s`
to fit in `max_length` characters.
## Function `cropped_repr(o, roffset=1, max_length=32, inner_max_length=None)`
Compute a cropped `repr()` of `o`.
Parameters:
* `o`: the object to represent
* `max_length`: the maximum length of the representation, default `32`
* `inner_max_length`: the maximum length of the representations
of members of `o`, default `max_length//2`
* `roffset`: the number of trailing characters to preserve, default `1`
## Function `cutprefix(s, prefix)`
Strip a `prefix` from the front of `s`.
Return the suffix if `s.startswith(prefix)`, else `s`.
Example:
>>> abc_def = 'abc.def'
>>> cutprefix(abc_def, 'abc.')
'def'
>>> cutprefix(abc_def, 'zzz.')
'abc.def'
>>> cutprefix(abc_def, '.zzz') is abc_def
True
## Function `cutsuffix(s, suffix)`
Strip a `suffix` from the end of `s`.
Return the prefix if `s.endswith(suffix)`, else `s`.
Example:
>>> abc_def = 'abc.def'
>>> cutsuffix(abc_def, '.def')
'abc'
>>> cutsuffix(abc_def, '.zzz')
'abc.def'
>>> cutsuffix(abc_def, '.zzz') is abc_def
True
## Class `FFloat(FNumericMixin, FormatableMixin, FormatableFormatter, string.Formatter, builtins.float)`
Formattable `float`.
## Class `FInt(FNumericMixin, FormatableMixin, FormatableFormatter, string.Formatter, builtins.int)`
Formattable `int`.
## Class `FNumericMixin(FormatableMixin, FormatableFormatter, string.Formatter)`
A `FormatableMixin` subclass.
## Function `format_as(format_s: str, format_mapping, formatter=None, error_sep=None, strict=None)`
Format the string `format_s` using `Formatter.vformat`,
return the formatted result.
This is a wrapper for `str.format_map`
which raises a more informative `FormatAsError` exception on failure.
Parameters:
* `format_s`: the format string to use as the template
* `format_mapping`: the mapping of available replacement fields
* `formatter`: an optional `string.Formatter`-like instance
with a `.vformat(format_string,args,kwargs)` method,
usually a subclass of `string.Formatter`;
if not specified then `FormatableFormatter` is used
* `error_sep`: optional separator for the multipart error message,
default from `FormatAsError.DEFAULT_SEPARATOR`:
`'; '`
* `strict`: optional flag (default `False`)
indicating that an unresolveable field should raise a
`KeyError` instead of inserting a placeholder
## Function `format_attribute(method)`
A decorator to mark a method as available as a format method.
Requires the enclosing class to be decorated with `@has_format_attributes`.
For example,
the `FormatableMixin.json` method is defined like this:
@format_attribute
def json(self):
return self.FORMAT_JSON_ENCODER.encode(self)
which allows a `FormatableMixin` subclass instance
to be used in a format string like this:
{instance:json}
to insert a JSON transcription of the instance.
It is recommended that methods marked with `@format_attribute`
have no side effects and do not modify state,
as they are intended for use in ad hoc format strings
supplied by an end user.
## Function `format_escape(s)`
Escape `{}` characters in a string to protect them from `str.format`.
## Function `format_recover(*da, **dkw)`
Decorator for `__format__` methods which replaces failed formats
with `{self:format_spec}`.
## Class `FormatableFormatter(string.Formatter)`
A `string.Formatter` subclass interacting with objects
which inherit from `FormatableMixin`.
## Class `FormatableMixin(FormatableFormatter, string.Formatter)`
A subclass of `FormatableFormatter` which provides 2 features:
- a `__format__` method which parses the `format_spec` string
into multiple colon separated terms whose results chain
- a `format_as` method which formats a format string using `str.format_map`
with a suitable mapping derived from the instance
via its `format_kwargs` method
(whose default is to return the instance itself)
The `format_as` method is like an inside out `str.format` or
`object.__format__` method.
The `str.format` method is designed for formatting a string
from a variety of other objects supplied in the keyword arguments.
The `object.__format__` method is for filling out a single `str.format`
replacement field from a single object.
By contrast, `format_as` is designed to fill out an entire format
string from the current object.
For example, the `cs.tagset.TagSetMixin` class
uses `FormatableMixin` to provide a `format_as` method
whose replacement fields are derived from the tags in the tag set.
Subclasses wanting to provide additional `format_spec` terms
should:
- override `FormatableFormatter.format_field1` to implement
terms with no colons, letting `format_field` do the split into terms
- override `FormatableFormatter.get_format_subspecs` to implement
the parse of `format_spec` into a sequence of terms.
This might recognise a special additional syntax
and quietly fall back to `super().get_format_subspecs`
if that is not present.
## Class `FormatAsError(builtins.LookupError, builtins.Exception, builtins.BaseException)`
Subclass of `LookupError` for use by `format_as`.
## Class `FStr(FormatableMixin, FormatableFormatter, string.Formatter, builtins.str)`
A `str` subclass with the `FormatableMixin` methods,
particularly its `__format__` method
which uses `str` method names as valid formats.
It also has a bunch of utility methods which are available
as `:`*method* in format strings.
## Function `get_chars(s, offset, gochars)`
Scan the string `s` for characters in `gochars` starting at `offset`.
Return `(match,new_offset)`.
`gochars` may also be a callable, in which case a character
`ch` is accepted if `gochars(ch)` is true.
## Function `get_decimal(s, offset=0)`
Scan the string `s` for decimal characters starting at `offset` (default `0`).
Return `(dec_string,new_offset)`.
## Function `get_decimal_or_float_value(s, offset=0)`
Fetch a decimal or basic float (nnn.nnn) value
from the str `s` at `offset` (default `0`).
Return `(value,new_offset)`.
## Function `get_decimal_value(s, offset=0)`
Scan the string `s` for a decimal value starting at `offset` (default `0`).
Return `(value,new_offset)`.
## Function `get_delimited(s, offset, delim)`
Collect text from the string `s` from position `offset` up
to the first occurence of delimiter `delim`; return the text
excluding the delimiter and the offset after the delimiter.
## Function `get_dotted_identifier(s, offset=0, **kw)`
Scan the string `s` for a dotted identifier (by default an
ASCII letter or underscore followed by letters, digits or
underscores) with optional trailing dot and another dotted
identifier, starting at `offset` (default `0`).
Return `(match,new_offset)`.
Note: the empty string and an unchanged offset will be returned if
there is no leading letter/underscore.
Keyword arguments are passed to `get_identifier`
(used for each component of the dotted identifier).
## Function `get_envvar(s, offset=0, environ=None, default=None, specials=None)`
Parse a simple environment variable reference to $varname or
$x where "x" is a special character.
Parameters:
* `s`: the string with the variable reference
* `offset`: the starting point for the reference
* `default`: default value for missing environment variables;
if `None` (the default) a `ValueError` is raised
* `environ`: the environment mapping, default `os.environ`
* `specials`: the mapping of special single character variables
## Function `get_hexadecimal(s, offset=0)`
Scan the string `s` for hexadecimal characters starting at `offset` (default `0`).
Return `(hex_string,new_offset)`.
## Function `get_hexadecimal_value(s, offset=0)`
Scan the string `s` for a hexadecimal value starting at `offset` (default `0`).
Return `(value,new_offset)`.
## Function `get_identifier(s, offset=0, alpha='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ', number='0123456789', extras='_')`
Scan the string `s` for an identifier (by default an ASCII
letter or underscore followed by letters, digits or underscores)
starting at `offset` (default 0).
Return `(match,new_offset)`.
*Note*: the empty string and an unchanged offset will be returned if
there is no leading letter/underscore.
Parameters:
* `s`: the string to scan
* `offset`: the starting offset, default `0`.
* `alpha`: the characters considered alphabetic,
default `string.ascii_letters`.
* `number`: the characters considered numeric,
default `string.digits`.
* `extras`: extra characters considered part of an identifier,
default `'_'`.
## Function `get_ini_clause_entryname(s, offset=0)`
Parse a `[`*clausename*`]`*entryname* string
from `s` at `offset` (default `0`).
Return `(clausename,entryname,new_offset)`.
## Function `get_ini_clausename(s, offset=0)`
Parse a `[`*clausename*`]` string from `s` at `offset` (default `0`).
Return `(clausename,new_offset)`.
## Function `get_nonwhite(s, offset=0)`
Scan the string `s` for characters not in `string.whitespace`
starting at `offset` (default `0`).
Return `(match,new_offset)`.
## Function `get_other_chars(s, offset=0, stopchars=None)`
Scan the string `s` for characters not in `stopchars` starting
at `offset` (default `0`).
Return `(match,new_offset)`.
## Function `get_prefix_n(s, prefix, n=None, *, offset=0)`
Strip a leading `prefix` and numeric value `n` from the string `s`
starting at `offset` (default `0`).
Return the matched prefix, the numeric value and the new offset.
Returns `(None,None,offset)` on no match.
Parameters:
* `s`: the string to parse
* `prefix`: the prefix string which must appear at `offset`
or an object with a `match(str,offset)` method
such as an `re.Pattern` regexp instance
* `n`: optional integer value;
if omitted any value will be accepted, otherwise the numeric
part must match `n`
If `prefix` is a `str`, the "matched prefix" return value is `prefix`.
Otherwise the "matched prefix" return value is the result of
the `prefix.match(s,offset)` call. The result must also support
a `.end()` method returning the offset in `s` beyond the match,
used to locate the following numeric portion.
Examples:
>>> import re
>>> get_prefix_n('s03e01--', 's')
('s', 3, 3)
>>> get_prefix_n('s03e01--', 's', 3)
('s', 3, 3)
>>> get_prefix_n('s03e01--', 's', 4)
(None, None, 0)
>>> get_prefix_n('s03e01--', re.compile('[es]',re.I))
(<re.Match object; span=(0, 1), match='s'>, 3, 3)
>>> get_prefix_n('s03e01--', re.compile('[es]',re.I), offset=3)
(<re.Match object; span=(3, 4), match='e'>, 1, 6)
## Function `get_qstr(s, offset=0, q='"', environ=None, default=None, env_specials=None)`
Get quoted text with slosh escapes and optional environment substitution.
Parameters:
* `s`: the string containg the quoted text.
* `offset`: the starting point, default `0`.
* `q`: the quote character, default `'"'`. If `q` is `None`,
do not expect the string to be delimited by quote marks.
* `environ`: if not `None`, also parse and expand `$`*envvar* references.
* `default`: passed to `get_envvar`
## Function `get_qstr_or_identifier(s, offset)`
Parse a double quoted string or an identifier.
## Function `get_sloshed_text(s, delim, offset=0, slosh='\\', mapper=<function slosh_mapper at 0x104903eb0>, specials=None)`
Collect slosh escaped text from the string `s` from position
`offset` (default `0`) and return the decoded unicode string and
the offset of the completed parse.
Parameters:
* `delim`: end of string delimiter, such as a single or double quote.
* `offset`: starting offset within `s`, default `0`.
* `slosh`: escape character, default a slosh ('\').
* `mapper`: a mapping function which accepts a single character
and returns a replacement string or `None`; this is used the
replace things such as '\t' or '\n'. The default is the
`slosh_mapper` function, whose default mapping is `SLOSH_CHARMAP`.
* `specials`: a mapping of other special character sequences and parse
functions for gathering them up. When one of the special
character sequences is found in the string, the parse
function is called to parse at that point.
The parse functions accept
`s` and the offset of the special character. They return
the decoded string and the offset past the parse.
The escape character `slosh` introduces an encoding of some
replacement text whose value depends on the following character.
If the following character is:
* the escape character `slosh`, insert the escape character.
* the string delimiter `delim`, insert the delimiter.
* the character 'x', insert the character with code from the following
2 hexadecimal digits.
* the character 'u', insert the character with code from the following
4 hexadecimal digits.
* the character 'U', insert the character with code from the following
8 hexadecimal digits.
* a character from the keys of `mapper`
## Function `get_tokens(s, offset, getters)`
Parse the string `s` from position `offset` using the supplied
tokeniser functions `getters`.
Return the list of tokens matched and the final offset.
Parameters:
* `s`: the string to parse.
* `offset`: the starting position for the parse.
* `getters`: an iterable of tokeniser specifications.
Each tokeniser specification `getter` is either:
* a callable expecting `(s,offset)` and returning `(token,new_offset)`
* a literal string, to be matched exactly
* a `tuple` or `list` with values `(func,args,kwargs)`;
call `func(s,offset,*args,**kwargs)`
* an object with a `.match` method such as a regex;
call `getter.match(s,offset)` and return a match object with
a `.end()` method returning the offset of the end of the match
## Function `get_uc_identifier(s, offset=0, number='0123456789', extras='_')`
Scan the string `s` for an identifier as for `get_identifier`,
but require the letters to be uppercase.
## Function `get_white(s, offset=0)`
Scan the string `s` for characters in `string.whitespace`
starting at `offset` (default `0`).
Return `(match,new_offset)`.
## Function `has_format_attributes(*da, **dkw)`
Class decorator to walk this class for direct methods
marked as for use in format strings
and to include them in `cls.format_attributes()`.
Methods are normally marked with the `@format_attribute` decorator.
If `inherit` is true the base format attributes will be
obtained from other classes:
* `inherit` is `True`: use `cls.__mro__`
* `inherit` is a class: use that class
* otherwise assume `inherit` is an iterable of classes
For each class `otherclass`, update the initial attribute
mapping from `otherclass.get_format_attributes()`.
## Function `hexify(bs)`
A flavour of `binascii.hexlify` returning a `str`.
## Function `htmlify(s, nbsp=False)`
Convert a string for safe transcription in HTML.
Parameters:
* `s`: the string
* `nbsp`: replaces spaces with `" "` to prevent word folding,
default `False`.
## Function `htmlquote(s)`
Quote a string for use in HTML.
## Function `is_dotted_identifier(s, offset=0, **kw)`
Test if the string `s` is an identifier from position `offset` onward.
## Function `is_identifier(s, offset=0, **kw)`
Test if the string `s` is an identifier
from position `offset` (default `0`) onward.
## Function `is_uc_identifier(s, offset=0, **kw)`
Test if the string `s` is an uppercase identifier
from position `offset` (default `0`) onward.
## Function `isUC_(s)`
Check that a string matches the regular expression `^[A-Z][A-Z_0-9]*$`.
## Function `jsquote(s)`
Quote a string for use in JavaScript.
## Function `lc_(value)`
Return `value.lower()`
with `'-'` translated into `'_'` and `' '` translated into `'-'`.
I use this to construct lowercase filenames containing a
readable transcription of a title string.
See also `titleify_lc()`, an imperfect reversal of this.
## Function `match_tokens(s, offset, getters)`
Wrapper for `get_tokens` which catches `ValueError` exceptions
and returns `(None,offset)`.
## Function `parseUC_sAttr(attr)`
Take an attribute name `attr` and return `(key,is_plural)`.
Examples:
* `'FOO'` returns `('FOO',False)`.
* `'FOOs'` or `'FOOes'` returns `('FOO',True)`.
Otherwise return `(None,False)`.
## Function `phpquote(s)`
Quote a string for use in PHP code.
## Function `r(o, max_length=None, *, use_cls=False)`
Like `typed_str` but using `repr` instead of `str`.
This is available as both `typed_repr` and `r`.
## Function `s(o, use_cls=False, use_repr=False, max_length=32)`
Return "type(o).__name__:str(o)" for some object `o`.
This is available as both `typed_str` and `s`.
Parameters:
* `use_cls`: default `False`;
if true, use `str(type(o))` instead of `type(o).__name__`
* `use_repr`: default `False`;
if true, use `repr(o)` instead of `str(o)`
I use this a lot when debugging. Example:
from cs.lex import typed_str as s
......
X("foo = %s", s(foo))
## Function `skipwhite(s, offset=0)`
Convenience routine for skipping past whitespace;
returns the offset of the next nonwhitespace character.
## Function `slosh_mapper(c, charmap=None)`
Return a string to replace backslash-`c`, or `None`.
## Function `snakecase(camelcased)`
Convert a camel cased string `camelcased` into snake case.
Parameters:
* `cameelcased`: the cameel case string to convert
* `first_letter_only`: optional flag (default `False`);
if true then just ensure that the first character of a word
is uppercased, otherwise use `str.title`
Example:
>>> snakecase('abcDef')
'abc_def'
>>> snakecase('abcDEf')
'abc_def'
>>> snakecase('AbcDef')
'abc_def'
## Function `split_remote_path(remotepath: str) -> Tuple[Optional[str], str]`
Split a path with an optional leading `[user@]rhost:` prefix
into the prefix and the remaining path.
`None` is returned for the prefix is there is none.
This is useful for things like `rsync` targets etc.
## Function `stripped_dedent(s)`
Slightly smarter dedent which ignores a string's opening indent.
Algorithm:
strip the supplied string `s`, pull off the leading line,
dedent the rest, put back the leading line.
This supports my preferred docstring layout, where the opening
line of text is on the same line as the opening quote.
Example:
>>> def func(s):
... """ Slightly smarter dedent which ignores a string's opening indent.
... Strip the supplied string `s`. Pull off the leading line.
... Dedent the rest. Put back the leading line.
... """
... pass
...
>>> from cs.lex import stripped_dedent
>>> print(stripped_dedent(func.__doc__))
Slightly smarter dedent which ignores a string's opening indent.
Strip the supplied string `s`. Pull off the leading line.
Dedent the rest. Put back the leading line.
## Function `strlist(ary, sep=', ')`
Convert an iterable to strings and join with `sep` (default `', '`).
## Function `tabpadding(padlen, tabsize=8, offset=0)`
Compute some spaces to use a tab padding at an offfset.
## Function `texthexify(bs, shiftin='[', shiftout=']', whitelist=None)`
Transcribe the bytes `bs` to text using compact text runs for
some common text values.
This can be reversed with the `untexthexify` function.
This is an ad doc format devised to be compact but also to
expose "text" embedded within to the eye. The original use
case was transcribing a binary directory entry format, where
the filename parts would be somewhat visible in the transcription.
The output is a string of hexadecimal digits for the encoded
bytes except for runs of values from the whitelist, which are
enclosed in the shiftin and shiftout markers and transcribed
as is. The default whitelist is values of the ASCII letters,
the decimal digits and the punctuation characters '_-+.,'.
The default shiftin and shiftout markers are '[' and ']'.
String objects converted with either `hexify` and `texthexify`
output strings may be freely concatenated and decoded with
`untexthexify`.
Example:
>>> texthexify(b'&^%&^%abcdefghi)(*)(*')
'265e25265e25[abcdefghi]29282a29282a'
Parameters:
* `bs`: the bytes to transcribe
* `shiftin`: Optional. The marker string used to indicate a shift to
direct textual transcription of the bytes, default: `'['`.
* `shiftout`: Optional. The marker string used to indicate a
shift from text mode back into hexadecimal transcription,
default `']'`.
* `whitelist`: an optional bytes or string object indicating byte
values which may be represented directly in text;
the default value is the ASCII letters, the decimal digits
and the punctuation characters `'_-+.,'`.
## Function `titleify_lc(value_lc)`
Translate `'-'` into `' '` and `'_'` translated into `'-'`,
then titlecased.
See also `lc_()`, which this reverses imperfectly.
## Function `typed_repr(o, max_length=None, *, use_cls=False)`
Like `typed_str` but using `repr` instead of `str`.
This is available as both `typed_repr` and `r`.
## Function `typed_str(o, use_cls=False, use_repr=False, max_length=32)`
Return "type(o).__name__:str(o)" for some object `o`.
This is available as both `typed_str` and `s`.
Parameters:
* `use_cls`: default `False`;
if true, use `str(type(o))` instead of `type(o).__name__`
* `use_repr`: default `False`;
if true, use `repr(o)` instead of `str(o)`
I use this a lot when debugging. Example:
from cs.lex import typed_str as s
......
X("foo = %s", s(foo))
## Function `unctrl(s, tabsize=8)`
Return the string `s` with `TAB`s expanded and control characters
replaced with printable representations.
## Function `untexthexify(s, shiftin='[', shiftout=']')`
Decode a textual representation of binary data into binary data.
This is the reverse of the `texthexify` function.
Outside of the `shiftin`/`shiftout` markers the binary data
are represented as hexadecimal. Within the markers the bytes
have the values of the ordinals of the characters.
Example:
>>> untexthexify('265e25265e25[abcdefghi]29282a29282a')
b'&^%&^%abcdefghi)(*)(*'
Parameters:
* `s`: the string containing the text representation.
* `shiftin`: Optional. The marker string commencing a sequence
of direct text transcription, default `'['`.
* `shiftout`: Optional. The marker string ending a sequence
of direct text transcription, default `']'`.
# Release Log
*Release 20240316*:
Fixed release upload artifacts.
*Release 20240211*:
New split_remote_path() function to recognise [[user@]host]:path.
*Release 20231018*:
New is_uc_identifier function.
*Release 20230401*:
Import update.
*Release 20230217.1*:
Fix package requirements.
*Release 20230217*:
* New get_prefix_n function to parse a numeric value preceeded by a prefix.
* Drop strip_prefix_n, get_prefix_n is more general and I had not got around to using strip_prefix_n yet - when I did, I ended up writing get_prefix_n.
*Release 20230210*:
* @has_format_attributes: new optional inherit parameter to inherit superclass (or other) format attributes, default False.
* New FNumericMixin, FFloat, FInt FormatableMixin subclasses like FStr - they add .localtime and .utctime formattable attributes.
*Release 20220918*:
typed_str(): crop the value part, default max_length=32, bugfix message cropping.
*Release 20220626*:
* Remove dependency on cs.py3, we've been Python 2 incompatible for a while.
* FormatableFormatter.format_field: promote None to FStr(None).
*Release 20220227*:
* typed_str,typed_repr: make max_length the first optional positional parameter, make other parameters keyword only.
* New camelcase() and snakecase() functions.
*Release 20211208*:
Docstring updates.
*Release 20210913*:
* FormatableFormatter.FORMAT_RE_ARG_NAME_s: strings commencing with digits now match \d+(\.\d+)[a-z]+, eg "02d".
* Alias typed_str as s and typed_repr as r.
* FormatableFormatter: new .format_mode thread local state object initially with strict=False, used to control whether unknown fields leave a placeholder or raise KeyError.
* FormatableFormatter.format_field: assorted fixes.
*Release 20210906*:
New strip_prefix_n() function to strip a leading `prefix` and numeric value `n` from the start of a string.
*Release 20210717*:
* Many many changes to FormatableMixin, FormatableFormatter and friends around supporting {foo|conv1|con2|...} instead of {foo!conv}. Still in flux.
* New typed_repr like typed_str but using repr.
*Release 20210306*:
* New cropped() function to crop strings.
* Rework cropped_repr() to do the repr() itself, and to crop the interiors of tuples and lists.
* cropped_repr: new inner_max_length for cropping the members of collections.
* cropped_repr: special case for length=1 tuples.
* New typed_str(o) object returning type(o).__name__:str(o) in the default case, useful for debugging.
*Release 20201228*:
Minor doc updates.
*Release 20200914*:
* Hide terribly special purpose lastlinelen() in cs.hier under a private name.
* New common_prefix and common_suffix function to compare strings.
*Release 20200718*:
get_chars: accept a callable for gochars, indicating a per character test function.
*Release 20200613*:
cropped_repr: replace hardwired 29 with computed length
*Release 20200517*:
* New get_ini_clausename to parse "[clausename]".
* New get_ini_clause_entryname parsing "[clausename]entryname".
* New cropped_repr for returning a shortened repr()+"..." if the length exceeds a threshold.
* New format_escape function to double {} characters to survive str.format.
*Release 20200318*:
* New lc_() function to lowercase and dash a string, new titleify_lc() to mostly reverse lc_().
* New format_as function, FormatableMixin and related FormatAsError.
*Release 20200229*:
New cutprefix and cutsuffix functions.
*Release 20190812*:
Fix bad slosh escapes in strings.
*Release 20190220*:
New function get_qstr_or_identifier.
*Release 20181108*:
new function get_decimal_or_float_value to read a decimal or basic float
*Release 20180815*:
No semantic changes; update some docstrings and clean some lint, fix a unit test.
*Release 20180810*:
* New get_decimal_value and get_hexadecimal_value functions.
* New stripped_dedent function, a slightly smarter textwrap.dedent.
*Release 20171231*:
New function get_decimal. Drop unused function dict2js.
*Release 20170904*:
Python 2/3 ports, move rfc2047 into new cs.rfc2047 module.
*Release 20160828*:
* Use "install_requires" instead of "requires" in DISTINFO.
* Discard str1(), pointless optimisation.
* unrfc2047: map _ to SPACE, improve exception handling.
* Add phpquote: quote a string for use in PHP code; add docstring to jsquote.
* Add is_identifier test.
* Add get_dotted_identifier.
* Add is_dotted_identifier.
* Add get_hexadecimal.
* Add skipwhite, convenince wrapper for get_white returning just the next offset.
* Assorted bugfixes and improvements.
*Release 20150120*:
cs.lex: texthexify: backport to python 2 using cs.py3 bytes type
*Release 20150118*:
metadata updates
*Release 20150116*:
PyPI metadata and slight code cleanup.
Raw data
{
"_id": null,
"home_page": "",
"name": "cs.lex",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "python2,python3",
"author": "",
"author_email": "Cameron Simpson <cs@cskk.id.au>",
"download_url": "https://files.pythonhosted.org/packages/82/a9/27445ea2dcd7cfa1d2b3f4d07f342e3e7d035048a5a24402e66e5db20c16/cs.lex-20240316.tar.gz",
"platform": null,
"description": "Lexical analysis functions, tokenisers, transcribers:\nan arbitrary assortment of lexical and tokenisation functions useful\nfor writing recursive descent parsers, of which I have several.\nThere are also some transcription functions for producing text\nfrom various objects, such as `hexify` and `unctrl`.\n\n*Latest release 20240316*:\nFixed release upload artifacts.\n\nGenerally the get_* functions accept a source string and an offset\n(usually optional, default `0`) and return a token and the new offset,\nraising `ValueError` on failed tokenisation.\n\n## Function `as_lines(chunks, partials=None)`\n\nGenerator yielding complete lines from arbitrary pieces of text from\nthe iterable of `str` `chunks`.\n\nAfter completion, any remaining newline-free chunks remain\nin the partials list; they will be unavailable to the caller\nunless the list is presupplied.\n\n## Function `camelcase(snakecased, first_letter_only=False)`\n\nConvert a snake cased string `snakecased` into camel case.\n\nParameters:\n* `snakecased`: the snake case string to convert\n* `first_letter_only`: optional flag (default `False`);\n if true then just ensure that the first character of a word\n is uppercased, otherwise use `str.title`\n\nExample:\n\n >>> camelcase('abc_def')\n 'abcDef'\n >>> camelcase('ABc_def')\n 'abcDef'\n >>> camelcase('abc_dEf')\n 'abcDef'\n >>> camelcase('abc_dEf', first_letter_only=True)\n 'abcDEf'\n\n## Function `common_prefix(*strs)`\n\nReturn the common prefix of the strings `strs`.\n\nExamples:\n\n >>> common_prefix('abc', 'def')\n ''\n >>> common_prefix('abc', 'abd')\n 'ab'\n >>> common_prefix('abc', 'abcdef')\n 'abc'\n >>> common_prefix('abc', 'abcdef', 'abz')\n 'ab'\n >>> # contrast with cs.fileutils.common_path_prefix\n >>> common_prefix('abc/def', 'abc/def1', 'abc/def2')\n 'abc/def'\n\n## Function `common_suffix(*strs)`\n\nReturn the common suffix of the strings `strs`.\n\n## Function `cropped(s: str, max_length: int = 32, roffset: int = 1, ellipsis: str = '...')`\n\nIf the length of `s` exceeds `max_length` (default `32`),\nreplace enough of the tail with `ellipsis`\nand the last `roffset` (default `1`) characters of `s`\nto fit in `max_length` characters.\n\n## Function `cropped_repr(o, roffset=1, max_length=32, inner_max_length=None)`\n\nCompute a cropped `repr()` of `o`.\n\nParameters:\n* `o`: the object to represent\n* `max_length`: the maximum length of the representation, default `32`\n* `inner_max_length`: the maximum length of the representations\n of members of `o`, default `max_length//2`\n* `roffset`: the number of trailing characters to preserve, default `1`\n\n## Function `cutprefix(s, prefix)`\n\nStrip a `prefix` from the front of `s`.\nReturn the suffix if `s.startswith(prefix)`, else `s`.\n\nExample:\n\n >>> abc_def = 'abc.def'\n >>> cutprefix(abc_def, 'abc.')\n 'def'\n >>> cutprefix(abc_def, 'zzz.')\n 'abc.def'\n >>> cutprefix(abc_def, '.zzz') is abc_def\n True\n\n## Function `cutsuffix(s, suffix)`\n\nStrip a `suffix` from the end of `s`.\nReturn the prefix if `s.endswith(suffix)`, else `s`.\n\nExample:\n\n >>> abc_def = 'abc.def'\n >>> cutsuffix(abc_def, '.def')\n 'abc'\n >>> cutsuffix(abc_def, '.zzz')\n 'abc.def'\n >>> cutsuffix(abc_def, '.zzz') is abc_def\n True\n\n## Class `FFloat(FNumericMixin, FormatableMixin, FormatableFormatter, string.Formatter, builtins.float)`\n\nFormattable `float`.\n\n## Class `FInt(FNumericMixin, FormatableMixin, FormatableFormatter, string.Formatter, builtins.int)`\n\nFormattable `int`.\n\n## Class `FNumericMixin(FormatableMixin, FormatableFormatter, string.Formatter)`\n\nA `FormatableMixin` subclass.\n\n## Function `format_as(format_s: str, format_mapping, formatter=None, error_sep=None, strict=None)`\n\nFormat the string `format_s` using `Formatter.vformat`,\nreturn the formatted result.\nThis is a wrapper for `str.format_map`\nwhich raises a more informative `FormatAsError` exception on failure.\n\nParameters:\n* `format_s`: the format string to use as the template\n* `format_mapping`: the mapping of available replacement fields\n* `formatter`: an optional `string.Formatter`-like instance\n with a `.vformat(format_string,args,kwargs)` method,\n usually a subclass of `string.Formatter`;\n if not specified then `FormatableFormatter` is used\n* `error_sep`: optional separator for the multipart error message,\n default from `FormatAsError.DEFAULT_SEPARATOR`:\n `'; '`\n* `strict`: optional flag (default `False`)\n indicating that an unresolveable field should raise a\n `KeyError` instead of inserting a placeholder\n\n## Function `format_attribute(method)`\n\nA decorator to mark a method as available as a format method.\nRequires the enclosing class to be decorated with `@has_format_attributes`.\n\nFor example,\nthe `FormatableMixin.json` method is defined like this:\n\n @format_attribute\n def json(self):\n return self.FORMAT_JSON_ENCODER.encode(self)\n\nwhich allows a `FormatableMixin` subclass instance\nto be used in a format string like this:\n\n {instance:json}\n\nto insert a JSON transcription of the instance.\n\nIt is recommended that methods marked with `@format_attribute`\nhave no side effects and do not modify state,\nas they are intended for use in ad hoc format strings\nsupplied by an end user.\n\n## Function `format_escape(s)`\n\nEscape `{}` characters in a string to protect them from `str.format`.\n\n## Function `format_recover(*da, **dkw)`\n\nDecorator for `__format__` methods which replaces failed formats\nwith `{self:format_spec}`.\n\n## Class `FormatableFormatter(string.Formatter)`\n\nA `string.Formatter` subclass interacting with objects\nwhich inherit from `FormatableMixin`.\n\n## Class `FormatableMixin(FormatableFormatter, string.Formatter)`\n\nA subclass of `FormatableFormatter` which provides 2 features:\n- a `__format__` method which parses the `format_spec` string\n into multiple colon separated terms whose results chain\n- a `format_as` method which formats a format string using `str.format_map`\n with a suitable mapping derived from the instance\n via its `format_kwargs` method\n (whose default is to return the instance itself)\n\nThe `format_as` method is like an inside out `str.format` or\n`object.__format__` method.\n\nThe `str.format` method is designed for formatting a string\nfrom a variety of other objects supplied in the keyword arguments.\n\nThe `object.__format__` method is for filling out a single `str.format`\nreplacement field from a single object.\n\nBy contrast, `format_as` is designed to fill out an entire format\nstring from the current object.\n\nFor example, the `cs.tagset.TagSetMixin` class\nuses `FormatableMixin` to provide a `format_as` method\nwhose replacement fields are derived from the tags in the tag set.\n\nSubclasses wanting to provide additional `format_spec` terms\nshould:\n- override `FormatableFormatter.format_field1` to implement\n terms with no colons, letting `format_field` do the split into terms\n- override `FormatableFormatter.get_format_subspecs` to implement\n the parse of `format_spec` into a sequence of terms.\n This might recognise a special additional syntax\n and quietly fall back to `super().get_format_subspecs`\n if that is not present.\n\n## Class `FormatAsError(builtins.LookupError, builtins.Exception, builtins.BaseException)`\n\nSubclass of `LookupError` for use by `format_as`.\n\n## Class `FStr(FormatableMixin, FormatableFormatter, string.Formatter, builtins.str)`\n\nA `str` subclass with the `FormatableMixin` methods,\nparticularly its `__format__` method\nwhich uses `str` method names as valid formats.\n\nIt also has a bunch of utility methods which are available\nas `:`*method* in format strings.\n\n## Function `get_chars(s, offset, gochars)`\n\nScan the string `s` for characters in `gochars` starting at `offset`.\nReturn `(match,new_offset)`.\n\n`gochars` may also be a callable, in which case a character\n`ch` is accepted if `gochars(ch)` is true.\n\n## Function `get_decimal(s, offset=0)`\n\nScan the string `s` for decimal characters starting at `offset` (default `0`).\nReturn `(dec_string,new_offset)`.\n\n## Function `get_decimal_or_float_value(s, offset=0)`\n\nFetch a decimal or basic float (nnn.nnn) value\nfrom the str `s` at `offset` (default `0`).\nReturn `(value,new_offset)`.\n\n## Function `get_decimal_value(s, offset=0)`\n\nScan the string `s` for a decimal value starting at `offset` (default `0`).\nReturn `(value,new_offset)`.\n\n## Function `get_delimited(s, offset, delim)`\n\nCollect text from the string `s` from position `offset` up\nto the first occurence of delimiter `delim`; return the text\nexcluding the delimiter and the offset after the delimiter.\n\n## Function `get_dotted_identifier(s, offset=0, **kw)`\n\nScan the string `s` for a dotted identifier (by default an\nASCII letter or underscore followed by letters, digits or\nunderscores) with optional trailing dot and another dotted\nidentifier, starting at `offset` (default `0`).\nReturn `(match,new_offset)`.\n\nNote: the empty string and an unchanged offset will be returned if\nthere is no leading letter/underscore.\n\nKeyword arguments are passed to `get_identifier`\n(used for each component of the dotted identifier).\n\n## Function `get_envvar(s, offset=0, environ=None, default=None, specials=None)`\n\nParse a simple environment variable reference to $varname or\n$x where \"x\" is a special character.\n\nParameters:\n* `s`: the string with the variable reference\n* `offset`: the starting point for the reference\n* `default`: default value for missing environment variables;\n if `None` (the default) a `ValueError` is raised\n* `environ`: the environment mapping, default `os.environ`\n* `specials`: the mapping of special single character variables\n\n## Function `get_hexadecimal(s, offset=0)`\n\nScan the string `s` for hexadecimal characters starting at `offset` (default `0`).\nReturn `(hex_string,new_offset)`.\n\n## Function `get_hexadecimal_value(s, offset=0)`\n\nScan the string `s` for a hexadecimal value starting at `offset` (default `0`).\nReturn `(value,new_offset)`.\n\n## Function `get_identifier(s, offset=0, alpha='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ', number='0123456789', extras='_')`\n\nScan the string `s` for an identifier (by default an ASCII\nletter or underscore followed by letters, digits or underscores)\nstarting at `offset` (default 0).\nReturn `(match,new_offset)`.\n\n*Note*: the empty string and an unchanged offset will be returned if\nthere is no leading letter/underscore.\n\nParameters:\n* `s`: the string to scan\n* `offset`: the starting offset, default `0`.\n* `alpha`: the characters considered alphabetic,\n default `string.ascii_letters`.\n* `number`: the characters considered numeric,\n default `string.digits`.\n* `extras`: extra characters considered part of an identifier,\n default `'_'`.\n\n## Function `get_ini_clause_entryname(s, offset=0)`\n\nParse a `[`*clausename*`]`*entryname* string\nfrom `s` at `offset` (default `0`).\nReturn `(clausename,entryname,new_offset)`.\n\n## Function `get_ini_clausename(s, offset=0)`\n\nParse a `[`*clausename*`]` string from `s` at `offset` (default `0`).\nReturn `(clausename,new_offset)`.\n\n## Function `get_nonwhite(s, offset=0)`\n\nScan the string `s` for characters not in `string.whitespace`\nstarting at `offset` (default `0`).\nReturn `(match,new_offset)`.\n\n## Function `get_other_chars(s, offset=0, stopchars=None)`\n\nScan the string `s` for characters not in `stopchars` starting\nat `offset` (default `0`).\nReturn `(match,new_offset)`.\n\n## Function `get_prefix_n(s, prefix, n=None, *, offset=0)`\n\nStrip a leading `prefix` and numeric value `n` from the string `s`\nstarting at `offset` (default `0`).\nReturn the matched prefix, the numeric value and the new offset.\nReturns `(None,None,offset)` on no match.\n\nParameters:\n* `s`: the string to parse\n* `prefix`: the prefix string which must appear at `offset`\n or an object with a `match(str,offset)` method\n such as an `re.Pattern` regexp instance\n* `n`: optional integer value;\n if omitted any value will be accepted, otherwise the numeric\n part must match `n`\n\nIf `prefix` is a `str`, the \"matched prefix\" return value is `prefix`.\nOtherwise the \"matched prefix\" return value is the result of\nthe `prefix.match(s,offset)` call. The result must also support\na `.end()` method returning the offset in `s` beyond the match,\nused to locate the following numeric portion.\n\nExamples:\n\n >>> import re\n >>> get_prefix_n('s03e01--', 's')\n ('s', 3, 3)\n >>> get_prefix_n('s03e01--', 's', 3)\n ('s', 3, 3)\n >>> get_prefix_n('s03e01--', 's', 4)\n (None, None, 0)\n >>> get_prefix_n('s03e01--', re.compile('[es]',re.I))\n (<re.Match object; span=(0, 1), match='s'>, 3, 3)\n >>> get_prefix_n('s03e01--', re.compile('[es]',re.I), offset=3)\n (<re.Match object; span=(3, 4), match='e'>, 1, 6)\n\n## Function `get_qstr(s, offset=0, q='\"', environ=None, default=None, env_specials=None)`\n\nGet quoted text with slosh escapes and optional environment substitution.\n\nParameters:\n* `s`: the string containg the quoted text.\n* `offset`: the starting point, default `0`.\n* `q`: the quote character, default `'\"'`. If `q` is `None`,\n do not expect the string to be delimited by quote marks.\n* `environ`: if not `None`, also parse and expand `$`*envvar* references.\n* `default`: passed to `get_envvar`\n\n## Function `get_qstr_or_identifier(s, offset)`\n\nParse a double quoted string or an identifier.\n\n## Function `get_sloshed_text(s, delim, offset=0, slosh='\\\\', mapper=<function slosh_mapper at 0x104903eb0>, specials=None)`\n\nCollect slosh escaped text from the string `s` from position\n`offset` (default `0`) and return the decoded unicode string and\nthe offset of the completed parse.\n\nParameters:\n* `delim`: end of string delimiter, such as a single or double quote.\n* `offset`: starting offset within `s`, default `0`.\n* `slosh`: escape character, default a slosh ('\\').\n* `mapper`: a mapping function which accepts a single character\n and returns a replacement string or `None`; this is used the\n replace things such as '\\t' or '\\n'. The default is the\n `slosh_mapper` function, whose default mapping is `SLOSH_CHARMAP`.\n* `specials`: a mapping of other special character sequences and parse\n functions for gathering them up. When one of the special\n character sequences is found in the string, the parse\n function is called to parse at that point.\n The parse functions accept\n `s` and the offset of the special character. They return\n the decoded string and the offset past the parse.\n\nThe escape character `slosh` introduces an encoding of some\nreplacement text whose value depends on the following character.\nIf the following character is:\n* the escape character `slosh`, insert the escape character.\n* the string delimiter `delim`, insert the delimiter.\n* the character 'x', insert the character with code from the following\n 2 hexadecimal digits.\n* the character 'u', insert the character with code from the following\n 4 hexadecimal digits.\n* the character 'U', insert the character with code from the following\n 8 hexadecimal digits.\n* a character from the keys of `mapper`\n\n## Function `get_tokens(s, offset, getters)`\n\nParse the string `s` from position `offset` using the supplied\ntokeniser functions `getters`.\nReturn the list of tokens matched and the final offset.\n\nParameters:\n* `s`: the string to parse.\n* `offset`: the starting position for the parse.\n* `getters`: an iterable of tokeniser specifications.\n\nEach tokeniser specification `getter` is either:\n* a callable expecting `(s,offset)` and returning `(token,new_offset)`\n* a literal string, to be matched exactly\n* a `tuple` or `list` with values `(func,args,kwargs)`;\n call `func(s,offset,*args,**kwargs)`\n* an object with a `.match` method such as a regex;\n call `getter.match(s,offset)` and return a match object with\n a `.end()` method returning the offset of the end of the match\n\n## Function `get_uc_identifier(s, offset=0, number='0123456789', extras='_')`\n\nScan the string `s` for an identifier as for `get_identifier`,\nbut require the letters to be uppercase.\n\n## Function `get_white(s, offset=0)`\n\nScan the string `s` for characters in `string.whitespace`\nstarting at `offset` (default `0`).\nReturn `(match,new_offset)`.\n\n## Function `has_format_attributes(*da, **dkw)`\n\nClass decorator to walk this class for direct methods\nmarked as for use in format strings\nand to include them in `cls.format_attributes()`.\n\nMethods are normally marked with the `@format_attribute` decorator.\n\nIf `inherit` is true the base format attributes will be\nobtained from other classes:\n* `inherit` is `True`: use `cls.__mro__`\n* `inherit` is a class: use that class\n* otherwise assume `inherit` is an iterable of classes\nFor each class `otherclass`, update the initial attribute\nmapping from `otherclass.get_format_attributes()`.\n\n## Function `hexify(bs)`\n\nA flavour of `binascii.hexlify` returning a `str`.\n\n## Function `htmlify(s, nbsp=False)`\n\nConvert a string for safe transcription in HTML.\n\nParameters:\n* `s`: the string\n* `nbsp`: replaces spaces with `\" \"` to prevent word folding,\n default `False`.\n\n## Function `htmlquote(s)`\n\nQuote a string for use in HTML.\n\n## Function `is_dotted_identifier(s, offset=0, **kw)`\n\nTest if the string `s` is an identifier from position `offset` onward.\n\n## Function `is_identifier(s, offset=0, **kw)`\n\nTest if the string `s` is an identifier\nfrom position `offset` (default `0`) onward.\n\n## Function `is_uc_identifier(s, offset=0, **kw)`\n\nTest if the string `s` is an uppercase identifier\nfrom position `offset` (default `0`) onward.\n\n## Function `isUC_(s)`\n\nCheck that a string matches the regular expression `^[A-Z][A-Z_0-9]*$`.\n\n## Function `jsquote(s)`\n\nQuote a string for use in JavaScript.\n\n## Function `lc_(value)`\n\nReturn `value.lower()`\nwith `'-'` translated into `'_'` and `' '` translated into `'-'`.\n\nI use this to construct lowercase filenames containing a\nreadable transcription of a title string.\n\nSee also `titleify_lc()`, an imperfect reversal of this.\n\n## Function `match_tokens(s, offset, getters)`\n\nWrapper for `get_tokens` which catches `ValueError` exceptions\nand returns `(None,offset)`.\n\n## Function `parseUC_sAttr(attr)`\n\nTake an attribute name `attr` and return `(key,is_plural)`.\n\nExamples:\n* `'FOO'` returns `('FOO',False)`.\n* `'FOOs'` or `'FOOes'` returns `('FOO',True)`.\nOtherwise return `(None,False)`.\n\n## Function `phpquote(s)`\n\nQuote a string for use in PHP code.\n\n## Function `r(o, max_length=None, *, use_cls=False)`\n\nLike `typed_str` but using `repr` instead of `str`.\nThis is available as both `typed_repr` and `r`.\n\n## Function `s(o, use_cls=False, use_repr=False, max_length=32)`\n\nReturn \"type(o).__name__:str(o)\" for some object `o`.\nThis is available as both `typed_str` and `s`.\n\nParameters:\n* `use_cls`: default `False`;\n if true, use `str(type(o))` instead of `type(o).__name__`\n* `use_repr`: default `False`;\n if true, use `repr(o)` instead of `str(o)`\n\nI use this a lot when debugging. Example:\n\n from cs.lex import typed_str as s\n ......\n X(\"foo = %s\", s(foo))\n\n## Function `skipwhite(s, offset=0)`\n\nConvenience routine for skipping past whitespace;\nreturns the offset of the next nonwhitespace character.\n\n## Function `slosh_mapper(c, charmap=None)`\n\nReturn a string to replace backslash-`c`, or `None`.\n\n## Function `snakecase(camelcased)`\n\nConvert a camel cased string `camelcased` into snake case.\n\nParameters:\n* `cameelcased`: the cameel case string to convert\n* `first_letter_only`: optional flag (default `False`);\n if true then just ensure that the first character of a word\n is uppercased, otherwise use `str.title`\n\nExample:\n\n >>> snakecase('abcDef')\n 'abc_def'\n >>> snakecase('abcDEf')\n 'abc_def'\n >>> snakecase('AbcDef')\n 'abc_def'\n\n## Function `split_remote_path(remotepath: str) -> Tuple[Optional[str], str]`\n\nSplit a path with an optional leading `[user@]rhost:` prefix\ninto the prefix and the remaining path.\n`None` is returned for the prefix is there is none.\nThis is useful for things like `rsync` targets etc.\n\n## Function `stripped_dedent(s)`\n\nSlightly smarter dedent which ignores a string's opening indent.\n\nAlgorithm:\nstrip the supplied string `s`, pull off the leading line,\ndedent the rest, put back the leading line.\n\nThis supports my preferred docstring layout, where the opening\nline of text is on the same line as the opening quote.\n\nExample:\n\n >>> def func(s):\n ... \"\"\" Slightly smarter dedent which ignores a string's opening indent.\n ... Strip the supplied string `s`. Pull off the leading line.\n ... Dedent the rest. Put back the leading line.\n ... \"\"\"\n ... pass\n ...\n >>> from cs.lex import stripped_dedent\n >>> print(stripped_dedent(func.__doc__))\n Slightly smarter dedent which ignores a string's opening indent.\n Strip the supplied string `s`. Pull off the leading line.\n Dedent the rest. Put back the leading line.\n\n## Function `strlist(ary, sep=', ')`\n\nConvert an iterable to strings and join with `sep` (default `', '`).\n\n## Function `tabpadding(padlen, tabsize=8, offset=0)`\n\nCompute some spaces to use a tab padding at an offfset.\n\n## Function `texthexify(bs, shiftin='[', shiftout=']', whitelist=None)`\n\nTranscribe the bytes `bs` to text using compact text runs for\nsome common text values.\n\nThis can be reversed with the `untexthexify` function.\n\nThis is an ad doc format devised to be compact but also to\nexpose \"text\" embedded within to the eye. The original use\ncase was transcribing a binary directory entry format, where\nthe filename parts would be somewhat visible in the transcription.\n\nThe output is a string of hexadecimal digits for the encoded\nbytes except for runs of values from the whitelist, which are\nenclosed in the shiftin and shiftout markers and transcribed\nas is. The default whitelist is values of the ASCII letters,\nthe decimal digits and the punctuation characters '_-+.,'.\nThe default shiftin and shiftout markers are '[' and ']'.\n\nString objects converted with either `hexify` and `texthexify`\noutput strings may be freely concatenated and decoded with\n`untexthexify`.\n\nExample:\n\n >>> texthexify(b'&^%&^%abcdefghi)(*)(*')\n '265e25265e25[abcdefghi]29282a29282a'\n\nParameters:\n* `bs`: the bytes to transcribe\n* `shiftin`: Optional. The marker string used to indicate a shift to\n direct textual transcription of the bytes, default: `'['`.\n* `shiftout`: Optional. The marker string used to indicate a\n shift from text mode back into hexadecimal transcription,\n default `']'`.\n* `whitelist`: an optional bytes or string object indicating byte\n values which may be represented directly in text;\n the default value is the ASCII letters, the decimal digits\n and the punctuation characters `'_-+.,'`.\n\n## Function `titleify_lc(value_lc)`\n\nTranslate `'-'` into `' '` and `'_'` translated into `'-'`,\nthen titlecased.\n\nSee also `lc_()`, which this reverses imperfectly.\n\n## Function `typed_repr(o, max_length=None, *, use_cls=False)`\n\nLike `typed_str` but using `repr` instead of `str`.\nThis is available as both `typed_repr` and `r`.\n\n## Function `typed_str(o, use_cls=False, use_repr=False, max_length=32)`\n\nReturn \"type(o).__name__:str(o)\" for some object `o`.\nThis is available as both `typed_str` and `s`.\n\nParameters:\n* `use_cls`: default `False`;\n if true, use `str(type(o))` instead of `type(o).__name__`\n* `use_repr`: default `False`;\n if true, use `repr(o)` instead of `str(o)`\n\nI use this a lot when debugging. Example:\n\n from cs.lex import typed_str as s\n ......\n X(\"foo = %s\", s(foo))\n\n## Function `unctrl(s, tabsize=8)`\n\nReturn the string `s` with `TAB`s expanded and control characters\nreplaced with printable representations.\n\n## Function `untexthexify(s, shiftin='[', shiftout=']')`\n\nDecode a textual representation of binary data into binary data.\n\nThis is the reverse of the `texthexify` function.\n\nOutside of the `shiftin`/`shiftout` markers the binary data\nare represented as hexadecimal. Within the markers the bytes\nhave the values of the ordinals of the characters.\n\nExample:\n\n >>> untexthexify('265e25265e25[abcdefghi]29282a29282a')\n b'&^%&^%abcdefghi)(*)(*'\n\nParameters:\n* `s`: the string containing the text representation.\n* `shiftin`: Optional. The marker string commencing a sequence\n of direct text transcription, default `'['`.\n* `shiftout`: Optional. The marker string ending a sequence\n of direct text transcription, default `']'`.\n\n# Release Log\n\n\n\n*Release 20240316*:\nFixed release upload artifacts.\n\n*Release 20240211*:\nNew split_remote_path() function to recognise [[user@]host]:path.\n\n*Release 20231018*:\nNew is_uc_identifier function.\n\n*Release 20230401*:\nImport update.\n\n*Release 20230217.1*:\nFix package requirements.\n\n*Release 20230217*:\n* New get_prefix_n function to parse a numeric value preceeded by a prefix.\n* Drop strip_prefix_n, get_prefix_n is more general and I had not got around to using strip_prefix_n yet - when I did, I ended up writing get_prefix_n.\n\n*Release 20230210*:\n* @has_format_attributes: new optional inherit parameter to inherit superclass (or other) format attributes, default False.\n* New FNumericMixin, FFloat, FInt FormatableMixin subclasses like FStr - they add .localtime and .utctime formattable attributes.\n\n*Release 20220918*:\ntyped_str(): crop the value part, default max_length=32, bugfix message cropping.\n\n*Release 20220626*:\n* Remove dependency on cs.py3, we've been Python 2 incompatible for a while.\n* FormatableFormatter.format_field: promote None to FStr(None).\n\n*Release 20220227*:\n* typed_str,typed_repr: make max_length the first optional positional parameter, make other parameters keyword only.\n* New camelcase() and snakecase() functions.\n\n*Release 20211208*:\nDocstring updates.\n\n*Release 20210913*:\n* FormatableFormatter.FORMAT_RE_ARG_NAME_s: strings commencing with digits now match \\d+(\\.\\d+)[a-z]+, eg \"02d\".\n* Alias typed_str as s and typed_repr as r.\n* FormatableFormatter: new .format_mode thread local state object initially with strict=False, used to control whether unknown fields leave a placeholder or raise KeyError.\n* FormatableFormatter.format_field: assorted fixes.\n\n*Release 20210906*:\nNew strip_prefix_n() function to strip a leading `prefix` and numeric value `n` from the start of a string.\n\n*Release 20210717*:\n* Many many changes to FormatableMixin, FormatableFormatter and friends around supporting {foo|conv1|con2|...} instead of {foo!conv}. Still in flux.\n* New typed_repr like typed_str but using repr.\n\n*Release 20210306*:\n* New cropped() function to crop strings.\n* Rework cropped_repr() to do the repr() itself, and to crop the interiors of tuples and lists.\n* cropped_repr: new inner_max_length for cropping the members of collections.\n* cropped_repr: special case for length=1 tuples.\n* New typed_str(o) object returning type(o).__name__:str(o) in the default case, useful for debugging.\n\n*Release 20201228*:\nMinor doc updates.\n\n*Release 20200914*:\n* Hide terribly special purpose lastlinelen() in cs.hier under a private name.\n* New common_prefix and common_suffix function to compare strings.\n\n*Release 20200718*:\nget_chars: accept a callable for gochars, indicating a per character test function.\n\n*Release 20200613*:\ncropped_repr: replace hardwired 29 with computed length\n\n*Release 20200517*:\n* New get_ini_clausename to parse \"[clausename]\".\n* New get_ini_clause_entryname parsing \"[clausename]entryname\".\n* New cropped_repr for returning a shortened repr()+\"...\" if the length exceeds a threshold.\n* New format_escape function to double {} characters to survive str.format.\n\n*Release 20200318*:\n* New lc_() function to lowercase and dash a string, new titleify_lc() to mostly reverse lc_().\n* New format_as function, FormatableMixin and related FormatAsError.\n\n*Release 20200229*:\nNew cutprefix and cutsuffix functions.\n\n*Release 20190812*:\nFix bad slosh escapes in strings.\n\n*Release 20190220*:\nNew function get_qstr_or_identifier.\n\n*Release 20181108*:\nnew function get_decimal_or_float_value to read a decimal or basic float\n\n*Release 20180815*:\nNo semantic changes; update some docstrings and clean some lint, fix a unit test.\n\n*Release 20180810*:\n* New get_decimal_value and get_hexadecimal_value functions.\n* New stripped_dedent function, a slightly smarter textwrap.dedent.\n\n*Release 20171231*:\nNew function get_decimal. Drop unused function dict2js.\n\n*Release 20170904*:\nPython 2/3 ports, move rfc2047 into new cs.rfc2047 module.\n\n*Release 20160828*:\n* Use \"install_requires\" instead of \"requires\" in DISTINFO.\n* Discard str1(), pointless optimisation.\n* unrfc2047: map _ to SPACE, improve exception handling.\n* Add phpquote: quote a string for use in PHP code; add docstring to jsquote.\n* Add is_identifier test.\n* Add get_dotted_identifier.\n* Add is_dotted_identifier.\n* Add get_hexadecimal.\n* Add skipwhite, convenince wrapper for get_white returning just the next offset.\n* Assorted bugfixes and improvements.\n\n*Release 20150120*:\ncs.lex: texthexify: backport to python 2 using cs.py3 bytes type\n\n*Release 20150118*:\nmetadata updates\n\n*Release 20150116*:\nPyPI metadata and slight code cleanup.\n\n",
"bugtrack_url": null,
"license": "GNU General Public License v3 or later (GPLv3+)",
"summary": "Lexical analysis functions, tokenisers, transcribers: an arbitrary assortment of lexical and tokenisation functions useful for writing recursive descent parsers, of which I have several. There are also some transcription functions for producing text from various objects, such as `hexify` and `unctrl`.",
"version": "20240316",
"project_urls": {
"URL": "https://bitbucket.org/cameron_simpson/css/commits/all"
},
"split_keywords": [
"python2",
"python3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "759ec3173944ef3b1cf5b0b085f652ec9e0faa85aba1cd876674d622f87222cf",
"md5": "4e7de3356d1ffe3f0d604cccd69e4416",
"sha256": "1b04dfa4585d26bcb48b1e0427fefb3affdaea68e2129e323f1a715f0c3f1582"
},
"downloads": -1,
"filename": "cs.lex-20240316-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4e7de3356d1ffe3f0d604cccd69e4416",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 27535,
"upload_time": "2024-03-16T06:59:25",
"upload_time_iso_8601": "2024-03-16T06:59:25.846505Z",
"url": "https://files.pythonhosted.org/packages/75/9e/c3173944ef3b1cf5b0b085f652ec9e0faa85aba1cd876674d622f87222cf/cs.lex-20240316-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "82a927445ea2dcd7cfa1d2b3f4d07f342e3e7d035048a5a24402e66e5db20c16",
"md5": "702250d0beddc21b42b153ad873fae5b",
"sha256": "56e622ad693c7d44fb82f1924764fb822463dd4c0ea99bbf89ef2b0b23f2476e"
},
"downloads": -1,
"filename": "cs.lex-20240316.tar.gz",
"has_sig": false,
"md5_digest": "702250d0beddc21b42b153ad873fae5b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 50394,
"upload_time": "2024-03-16T06:59:28",
"upload_time_iso_8601": "2024-03-16T06:59:28.496346Z",
"url": "https://files.pythonhosted.org/packages/82/a9/27445ea2dcd7cfa1d2b3f4d07f342e3e7d035048a5a24402e66e5db20c16/cs.lex-20240316.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-16 06:59:28",
"github": false,
"gitlab": false,
"bitbucket": true,
"codeberg": false,
"bitbucket_user": "cameron_simpson",
"bitbucket_project": "css",
"lcname": "cs.lex"
}
