DictOfSeries

class dios.DictOfSeries(data=None, columns=None, index=None, itype=None, cast_policy='save', fastpath=False)

Bases: dios.base._DiosBase

A data frame where every column has its own index.

DictOfSeries is a collection of pd.Series’s which aim to be as close as possible similar to pd.DataFrame. The advantage over pd.DataFrame is, that every column has its own row-index, unlike the former, which provide a single row-index for all columns. This solves problems with unaligned data and data which varies widely in length.

Indexing with di[], di.loc[] and di.iloc[] should work analogous to these methods from pd.DataFrame. The indexer can be a single label, a slice, a list-like, a boolean list-like, or a boolean DictOfSeries/pd.DataFrame and can be used to selectively get or set data.

Parameters

• data (array-like, Iterable, dict, or scalar value) – Contains data stored in Series.

• columns (array-like) – Column labels to use for resulting frame. Will default to RangeIndex(0, 1, 2, …, n) if no column labels are provided.

• index (Index or array-like) – Index to use to reindex every given series during init. Ignored if omitted.

• itype (Itype, pd.Index, Itype-string-repr or type) – Every series that is inserted, must have an index of this type or any of this types subtypes. If None, the itype is inferred as soon as the first non-empty series is inserted.

• cast_policy ({'save', 'force', 'never'}, default 'save') – Policy used for (down-)casting the index of a series if its type does not match the itype.

Attributes Summary

aloc	Access a group of rows and columns by label(s) or a boolean array with automatic alignment of indexers.
at	Access a single value for a row/column label pair.
cast_policy	The policy to use for casting new columns if its initial itype does not fit.
columns	The column labels of the DictOfSeries
debugDf	Alias for to_df() as property, for debugging purpose.
dtypes	Return pandas.Series with the dtypes of all columns.
empty	Indicator whether DictOfSeries is empty.
iat	Access a single value for a row/column pair by integer position.
iloc	Purely integer-location based indexing for selection by position.
indexes	Return pandas.Series with the indexes of all columns.
itype	The Itype of the DictOfSeries.
lengths	Return pandas.Series with the lenght of all columns.
loc	Access a group of rows and columns by label(s) or a boolean array.
size
values	Return a numpy.array of numpy.arrays with the values of all columns.

Methods Summary

all([axis])	Return whether all elements are True, potentially over an axis.
any([axis])	Return whether any element is True, potentially over an axis.
apply(func[, axis, raw, args])	Apply a function along an axis of the DictOfSeries.
astype(dtype[, copy, errors])	Cast the data to the given data type.
clear()
copy([deep])	Make a copy of this DictOfSeries’ indices and data.
copy_empty([columns])	Return a new DictOfSeries object, with same properties than the original.
dropempty()	Drop empty columns.
dropna([inplace])	Return a bolean array that is True if the value is a Nan-value
equals(other)
for_each(attr_or_callable, **kwds)	Apply a callable or a pandas.Series method or property on each column.
get(key[, default])
hasnans([axis, drop_empty])	Returns a boolean Series along an axis, which indicates if it contains NA-entries.
index_of([method])	Return an single index with indices from all columns.
isdata()	Alias for notna(drop_empty=True).
isempty()	Returns a boolean Series, which indicates if an column is empty
isin(values)	Return a boolean dios, that indicates if the corresponding value is in the given array-like.
isna([drop_empty])	Return a boolean DictOfSeries which indicates NA positions.
isnull([drop_empty])	Alias for isna()
items()
iteritems()
iterrows([fill_value, squeeze])	Iterate over DictOfSeries rows as (index, pandas.Series/DictOfSeries) pairs.
keys()
max([axis, skipna])
memory_usage([index, deep])
min([axis, skipna])
notempty()	Returns a boolean Series, which indicates if an column is not empty
notna([drop_empty])	Return a boolean DictOfSeries which indicates non-NA positions.
notnull([drop_empty])	Alias, see notna().
pop(*args)
popitem()
reduce_columns(func[, initial, skipna])	Reduce all columns to a single pandas.Series by a given function.
setdefault(key[, default])
squeeze([axis])	Squeeze a 1-dimensional axis objects into scalars.
to_csv(*args, **kwargs)	Write object to a comma-separated values (csv) file.
to_df()	Transform DictOfSeries to a pandas.DataFrame.
to_string([max_rows, min_rows, max_cols, …])	Pretty print a dios.
update(other)

Attributes Documentation

aloc

Access a group of rows and columns by label(s) or a boolean array with automatic alignment of indexers.

See indexing docs

at

Access a single value for a row/column label pair.

See indexing docs

cast_policy

The policy to use for casting new columns if its initial itype does not fit.

See Itype documentation for more info.

columns

The column labels of the DictOfSeries

debugDf

Alias for to_df() as property, for debugging purpose.

dtypes

Return pandas.Series with the dtypes of all columns.

empty

Indicator whether DictOfSeries is empty.

Returns

If DictOfSeries is empty, return True, if not return False.

Return type

bool

See also

DictOfSeries.dropempty

drop empty columns

DictOfSeries.dropna

drop NAN’s from a DictOfSeries

pandas.Series.dropna

drop NAN’s from a Series

Notes

If DictOfSeries contains only NaNs, it is still not considered empty. See the example below.

Examples

An example of an actual empty DictOfSeries.

>>> di_empty = DictOfSeries(columns=['A'])
>>> di_empty
Empty DictOfSeries
Columns: ['A']
>>> di_empty.empty
True

If we only have NaNs in our DictOfSeries, it is not considered empty! We will need to drop the NaNs to make the DictOfSeries empty:

>>> di = pd.DictOfSeries({'A' : [np.nan]})
>>> di
    A |
===== |
0 NaN |
>>> di.empty
False
>>> di.dropna().empty
True

iat

Access a single value for a row/column pair by integer position.

See indexing docs

iloc

Purely integer-location based indexing for selection by position.

See indexing docs

indexes

Return pandas.Series with the indexes of all columns.

itype

The Itype of the DictOfSeries.

See Itype documentation for more info.

lengths

Return pandas.Series with the lenght of all columns.

loc

Access a group of rows and columns by label(s) or a boolean array.

See indexing docs

size

values

Return a numpy.array of numpy.arrays with the values of all columns.

The outer has the length of columns, the inner holds the values of the column.

Methods Documentation

all(axis=0)

Return whether all elements are True, potentially over an axis.

Returns True unless there at least one element within a series or along a DictOfSeries axis that is False or equivalent (e.g. zero or empty).

Parameters

axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) –

Indicate which axis or axes should be reduced.

• 0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.

• 1 / ‘columns’ : reduce the columns, return a Series whose index is the union of all columns indexes.

• None : reduce all axes, return a scalar.

Returns

Return type

pandas.Series

See also

pandas.Series.all()

Return True if all elements are True.

any()

Return True if one (or more) elements are True.

any(axis=0)

Return whether any element is True, potentially over an axis.

Returns False unless there at least one element within a series or along a DictOfSeries axis that is True or equivalent (e.g. non-zero or non-empty).

Parameters

axis ({0 or ‘index’, 1 or ‘columns’, None}, default 0) –

Indicate which axis or axes should be reduced.

• 0 / ‘index’ : reduce the index, return a Series whose index is the original column labels.

• 1 / ‘columns’ : reduce the columns, return a Series whose index is the union of all columns indexes.

• None : reduce all axes, return a scalar.

Returns

Return type

pandas.Series

See also

pandas.Series.any()

Return whether any element is True.

all()

Return True if all elements are True.

apply(func, axis=0, raw=False, args=(), **kwds)

Apply a function along an axis of the DictOfSeries.

Parameters

• func (callable) – Function to apply on each column.

• axis ({0 or 'index', 1 or 'columns'}, default 0) –

  Axis along which the function is applied:

  • 0 or ‘index’: apply function to each column.

  • 1 or ‘columns’: NOT IMPLEMENTED

• raw (bool, default False) –

  Determines if row or column is passed as a Series or ndarray object:

  • False : passes each row or column as a Series to the function.

  • True : the passed function will receive ndarray objects instead. If you are just applying a NumPy reduction function this will achieve much better performance.

• args (tuple) – Positional arguments to pass to func in addition to the array/series.

• **kwds – Additional keyword arguments to pass as keywords arguments to func.

Returns

Result of applying func along the given axis of the DataFrame.

Return type

Series or DataFrame

Raises

NotImplementedError –

• if axis is ‘columns’ or 1

See also

DictOfSeries.for_each()

apply pd.Series methods or properties to each column

Examples

We use the example DictOfSeries from indexing.

>>> di = di[:5]
    a |    b |     c |     d |
===== | ==== | ===== | ===== |
0   0 | 2  5 | 4   7 | 6   0 |
1   7 | 3  6 | 5  17 | 7   1 |
2  14 | 4  7 | 6  27 | 8   2 |
3  21 | 5  8 | 7  37 | 9   3 |
4  28 | 6  9 | 8  47 | 10  4 |

>>> di.apply(max)
columns
a    28
b     9
c    47
d     4
dtype: int64

>>> di.apply(pd.Series.count)
columns
a    5
b    5
c    5
d    5
dtype: int64

One can pass keyword arguments directly..

>>> di.apply(pd.Series.value_counts, normalize=True)
      a |      b |       c |      d |
======= | ====== | ======= | ====== |
7   0.2 | 7  0.2 | 7   0.2 | 4  0.2 |
14  0.2 | 6  0.2 | 37  0.2 | 3  0.2 |
21  0.2 | 5  0.2 | 47  0.2 | 2  0.2 |
28  0.2 | 9  0.2 | 27  0.2 | 1  0.2 |
0   0.2 | 8  0.2 | 17  0.2 | 0  0.2 |

Or define a own funtion..

>>> di.apply(lambda s : 'high' if max(s) > 10 else 'low')
columns
a    high
b     low
c    high
d     low
dtype: object

And also more advanced functions that return a list-like can be given. Note that the returned lists not necessarily must have the same length.

>>> func = lambda s : ('high', max(s), min(s)) if min(s) > (max(s)//2) else ('low',max(s))
>>> di.apply(func)
     a |       b |      c |      d |
====== | ======= | ====== | ====== |
0  low | 0  high | 0  low | 0  low |
1   28 | 1     9 | 1   47 | 1    4 |
       | 2     5 |        |        |

astype(dtype, copy=True, errors='raise')

Cast the data to the given data type.

clear()

copy(deep=True)

Make a copy of this DictOfSeries’ indices and data.

Parameters

deep (bool, default True) – Make a deep copy, including a copy of the data and the indices. With deep=False neither the indices nor the data are copied.

Returns

copy

Return type

DictOfSeries

See also

pandas.DataFrame.copy()

copy_empty(columns=True)

Return a new DictOfSeries object, with same properties than the original. :param columns: If True, the copy will have the same, but empty columns like the original. :type columns: bool, default True

Returns

DictOfSeries

Return type

empty copy

Examples

>>> di = DictOfSeries({'A': range(2), 'B': range(3)})
>>> di
   A |    B |
==== | ==== |
0  0 | 0  0 |
1  1 | 1  1 |
     | 2  2 |

>>> empty = di.copy_empty()
>>> empty
Empty DictOfSeries
Columns: ['A', 'B']

The properties are the same, eg.

>>> empty.itype == di.itype
True
>>> empty.cast_policy == di.cast_policy
True
>>> empty.dtypes == di.dtypes
columns
A    True
B    True
dtype: bool

dropempty()

Drop empty columns. Return copy.

dropna(inplace=False)

Return a bolean array that is True if the value is a Nan-value

equals(other)

for_each(attr_or_callable, **kwds)

Apply a callable or a pandas.Series method or property on each column.

Parameters

• attr_or_callable (Any) – A pandas.Series attribute or any callable, to apply on each column. A series attribute can be any property, field or method and also could be specified as string. If a callable is given it must take pandas.Series as the only positional argument.

• **kwds (any) – kwargs to passed to callable

Returns

A series with the results, indexed by the column labels.

Return type

pandas.Series

See also

DictOfSeries.apply()

Apply functions to columns and convert result to DictOfSeries.

Examples

>>> d = DictOfSeries([range(3), range(4)], columns=['a', 'b'])
>>> d
   a |    b |
==== | ==== |
0  0 | 0  0 |
1  1 | 1  1 |
2  2 | 2  2 |
     | 3  3 |

Use with a callable..

>>> d.for_each(max)
columns
a    2
b    3
dtype: object

..or with a string, denoting a pd.Series attribute and therefor is the same as giving the latter.

>>> d.for_each('max')
columns
a    2
b    3
dtype: object

>>> d.for_each(pd.Series.max)
columns
a    2
b    3
dtype: object

Both also works with properties:

>>> d.for_each('dtype')
columns
a    int64
b    int64
dtype: object

get(key, default=None)

hasnans(axis=0, drop_empty=False)

Returns a boolean Series along an axis, which indicates if it contains NA-entries.

index_of(method='all')

Return an single index with indices from all columns.

Parameters

method (string, default 'all') –

• ‘all’ : get all indices from all columns

• ’union’ : alias for ‘all’

• ’shared’ : get indices that are present in every columns

• ’intersection’ : alias for ‘shared’

• ’uniques’ : get indices that are only present in a single column

• ’non-uniques’ : get indices that are present in more than one column

Returns

A single duplicate-free index, somehow representing indices of all columns.

Return type

pd.Index

Examples

We use the example DictOfSeries from indexing.

>>> di
    a |      b |      c |     d |
===== | ====== | ====== | ===== |
0   0 | 2    5 | 4    7 | 6   0 |
1   7 | 3    6 | 5   17 | 7   1 |
2  14 | 4    7 | 6   27 | 8   2 |
3  21 | 5    8 | 7   37 | 9   3 |
4  28 | 6    9 | 8   47 | 10  4 |
5  35 | 7   10 | 9   57 | 11  5 |
6  42 | 8   11 | 10  67 | 12  6 |
7  49 | 9   12 | 11  77 | 13  7 |
8  56 | 10  13 | 12  87 | 14  8 |
9  63 | 11  14 | 13  97 | 15  9 |

>>> di.index_of()
RangeIndex(start=0, stop=16, step=1)

>>> di.index_of("shared")
Int64Index([6, 7, 8, 9], dtype='int64')

>>> di.index_of("uniques")
Int64Index([0, 1, 14, 15], dtype='int64')

isdata()

Alias for notna(drop_empty=True).

isempty()

Returns a boolean Series, which indicates if an column is empty

isin(values)

Return a boolean dios, that indicates if the corresponding value is in the given array-like.

isna(drop_empty=False)

Return a boolean DictOfSeries which indicates NA positions.

isnull(drop_empty=False)

Alias for isna()

items()

iteritems()

iterrows(fill_value=nan, squeeze=True)

Iterate over DictOfSeries rows as (index, pandas.Series/DictOfSeries) pairs. MAY BE VERY PERFORMANCE AND/OR MEMORY EXPENSIVE

Parameters

• fill_value (scalar, default numpy.nan) –

  Fill value for row entry, if the column does not have an entry at the current index location. This ensures that the returned Row always contain all columns. If None is given no value is filled.

  If fill_value=None and squeeze=True the resulting Row (a pandas.Series) may differ in length between iterator calls. That’s because an entry, that is not present in a column, will also not be present in the resulting Row.

• squeeze (bool, default False) –

  • True : A pandas.Series is returned for each row.

  • False : A single-rowed DictOfSeries is returned for each row.

Yields

• index (label) – The index of the row.

• data (Series or DictOfSeries) – The data of the row as a Series if squeeze is True, as a DictOfSeries otherwise.

See also

DictOfSeries.iteritems()

Iterate over (column name, Series) pairs.

keys()

max(axis=None, skipna=None)

memory_usage(index=True, deep=False)

min(axis=0, skipna=True)

notempty()

Returns a boolean Series, which indicates if an column is not empty

notna(drop_empty=False)

Return a boolean DictOfSeries which indicates non-NA positions.

notnull(drop_empty=False)

Alias, see notna().

pop(*args)

popitem()

reduce_columns(func, initial=None, skipna=False)

Reduce all columns to a single pandas.Series by a given function.

Apply a function of two pandas.Series as arguments, cumulatively to all columns, from left to right, so as to reduce the columns to a single pandas.Series. If initial is present, it is placed before the columns in the calculation, and serves as a default when the columns are empty.

Parameters

• func (function) – The function must take two identically indexed pandas.Series and should return a single pandas.Series with the same index.

• initial (column-label or pd.Series, default None) – The series to start with. If None a dummy series is created, with the indices of all columns and the first seen values.

• skipna (bool, default False) – If True, skip NaN values.

Returns

A series with the reducing result and the index of the start series, defined by initializer.

Return type

pandas.Series

setdefault(key, default=None)

squeeze(axis=None)

Squeeze a 1-dimensional axis objects into scalars.

to_csv(*args, **kwargs)

Write object to a comma-separated values (csv) file.

Changed in version 0.24.0: The order of arguments for Series was changed.

Parameters

• path_or_buf (str or file handle, default None) –

  File path or object, if None is provided the result is returned as a string. If a file object is passed it should be opened with newline=’’, disabling universal newlines.

  Changed in version 0.24.0: Was previously named “path” for Series.

• sep (str, default ',') – String of length 1. Field delimiter for the output file.

• na_rep (str, default '') – Missing data representation.

• float_format (str, default None) – Format string for floating point numbers.

• columns (sequence, optional) – Columns to write.

• header (bool or list of str, default True) –

  Write out the column names. If a list of strings is given it is assumed to be aliases for the column names.

  Changed in version 0.24.0: Previously defaulted to False for Series.

• index (bool, default True) – Write row names (index).

• index_label (str or sequence, or False, default None) – Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the object uses MultiIndex. If False do not print fields for index names. Use index_label=False for easier importing in R.

• mode (str) – Python write mode, default ‘w’.

• encoding (str, optional) – A string representing the encoding to use in the output file, defaults to ‘utf-8’.

• compression (str or dict, default 'infer') –

  If str, represents compression mode. If dict, value at ‘method’ is the compression mode. Compression mode may be any of the following possible values: {‘infer’, ‘gzip’, ‘bz2’, ‘zip’, ‘xz’, None}. If compression mode is ‘infer’ and path_or_buf is path-like, then detect compression mode from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’ or ‘.xz’. (otherwise no compression). If dict given and mode is ‘zip’ or inferred as ‘zip’, other entries passed as additional compression options.

  Changed in version 1.0.0: May now be a dict with key ‘method’ as compression mode and other entries as additional compression options if compression mode is ‘zip’.

• quoting (optional constant from csv module) – Defaults to csv.QUOTE_MINIMAL. If you have set a float_format then floats are converted to strings and thus csv.QUOTE_NONNUMERIC will treat them as non-numeric.

• quotechar (str, default '"') – String of length 1. Character used to quote fields.

• line_terminator (str, optional) –

  The newline character or character sequence to use in the output file. Defaults to os.linesep, which depends on the OS in which this method is called (‘n’ for linux, ‘rn’ for Windows, i.e.).

  Changed in version 0.24.0.

• chunksize (int or None) – Rows to write at a time.

• date_format (str, default None) – Format string for datetime objects.

• doublequote (bool, default True) – Control quoting of quotechar inside a field.

• escapechar (str, default None) – String of length 1. Character used to escape sep and quotechar when appropriate.

• decimal (str, default '.') – Character recognized as decimal separator. E.g. use ‘,’ for European data.

Returns

If path_or_buf is None, returns the resulting csv format as a string. Otherwise returns None.

Return type

None or str

See also

read_csv()

Load a CSV file into a DataFrame.

to_excel()

Write DataFrame to an Excel file.

Examples

>>> df = pd.DataFrame({'name': ['Raphael', 'Donatello'],
...                    'mask': ['red', 'purple'],
...                    'weapon': ['sai', 'bo staff']})
>>> df.to_csv(index=False)
'name,mask,weapon\nRaphael,red,sai\nDonatello,purple,bo staff\n'

Create ‘out.zip’ containing ‘out.csv’

>>> compression_opts = dict(method='zip',
...                         archive_name='out.csv')  
>>> df.to_csv('out.zip', index=False,
...           compression=compression_opts)  

to_df()

Transform DictOfSeries to a pandas.DataFrame.

Because a pandas.DataFrame can not handle Series of different length, but DictOfSeries can, the missing data is filled with NaNs.

Returns

pandas.DataFrame

Return type

transformed data

Examples

Missing data locations are filled with NaN’s

>>> a = pd.Series(11, index=range(2))
>>> b = pd.Series(22, index=range(3))
>>> c = pd.Series(33, index=range(1,9,3))
>>> di = DictOfSeries(dict(a=a, b=b, c=c))
>>> di
    a |     b |     c |
===== | ===== | ===== |
0  11 | 0  22 | 1  33 |
1  11 | 1  22 | 4  33 |
      | 2  22 | 7  33 |
>>> di.to_df()
columns     a     b     c
0        11.0  22.0   NaN
1        11.0  22.0  33.0
2         NaN  22.0   NaN
4         NaN   NaN  33.0
7         NaN   NaN  33.0

to_string(max_rows=None, min_rows=None, max_cols=None, na_rep='NaN', show_dimensions=False, method='indexed', no_value=' ', empty_series_rep='no data', col_delim=' | ', header_delim='=', col_space=None)

Pretty print a dios.

if method == indexed (default):

every column is represented by a own index and corresponding values

if method == aligned [2]:

one(!) global index is generated and values from a column appear at the corresponding index-location.

Parameters

• max_cols – not more column than max_cols are printed [1]

• max_rows – see min_rows [1]

• min_rows – not more rows than min_rows are printed, if rows of any series exceed max_rows [1]

• na_rep – all NaN-values are replaced by na_rep. Default NaN

• empty_series_rep – Ignored if not method=’indexed’. Empty series are represented by the string in empty_series_rep

• col_delim (str) – Ignored if not method=’indexed’. between all columns col_delim is inserted.

• header_delim – Ignored if not method=’indexed’. between the column names (header) and the data, header_delim is inserted, if not None. The string is repeated, up to the width of the column. (str or None).

• no_value – Ignored if not method=’aligned’. value that indicates, that no entry in the underling series is present. Bear in mind that this should differ from na_rep, otherwise you cannot differ missing- from NaN- values.

Notes

[1]: defaults to the corresponding value in dios_options [2]: the common-params are directly passed to pd.DataFrame.to_string(..) under the hood, if method is aligned

update(other)