Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dfview #297

Open
wants to merge 239 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
239 commits
Select commit Hold shift + click to select a range
a2d7008
fixing issue #86 from upstream:
Mar 11, 2021
62925bb
add unit test for Field get_spans() function
Mar 12, 2021
0e313dc
remove unuseful line comments
Mar 12, 2021
e211371
add dataset, datafreame class
deng113jie Mar 15, 2021
39e4535
Merge remote-tracking branch 'upstream/master'
deng113jie Mar 15, 2021
329a7cc
closing issue 92, reset the dataset when call field.data.clear
deng113jie Mar 15, 2021
d9d8b02
closing issue 92, reset the dataset when call field.data.clear
deng113jie Mar 15, 2021
f7ba342
Merge branch 'master' into patch92
deng113jie Mar 15, 2021
21f0fa9
add unittest for field.data.clear function
deng113jie Mar 15, 2021
c9363ef
recover the dataset file to avoid merge error when fixing issue 92
deng113jie Mar 15, 2021
14fc1f3
fix end_of_file char in dataset.py
deng113jie Mar 15, 2021
2d13342
add get_span for index string field
deng113jie Mar 16, 2021
666073e
unittest for get_span functions on different types of field, eg. fixe…
deng113jie Mar 17, 2021
73aa50e
Merge remote-tracking branch 'upstream/master'
deng113jie Mar 18, 2021
689cc3f
Merge remote-tracking branch 'upstream/master' into dataframe
deng113jie Mar 18, 2021
8ba818f
dataframe basic methods and unittest
deng113jie Mar 19, 2021
abb3337
more dataframe operations
deng113jie Mar 22, 2021
3180cbd
fix upstream merge conflict
deng113jie Mar 24, 2021
9b9c420
minor fixing
deng113jie Mar 24, 2021
55989d6
update get_span to field subclass
deng113jie Mar 24, 2021
cd69d04
solve conflict
deng113jie Mar 24, 2021
f2136d5
intermedia commit due to test pr 118
deng113jie Mar 24, 2021
30953e3
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Mar 24, 2021
0dccc6e
Merge remote-tracking branch 'upstream/master' into dataframe
deng113jie Mar 24, 2021
000463d
Implementate get_spans(ndarray) and get_spans(ndarray1, ndarray2) fun…
deng113jie Mar 24, 2021
37972b5
Merge branch 'dataframe'
deng113jie Mar 24, 2021
74c1dad
Move the get_spans functions from persistence to operations.
deng113jie Mar 25, 2021
bf210c4
Merge branch 'dataframe'
deng113jie Mar 25, 2021
95c1645
minor edits for pull request
deng113jie Mar 25, 2021
5db42d2
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Mar 25, 2021
664e255
remove dataframe for pull request
deng113jie Mar 25, 2021
02265fe
remove dataframe test for pr
deng113jie Mar 25, 2021
f536652
add dataframe
deng113jie Mar 25, 2021
bafe9cf
Merge remote-tracking branch 'upstream/master' into dataframe
deng113jie Mar 25, 2021
223dbe9
fix get_spans_for_2_fields_by_spans, fix the unittest
deng113jie Mar 25, 2021
cc48016
Merge branch 'master' into dataframe
deng113jie Mar 25, 2021
948ce1a
Initial commit for is_sorted method on Field
atbenmurray Mar 25, 2021
37b8ac2
minor edits for the pr
deng113jie Mar 26, 2021
0369c92
fix minor edit error for pr
deng113jie Mar 26, 2021
2096828
Merge branch 'master' into dataframe
deng113jie Mar 26, 2021
f213240
add apply_index and apply_filter methods on fields
deng113jie Mar 26, 2021
b050d74
Merging from recent PRs
atbenmurray Mar 26, 2021
76b5ff1
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera into da…
deng113jie Mar 26, 2021
fe36b94
Adding in missing tests for all field types for is_sorted
atbenmurray Mar 26, 2021
daa6012
update the apply filter and apply index on Fields
deng113jie Mar 26, 2021
5c43f38
minor updates to line up w/ upstream
deng113jie Mar 26, 2021
459b91c
update apply filter & apply index methods in fields that differ if de…
deng113jie Mar 26, 2021
c0ac960
updated the apply_index and apply_filter methods in fields. Use oldda…
deng113jie Mar 29, 2021
dd0867d
add dataframe basic functions and operations; working on dataset to e…
deng113jie Mar 30, 2021
e52d825
add functions in dataframe
deng113jie Apr 1, 2021
463ea70
integrates the dataset, dataframe into the session
deng113jie Apr 6, 2021
76d1952
update the fieldsimporter and field.create_like methods to call dataf…
deng113jie Apr 7, 2021
7cfeceb
add license info to a few files
deng113jie Apr 7, 2021
b1cb082
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 8, 2021
eaac2b6
csv_reader_with_njit
Liyuan-Chen-1024 Apr 8, 2021
a9ce1fb
change output_excel from string to int
Liyuan-Chen-1024 Apr 9, 2021
113a83f
Merge branch 'master' of github.com:KCL-BMEIS/ExeTera into importer_c…
Liyuan-Chen-1024 Apr 9, 2021
375982c
solve merge conflict
Liyuan-Chen-1024 Apr 9, 2021
e9d1053
initialize column_idx matrix outside of the njit function
Liyuan-Chen-1024 Apr 9, 2021
e1ed80d
use np.fromfile to load the file into byte array
Liyuan-Chen-1024 Apr 9, 2021
f4fe394
Merge branch 'master' into field_is_sorted_method
atbenmurray Apr 11, 2021
a057677
Refactoring and reformatting of some of the dataset / dataframe code;…
atbenmurray Apr 11, 2021
0845a63
Merge branch 'issort' into dataframe
deng113jie Apr 12, 2021
4d2886a
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera into da…
deng113jie Apr 12, 2021
db3ec9f
Work on fast csv reading
atbenmurray Apr 12, 2021
f2efedc
Address issue #138 on minor tweaks
deng113jie Apr 12, 2021
4926330
remove draft group.py from repo
deng113jie Apr 12, 2021
56bb190
Improved performance from the fast csv reader through avoiding ndarra…
atbenmurray Apr 12, 2021
04d810b
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 13, 2021
f0b7e37
fix dataframe api
deng113jie Apr 13, 2021
18d49a6
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 13, 2021
737eeed
fixing #13 and #14, add dest parameter to get_spans(), tidy up the fi…
deng113jie Apr 13, 2021
732762d
minor fix remove dataframe and file property from dataset, as not use…
deng113jie Apr 13, 2021
ab6508c
minor fix on unittest
deng113jie Apr 13, 2021
39027f7
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 14, 2021
358d82b
add docstring for dataset
deng113jie Apr 14, 2021
98a4d7f
copy/move for dataframe; docstrings
deng113jie Apr 15, 2021
e6b1a57
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 15, 2021
a0e0167
categorical field: convert from byte int to value int within njit fun…
Liyuan-Chen-1024 Apr 15, 2021
204bd39
merge
Liyuan-Chen-1024 Apr 15, 2021
c788b96
Adding in of pseudocode version of fast categorical lookup
atbenmurray Apr 15, 2021
60f2ba9
clean up the comments
Liyuan-Chen-1024 Apr 15, 2021
bba4829
Merge branch 'importer_csv_reader' of github.com:KCL-BMEIS/ExeTera in…
Liyuan-Chen-1024 Apr 15, 2021
c341eb2
docstrings for dataframe
deng113jie Apr 16, 2021
b23f1d8
Major reworking of apply_filter / apply_index for fields; they should…
atbenmurray Apr 16, 2021
63bd5a0
add unittest for various fields in dataframe
deng113jie Apr 16, 2021
650014e
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 16, 2021
cb9f2a2
add unittest for Dataframe.add/drop/move
deng113jie Apr 16, 2021
013f401
minor change on name to make sure name in consistent over dataframe, …
deng113jie Apr 16, 2021
18ce7ce
minor fixed of adding prefix b to string in test_session and test_dat…
deng113jie Apr 16, 2021
8657081
minor fixed of adding prefix b to string in test_session and test_dat…
deng113jie Apr 16, 2021
51e2fec
Completed initial pass of memory fields for all types
atbenmurray Apr 16, 2021
955aede
categloric field.keys will return byte key as string, thus minor chan…
deng113jie Apr 16, 2021
039d8ee
solved the byte to string issue, problem is dof python 3.7 and 3.8
deng113jie Apr 16, 2021
547bb88
Miscellaneous field fixes; fixed issues with dataframe apply_filter /…
atbenmurray Apr 16, 2021
700635f
Moving most binary op logic out into a static method in FieldDataOps
atbenmurray Apr 16, 2021
dec92ca
Resolved conflicts in dataframe.py
atbenmurray Apr 16, 2021
b631932
Dataframe copy, move and drop operations have been moved out of the D…
atbenmurray Apr 17, 2021
4804417
Fixing accidental introduction of CRLF to abstract_types
atbenmurray Apr 17, 2021
f16cb09
Fixed bug where apply_filter and apply_index weren't returning a fiel…
atbenmurray Apr 17, 2021
37dac08
Fixed issue in timestamp_field_create_like when group is set and is a…
atbenmurray Apr 17, 2021
8c62e0a
persistence.filter_duplicate_fields now supports fields as well as nd…
atbenmurray Apr 17, 2021
cfcb69b
sort_on message now shows in verbose mode under all circumstances
atbenmurray Apr 17, 2021
22504ef
Fixed bug in apply filter when a destination dataset is applied
atbenmurray Apr 17, 2021
23c373d
Added a test to catch dataframe.apply_filter bug
atbenmurray Apr 17, 2021
98624e6
Bug fix: categorical_field_constructor in fields.py was returning num…
atbenmurray Apr 17, 2021
76d8717
Copying data before filtering, as filtering in h5py is very slow
atbenmurray Apr 17, 2021
44a9c3d
Adding apply_spans functions to fields
atbenmurray Apr 18, 2021
210f847
Fixed TestFieldApplySpansCount.test_timestamp_apply_spans that had be…
atbenmurray Apr 18, 2021
f8829ae
Merge commit 'refs/pull/149/head' of https://github.com/KCL-BMEIS/Exe…
deng113jie Apr 19, 2021
a7d6673
Issues found with indexed strings and merging; fixes found for apply_…
atbenmurray Apr 19, 2021
3d322c2
Updated merge functions to consistently return memory fields if not p…
atbenmurray Apr 19, 2021
294ec3a
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 20, 2021
e8edd9d
concate cat keys instead of padding
Liyuan-Chen-1024 Apr 20, 2021
c2ba9ff
some docstring for fields
deng113jie Apr 20, 2021
1a19815
dataframe copy/move/drop and unittest
deng113jie Apr 20, 2021
1fb0362
Fixing issue with dataframe move/copy being static
atbenmurray Apr 20, 2021
937368e
Updating HDF5Field writeable methods to account for prior changes
atbenmurray Apr 20, 2021
cddcf66
Adding merge functionality for dataframes
atbenmurray Apr 20, 2021
534cbd4
dataset.drop is a member method of Dataset as it did not make sense f…
atbenmurray Apr 20, 2021
e5dc536
Added missing methods / properties to DataFrame ABC
atbenmurray Apr 20, 2021
9b1a4a9
minor update on dataframe static function
deng113jie Apr 20, 2021
1967685
minor update
deng113jie Apr 20, 2021
6c3270a
Merge commit 'refs/pull/157/head' of https://github.com/KCL-BMEIS/Exe…
deng113jie Apr 20, 2021
6bdb08e
minor update session
deng113jie Apr 21, 2021
3680436
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 21, 2021
cf5f5a6
minor comments update
deng113jie Apr 21, 2021
23ad71a
minor comments update
deng113jie Apr 21, 2021
75eefc0
add unittest for csv_reader_speedup.py
Liyuan-Chen-1024 Apr 21, 2021
3ddc916
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 21, 2021
3a6dc51
Merge commit 'refs/pull/137/head' of https://github.com/KCL-BMEIS/Exe…
deng113jie Apr 22, 2021
c02fe32
count operation; logical not for numeric fields
deng113jie Apr 26, 2021
58159d0
remove csv speed up work from commit
deng113jie Apr 27, 2021
a7b477d
minor update
deng113jie Apr 27, 2021
903f3b4
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 27, 2021
29f736d
unit test for logical not in numeric field
deng113jie Apr 27, 2021
7fd9bdc
patch for get_spans for datastore
deng113jie Apr 28, 2021
04df757
tests for two fields
deng113jie Apr 28, 2021
e47e15c
add as type to numeric field
deng113jie Apr 29, 2021
a4b14fb
Merge branch 'master' of https://github.com/deng113jie/ExeTera
deng113jie Apr 29, 2021
5492b94
seperate the unittest of get_spans by datastore reader
deng113jie Apr 29, 2021
25320bd
unittest for astype
deng113jie Apr 29, 2021
e289c6b
Merge branch 'dspatch'
deng113jie May 4, 2021
a59c13a
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie May 4, 2021
87df0bc
update astype for fields, update logical_not for numeric fields
deng113jie May 10, 2021
0875149
remove dataframe view commits
deng113jie May 10, 2021
c335831
remove kwargs in get_spans in session, add fields back for backward c…
deng113jie May 11, 2021
bdf783a
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie May 11, 2021
ea20c60
remove filter view tests
deng113jie May 11, 2021
778d56c
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie May 27, 2021
611601a
partial commit on viewer
deng113jie Jun 10, 2021
66867b7
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Sep 20, 2021
fbe396f
remote view from git
deng113jie Sep 20, 2021
c2c7185
add df.describe unittest
deng113jie Sep 22, 2021
78cc222
sync with upstream
deng113jie Sep 23, 2021
001134c
Delete python-publish.yml
deng113jie Sep 23, 2021
eb0bb76
Update python-app.yml
deng113jie Sep 23, 2021
d646ac2
Update python-app.yml
deng113jie Sep 23, 2021
b55775b
dataframe describe function
deng113jie Sep 23, 2021
0d23098
Merge branch 'master' of https://github.com/deng113jie/ExeTera
deng113jie Sep 23, 2021
7774c6f
sync with upstream
deng113jie Sep 23, 2021
ae1d621
Update python-app.yml
deng113jie Sep 30, 2021
3d5738e
alternative get_timestamp notebook for discussion
deng113jie Oct 5, 2021
4685c6b
update the notebook output of linux and mac
deng113jie Oct 5, 2021
dc38d28
update format
deng113jie Oct 5, 2021
0df34bc
update the to_timestamp and to_timestamp function in utils
deng113jie Oct 11, 2021
87353e3
add unittest for utils to_timestamp and to_datetimie
deng113jie Oct 11, 2021
87abe47
fix for pr
deng113jie Oct 11, 2021
a3719ef
setup github action specific for windows for cython
deng113jie Oct 12, 2021
ed42f70
minor workflow fix
deng113jie Oct 12, 2021
2157da2
add example pyx file
deng113jie Oct 12, 2021
1abeaa7
fix package upload command on win; as the git action
deng113jie Oct 12, 2021
e77562e
add twine as tools
deng113jie Oct 12, 2021
03208aa
add linux action file
deng113jie Oct 12, 2021
35430f2
update the linux build command
deng113jie Oct 12, 2021
de3e7e5
build workflow for macos
deng113jie Oct 12, 2021
a8af750
minor update the macos workflow
deng113jie Oct 12, 2021
d41a24b
fixed timestamp issue on windows by add timezone info to datetime
deng113jie Oct 14, 2021
c98b87c
finanlize workflow file, compile react to publish action only
deng113jie Oct 14, 2021
a57c413
avoid the bytearray vs string error in windows by converting result to
deng113jie Oct 14, 2021
764650b
fixing string vs bytesarray issue
deng113jie Oct 14, 2021
4676901
update categorical field key property, change the key, value to bytes if
deng113jie Oct 15, 2021
e5d74c6
solved index must be np.int64 error
deng113jie Oct 15, 2021
030d587
all unittest error on windoes removed
deng113jie Oct 15, 2021
55e62eb
Merge branch 'master' into win_actions
deng113jie Oct 15, 2021
7cf7bae
minor update on workflow file
deng113jie Oct 15, 2021
521142e
minor update workflow file
deng113jie Oct 15, 2021
9373fd2
minor fix: use pip install -r ; remove unused import in utils.py
deng113jie Oct 15, 2021
6f67ac4
update action file
deng113jie Oct 15, 2021
703a19a
remove change on test_presistence on uint32 to int32
deng113jie Oct 18, 2021
613532a
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Nov 22, 2021
b981bb9
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Jan 17, 2022
e35c1c4
Merge branch 'KCL-BMEIS:master' into master
deng113jie Jan 25, 2022
a7ee946
Merge branch 'master' of https://github.com/deng113jie/ExeTera
deng113jie Jan 25, 2022
0f319d1
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Feb 1, 2022
a5ab148
add output argument for describe function in dataframe, so that the r…
deng113jie Feb 1, 2022
f50ab1f
comment all print function in unittest
deng113jie Feb 1, 2022
94a074a
modify the remap function for categorical field and categorical mem f…
deng113jie Feb 1, 2022
ef563e9
Added check to ensure ExeTera entry point actually works after pip in…
ericspod Feb 8, 2022
e0caad0
Attempted Fix
ericspod Feb 9, 2022
ba06c9f
find_packages
ericspod Feb 9, 2022
25a3cc3
Tweak
ericspod Feb 9, 2022
2e03557
Merge commit 'refs/pull/253/head' of https://github.com/KCL-BMEIS/Exe…
deng113jie Feb 9, 2022
5f83d7e
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Feb 9, 2022
d932731
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Feb 14, 2022
fff9a19
fixing issue 214
deng113jie Feb 14, 2022
ffbc4f0
fixing bug on dataset set item
deng113jie Feb 17, 2022
7f82d58
fixing apply_span_src in fields.py
deng113jie Feb 17, 2022
474425a
revert change on field
deng113jie Feb 17, 2022
1c59265
add unittest for dataset setitem bug
deng113jie Feb 17, 2022
5002c65
examples using dataset generated by randomdataset
deng113jie Feb 18, 2022
efcde7c
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Feb 18, 2022
825aaf1
update on examples
deng113jie Feb 23, 2022
47e2e7e
update example: added two csv files and one json files and one import…
deng113jie Feb 23, 2022
071be03
minor update on readme
deng113jie Feb 23, 2022
c908397
remove output from notebooks
deng113jie Feb 23, 2022
72cca74
minor update on example notebooks
deng113jie Feb 25, 2022
bd9a85f
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Mar 3, 2022
62d0d69
update examples
deng113jie Mar 7, 2022
40e8770
update the example notebooks
deng113jie Mar 8, 2022
81bbbcd
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Mar 24, 2022
3507595
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera
deng113jie Apr 21, 2022
099c8f6
df view init commit
deng113jie Apr 27, 2022
3d6966e
dataframe view init commit:
deng113jie May 5, 2022
735b7a5
dataframe view updates 3:
deng113jie May 6, 2022
85169e6
change filtered data presentation from data array to field __getitem__
deng113jie May 9, 2022
9667dd7
minor update
deng113jie May 9, 2022
c333f6d
updated view functions,
deng113jie May 11, 2022
bb764bd
modify the association between view fields with field array, by assig…
deng113jie May 12, 2022
7803d76
update the view:
deng113jie May 18, 2022
dfb36ab
fixed the data[:] for indexed string fields
deng113jie May 18, 2022
5c93b43
Merge branch 'master' of https://github.com/KCL-BMEIS/ExeTera into df…
deng113jie May 18, 2022
fe9cee8
update Eric's comments
deng113jie May 23, 2022
c1ad9ba
minor update
deng113jie May 23, 2022
80c0339
update unittests for dataframe view
deng113jie May 24, 2022
6cb1d3e
minor update
deng113jie May 24, 2022
e8cf7f2
add persistence over view so that view
deng113jie May 25, 2022
3153f2b
add unittest for view presistence
deng113jie May 26, 2022
135260e
documents on future work
deng113jie May 26, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixing issue with dataframe move/copy being static
  • Loading branch information
atbenmurray committed Apr 20, 2021
commit 1fb036225bb7de0336fdb960010670510eb224fb
5 changes: 5 additions & 0 deletions exetera/core/abstract_types.py
Original file line number Diff line number Diff line change
@@ -25,6 +25,11 @@ def name(self):
def timestamp(self):
raise NotImplementedError()

@property
@abstractmethod
def dataframe(self):
raise NotImplementedError()

@property
@abstractmethod
def chunksize(self):
118 changes: 64 additions & 54 deletions exetera/core/dataframe.py
Original file line number Diff line number Diff line change
@@ -80,6 +80,10 @@ def add(self, field):
nfield.data.write(field.data[:])
self._columns[dname] = nfield

def drop(self, name):
del self._columns[name]
del self._h5group[name]

def create_group(self, name):
"""
Create a group object in HDF5 file for field to use. Please note, this function is for
@@ -91,22 +95,13 @@ def create_group(self, name):
self._h5group.create_group(name)
return self._h5group[name]

def create_numeric(self, name, nformat, timestamp=None, chunksize=None):
"""
Create a numeric type field.
"""
fld.numeric_field_constructor(self._dataset.session, self, name, nformat, timestamp, chunksize)
field = fld.NumericField(self._dataset.session, self._h5group[name],
write_enabled=True)
self._columns[name] = field
return self._columns[name]

def create_indexed_string(self, name, timestamp=None, chunksize=None):
"""
Create a indexed string type field.
"""
fld.indexed_string_field_constructor(self._dataset.session, self, name, timestamp, chunksize)
field = fld.IndexedStringField(self._dataset.session, self._h5group[name],
fld.indexed_string_field_constructor(self._dataset.session, self, name,
timestamp, chunksize)
field = fld.IndexedStringField(self._dataset.session, self._h5group[name], self, name,
write_enabled=True)
self._columns[name] = field
return self._columns[name]
@@ -115,19 +110,31 @@ def create_fixed_string(self, name, length, timestamp=None, chunksize=None):
"""
Create a fixed string type field.
"""
fld.fixed_string_field_constructor(self._dataset.session, self, name, length, timestamp, chunksize)
field = fld.FixedStringField(self._dataset.session, self._h5group[name],
fld.fixed_string_field_constructor(self._dataset.session, self, name,
length, timestamp, chunksize)
field = fld.FixedStringField(self._dataset.session, self._h5group[name], self, name,
write_enabled=True)
self._columns[name] = field
return self._columns[name]

def create_numeric(self, name, nformat, timestamp=None, chunksize=None):
"""
Create a numeric type field.
"""
fld.numeric_field_constructor(self._dataset.session, self, name,
nformat, timestamp, chunksize)
field = fld.NumericField(self._dataset.session, self._h5group[name], self, name,
write_enabled=True)
self._columns[name] = field
return self._columns[name]

def create_categorical(self, name, nformat, key, timestamp=None, chunksize=None):
"""
Create a categorical type field.
"""
fld.categorical_field_constructor(self._dataset.session, self, name, nformat, key,
timestamp, chunksize)
field = fld.CategoricalField(self._dataset.session, self._h5group[name],
field = fld.CategoricalField(self._dataset.session, self._h5group[name], self, name,
write_enabled=True)
self._columns[name] = field
return self._columns[name]
@@ -136,8 +143,9 @@ def create_timestamp(self, name, timestamp=None, chunksize=None):
"""
Create a timestamp type field.
"""
fld.timestamp_field_constructor(self._dataset.session, self, name, timestamp, chunksize)
field = fld.TimestampField(self._dataset.session, self._h5group[name],
fld.timestamp_field_constructor(self._dataset.session, self, name,
timestamp, chunksize)
field = fld.TimestampField(self._dataset.session, self._h5group[name], self, name,
write_enabled=True)
self._columns[name] = field
return self._columns[name]
@@ -225,7 +233,9 @@ def delete_field(self, field):

:param field: The field to delete from this dataframe.
"""
name = field.name[field.name.index('/', 1)+1:]
if field.dataframe != self:
raise ValueError("This field is owned by a different dataframe")
name = field.name
if name is None:
raise ValueError("This dataframe does not contain the field to delete.")
else:
@@ -300,42 +310,42 @@ def apply_index(self, index_to_apply, ddf=None):
field.apply_index(index_to_apply, in_place=True)
return self

@staticmethod
def copy(field: fld.Field, dataframe: DataFrame, name: str):
"""
Copy a field to another dataframe as well as underlying dataset.

:param field: The source field to copy.
:param dataframe: The destination dataframe to copy to.
:param name: The name of field under destination dataframe.
"""
dfield = field.create_like(dataframe, name)
if field.indexed:
dfield.indices.write(field.indices[:])
dfield.values.write(field.values[:])
else:
dfield.data.write(field.data[:])
dataframe.columns[name] = dfield

@staticmethod
def drop(dataframe: DataFrame, field: fld.Field):
"""
Drop a field from a dataframe.

:param dataframe: The dataframe where field is located.
:param field: The field to delete.
"""
dataframe.delete_field(field)
def copy(field: fld.Field, dataframe: DataFrame, name: str):
"""
Copy a field to another dataframe as well as underlying dataset.

@staticmethod
def move(src_df: DataFrame, field: fld.Field, dest_df: DataFrame, name: str):
"""
Move a field to another dataframe as well as underlying dataset.
:param field: The source field to copy.
:param dataframe: The destination dataframe to copy to.
:param name: The name of field under destination dataframe.
"""
dfield = field.create_like(dataframe, name)
if field.indexed:
dfield.indices.write(field.indices[:])
dfield.values.write(field.values[:])
else:
dfield.data.write(field.data[:])
dataframe.columns[name] = dfield


# def drop(dataframe: DataFrame, field: fld.Field):
# """
# Drop a field from a dataframe.
#
# :param dataframe: The dataframe where field is located.
# :param field: The field to delete.
# """
# dataframe.delete_field(field)


def move(field: fld.Field, dest_df: DataFrame, name: str):
"""
Move a field to another dataframe as well as underlying dataset.

:param src_df: The source dataframe where the field is located.
:param field: The field to move.
:param dest_df: The destination dataframe to move to.
:param name: The name of field under destination dataframe.
"""
HDF5DataFrame.copy(field, dest_df, name)
HDF5DataFrame.drop(src_df, field)
:param src_df: The source dataframe where the field is located.
:param field: The field to move.
:param dest_df: The destination dataframe to move to.
:param name: The name of field under destination dataframe.
"""
copy(field, dest_df, name)
field.dataframe.drop(field.name)
70 changes: 41 additions & 29 deletions exetera/core/fields.py
Original file line number Diff line number Diff line change
@@ -22,22 +22,28 @@
from exetera.core import validation as val

class HDF5Field(Field):
def __init__(self, session, group, name=None, write_enabled=False):
def __init__(self, session, group, dataframe, name=None, write_enabled=False):
super().__init__()

if name is None:
field = group
else:
field = group[name]
# if name is None:
# field = group
# else:
# field = group[name]
self._session = session
self._field = field
self._field = group
self._name = name
self._fieldtype = self._field.attrs['fieldtype']
self._dataframe = dataframe
self._write_enabled = write_enabled
self._value_wrapper = None

@property
def name(self):
return self._field.name
return self._name

@property
def dataframe(self):
return self._dataframe

@property
def timestamp(self):
@@ -80,6 +86,10 @@ def __init__(self, session):
def name(self):
return None

@property
def dataframe(self):
return None

@property
def timestamp(self):
return None
@@ -506,7 +516,7 @@ def __init__(self, session, length):
self._length = length

def writeable(self):
return FixedStringField(self._session, self._field, write_enabled=True)
return FixedStringField(self._session, self._field, None, write_enabled=True)

def create_like(self, group=None, name=None, timestamp=None):
return FieldDataOps.fixed_string_field_create_like(self, group, name, timestamp)
@@ -1033,15 +1043,16 @@ def timestamp_field_constructor(session, group, name, timestamp=None, chunksize=


class IndexedStringField(HDF5Field):
def __init__(self, session, group, name=None, write_enabled=False):
super().__init__(session, group, name=name, write_enabled=write_enabled)
def __init__(self, session, group, dataframe, name=None, write_enabled=False):
super().__init__(session, group, dataframe, name=name, write_enabled=write_enabled)
self._session = session
self._dataframe = None
self._data_wrapper = None
self._index_wrapper = None
self._value_wrapper = None

def writeable(self):
return IndexedStringField(self._session, self._field, write_enabled=True)
return IndexedStringField(self._session, self._field, None, write_enabled=True)

def create_like(self, group=None, name=None, timestamp=None):
return FieldDataOps.indexed_string_create_like(self, group, name, timestamp)
@@ -1139,16 +1150,16 @@ def apply_spans_max(self, spans_to_apply, target=None, in_place=False):


class FixedStringField(HDF5Field):
def __init__(self, session, group, name=None, write_enabled=False):
super().__init__(session, group, name=name, write_enabled=write_enabled)
def __init__(self, session, group, dataframe, name=None, write_enabled=False):
super().__init__(session, group, dataframe, name=name, write_enabled=write_enabled)
# TODO: caution; we may want to consider the issues with long-lived field instances getting
# out of sync with their stored counterparts. Maybe a revision number of the stored field
# is required that we can check to see if we are out of date. That or just make this a
# property and have it always look the value up
self._length = self._field.attrs['strlen']

def writeable(self):
return FixedStringField(self._session, self._field, write_enabled=True)
return FixedStringField(self._session, self._field, None, write_enabled=True)

def create_like(self, group=None, name=None, timestamp=None):
return FieldDataOps.fixed_string_field_create_like(self, group, name, timestamp)
@@ -1220,12 +1231,12 @@ def apply_spans_max(self, spans_to_apply, target=None, in_place=False):


class NumericField(HDF5Field):
def __init__(self, session, group, name=None, mem_only=True, write_enabled=False):
super().__init__(session, group, name=name, write_enabled=write_enabled)
def __init__(self, session, group, dataframe, name, write_enabled=False):
super().__init__(session, group, dataframe, name=name, write_enabled=write_enabled)
self._nformat = self._field.attrs['nformat']

def writeable(self):
return NumericField(self._session, self._field, write_enabled=True)
return NumericField(self._session, self._field, None, self._name, write_enabled=True)

def create_like(self, group=None, name=None, timestamp=None):
return FieldDataOps.numeric_field_create_like(self, group, name, timestamp)
@@ -1375,9 +1386,8 @@ def __ge__(self, value):


class CategoricalField(HDF5Field):
def __init__(self, session, group,
name=None, write_enabled=False):
super().__init__(session, group, name=name, write_enabled=write_enabled)
def __init__(self, session, group, dataframe, name=None, write_enabled=False):
super().__init__(session, group, dataframe, name=name, write_enabled=write_enabled)
self._nformat = self._field.attrs['nformat'] if 'nformat' in self._field.attrs else 'int8'

def writeable(self):
@@ -1492,8 +1502,8 @@ def __ge__(self, value):


class TimestampField(HDF5Field):
def __init__(self, session, group, name=None, write_enabled=False):
super().__init__(session, group, name=name, write_enabled=write_enabled)
def __init__(self, session, group, dataframe, name=None, write_enabled=False):
super().__init__(session, group, dataframe, name=name, write_enabled=write_enabled)

def writeable(self):
return TimestampField(self._session, self._field, write_enabled=True)
@@ -1778,7 +1788,8 @@ def __init__(self, session, group, name,
if optional is True:
filter_name = '{}_set'.format(name)
numeric_field_constructor(group, filter_name, 'bool', timestamp, chunksize)
self._filter_field = NumericField(session, group, filter_name, write_enabled=True)
self._filter_field = NumericField(session, group, None, filter_name,
write_enabled=True)

def chunk_factory(self, length):
return np.zeros(length, dtype='U32')
@@ -1820,7 +1831,8 @@ def __init__(self, session, group, name,
filter_name = '{}_set'.format(name)
numeric_field_constructor(session, group, filter_name, 'bool',
timestamp, chunksize)
self._filter_field = NumericField(session, group, filter_name, write_enabled=True)
self._filter_field = NumericField(session, group, None, filter_name,
write_enabled=True)

def chunk_factory(self, length):
return np.zeros(length, dtype='U10')
@@ -2339,7 +2351,7 @@ def indexed_string_create_like(source, group, name, timestamp):

if isinstance(group, h5py.Group):
indexed_string_field_constructor(source._session, group, name, ts, source.chunksize)
return IndexedStringField(source._session, group[name], write_enabled=True)
return IndexedStringField(source._session, group[name], None, name, write_enabled=True)
else:
return group.create_indexed_string(name, ts, source.chunksize)

@@ -2356,7 +2368,7 @@ def fixed_string_field_create_like(source, group, name, timestamp):

if isinstance(group, h5py.Group):
fixed_string_field_constructor(source._session, group, name, length, ts, source.chunksize)
return FixedStringField(source._session, group[name], write_enabled=True)
return FixedStringField(source._session, group[name], None, name, write_enabled=True)
else:
return group.create_fixed_string(name, length, ts)

@@ -2373,7 +2385,7 @@ def numeric_field_create_like(source, group, name, timestamp):

if isinstance(group, h5py.Group):
numeric_field_constructor(source._session, group, name, nformat, ts, source.chunksize)
return NumericField(source._session, group[name], write_enabled=True)
return NumericField(source._session, group[name], None, name, write_enabled=True)
else:
return group.create_numeric(name, nformat, ts)

@@ -2394,7 +2406,7 @@ def categorical_field_create_like(source, group, name, timestamp):
if isinstance(group, h5py.Group):
categorical_field_constructor(source._session, group, name, nformat, keys,
ts, source.chunksize)
return CategoricalField(source._session, group[name], write_enabled=True)
return CategoricalField(source._session, group[name], None, name, write_enabled=True)
else:
return group.create_categorical(name, nformat, keys, ts)

@@ -2410,6 +2422,6 @@ def timestamp_field_create_like(source, group, name, timestamp):

if isinstance(group, h5py.Group):
timestamp_field_constructor(source._session, group, name, ts, source.chunksize)
return TimestampField(source._session, group[name], write_enabled=True)
return TimestampField(source._session, group[name], None, name, write_enabled=True)
else:
return group.create_timestamp(name, ts)
Loading