Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError: 'NoneType' object has no attribute 'dropna' #7

Closed
gohingsiang opened this issue Jan 17, 2018 · 50 comments
Closed

AttributeError: 'NoneType' object has no attribute 'dropna' #7

gohingsiang opened this issue Jan 17, 2018 · 50 comments

Comments

@gohingsiang
Copy link

gohingsiang commented Jan 17, 2018

Hello, after i try to run the run_experiments.py from /neuralnilm/scripts , it shows "AttributeError: 'NoneType' object has no attribute 'dropna'". Does anyone know how to solve it?

The version of Theano installed is (0.8.0) and Lasagne (0.1).

(my_project) ubuntu@ubuntu:~/Desktop/Goh_new/neuralnilm/scripts$ python run_experiments.py
Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is disabled, CuDNN 6021)
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/sandbox/cuda/init.py:600: UserWarning: Your CuDNN version is more recent then Theano. If you see problems, try updating Theano or downgrading CuDNN to version 4.
warnings.warn(warn)
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.
"downsample module has been moved to the theano.tensor.signal.pool module.")
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/init.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._conv import register_converters as _register_converters
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/init.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import h5a, h5d, h5ds, h5f, h5fd, h5g, h5r, h5s, h5t, h5p, h5z
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/_hl/group.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .. import h5g, h5i, h5o, h5r, h5t, h5l, h5p
WARNING (theano.sandbox.cuda): Ignoring call to use(1), GPU number 0 is already in use.
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/utils/init.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .murmurhash import murmurhash3_32
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/utils/extmath.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._logistic_sigmoid import _log_logistic_sigmoid
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/utils/extmath.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .sparsefuncs_fast import csr_row_norms
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/metrics/cluster/supervised.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .expected_mutual_info_fast import expected_mutual_information
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/sklearn/metrics/pairwise.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan
Loading NILMTK activations...
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/init.py:90: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .utilsextension import (
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/file.py:35: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import hdf5extension
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/link.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import linkextension
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/table.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import tableextension
/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/tables/index.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import indexesextension
Loading kettle for UK-DALE_building_1...
Loaded 2856 kettle activations from UK-DALE_building_1.
Loading microwave for UK-DALE_building_1...
Loaded 3181 microwave activations from UK-DALE_building_1.
Loading washing machine for UK-DALE_building_1...
Loaded 547 washing machine activations from UK-DALE_building_1.
Loading dish washer for UK-DALE_building_1...
Loaded 206 dish washer activations from UK-DALE_building_1.
Loading fridge for UK-DALE_building_1...
Loaded 16595 fridge activations from UK-DALE_building_1.
Loading kettle for UK-DALE_building_2...
Loaded 534 kettle activations from UK-DALE_building_2.
Loading microwave for UK-DALE_building_2...
Loaded 412 microwave activations from UK-DALE_building_2.
Loading washing machine for UK-DALE_building_2...
Loaded 53 washing machine activations from UK-DALE_building_2.
Loading dish washer for UK-DALE_building_2...
Loaded 98 dish washer activations from UK-DALE_building_2.
Loading fridge for UK-DALE_building_2...
Loaded 2 fridge activations from UK-DALE_building_2.
Loading kettle for UK-DALE_building_3...
Loaded 40 kettle activations from UK-DALE_building_3.
Loading microwave for UK-DALE_building_3...
UK-DALE_building_3 has no microwave. Full exception: {'instance': 1, 'type': 'microwave'}
Loading washing machine for UK-DALE_building_3...
UK-DALE_building_3 has no washing machine. Full exception: {'instance': 1, 'type': 'washing machine'}
Loading dish washer for UK-DALE_building_3...
UK-DALE_building_3 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'}
Loading fridge for UK-DALE_building_3...
UK-DALE_building_3 has no fridge. Full exception: {'instance': 1, 'type': 'fridge'}
Loading kettle for UK-DALE_building_4...
Loaded 689 kettle activations from UK-DALE_building_4.
Loading microwave for UK-DALE_building_4...
Loaded 154 microwave activations from UK-DALE_building_4.
Loading washing machine for UK-DALE_building_4...
Loaded 154 washing machine activations from UK-DALE_building_4.
Loading dish washer for UK-DALE_building_4...
UK-DALE_building_4 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'}
Loading fridge for UK-DALE_building_4...
Loaded 4680 fridge activations from UK-DALE_building_4.
Loading kettle for UK-DALE_building_5...
Loaded 173 kettle activations from UK-DALE_building_5.
Loading microwave for UK-DALE_building_5...
Loaded 0 microwave activations from UK-DALE_building_5.
Loading washing machine for UK-DALE_building_5...
Loaded 6 washing machine activations from UK-DALE_building_5.
Loading dish washer for UK-DALE_building_5...
Loaded 23 dish washer activations from UK-DALE_building_5.
Loading fridge for UK-DALE_building_5...
Loaded 1632 fridge activations from UK-DALE_building_5.
Loading kettle for UK-DALE_building_2...
Loaded 563 kettle activations from UK-DALE_building_2.
Loading microwave for UK-DALE_building_2...
Loaded 422 microwave activations from UK-DALE_building_2.
Loading washing machine for UK-DALE_building_2...
Loaded 57 washing machine activations from UK-DALE_building_2.
Loading dish washer for UK-DALE_building_2...
Loaded 105 dish washer activations from UK-DALE_building_2.
Loading fridge for UK-DALE_building_2...
Loaded 2 fridge activations from UK-DALE_building_2.
Loading kettle for UK-DALE_building_5...
Loaded 192 kettle activations from UK-DALE_building_5.
Loading microwave for UK-DALE_building_5...
Loaded 1 microwave activations from UK-DALE_building_5.
Loading washing machine for UK-DALE_building_5...
Loaded 6 washing machine activations from UK-DALE_building_5.
Loading dish washer for UK-DALE_building_5...
Loaded 46 dish washer activations from UK-DALE_building_5.
Loading fridge for UK-DALE_building_5...
Loaded 3232 fridge activations from UK-DALE_building_5.
Loading kettle for UK-DALE_building_1...
Loaded 0 kettle activations from UK-DALE_building_1.
Loading microwave for UK-DALE_building_1...
Loaded 0 microwave activations from UK-DALE_building_1.
Loading washing machine for UK-DALE_building_1...
Loaded 0 washing machine activations from UK-DALE_building_1.
Loading dish washer for UK-DALE_building_1...
Loaded 0 dish washer activations from UK-DALE_building_1.
Loading fridge for UK-DALE_building_1...
Loaded 0 fridge activations from UK-DALE_building_1.
Loading kettle for UK-DALE_building_2...
Loaded 29 kettle activations from UK-DALE_building_2.
Loading microwave for UK-DALE_building_2...
Loaded 10 microwave activations from UK-DALE_building_2.
Loading washing machine for UK-DALE_building_2...
Loaded 4 washing machine activations from UK-DALE_building_2.
Loading dish washer for UK-DALE_building_2...
Loaded 7 dish washer activations from UK-DALE_building_2.
Loading fridge for UK-DALE_building_2...
Loaded 0 fridge activations from UK-DALE_building_2.
Loading kettle for UK-DALE_building_3...
Loaded 20 kettle activations from UK-DALE_building_3.
Loading microwave for UK-DALE_building_3...
UK-DALE_building_3 has no microwave. Full exception: {'instance': 1, 'type': 'microwave'}
Loading washing machine for UK-DALE_building_3...
UK-DALE_building_3 has no washing machine. Full exception: {'instance': 1, 'type': 'washing machine'}
Loading dish washer for UK-DALE_building_3...
UK-DALE_building_3 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'}
Loading fridge for UK-DALE_building_3...
UK-DALE_building_3 has no fridge. Full exception: {'instance': 1, 'type': 'fridge'}
Loading kettle for UK-DALE_building_4...
Loaded 44 kettle activations from UK-DALE_building_4.
Loading microwave for UK-DALE_building_4...
Loaded 4 microwave activations from UK-DALE_building_4.
Loading washing machine for UK-DALE_building_4...
Loaded 4 washing machine activations from UK-DALE_building_4.
Loading dish washer for UK-DALE_building_4...
UK-DALE_building_4 has no dish washer. Full exception: {'instance': 1, 'type': 'dish washer'}
Loading fridge for UK-DALE_building_4...
Loaded 145 fridge activations from UK-DALE_building_4.
Loading kettle for UK-DALE_building_5...
Loaded 19 kettle activations from UK-DALE_building_5.
Loading microwave for UK-DALE_building_5...
Loaded 1 microwave activations from UK-DALE_building_5.
Loading washing machine for UK-DALE_building_5...
Loaded 1 washing machine activations from UK-DALE_building_5.
Loading dish washer for UK-DALE_building_5...
Loaded 23 dish washer activations from UK-DALE_building_5.
Loading fridge for UK-DALE_building_5...
Loaded 1600 fridge activations from UK-DALE_building_5.
Done loading NILMTK activations.
Starting training for net ae, appliance washing machine.
------------- INITIALISING SyntheticAggregateSource --------------
------------- INITIALISING RealAggregateSource --------------
Loading NILMTK mains...
Loading mains for UK-DALE_building_1...
Loaded mains data from building UK-DALE_building_1 for fold train from 2013-04-12 00:00:00+01:00 to 2015-01-05 06:27:12+00:00.
Loading mains for UK-DALE_building_5...
Loaded mains data from building UK-DALE_building_5 for fold train from 2014-06-29 17:23:42+01:00 to 2014-08-31 23:59:54+01:00.
Loading mains for UK-DALE_building_1...
Traceback (most recent call last):
File "run_experiments.py", line 60, in
main()
File "run_experiments.py", line 22, in main
_run_job(next_job)
File "run_experiments.py", line 48, in _run_job
eval("{next_job}.run('{next_job}')".format(next_job=next_job))
File "", line 1, in
File "/home/ubuntu/Desktop/Goh_new/neuralnilm/experiment_definitions/e578.py", line 63, in run
pipeline = get_pipeline(target_appliance, activations)
File "/home/ubuntu/Desktop/Goh_new/neuralnilm/experiment_definitions/e578.py", line 185, in get_pipeline
sample_period=SAMPLE_PERIOD
File "build/bdist.linux-x86_64/egg/neuralnilm/data/realaggregatesource.py", line 65, in init
File "build/bdist.linux-x86_64/egg/neuralnilm/data/realaggregatesource.py", line 93, in _load_mains_into_memory
AttributeError: 'NoneType' object has no attribute 'dropna'
Closing remaining open files:/home/ubuntu/Desktop/Goh/Data/ukdale.h5...done

@Jacoobr
Copy link

Jacoobr commented Jan 17, 2018

Hello, gohingsiang. I met the same problem too. Have you fix the problem?

@gohingsiang
Copy link
Author

Hi, Jacoobr, I still struggle with it, however I suspect this problem is due to the version of Lasagne or Theano that we used. Therefore I did ask Jack Kelly regarding the version of Lasagne and Theano that he used that time but unfortunately he don't remember about it. Then, after that i try to use Theano 0.6.0 and Theano 0.7.0 I'm still getting the same error.

@Jacoobr
Copy link

Jacoobr commented Jan 18, 2018

Hi, gohingsiang, I think this may caused by pandas version, so I try to down my pandas version from 0.22 to 0.19.2. But I still get this 'dropna' error. By the way, I also run a similar Energy Disaggregation project that comes from hy10327's project: https://github.com/hyl0327/neuralnilmtp . I get the same error too. I hope you can try hy10327's project too, may be the code work nicely for you. The problem confused me many days. What's more important, Do you get the dataset from UK-DALE : http://data.ukedc.rl.ac.uk/simplebrowse/edc/efficiency/residential/EnergyConsumption/Domestic/UK-DALE-2015/UK-DALE-disaggregated/ukdale.h5.tgz ?

@gohingsiang
Copy link
Author

Thanks for the information! I will let you know the result after i tried. And yes that is where i get the dataset from UK-DALE and currently I'm using the file extracted from ukdale.h5.tgz.

@Jacoobr
Copy link

Jacoobr commented Jan 18, 2018

Thank you, gohingsiang. May i see your code in the 'neuralnilm/experiment_definitions/e578.py"' script file? Have you edit the author's code?

@gohingsiang
Copy link
Author

Following are the code for the neuralnilm/experiment_definitions/e578.py that I used. I doesn't change the code inside it except for the NILMTK_FILENAME which has been changed to NILMTK_FILENAME = '/home/ubuntu/Desktop/Goh/Data/ukdale.h5'

#!/usr/bin/env python0
from future import print_function, division
from os.path import join
import sys
from neuralnilm.config import config

import logging
logger = logging.getLogger('neuralnilm')

experiment_definition_path = config.get('Paths', 'experiment_definitions')
job_list_filename = join(experiment_definition_path, 'job_list.txt')

def main():
if experiment_definition_path not in sys.path:
sys.path.append(sc experiment_definition_path)
#sys.path.insert(0, experiment_definition_path) #original
testing = sys.path.insert(0, experiment_definition_path)#testing
print ("\nthis is testing ",testing)
print ("\nthis is sys.path ",sys.path)
next_job = _get_next_job()

while next_job:
    try:
        _run_job(next_job)

    except KeyboardInterrupt:
        delete_this_job = raw_input(
            "Delete this job from job list [Y/n]? ").lower()
        if delete_this_job != "n":
            _delete_completed_job()
        continue_running = raw_input(
            "Continue running other experiments [N/y]? ").lower()
        if continue_running != "y":
            break
    else:
        _delete_completed_job()
        next_job = _get_next_job()

def _get_next_job():
with open(job_list_filename, 'r') as fh:
next_job = fh.readline().strip()
return next_job

def _run_job(next_job):
logger.info("Running {}".format(next_job))
exec("import {:s}".format(next_job))
print ("\nthis is import{:s} \n","import {:s}")
from neuralnilm.utils import configure_logger
configure_logger('neuralnilm.log')
eval("{next_job}.run('{next_job}')".format(next_job=next_job))

def _delete_completed_job():
with open(job_list_filename, 'r') as fh:
remaining_jobs = fh.readlines()[1:]
with open(job_list_filename, 'w') as fh:
fh.writelines(remaining_jobs)
logger.info("Remaining jobs = {}".format(remaining_jobs))

if name == "main":
main()

@gohingsiang
Copy link
Author

Hi Jacoobr, after I try it with the pandas 0.19.2 I getting the same error as you mentioned before.

ValueError("cannot reindex a non-unique index "
ValueError: cannot reindex a non-unique index with a method or limit

@Jacoobr
Copy link

Jacoobr commented Jan 18, 2018

Thank you gohingsiang, Can you tell me which pandas version that you use before you test with the version 0.19.2?

@gohingsiang
Copy link
Author

gohingsiang commented Jan 18, 2018

I was using pandas 0.16.2 before due to the suggestion from #1.

Beside from 0.16.2 and 0.19.2 as you suggested, I also tried pandas 0.16.0 and 0.17.1. But unfortunately I'm still getting the AttributeError: 'NoneType' object has no attribute 'dropna'.

I also tried different version of Theano such as 0.6.0, 0.7.0 and 0.8.0 but nothing jus work.

@gohingsiang
Copy link
Author

By the way, I tried to run the code Neural NILM test.ipynb and NeuralNILM design.ipynb from the neuralnilm/notebooks/ and it's working, but the structure of the network is not the same as writen in Jack Kelly paper

@Jacoobr
Copy link

Jacoobr commented Jan 19, 2018

Thank you very much, gohingsiang. Your message help me a lot. Now i try to go back to Jack Kelly's paper and his code to see what exactly he did in his project. Any progress about this AttributeError: 'NoneType' object has no attribute 'dropna', i will tell you. Thank you .

@gohingsiang
Copy link
Author

gohingsiang commented Jan 19, 2018

My pleasure. Any progress regarding AttributeError: 'NoneType' object has no attribute 'dropna', i will inform you as well. Thanks in advance!

By the way, the result of the paper is written based on e567.py from his previous github Neural NILM Prototype. I tried to run the code but I'm gettng another error regarding the keyword argument for "border_mode". Maybe you can try on it.

@Jacoobr
Copy link

Jacoobr commented Jan 19, 2018

Thank you. I will try the experiment based on e567.py later. Then, i will tell you the result of mine. By the way, i faced a new error 'NameError: name 'Plotter' is not defined'. I guess this is the version of my matplotlib 1.4.3 caused. Can you tell me what's your matplotlib's version that you used? Thank you.

@Jacoobr
Copy link

Jacoobr commented Jan 19, 2018

Hi, gohingsiang. Would you mind help me see which package the Plotter() dependency. In the code neuralnilm-master/notebooks/NeuralNILM design.ipynb
`plotter = Plotter() # this is the code that caused NameError:name 'Plotter' is not defined
disaggregator = Disaggregator()

trainer = Trainer(
net,
data_pipeline=None,
repeat_callbacks=[
( 10, Trainer.validate),
(1000, plotter.plot),
(1000, disaggregator.disaggregate),
(1000, disag_metrics.run_metrics)
]
)
`
Thank you.

@gohingsiang
Copy link
Author

I removed this part of the code and I'm getting the result, I think that this part of the code is one of the code that Jack Kelly develop at that moment, but the code work fine even if you removed this part of the code.

@Jacoobr
Copy link

Jacoobr commented Jan 19, 2018

Thanks for your reply. The code works just like you said.

@gohingsiang
Copy link
Author

But please take note that the structure of the network used is not the same as in Jack Kelly's paper. I was thinking to change the coding in the ipynb but i notice that they way how it handle the data is slightly different with the e567.py and e578.py.

@Jacoobr
Copy link

Jacoobr commented Jan 20, 2018

Hi, gohingsiang. Do you have any progress about training the net? When i try to run the script of e567.py, an error raised like this: OSError: [Errno 2] No such file or directory: '/data/dk3810/figures/e567_microwave_ae'. Do you have any idea about this error? By the way , i find nothing about the 'e567_microwave_ae' in Jack Kelly's previous project.

@gohingsiang
Copy link
Author

Hi, Jacoobr, for the e567.py, I was facing some error regarding keyword argument for the boder_mode, did you face this problem before?

@Jacoobr
Copy link

Jacoobr commented Jan 21, 2018

Hello, gohingsiang, i was so busy that i didn't reply your question yesterday. I'm sorry about that. For your question, i also faced this error by running the script e567.py. It raised an error the keyword argument for "border_mode" like you said. I fixed this error by annotating the code 'border_mode' in the script e567.py temporary. Then i ran the script again, this time it worked. I see the net begin to train with many out put like this:
`
.......Running net.fit for e567_microwave_rectangles
Starting training for 3000 iterations.

Update Train cost Valid cost Train / Val Secs per update
Saving plots...
/usr/local/anaconda2.4.2b/python2.7/site-packages/matplotlib/collections.py:590

: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
if self._edgecolors == str('face'):
Saving params...
Saving activations...
Finished saving.
0 | 0.220820 | 0.194270 | 1.136669 | 63.896s
1 | 0.317625 | 0.194270 | 1.634969 | 0.989s
2 | 0.217491 | 0.194270 | 1.119533 | 0.987s
3 | 0.109623 | 0.194270 | 0.564285 | 1.005s
4 | 0.258685 | 0.194270 | 1.331575 | 0.990s
5 | 0.159591 | 0.194270 | 0.821490 | 1.050s
......`
By the way, i use Jack Kelly's neuralnilm_prototype project for this test. Hope this can help you.

@gohingsiang
Copy link
Author

gohingsiang commented Jan 22, 2018

Hi, Jacoobr, I tried to replace the word "border_mode" with "pad" inside the coding after I referred http://lasagne.readthedocs.io/en/latest/modules/layers/conv.html#lasagne.layers.Conv1DLayer, however I'm getting another error.

Exception: ('The following error happened while compiling the node', GpuDnnConvDesc{border_mode='full', subsample=(1, 1), conv_mode='conv', precision='float32'}(MakeVector{dtype='int64'}.0, MakeVector{dtype='int64'}.0), '\n', 'nvcc return status', 2, 'for cmd', '/usr/local/cuda-8.0/bin/nvcc -shared -O3 -arch=sm_61 -m64 -Xcompiler -fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden -Xlinker -rpath,/home/ubuntu/.theano/compiledir_Linux-4.13--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.12-64/cuda_ndarray -I/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/sandbox/cuda -I/usr/local/cuda-8.0/include -I/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -I/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/theano/gof -o /home/ubuntu/.theano/compiledir_Linux-4.13--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.12-64/tmppHh_Rr/358ed08bfb8a798e5d73c5258b893f50.so mod.cu -L/usr/lib -lcudnn -lpython2.7 -lcudart', "[GpuDnnConvDesc{border_mode='full', subsample=(1, 1), conv_mode='conv', precision='float32'}(<TensorType(int64, vector)>, <TensorType(int64, vector)>)]")

May I have a look on your code that annotating the code 'border_mode' in the script e567.py temporary?

@Jacoobr
Copy link

Jacoobr commented Jan 23, 2018

Hi, gohingsiang. The code that i annotated in the script e567.py like this:
287 # 'border_mode': 'valid' ...... 295 #'border_mode': 'valid' ...... 374 #'border_mode': 'same' ...... 443 #'border_mode': 'valid' ...... 478 #'border_mode': 'full'
But please take note that this is only a temporary way to run the code. In my case, i tried to run the three nets that Jack Kelly said in his paper.But i only ran the rectangle net successfully. The other two nets got a 'reshape error'. I guess this error was caused by the five lines code that i annotated in the script e567.py.

@gohingsiang
Copy link
Author

gohingsiang commented Jan 23, 2018

Hi, Jacoobr, maybe you can try to replace the 'border_mode': 'valid' with 'pad': 'valid'.

            'type': Conv1DLayer,  # convolve over the time axis
            'num_filters': 16,
            'filter_size': 4,
            'stride': 1,
            'nonlinearity': None,
            'pad': 'valid', #addon
	#'border_mode': 'valid' #original code

It works for me and hopefully it works for you, but right now I was trying to find the code for the metric but I'm uncertain about which code i should used in Neural NILM Prototype. Do you have any idea?

@Jacoobr
Copy link

Jacoobr commented Jan 24, 2018

Hi gohingsiang, thanks for your suggestion.I will try it later. What's more, maybe you can find something ahout the metric from Where the code send the metric to MongDB in Jack Kelly's project. There are some message about the metric as follow in his README.md :
`trainer.py includes a Trainer class which trains each net and sends metrics to the Mongo database.

neuralnilm/neuralnilm/data includes the mechanisms for loading and transforming data. This is designed to be very modular so you can easily mix and match data sources, pre-processing steps and networks. For a quick overview of how this all works, take a look at some of the python files in the experiment_definitions directory.

neuralnilm/neuralnilm/monitor contains the mechanisms for loading metrics and metadata from the Mongo database and plotting it.`
Hope this can help you.

@gohingsiang
Copy link
Author

Hi, Jacoobr, thanks for your information, in fact now I'm trying to modified the code for the e578.py especially in real_agg_source in line 179. By the way, I can't found any folder regarding the scores or params after i run the code for e567.py, did you get any file regarding it?

@Jacoobr
Copy link

Jacoobr commented Jan 24, 2018

Did you mean where the output stored by running the script e567.py? When i finished running the script, i can see the output of the params and many output *.pdf file in the file where i set it in e567.py like this:
`NAME = os.path.splitext(os.path.split(main.file)[1])[0]
#PATH = "/homes/dk3810/workspace/python/neuralnilm/figures"
PATH = "/home/jacoob/workStation/extraProject/figures/" #the path that the output stored
#PATH = "/home/jack/experiments/neuralnilm/figures"

UKDALE_FILENAME = '/home/jacoob/workStation/extraProject/dataSet/ukdale.h5'
`

@gohingsiang
Copy link
Author

gohingsiang commented Jan 24, 2018

I did found those figures and files, but when I run the code in the Neural NILM test.ipynb and NeuralNILM design.ipynb from the neuralnilm/notebooks/, I found there is a file name net_params.h5 in the same folder with the figures, however, I couldn't found any file in that particular folder after running the e567.py code.

@Jacoobr
Copy link

Jacoobr commented Jan 25, 2018

Hello gohingsiang, Did you mean where the net params stored when the e567.py finished ? I can see the output file like : e567_microwave_rnn.hdf5,e567_microwave_rectangles.hdf5 ,etc. Maybe JackKelly changed the file format which he used in his neuralnilm project.And i open the net_params.h5 file and e567_microwave_rnn.hdf5(the file that output by running e567.py script.) file with hdfview software. I can see the same data format.So i guess the *.hdf5 is the file where params stored.

@gohingsiang
Copy link
Author

Okay, thanks for your information.

@Jacoobr
Copy link

Jacoobr commented Jan 28, 2018

Hi goingsiang, Have you test the script disag_567.py in Jack Kelly's neuralnilm-prototype project. I try to test this script and i get an error like this:
return np.concatenate((np.zeros(padshape, dtype=arr.dtype), arr), MemoryError Exception TypeError: TypeError("'NoneType' object is not callable",) in <function _remove at 0x7f0dab849758> ignored
Do you have any idea about this error?

@gohingsiang
Copy link
Author

gohingsiang commented Jan 29, 2018

Hi Jacoobr, after running the code I get this following error:

Loading params from /home/ubuntu/Desktop/Goh_new/Data/e567_output/e567_modified_washing machine_rectangles/e567_modified_washing machine_rectangles.hdf5...
Traceback (most recent call last):
File "disag_567.py", line 280, in
neural_nilm_disag()
File "disag_567.py", line 263, in neural_nilm_disag
net = get_net(appliance, architecture)
File "disag_567.py", line 131, in get_net
path=join(NET_BASE_PATH, experiment_name))
File "/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/NeuralNILM-0.0.1-py2.7.egg/neuralnilm/net.py", line 515, in load_params
epoch_group = f[epoch_name]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/home/ubuntu/Desktop/Goh_new/my_project/local/lib/python2.7/site-packages/h5py/_hl/group.py", line 167, in getitem
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'epoch300000' doesn't exist)"

I check the content inside the e567_modified_kettle_ae.hdf5 but inside only content epoch000000, it seem like it doesn't save the param for my case.
Did you face this problem before?

@Jacoobr
Copy link

Jacoobr commented Jan 29, 2018

I faced this problem before? There is only epoch000000 in the .hdf5 file like you said. I guess this may caused by the iteration(epochs) number(3000) that i set when i training the net. So i changed it to 0, it means let she script only load epoch000000. Then, this problem gone.Hope the message can help you.

@gohingsiang
Copy link
Author

I tried to KeyboardInterrupt during the training, then it came out with a menu, then i choice the Save plots and params, then i was able to obtain the epoch000000 and epoch003100 inside the hdf file, I think that it's due to the method for saving the params. I think that we should change the way to save the params as in /neuralnilm_prototype/neuralnilm/experiment.py (show as below). I'm afraid the data from epoch000000 might not be the params obtained from the last training.

def menu(net, epochs):
# Print menu
print("")
print("------------------ OPTIONS ------------------")
print("d: Enter debugger.")
print("s: Save plots and params.")
print("q: Quit all experiments.")
print("e: Change number of epochs to train this net (currently {})."
.format(epochs))
print("c: Continue training.")
print("")

# Get input
selection_str = raw_input("Please enter one or more letters: ")

# Handle input
for selection in selection_str:
    if selection == 'd':
        import ipdb
        ipdb.set_trace()
    elif selection == 's':
        net.save()
    elif selection == 'q':
        sure = raw_input("Are you sure you want to quit [Y/n]? ")
        if sure.lower() != 'n':
            raise
    elif selection == 'e':
        new_epochs = raw_input("New number of epochs (or 'None'): ")
        if new_epochs == 'None':
            epochs = None
        else:
            try:
                epochs = int(new_epochs)
            except:
                print("'{}' not an integer!".format(new_epochs))
    elif selection == 'c':
        break
    else:
        print("Selection '{}' not recognised!".format(selection))
        break
print("Continuing training for {} epochs...".format(epochs))
fit(net, epochs)

@Jacoobr
Copy link

Jacoobr commented Jan 30, 2018

Hello,gohingsiang. You are right. The number of iteration that you set when training the net should be bigger than that i set 3000. The more iteration we training, the better feature we will get when we disaggregate a single application from aggregate data.Now i'm trying to disaggregate one application like kettle, washing machine use the script disag_567.py.But when the script load the main data from building_{:d}_mains.csv, i get MemoryError like i said before. Maybe the code works for your test. By the way, i use 100000 rows data(the data i stored in the building_1_mains.csv) of the mains.data that i get from UK-DALE dataset to test the script.

@gohingsiang
Copy link
Author

gohingsiang commented Jan 30, 2018

May i know where is your location of the building_1_mains.csv aforementioned? Because i couldn't found any related .csv file. Is that the file generated from the disag_567.py?

@gohingsiang
Copy link
Author

gohingsiang commented Jan 30, 2018

Hi Jacoobr, may I know where you get the ground_truth_and_mains?

@Jacoobr
Copy link

Jacoobr commented Jan 30, 2018

I get the file that in the ground_truth_and_mains file are as follow::
building_1_mains.csv、building_2_mains.csv、building_5_mains.csv
These 'building_i_mains.csv' files made by 'mains.dat' that come from house1, house2, house5 in dataset.Down the end of the page, you will see the mains.dat data. I get the .csv file by Excel, so the .csv file only have 1048576 rows. Because the lines of mains.dat file over the limit of Excel can handled.

@gohingsiang
Copy link
Author

Maybe you can try to use the ground_truth_and_mains from http://jack-kelly.com/neuralnilm/ and I was able to run without error, I hope that this will able to solve your MemoryError.

@Jacoobr
Copy link

Jacoobr commented Jan 31, 2018

Hi, gohingsiang. Thank you very much.I just forget the site that you mentioned.Thank you.

@Jacoobr
Copy link

Jacoobr commented Feb 1, 2018

Hello, gohingsiang.Hope the message as follows can help you fix the problem that you mentioned yesterday.
some latex package that i installed in my Ubuntu16.04.
And this works for my to run the BuildSys_results_plot.ipynb nicely. Hope this can help you.

@gohingsiang
Copy link
Author

gohingsiang commented Feb 1, 2018

Thanks for you reply, I did found this post and the error has been solved! Thanks for all the information!

@gohingsiang
Copy link
Author

Hi Jacoobr, may I have your email in case there something confusing, we can discuss individually.

@Jacoobr
Copy link

Jacoobr commented Feb 3, 2018

Hi gohingsiang, you can contact me with this email: .I's my pleasure to discuss problem with you.

@changfangyi
Copy link

Do u guys use ukdale.h5 for your project ?
u can try redd.h5 and https://github.com/hyl0327/neuralnilmtp is designed for redd actually.

@gohingsiang
Copy link
Author

Yes, changfangyi, we are using the ukdale.h5 for this project. But for me, I think for this particular moment I'm gonna put more attention on UK-DALE for this project. Anyway thanks for the information shared. I really appreciated it!

@changfangyi
Copy link

I wish that your problem was solved. I use mac or linux to perform my project, and I don't face the bug "AttributeError: 'NoneType' object has no attribute 'dropna'", even using ukdale.

@gohingsiang
Copy link
Author

Hi, @changfangyi ,may i know your reproduced project is based on which github? Is it the neuralnilm or the neuralnilm_prototype?

@changfangyi
Copy link

My project is based on neuralnilm.
Here is the code:
https://github.com/changfangyi/III_NILM_DeepLearning

@Jacoobr
Copy link

Jacoobr commented Mar 8, 2018

Hi, @changfangyi , Can you provide me with REDD dataset for the future work with NILM project? I can't download the REDD dataset from the site that you mentioned for some reasons. Thank you very much.

@changfangyi
Copy link

ok,
you can download the data from my one drive

https://1drv.ms/f/s!AkX-2oVmhJj3gbc3dM3EFlIgblmbVw

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants