Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dropout WIP #535

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Dropout WIP #535

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
72067a8
SDR:addNoice2 : adds random noice with probability
breznak Jul 3, 2019
bfbb417
SP apply dropout / noise WIP
breznak Jul 3, 2019
7d43342
update some exact output tests
breznak Jul 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion src/htm/algorithms/SpatialPooler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,9 +468,16 @@ void SpatialPooler::initialize(
}


void SpatialPooler::compute(const SDR &input, const bool learn, SDR &active) {
void SpatialPooler::compute(SDR &input, const bool learn, SDR &active) { //TODO make input const again
input.reshape( inputDimensions_ );
active.reshape( columnDimensions_ );

//dropout, apply noise to input
input.addNoise2(0.01f, rng_); //TODO apply at synapse level in Conn?
//TODO fix for probability << input.size
//TODO apply killCells to active output?
//TODO apply dropout to segments? (so all are: synapse, segment, cell/column)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proof of concept dropout applied to input (as noise) and output (as killCells).

  • I'd prefer this be applied in Connections (in adaptSegment?)
  • where to apply?
    • ideally all of: SP, TM. & synapse, segment, cell, column
    • but that would be computationally infeasible, so..?


updateBookeepingVars_(learn);
calculateOverlap_(input, overlaps_);
calculateOverlapPct_(overlaps_, overlapsPct_);
Expand All @@ -495,6 +502,9 @@ void SpatialPooler::compute(const SDR &input, const bool learn, SDR &active) {
updateMinDutyCycles_();
}
}

//dropout output
active.addNoise2(0.001f, rng_);
}


Expand Down
2 changes: 1 addition & 1 deletion src/htm/algorithms/SpatialPooler.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ class SpatialPooler : public Serializable
inhibition. The size of the SDR is equal to the number of
columns (also returned by the method getNumColumns).
*/
virtual void compute(const SDR &input, const bool learn, SDR &active);
virtual void compute(SDR &input, const bool learn, SDR &active);


/**
Expand Down
16 changes: 16 additions & 0 deletions src/htm/types/Sdr.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,22 @@ namespace htm {
}


void SparseDistributedRepresentation::addNoise2(const Real probability, Random& rng) {
NTA_ASSERT( probability >= 0.0f and probability <= 1.0f );
const ElemSparse numFlip = static_cast<ElemSparse>(size * probability);
if (numFlip == 0) return;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to write an effective implementation, but this has problem with p << size. Should we bother with such cases? return/assert?


// instead of applying probability `p` `n` (=size) times to each bit,
// flip `p*n` random bits.
getDense(); //create dense_ array
for (ElemSparse i=0; i < numFlip; i++) {
const ElemSparse toggle = static_cast<ElemSparse>(rng.getUInt32(size));
dense_[toggle] ^= true; //XOR, flip
}
setDenseInplace();
}


void SparseDistributedRepresentation::addNoise(Real fractionNoise) {
Random rng( 0 );
addNoise( fractionNoise, rng );
Expand Down
8 changes: 7 additions & 1 deletion src/htm/types/Sdr.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,10 +471,16 @@ class SparseDistributedRepresentation : public Serializable
* @param rng The random number generator to draw from. If not given, this
* makes one using the magic seed 0.
*/
void addNoise(Real fractionNoise);
void addNoise(Real fractionNoise); //TODO the name is confusing, rename to shuffle ?
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it OK to rename this to shuffle() and have addNoise for the new fn? @ctrl-z-9000-times

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at your new addNoise function and I think it will have issues with keeping the sparsity at a reasonable level. I think that the sparsity of an SDR after this method is called on it will always tend towards 50%.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that would be indeed wrong. What I intended:

  • have SDR of current input
    • flip 0.01% bits
  • have a new SDR
    • flip 0.01%bits

So the sparsity would remain the same (actially grow, because we have much more off bits, so flipping on would be more probable). But it should remain the x% (2%) + 0.001%


void addNoise(Real fractionNoise, Random &rng);

/**
* actual add noise :)
* TODO doc
*/
void addNoise2(const Real probability, Random& rng);

/**
* Modify the SDR by setting a fraction of the bits to zero.
*
Expand Down